Backup Zoom Servers

Zoom Database Server Backup

It is always important when installing Zoom to have a backup plan for key files in the unfortunate event that a system failure occurs. Fortunately, backing up the Zoom system can be configuring any enterprise backup system. Zoom’s RevDB database allows the database to be backed up even while Zoom is running. Every time Zoom Server starts, it performs a checkpoint of database to ensure the database is correct. All the database files as well as the check-pointed state of the repository is saved under the db/ folder. You can setup a periodic job that does an incremental backup of this folder.

What files to Backup

For an effective backup strategy, the following items need to be backed up:

conf folder

The conf folder contains all of the information for how the Zoom Server is configured and your metadata specifications. It is important to keep a backup of this folder so that in the event of having to reinstall the software, all of your settings are preserved. It can be found in the following locations:

OS	Path
Linux	[ZoomDir]/conf
Windows Server	C:\Program Files (x86)\Evolphin\DAM\conf

db folder

The db folder contains the RevDB database and is one of the most crucial items to backup on a regular basis as this contains the contents of the entire Zoom repository.

The following are the default locations of the db folder:

OS	Path
Linux	[ZoomDir]/db
Windows Server	C:\Program Files (x86)\Evolphin\DAM\db

logs folder

audit logs under the <zoom-server--install-dir>/logs/ folder are recommended. It is not necessary to backup rest of the log files.

zm.conf

For older Zoom installations, the zm.conf file is the configuration file for the Preview Server. This allows the server to be distributed on different servers than the Zoom Server. By default, this file is not created but should be backed up in the even this file is created.

OS	Path
Linux	[ZoomDir]/webcontext/servers/PreviewServer
Windows Server	C:\Program Files (x86)\Evolphin\DAM\webcontext\servers\PreviewServer

Pre-backup procedure

Zoom server checkpoints the database periodically from its transactional redo logs based on a schedule specified via the Zoom web console. To avoid copying both redo logs and check-pointed data twice, it is highly recommended but not necessary that a full backup be started post check-pointing.

When the check-pointing is over, Zoom will spit out a message in the server log similar to the log message below, that can be monitored by a script to kick off the full database backup. The time to complete a checkpoint depends upon the amount of data in the repository that was changed since the last check-point and the sustained disk IO throughput. Usually the initial import takes the longest to check-point as multi terabytes of data might have been imported. After that <10% of repository data might change every month because of Zoom’s de-duplication features.

Below is an example of check-point start and end logging performed by Zoom server:

=========== BEGIN Zoom LOG ==============
Start Time: Fri Jun 15 13:52:11 PDT 2012
Product Version: 3.6-b5004
===========  LOG HEADER =================

INFO: Fri, 15 Jun 2012 13:52:11.054 PDT  [TransactionRedoLogManager.setup main] REDO_CHECKPOINT_START
INFO: Fri, 15 Jun 2012 13:52:11.743 PDT  [TransactionRedoLogManager.setup main] REDO_CHECKPOINT_END 

Checkpointing Frequency

A checkpoint creates a known good point from which the Zoom RevDB database can start applying changes contained in the redo log during recovery after an unexpected shutdown or crash.  Without a checkpoint, the recovery will take forever as the server would have to look at every single redo log.

Checkpoint also help distribute the disk IO load across multiple files within the Zoom transnational file store. Over a longer period of time the system will perform better w.r.t disk IO with checkpointed redo log.

Eventually RevDB will implement background checkpoints currently these are done at server restart only to ensure no writers can come in while the redo logs are being garbage collected/deleted during checkpoint. This downtime is a non-issue with Zoom NonStop as there will be a node that can serve the readers & writers during the downtime of another node.

Check-pointing allows the Zoom server to apply the transactions written to the Zoom RevDB’s redo logs to the actual database on disk. Transactions in Zoom span both file data and metadata changes. Check-pointing ensures that changes in transactions to one or more asset is atomically written out to the database on disk.

Check-pointing occurs at either Zoom DAM server restart or periodically based on a schedule specified in the Zoom Web Console -> Server Control Panel -> Automatic Server restart Settings. Out of the box, periodic restart or checkpoint is disabled. Using the Web Management Console’s GUI the check-pointing via restart can be enabled, start date, start time, and a frequency (in days) can also be specified in the web-based form.

Frequency of restart and thus check-pointing is based on how much data in the transaction redo log can accumulate between restart time period. For example, if in a 7 day period, 250 GB of changes collect in the transaction log and the storage transfer rate is 50 Gbps, it could take around 40 seconds to finish the check-point. During the check-pointing, the server goes into a standby state and does not serve any client requests. If the system administrator requires a smaller checkpoint interval they may decide to change the frequency for instance from 7 days to 3 days based on the expected time for storage transfer IO rate.

With the Zoom NonStop product, system administrators can have a Zoom server perform a check-point while letting a second Zoom Server attached to the shared disk storage continue to serve client requests with 0 down-time during check-pointing.

Incremental backups

Incremental backups of the Zoom folders can be performed at any time. Usually, the best strategy is to perform incremental backups between two check-points. For instance, if the check-pointing is done on a weekly basis, say every Sunday morning, during the week from Mon-Sat, IT administrator can set up incremental backups with a full backup on Sunday night.

Zoom de-duplication and backups

Zoom clients send just deltas or changes whenever a file is edited on the end-user’s desktop. Beyond the first version, other versions are always stored as compressed deltas in the Zoom database folder. The backup strategy is not impacted by how the Zoom de-duplication mechanism stores the data on its back-end.

From the administrator perspective, the Zoom folders are to be treated as a black-box that needs to be periodically backed-up. The administrator need not know the internal data structure used to store the database records and file data.

Restore database from backup

In case of a disaster that wipes out the server storage or the storage disk are corrupted, the administrator needs to stop the Zoom MAM/DAM server, mount a new Zoom db/ folder on a clean storage device, copy the content of the last valid backup to this folder and restart the Zoom services. It would check-point a new database state after the restore and re-start operation complete.

1. Shutdown the Zoom servers (DAM/MAM, Preview, Curator and Transcoder Services
2. Delink all the working copies from the Zoom server to ensure there is no old state associated in the user’s desktop
3. Move the Zoom server’s current db/ folder to a safe location.
4. Restore the contents of the db/ folder from the backup.
5. Restoring the database from a prior point-in-time backup will result in data loss based on files ingested after the backup. This will result in a stale Preview and Curator Server database.
   1. So, it is recommended to restore the Preview cache and Curator Server database from a backup of same or earlier point in time of Zoom database backup. For example, if you restored the 5 days old Zoom database from backup, you must restore the Preview Server cache and Curator Server database from 5 days or older backup.
   2. In any case do not restore the Preview Server cache and Curator Server database from from a backup newer than the Zoom server backup.
   3. Alternatively, you can start from an empty state, if you don’t have the Preview Server cache and Curator Server database backup available. Note: Rebuilding the curator index can take days depending upon the size of your database.
6. Restart the Zoom Server, Preview, Curator & Transcoder servers.
7. Verify via Asset Browser the files in the restored database

Zoom Preview Server Backup

To speed up previews it is a good idea to backup the preview cache though it is not required. If the preview cache is deleted, Zoom can always recreate the missing previews at the cost of additional compute time. Please check the location of the preview cache as mentioned here.

Zoom Preview Server Restore

Preview Server cache can be restored from backup if available and will results in faster Preview response for already cached assets, otherwise Preview Server will build cache on demand as assets are accessed.
Important: If you are restoring the Preview Server cache from backup, it must be older or of same time as the age Zoom server database. This only applies when Zoom server database is also restored from past time backup. In that case you cannot use the Preview Server cache, you must either start from scratch or restore the Preview Server cache from from the same age as Zoom server database.

• Stop the Preview Server.
• Before you begin locate the Preview Server cache directory.
• Preview Server cache directory path can be found in the <preview server install folder>/conf/preview-server.xml file. Please see in this XML file the tag: <previewCacheLocation>
• If you are starting from scratch, then clean our the Preview Server cache directory.
• Else if you are restoring from database, then remove the current cache directory contents and copy the contents from backed up cache directory.
• Once the cache is restored, start the Preview Server.

Zoom Curator Service Restore

Curator Server database can be restored from backup if available and will results in faster indexing, otherwise Curator Server will re-build cache, which could take a long time depending on the data present in the Zoom Server.
Important: If you are restoring the Curator Server cache from backup, it must be older or of same time as the age Zoom server database. This only applies when Zoom server database is also restored from past time backup. In that case you cannot use the Curator Server cache, you must either start from scratch or restore the Curator Server cache from from the same age as Zoom server database.

• Stop the Curator Server.
• Before you begin locate the Curator Server cache directory.
• Curator Server cache directory path can be found in the <curator server install folder>/conf/curator-server.xml file. Please see in this XML file the tag: <solrDataDir>
• If you are starting from scratch, then clean our the Curator Server cache directory.
• Else if you are restoring from database, then remove the current cache directory contents and copy the contents from backed up cache directory.
• Once the cache is restored, start the Curator Server.

See Also:
Improve Curator Server Indexing when Metadata Changes are a lot

Improve Curator Server Indexing when Metadata Changes are a lot

Indexing the large Zoom Server database on Curator can take long time, sometimes upto days.
So if for any reason Curator Server database backup is not available, please follow the below given steps precisely and carefully to reduce the Curator Server indexing time.
Note: You don’t have to follow these steps if your Zoom Server database is not huge and we still recommend taking Curator Server database backup and restore it from backup if available.

1. Stop the Curator Server is running.
2. Delete the existing Curator Server database. Typically the solr-db directory.
3. Start the Zoom Server if not running.
4. Now we need to note down the highest available RRN on the Zoom Server. For this, either Changeset Browser GUI app or the ZM CLI can be used.
   1. To use the ZM CLI, run the follow command: zm -s https://your-zoom-server:9880 changeset -n 1
   2. Now note down the latest available RRN value on the Zoom server printed by the above command.
5. Start the Curator Server.
6. Monitor the Curator Server log to see when the server has started up and indicates the start of indexing.
7. Run the following command using “curl” from the Curator or Zoom server machine, to set the max SetMD indexed RRN to the RRN noted in step 4 above.   
   curl http://<curator-host>:8983/solr/BK/update?commit=true -H 'Content-Type: application/json' --data-binary '[{"id":"0_0_0", "max_setmd_indexed_rrn":{"set":RRN_VALUE_FROM_STEP_4}}]'
   Note: You must run the above curl command either on the Curator Server itself or from the Zoom Server machine.
8.  Ensure that a success response is received. Solr’s success status code is 0.
9. Restart the Curator service after 30 seconds or more of completing the above curl command successfully.
10. This will allow Curator’s Metadata indexing jump ahead to the point up till which the VCS indexing would have already handled the changes, i.e. the time when you started the new Curator service. The Non-Content (or VCS) Indexing will handle indexing the metadata changes up till this point. Keep monitoring the indexing status of Non-Content and Content data on the Curator Dashboard.
11. In case, Content indexing is enabled, then it would be beneficial to increase the heap memory settings of the Curator service in curator-server.conf. The heap memory settings can be reverted to earlier levels after the reindexing is completed.