Setting up Archive/Restore with VideoLX (Zoom 7.1+)

Storage Models

Zoom has two storage models for assets as described here. Please familiarize yourself with these before proceeding with the steps below. With VideoLX or VideoFX modules, both storage models are used in a Zoom deployment.

Direct Ingested Assets – To archive/restore these assets in the Zoom Asset Browser (VAB),  use the VAB > Selected Files > Right-click context menu > Archive/Restore or use Web Management Console.

External Assets – To archive/restore these assets in the Zoom Asset Browser (VAB) currently it is required to use auto-tasks, VAB > Selected Files > Right-click context menu > Create Job

Overview

This post is about adding archive/restore functionality to an existing VideoLX setup. If you have to set up VideoLX first, then check here for details.

Zoom offers a built-in internal archive system. It also allows extending the internal module by using external archive modules like SGL, S3, and DIVA. You can skip these configuration steps if you do not need archive with VideoLX.

First, you need to install Perl modules for the installers of archive/restore hook files – follow this article to install Perl and its required modules.

We will set up the external archive modules by building on the basic archive functionality. These sections are covered:

Follow each section to go through the configuration steps in detail.

Run script installers

Follow this post to run script installers on your Zoom Server. This will install the scripts necessary to run archive & restore hooks. 

For external assets these hooks ensure that such assets that have a proxy (metadata field for the proxy is set to true) directly ingested in Zoom are accidentally archived using the direct Archive menu option, the proxies are skipped from getting archived.

Check Archive license

Archiving is an optional feature that is available with an appropriate license. You can check your license information in Zoom’s Web Management Console.

To check the license information, log on to your Web Management Console, for example, http://MyZoomServer:8443/

Open the License Management node under the System node in the Admin Menu sidebar. Click License Information to view your license.

If you hold a license that allows archive operations then the Archive Module would be enabled here.

From Zoom 7.3.1 onwards, for customers that hold a Hierarchical Archive license, it would also be enabled here as Hierarchical Archive. For Zoom 7.3 only, Hierarchical Archive is listed as Multi-tiered Archive here.

If you would like to purchase the Archive Module or a Hierarchical Archive license then contact Evolphin Support.

Enable basic archive in your setup

When the Archive Module is included as a part of the Zoom license, Zoom can be set up to archive assets to a single designated archive location using its internal Basic Archive Module. This folder location must exist on the Zoom MAM server in order for the configuration to work. Enable and set up the Basic Archive by following these steps:

  1. Open the Web Management Console for your Zoom Server.
    ex. http://localhost:8443 or http://<zoomserver>:8443
  2. Log in using your admin credentials.
  3. In the Admin Menu sidebar, click Server Control Panel under the System node.
  4. Click Archive Management on the Server Control Panel page.

  5. Click Enable Archive to enable it.
  6. Specify a local path on Zoom MAM Server as the Archive Location.
    If using an external archive module, like SGL or S3, the archived files are moved to the external archive systems from this path. Check Getting started with Hierarchical Archive for more details.
    Ex. E:\zoom\archive\ or /mnt/Archive on the Zoom MAM Server.
  7. If you are not setting up External Archive then pre and post scripts for archive and restore are optional. You can still set up scripts here to execute before and after the archive/restore operations.
  8. If you are setting up External Archive, then you need these script files:
    1. Specify the path for Pre-script for Archive from the location where the script installer placed the script files (as described here). For example, if you had specified /home/evolphin as the root path for the script installer, then the path will be /home/evolphin/zoom-deploy/ArchivePreHook/archive-hook.pl.
    2. Similarly, specify the path for Pre-script for Restore from the location of the restore pre-script in the script files on Zoom MAM Server. For example, as per the above example, this path will be /home/evolphin/zoom-deploy/RestorePreHook/restore-hook.pl.
  9. Update Limit on Arguments on Command-line to be 0. This is needed to remove the limit on the number of asset IDs passed in a Zoom command while using archiving.
  10. Click Save.
  11. You will be prompted to restart the server. Click Yes.
  12. Refresh your web browser.

Basic Archive settings are now saved.

Archiving to multiple target locations 

This step is not needed for external assets. It is only applicable to direct ingest assets.

As a media manager, you may decide that assets from some projects in Zoom may be less relevant and could be archived to low-cost tape storage, while assets from other projects may have to be kept more accessible on a fast SAN. You could achieve this out of the box starting with Zoom 7.1 and above.

You can set up these configuration details from the Web Management Console. Log in as an admin to the Web Management Console. Open Server Control Panel under the System node in the Admin Menu sidebar. Click Archive Management

Path-specific archive targets

Enter the Zoom paths and their corresponding file system locations to which assets must be archived in the Path-specific Archive Targets box.

For example, the entry may look like this on a Linux server:

/defproj=/data/Archives/misc/ ; /Library = /mount/commons

And like this on a Windows server:  

/defproj=\\NY-NAS\backups\defproj; /Library=L:\Users Data\Users1\akila\Others

Click Save. Choose Yes to restart the Zoom Server to save the configuration changes.

At this point server.xml will show a new entry in <archivespec>, <targets>

<targets>/defproj=\\NY-NAS\backups\defproj; /Library=L:\Users Data\Users1\akila\Others</targets>

The paths may be a mapped drive letter path or a UNC path, or a local or network file system path. As long as the Zoom server process has the correct permissions into the target locations, the archive operations will work just fine. 

Any assets belonging to paths given here will get archived according to the locations configured here; the assets belonging to paths not mentioned in this field will still get archived into the default archive location. 

Though it is usual to put the paths corresponding to Zoom projects here, it is not mandatory. You could specify any folder path in the configuration and partition your archive locations even within the project-level boundaries. 

This configuration is only allowed when there are no assets that are currently archived. Therefore it is recommended that you configure this during the initial Archive setup.
Configuration of default archive location is still mandatory, and this path must also physically exist and be accessible from the Zoom Server. 
 

Check metadata schema

While setting up your Zoom MAM Server for VideoLX, we had set up some metadata fields to be used as the folder schema to save the various high-res, low-res, mid-res, or direct-ingest assets. We can also use the fields set up in a similar way to set up the folder schema for Archived assets. Check here to see how to set up a metadata schema on your Zoom MAM Server.

Add auto task job forms for external assets

We need to add job forms for Archive and Restore flow.

  1. Navigate to the dynamic-forms directory in the conf directory inside the default Zoom installation folder (Windows: [ZoomInstallDir]\conf\dynamic-forms\ and Linux: $home$/zoom/conf/dynamic-forms/)
  2. Open createjobfromtemplate.xml for editing.
  3. Add a new form each for archive and restore as follows:
    If using VideoLX instead of VideoFX, there is no need to set up a checkbox for mid-res as mid-res is not supported with VideoLX
    <form heading=”Archive” name=”Archive” tooltip=”This is tooltip” formFor=”*Archive*”>
    <!– Must specify name and id to each tag. Name field is used to store at backend and Id field is required by Flex form framework –>
    <checkbox id=”ArchiveHires” name=”archiveHires” enable=”true” label=”Archive Native File” checked=”true”/>
    <checkbox id=”ArchiveProres” name=”archiveMidres” enable=”true” label=”Archive Prores File” checked=”true”/>
    </form>
    <form heading=”Restore” name=”Restore” tooltip=”This is tooltip” formFor=”*Restore*”>
    <!– Must specify name and id to each tag. Name field is used to store at backend and Id field is required by Flex form framework –>
    <checkbox id=”RestoreHires” name=”restoreHires” enable=”true” label=”Restore Native File” checked=”true”/>
    <checkbox id=”RestoreProres” name=”restoreMidres” enable=”true” label=”Restore Prores File” checked=”true”/>
    </form>
  4. Save and close the file.

Update workflow executables folder for external assets

The executable/script files for Archive workflow are placed by the script installer when running on the Zoom MAM Server. These have to be set as default automatic workflow executables in the Web Management Console. 

  1. Open Web Management Console.
    ex. http://localhost:8443 or http://<zoomserver>:8443
  2. Log in using your admin credentials.
  3. In the Admin Menu sidebar, click on Server Control Panel under the System node.
  4. Click on Workflow Settings on the Server Control Panel page.
  5. Update the Automatic Work Executables Directory to point to the workflow executables folder placed by the script installer (as described here). For example, if you had specified /home/evolphin as the root path for the script installer, then the workflow executables path will be /home/evolphin/zoom-deploy/AutoTasks.

  6. Click Save.
  7. You will be prompted to restart the server. Click Yes.
  8. Refresh your web browser.

Add workflow templates for external assets

You need to create workflow templates for auto tasks in Web Management Console and assign archive and restore scripts to these. To learn more about workflows in Zoom, click here.

  1. Open Web Management Console.
    ex. http://localhost:8443 or http://<zoomserver>:8443
  2. Log in using your admin credentials.
  3. In the Admin Menu sidebar, click on Manage Workflow under the Team Workflow node.

  4. Click on the + sign at the lower bottom corner of the page to add a new workflow.

  5. Drag and drop the Auto task symbol into the design space and attach it to the Start and End tasks using arrowheads that appear while hovering over any task.

  6. Select the auto task and click on the drop-down for Project in the Owner box below the design space. Select your desired project.
  7. Next, click on the Executable drop-down and choose Archive/archiveHiRes.pl from the list.

  8. Carefully click on the Save as Template button on the top bar (not Save button) and save this workflow template as a user-friendly name that indicates the archive back-end in use such as Archive / ArchiveS3 / ArchiveSGL etc and click Save.

  9. Click on the + sign again to add another workflow.
  10. Again, drag and drop the Auto task and attach it with the Start and End tasks.
  11. Select the Auto task and select the desired project from the Project drop-down.
  12. Choose the executable as Restore/restoreHiRes.pl from the Executable drop-down.
  13. Carefully click Save as Template and save the workflow template name that indicates the archive back-end in use such as Restore / RestoreS3 / RestoreSGL

  14. The workflow templates for Archive and Restore flow are now set.
    The above template names need to match the formFor attribute in the dynamic job form you set up earlier as the VAB uses the pattern to load up the appropriate workflow template. 

You are now set to use the Archive module. To re-iterate based on the storage model mentioned at the start of this article you will use different menu options to archive & restore external versus direct ingest assets. The archive/restore statuses will be updated in the metadata fields and will be search-able once Curator has indexed those metadata values.