ID: 1186

Print Friendly, PDF & Email

Configuring Archive Module

An archive module allows the Zoom archive manager to delegate the actual archive/restore operation to a specific archive handler. The following are currently supported:

Archive Module Type Description
Basic Internal/Built-in Supports archiving to a single file system path, without ability to map Zoom project vaults to different file system dirs
FileSystemArchive External Similar to basic + ability to map Zoom project vaults to different file system dirs. Allows each brand/business unit to have their own archive storage
AwsS3 External Supports Object Storage or Cloud based Storage such as Google Cloud Storage, Aws S3, Cloudian S3  
FrontPorch External Supports Oracle FrontPorch DIVA
SGL External Supports SGL FlashNet

Archiving is an optional feature that is available only to deployments that hold an appropriate license. You can check your license information in Zoom’s web-based administration console.

To check the license information, log on to your administration console in a browser at a URL like this: http://zoomserver:8443/

From the navigation menu on the left, click open the “License Management” option under the “System” menu. If you hold a license that allows archive operations you will see a separate “Archive Management” menu in your navigation panel as shown below.

Archive License

 

 

For users who have archive as part of the Zoom license they have purchased, Zoom can archive assets to a single designated archive location using the Basic Module. This folder location must pre-exist at the Zoom DAM server in order for the configuration to work. You can configure location of the Archive by doing the following:

  1. In your preferred web browser, go to Web Administration Console.
    ex. http://localhost:8443 or http://<zoomserver>:8443
  2. Log in using your admin credentials.
  3. In the left sidebar, click on Server Control Panel under the Server section.
  4. Click on Archive Management.
  5. Under Archive Location, set the path to the archive folder as it exists on the Zoom DAM server.
    Ex. e:\zoom\archive\ or /mnt/Archive on Zoom DAM server.

     

     

  6. Restore Threshold: This is the file size in bytes which is a threshold for restore petition. Please check the section on “Restore Petition” for more details. The default threshold is -1.
  7. Click Save.
  8. You will be prompted to restart the server. Click Yes.
  9. Refresh your web browser.

This post explains the steps involved in setting up archiving to external systems like S3 / SGL / DIVA or dedicated File-System partitions. The external systems act as a secondary archive storage for the Zoom internal, and Zoom referenced (or indexed) assets.

Understanding the External Archive Flow

Archive Pre Hook Flow

  1. The user initiates an archive from Asset Browser, the control passes to the archive pre-hook installed on Zoom server. Metadata is updated on the asset as shown in the diagram below. The hook is responsible for marking assets in a  “Pending ..” state for them to be archived by the Archive Cron timer task

Checkpointing

If the asset is a Zoom direct asset (stored in the Zoom database), it needs to be checkpointed before it can be archived. Checkpointing will ensure all the versions are stored in an “Asset ID” file that will be copied to the external archive. Assets that have native or mid-res proxy on an external storage such as a SAN, can be copied without checkpointing but even for these assets the low-res proxy is often checked-in into Zoom database as a direct asset. if so the low-res proxy will also need to be check-pointed in order to archive.

Archive Cron Flow

The archive cron timer task will run periodically and submit assets that are ready for archiving to the external archive system such as S3 or SGL. The assets need to meet the following requirements:

  1. Metadata needs to indicate a “Pending..” status
  2. Direct asset/low-res proxy in Zoom database representing the asset needs to be check-pointed

Monitor Flow (for SGL and other Job Queue bases external archive systems)

Understanding the External Restore Flow

Restore Pre Hook Flow

  1. User initiates  a restore from Asset Browser, the control passes to the restore pre-hook installed on Zoom server. Metadata is updated on the asset as shown in the diagram below. The hook is responsible for marking assets in a  “Pending Restore..” state for them to be restored by the Archive Cron timer task

Archive Cron Flow

The archive cron timer task will run periodically and copy assets that are ready for restore from the external archive system such as S3 or SGL. The assets need to meet the following requirements:

  1. Metadata needs to indicate a “Pending restore..” status

SGL Monitor Cron Flow

For SGL external archive system the following flow is used:

 

Steps to install and configure the Archive components

  1. Install Perl library components.
  2. Install the Archive-specific modules.
  3. Configure the archive and restore pre-hooks 
  4. Update the various configuration files for your deployment.
  5. Configure the recurring archive timer – Cron for Linux and for Windows a timer. 

Installing the Perl libraries and modules

Review and follow the instructions for installing the Perl sdk. You only need Perl and the modules so just follow the instructions for “Perl Installation”. 

If the zip contains an “extras” folder, there is a perlmodules.bat file that you can just run from the command line to install all the modules which include some extras for Archive. If you do not use the script make sure you install Data::UUID 
Note: for Windows use Strawberry Perl 5.24.1.1 

Installing the Archive Modules

Download the archive.zip file from the support site or your mail. 

Usually our config files are defaulted to be organized under a root folder called “zoom-deploy”. 

 

After unzipping you should have the following folder structure. 

<your_root_folder>/

                                ArchiveCron/

                                                     conf/

                                                             config.ini

                                                             log4perl.conf

                                                     archiveCron.bat      Note: only for Windows deployment

                                                     archiveCron.pl

                                ArchivePreHook/

                                                           conf/

                                                                   config.ini

                                                                   log4perl.conf

                                                           archive-hook.bat  Note: only for Windows deployment

                                                           archive-hook.pl

                                lib/  Note: not exploded contains all the modules used by the system

                                logs/

                                        Archive-Cron.log

                                        Archive-Prehook.log

                                        archiveCronBat.log       Note: only for Windows deployment

                                        archiveHiresBat.log       Note: only for Windows deployment with VFX

                                        RestorePre-Hook.log

                                        RestorePreHookBat.log    Note: only for Windows deployment                               

                                RestorePreHook/

                                                            restorePreHook.bat    Note: only for Windows deployment

                                                            restorePreHook.pl

 

 

 

Configuring the Archive / Restore Pre-hooks

  • Login to web-based admin console
  • Open the “Server Control Panel” menu
  • Launch the Archive Settings menu option
  • Check “Enable Archive”
  • Create a folder on your server file system and copy/paste that path into “Archive Location”. 
  • Put the path for the “Pre-script for Archive” in the box. In our example we are using C:/zoom-deploy/ArchivePreHook/archive-hook.bat
  • Put the path for he “Pre-script for Restore” in the box. in our example we are using C:/zoom-deploy/RestorePreHook/RestorePreHook.bat
  • Leave both “Post-script” fields blank
  • Set your email list if you are using one, you can add it at a later date.
  • Set “Limit on Arguments on Command line” to 0. 
  • Click on the fields for archive and restore pre-processing, and point them to the Archive and Restore pre-hook batch files (if Windows) or perl scripts (if Linux). See screenshot below.
  • Save and restart server for the changes to take effect. 

 

Archive Panel view

Configuration Files

It is recommended that before you edit any config.ini file you make a copy and name it something unique first. 

System-wide Common Parameters

For every config.ini, ArchiveCron, ArchivePreHook and RestorePreHook we have common sections; these are detailed below:

[ZOOMSERVER]

serverURL=  http://<your host IP address>:8880 

IP and port for your server

webminURL= http://<your host IP address>:8443

IP and port for your server

serverUsername= <your admin account>

Zoom admin user account to use

serverPassword= <your admin password> Password for the above account

 

 [METADATA]

proxy= <your metadata group>_Proxy Indicates whether or not, an asset is a proxy for a hi-res external asset. [VideoFX]
hiResPath= <your metadata group>_NativeFilePath Indicates the path where the hi-res external asset is placed. [VideoFX]
hiResArchiveStatus <your metadata group>_HiResArchStatus Indicates the current status of the external archive operation on the hi-res asset. [VideoFX]
directArchiveStatus= <your metadata group>_DirectArchivetatus Indicates the current status of the external archive operation on the Zoom assets. [In VideoFX setup, specifically for non-proxy files]
lowresArchiveStatus= <your metadata group_DirectArchiveStatus Indicates the current status of the external archive operation on the low-resolution proxy asset. [VideoFX]
midresArchiveStatus= <your metadata group_MidResArhchiveStatus Indicates the current status of the external archive operation on the mid-res asset. [VideoFX]

 

Archive Pre-Hook config.ini

see notes on common section above

 

[TO ARCHIVE] – this section tells the system which categories of files are to be archived. Set 0 = disable and 1 = enable. 

hires= 0 or 1 Enable to archive hi-res assets
midres= 0 or 1 Enable to archive mid-res assets
lowres= 0 or 1 Enable to archive proxy of external hi-res assets
direct= 0 or 1 Enable to archive assets directly ingested into the Zoom repository

 

ArchiveCron config.ini

see notes on common section above

 

[DEFAULT UNC path]

root_UNC_PATH

Specify the location that the assets must be copied to, if no dedicated path is defined for the project to which an asset belongs

 

[Project UNC map]

<Zoom Project Root Path>  Location where assets belonging to this project must be archived to
   

 

Please use project Root Directory instead of Project Name. The Project Root Dir can be determined from the Zoom Project Settings in Web Administration Console or using Z -> Zoom Repository app from the desktop and mouse over the Project name in the left navigation.

[ARCHIVE]

archiveModule  Can be set to “AwsS3”, “FileSystemArchive”, “SGL”
   

Zoom Server Metadata.xml

If you choose to follow our metadata recommendation, add the following to your metadata.xml and restart the server. 

ARCH metadata example

It is recommended that you make these parameters as mandatory in your settings. 
If you have the extras folder, it is in ARCH_metadata_example.xml

The Cron or Timer Task

A key component of the external archive module is a script that needs to be periodically run continuously, without user intervention. To achieve this, we configure the target script using a Cron tab in Linux, or using the Windows Task Scheduler in Windows. See details below. 

For Linux

When set, this will cycle and call the archiveCron.pl causing archive and/or restore events to run. 

For Linux you set up cron for the user account and the following line can be copied and pasted into your crontab. 

*/1 * * * * <your root directory>/zoom-deploy/ArchiveCron/archiveCron.pl 

Note: the above will run every minute

For Windows 

To setup a task scheduler,  run the following steps as administrator:

  • Open the control panel
  • Choose “System and Security”
  • Under “Administrative Tools”, select “Schedule tasks” 
  • In the “Task Scheduler”, select “Create Task”

windows_task_create

 

  • in the “General” tab seen above, do the following:
  • Provide a Name, example “zoom archive” and description
  • Enter a Location \ do not change
  • Using the “Change User or Group…” Do not change
  • Select elect the “Run whether user is logged on or not”, we want it to run all the time 
  • Configure for <your version of the OS>
  • Now select the “Triggers” tab: and use the “New…” button to create a new task

windows_task_trigger

  • Select “Begin the task” and select “At startup”
  • Select the “Repeat task every” checkbox
  • Choose your duration, change “5 minutes” to “1 minute”
  • Change “for a duration of” to “Indefinitely”
  • Select “Ok”
  • Now using the Actions tab, we need to select the program to run:

windows_task_actions

 

  • Select “Start a program”
  • Browse to <Your root directory> and select the appropriate batch file. 
  • Leave the next two, optional fields blank
  • Select “OK”
  • Restart the system and your task should be active

 

A couple items to note: When ingesting new assets you will not be able to archive them until you checkpoint. Additionally, archived assets will not be removed from primary storage until you checkpoint.