ID: 1612

Print Friendly, PDF & Email

Making Hot Folders Solve Your Problems

At its core, Zoom hot folder ingest will take the assets brought into a specific folder on your machine and put them into a specific location on the Zoom server. But often times, this in itself will not meet your requirements. This post is about embellishing the hot folder set-up to make it serve your requirements fully.

Here is a post that outlines the configuration details of this mechanism, keep this handy for reference: http://help.evolphin.com/docs/v5/administration-docs/hot-folders/

Hot Folder Flow

What really happens once you set up the necessary configuration parameters for the hot folder ingest?

  • Zoom periodically checks the disk location that is marked as the “hot folder”. How often, is controlled by the “Sync every” field of the hot folder settings.
  • If there are files present in that folder, it runs the pre-script if one was configured.
  • After running the pre-script, Zoom goes on to check whether hot folder settings are valid and the contents of that folder are eligible for import just yet based on the “untouched interval” configured. This is to not accidentally import a partial file that is still being written to by another process.
  • After identifying the list of files that are eligible to be imported, Zoom triggers the actual import operation.
Make sure to specify the untouched time interval that is large enough to prevent accidental partial imports. For example, if it takes approximately 20 seconds to copy 1 GB file data to your hot folder location, and you expect that the maximum file size that you will encounter is only 1 GB, then you must specify the untouched interval slightly larger that 20 seconds.
  •  Once the import is successful, the post script will run to perform any post actions if one was configured.

Hot folder is not a manual process, there is no need for someone to trigger it. All you need to do is configure it with the details including how often you want to run this little exercise, and Zoom will take care of everything else.

Metadata on Ingested Files

Zoom supports applying a set of metadata on the files being imported via the hot folder mechanism. The metadata properties can be arranged in the form of

  •  key-value pairs to be applied on all files in the hot folder
  •  CSV format which will specify metadata per file.
The metadata file has to be present at the time the import operation is actually launched.

Hot Folder Pre-Scripts

Pre-script will run before the import of assets will begin provided that running of the script is enabled.The pre-script can control the flow of the hot folder mechanism on the basis of some return values from the script. The following table describes the status codes returned by the pre-script and how the hot folder mechanism responds to those status codes.

Return Status
Description
0 Successful exit of the script. The hot folder mechanism can proceed with the hot folder import.
1 This status will tell the hot folder mechanism to ignore the hot import and the post import  tasks and proceed with re-scheduling of the next run of the hot folder.
2 This status code means that the script failed due to some reason, so the hot folder mechanism will disable the hot folder until enabled by the user from the settings dialog

 

After the pre-script runs, the hot folder mechanism validates all the other input parameters for that hot folder. In case of any error during the validation, the hot folder will be disabled until manually enabled by the user.

 

Input Parameters to the script

The pre-import script gets some parameters as input, which are some of the parameters set for the hot folder. The following tables shows the input parameters to the script and their order:

Order
Name of Parameter
Description
1 Server url The zoom server url where the hot folder will import assets to
2 hot folder Is the hot folder path from where the assets will be picked
3 project Is the project on the zoom server
4 destination path Is the path on the server where the assets will be imported to
5 sync time( in millis) Is the time duration after which hot import will start. Note: The time duration will be in milliseconds
6 untouched time(in millis) Is the asset unmodified time which will also be in milliseconds
7 hot folder username is the username which is used to validate the hot folder at the time of creation. In case no hot folder user is logged in then “No username found” will be send as the value of the parameter

 

The user can use these parameters in the same order in which they are depicted and perform operations using those parameters.

The Role of Pre-scripts

The hot folder ingest mechanism will simply import the set of files found in that location, onto a location in Zoom repository. If this plain-vanilla flavour won’t work for you, you can always add more dimensions to the mechanism by means of the pre-scripts. Here are a few use-cases.

Transforming Assets Before Ingest

The files that are written to the hot folder may be in a format that you don’t prefer, and you want to convert it into another format before you store them in Zoom server. You could write a simple shell or perl script to apply the necessary transformations and configure that script as the pre-script for your hot folder. The actual import will simply pick up all the files that are available in the current folder after the script transformations have been executed.

For example, if a process copies image sequence files onto the hot folder, you can run a script to convert these image sequences into a video and store that also in the hot folder. You may keep the original image sequences or delete them from the script if you don’t want them cluttering up the repository.

 

The hook for the running a script prior to the actual import allows you to cascade many scripts – all you would need to do is string up the scripts you want to invoke in another batch or shell or perl script and configure that in the hot folder.

If you are applying transformations on the files, and you want to make sure that the transformed files will get imported in the current ingest cycle itself, add a waiting mechanism in your script after the transformations are done. This wait interval must exceed the “asset untouched time” in the hot folder settings in order to make the files eligible for import.

Re-organizing Assets Before Ingest

The original process that is copying files onto your hot folder may be putting the files flatly at the top-level. You may want to re-organize these into a multitude of sub-folders, or perhaps rename certain files with the timestamps etc. You could do this re-organization from within a script and set that as the pre-script for this hot folder.

Remember, the ingest mechanism will import into server, all the contents under the hot folder with its sub-folder hierarchy intact.

 

Dynamic Metadata Generation

It is possible that you may want to construct or generate metadata based on the files that reach the hot folder.

For example, you may want to set a metadata property called MEDIA_TYPE and set it to MULTIPAGE_DOCUMENT for PDF files or MS WORD files.

 

If you need to generate metadata dynamically, instead of applying the same old same old values on all the files getting ingested from that particular hot folder, you can control that through the pre-scripts again. The format of the metadata file is a simple key=value pair text file as shown in the previous section.

You can find several examples of Hot Folder pre-ingest scripts here to help with your Hot Folder automation needs.

 

Hot Folder Post-scripts

From Zoom 5.2 and higher, post-script support has been added to the hot folder settings. These post-scripts if configured, will run after the import operation is complete. The post-script will only execute if the Import is successful.

Input Parameters to the script

The post-script gets some parameters as input, which are some of the parameters set for the hot folder. The following tables shows the input parameters to the script and their order:

Order
Name of Parameter
Description
1 Server url The zoom server url where the hot folder will import assets to
2 hot folder Is the hot folder path from where the assets will be picked
3 project Is the project on the zoom server
4 destination path Is the path on the server where the assets will be imported to
5 sync time( in millis) Is the time duration after which hot import will start. Note: The time duration will be in milliseconds
6 untouched time(in millis) Is the asset unmodified time which will also be in milliseconds
7 hot folder username is the username which is used to validate the hot folder at the time of creation. In case no hot folder user is logged in then “No username found” will be send as the value of the parameter

The user can use these parameters in the same order in which they are depicted and perform operations using those parameters.

The Role of Post-scripts

Post-scripts are typically useful for clean-up activities after a successful ingest operation.

Zoom provides a way to move / delete the files that were successfully ingested.

 

However the use of post-scripts is not restricted to this alone. There may be requirements to trigger one or more operations on the availability of new assets into the Zoom server – for example, issuing notifications to other users who are waiting for the assets to get ingested, so they can start working on them. Because the scripts do not dictate what is allowed and what is not, this is really a powerful mechanism that can be crafted into solving very specific and critical problems with the right set of scripts.