ID: 13451

Print Friendly, PDF & Email

Getting started with AI

Zoom 7.5 comes with the capability to integrate with Artificial Intelligence (AI) providers to analyze files and media in Zoom, store the results from AI, make the results searchable, and display the results in an innovative UI. Data from AI engines can also be used to automatically run tasks based on certain triggers.

 

Before beginning the setup of AI with your Zoom server, please ensure these prerequisites are met:

  1. Your Zoom Server, Preview Server, and Curator Server are installed and running.
  2. Ensure you have received licenses for the AI Integration and any required engine licenses (such as Veritone) from Evolphin Sales.
  3. You have received your API token, Engine IDs, and Library IDs from your AI provider. In the case of Veritone, they will also let you know the AI types to be used with each of its engines.
  4. For Veritone integration, ensure that you have an exclusive Amazon S3 Bucket to share content with Veritone’s platform. Make sure to add expiration rules to clean up the bucket in a fixed number of days. The S3 Bucket Secret Access Key and Access Key ID should have read, write and list permission on the bucket. Keep the Secret Access Key and Access Key ID handy.
  5. Designate a server machine inside your network as the Hub. This could also be on the existing Zoom Server or Preview Server machine. This machine also needs Java 8 or higher.
  6. Plan on having an exclusive Zoom user account for the Hub. One Zoom user account per Hub will be needed.
  7. Choose a machine to install SQL Server or use an existing SQL Server for job reporting.

 

After these prerequisites are met, follow the below steps:

 

We need to enable AI in Zoom before beginning to configure it. Follow these steps to enable AI for the first time or update the settings later:

  1. Open the Web Management Console for your Zoom Server.
    ex. http://localhost:8443 or http://<zoomserver>:8443
  2. Log in using your admin credentials.
  3. In the Admin Menu sidebar, click Server Control Panel under the System node.
  4. Click AI Settings on the Server Control Panel page.

  5. When enabling for the first time, click Enable AI service and then click Save.
  6. Reload the browser page. A new node called AI Configuration is added to the Admin Menu sidebar.

  7. Open the Server Control Panel under the System node again. Now, click AI Settings on the Server Control Panel page to finish configuring AI in Zoom.
  8. If needed, then you can additionally set other AI settings here:
    1. Metadata AI data limit: controls the number of AI entries returned from the server in a single command for a single asset. Setting this value too high will affect the client performance. The default is 1000. We recommend setting it to a higher value if you are not seeing all the results. This value can easily be set to 10000.
    2. Metadata AI data group by: by default, Metadata AI data is grouped by AI Type, such as in the Zoom Web Client. For Zoom 7.5, the only values supported are type and engineName. Please use the exact case as mentioned here.
    3. Metadata AI data sort by: by default, Metadata AI data is sorted by values present in them. For Zoom 7.5, only value is supported as the sorting order.
    4. Maximum AI json file size: controls the maximum amount of AI data that can be saved on the Zoom server per asset in bytes. The default value is 10 megabytes (10485760 bytes) which is usually enough for storing 100,000 entries of AI data per document. This value is required, as exceeding it will throw an error in the system (if the AI data received is more than this size).
  9.  After updating the values, click Save. Click Ok at the prompt.
  10. The AI settings are saved in Zoom.

 

Zoom stores Hub configuration settings to connect with a Hub Server. These configuration values should be added to Zoom before installing the Hub Server.

If these Hub configuration values need to be added to Zoom after installing the Hub Server, then restart the Hub Server after adding the configuration values.
 

Log in to the Web Management Console and click the Hub Settings Panel under the System node in the Admin Menu sidebar.

 

On the Hub Settings Panel, click Add Hub and specify a name for the new Hub. If you want to copy the settings of an existing Hub to your new Hub, select the existing Hub name from the Copy Configuration From Hub drop-down, otherwise leave it blank. Click Add.

You can also select an existing hub from the list and its settings will be loaded in the Hub Settings Panel. You can delete a Hub configuration by selecting it from in the Hub box and clicking Delete.
For Zoom 7.3, while using multiple Hubs in a multi-location Zoom setup, if you want a Hub to only work for one location, name the Hub as that location (steps shown below).

Update the basic connectivity details for the hub, like Host and Port settings. For location-specific settings, also specify the location where the Hub should work.

Once the basic connectivity parameters are specified, enter values for the advanced configuration. It is categorized into two parts namely the Functional Configuration and the Archive/Restore Configuration. See the sections below to know about the values that should be entered for these:

If you are configuring Hub for AI, only Functional Configuration (SQL Settings and IP Whitelist) details are needed.

 

Functional Configuration

These values are needed for the basic working of the Hub. It covers Process Control values, SQL DB values, and IP Whitelisting values.

 

Process Control

These are configuration parameters that control how responsive the hub will be and must be filled in based on your assessment of the expected load that the hub is supposed to handle. If you are unsure about this, then use the default values.

Click Advance Settings to see additional Process Control options.

  • Max Retry Count: Number of times the failed jobs are retried automatically. 
  • Enable Hub Analytics: Flag to turn on or off the Analytics module on the Hub. 
  • Core Pool Size: Minimum number of threads that will be created by default for job execution.
  • Max Pool Size: Maximum number of threads that can be created for job execution.
  • Queue Blocking Limit: Number of jobs that can be queued in the Hub for processing. 
  • Keep Alive Time: Duration (in minutes) for which resources are kept when the Hub is not servicing any requests; after this duration, the resources are surrendered. 

 

SQL Settings

Hub uses a SQL Server to store job-related details.

The Zoom Server also uses SQL to store process data for some of its other modules too. If it is already configured for your Transcoder and you would like to use the same SQL database for the Hub as well, then simply select the Use Same as Transcoder option. If you want to create a new or separate SQL database for the Hub, then follow instructions here to download your OS-specific installer of MySQL. Run the installer and follow the instructions. Create a user and provide the host, username, password, driver (com.mysql.cj.jdbc.Driver) details here.

 

IP Whitelist

For security reasons, only IPs configured here will be allowed to communicate with the Hub Server. Typically, the Zoom Server sends requests to the Hub. In addition, anyone viewing the Hub Dashboard will also be sending queries and so the IPs of all clients who would need access to the Hub Dashboard must also be added here.

Add the list of IPs from which Hub will entertain requests.

If this is left blank, then the Hub will service requests from all machines.
 
 

Archive/Restore Configuration

These values are needed for the handling of Archive jobs with the Hub. It covers paths and projects covered under various tiers.

In this section, the details of all destination tiers are added.

All the paths should be with respect to the Hub Server.
 

FS Tier 1

  1. Default: Comma-separated list of TPM (Third Party Mount point) paths whose assets are expected to be archived using the current Hub.  
  2. Direct Asset Archive DB Mount Path: The default location where the Zoom Server archives the direct-ingest assets to. 
  3. Project-wise TPM Mapping: Project-specific archive destination locations as configured in the Zoom Server archive settings. 

FS Tier 2

  1. Default: Comma-separated list of TPM paths whose assets are expected to be archived using the current Hub.  
  2. Project-wise TPM Mapping: Project-specific archive destination locations for tier-2 archive

S3 Tier 2

Enter the details of the Amazon S3 bucket like the secret key, access key, and region. Here it is also possible to define unique S3 buckets for specific Zoom projects. For projects not assigned a unique bucket, the default one will be used. 

Click Save Hub after configuring the necessary details across various panels.

 

Any change in the Hub settings made via the Web Management Console requires the Hub to be restarted to take effect. Restart the Evolphin Job Hub service to see the changes.
Please be very careful when you try to edit the TPM paths or the Tier details. If you archive assets to a certain S3 bucket or a file-system location, and the configuration is changed after that then there is no way of accessing these assets from the previously-stored location. Thereafter, any attempt to restore assets from the lost configuration will fail with a data loss error. This will not be able to be corrected without a significant amount of manual intervention and copying of data. 

 

The Hub Server is installed to manage jobs for various Zoom modules. It can be used to manage Hierarchical Archive or AI jobs for Zoom.

You can install it on any server in your Zoom network. You need to have the Hub installer from Evolphin Support to proceed.

Hub is installed after Hub configuration has been added into Zoom.

 

Installing the Hub

Extract the shared Hub installer zip to any path on your designated Hub Server machine and follow the steps below to install the Hub:

On Windows

  1. Open Command Line as an administrator.
  2. Navigate to the /bin folder inside the extracted Hub build files.
  3. Run Hub install (this will register Evolphin Job Hub as a service)
  4. Run Hub start (this will start the Hub Server)
    To stop use Hub stop; for restart use Hub restart. To remove this service use Hub remove.

On Linux

  1. Navigate to the ../EJH/bin
  2. Run ./Hub start (this will start Hub server)
    To stop use ./Hub stop, for restart use ./Hub restart and to remove this service use ./Hub remove.
Hub runs as a service called Evolphin Job Hub on the Hub Server.

 

Registering with Zoom

After installing the Hub using the steps above, open the Hub Configuration page using the URL http://[HubIP]:[HubPort]/config/Hub-config in your browser to configure the remaining details and register the Hub with you Zoom Server. The Hub IP and port are the same as configured in the Hub Settings Panel inside the Web Management Console earlier.

Update the basic connectivity details like the Hub Host and Hub Port.

Update the Location with the location of the hub as specified in the Web Management Console’s Hub Settings Panel.

Update the Zoom Server’s Web Management Console’s URL under Webmin URL.

In order to run search queries on Zoom and to perform other operations throughout the course of the archive/restore operations, a dedicated Zoom user account is required. Update that Zoom User and Zoom Password here.

Select Enable Location Based Hub if you want this Hub to service archive/restore requests only on the assets that are in the same location as the Hub. If this is not selected, then the Hub will service all assets that need to be processed if it has physical access to them even if they are in a different location.

Click Save.

The Hub is now registered with Zoom. If needed, you can modify these settings here on the Hub Config page and update other Hub configuration parameters through the Zoom Web Management Console.

 

In particular, the DB settings could be updated using the file db-config.properties.

Open the conf folder inside the Hub installation directory. Inside the conf folder, locate db-config.properties. The values for these properties should be set:

    1. jdbc.url – URL of your SQL Server with the port.
    2. jdbc.user – Username of the SQL server. This user must have full access to the SQL server at least to the database named hub.
    3. jdbc.password – Password of the SQL user.
    4. jdbc.param – 

Save the file and restart the Hub.

 

Once you have configured Hub in Zoom and installed your Hub Server, you can add Veritone settings to it.

On the Hub Server where you want to run AI jobs, open the conf folder inside the Hub installation directory. Inside the conf folder, locate ai-spec.xml.

<AiSpec xmlSpecVersion=”4″>
<ScratchDirSpec>
<scratchDir>/home/evolphin/.ejh/tmp/ai</scratchDir>
</ScratchDirSpec>
<ExecutorSpec>
<maxThreadCount>4</maxThreadCount>
<maxRetryCount>5</maxRetryCount>
<TimeoutSpec>
<timeout>60</timeout>
<timeUnit>SECONDS</timeUnit>
</TimeoutSpec>
<queueSize>100</queueSize>
</ExecutorSpec>
<previewServerUrl>http://localhost:8873</previewServerUrl>
<PreviewFetchHttpCallTimeout>
<timeout>60</timeout>
<timeUnit>SECONDS</timeUnit>
</PreviewFetchHttpCallTimeout>
<RekognitionSpec>
<defaultMaxLabel>100</defaultMaxLabel>
<defaultMinConfidence>0.5</defaultMinConfidence>
<awsDefaultRegion>us-west-2</awsDefaultRegion>
</RekognitionSpec>
<batchSize>100</batchSize>
<VeritoneSpec>
<queriesDirectory>veritone-queries</queriesDirectory>
<awsAccessKeyId>XYZ</awsAccessKeyId>
<secretAccessKey>ABCDEF</secretAccessKey>
<bucketName>buckername</bucketName>
<region>bucker-region</region>
<defaultPublicLinkExpiryInDays>7</defaultPublicLinkExpiryInDays>
<defaultMinConfidence>0.0</defaultMinConfidence>
<minimumValueLength>0</minimumValueLength>
<maximumVeritoneJobs>2</maximumVeritoneJobs>
</VeritoneSpec>
<AiJobCountLimits>
<amazonAiProviderLimit>0</amazonAiProviderLimit>
<veritoneAiProviderLimit>10</veritoneAiProviderLimit>
</AiJobCountLimits>
<rrnToStartPollingFrom>0</rrnToStartPollingFrom>
</AiSpec>

 

You need to set a few tags here:

  • scratchDir – set this for a temporary directory which is used to hold data in an intermediate state. For example, file proxies are kept in this directory until they are sent for AI analysis. Another use of it is to store AI data in the intermediate state. It is recommended to point this directory to a large storage volume where hundreds of proxy files can be stored easily. These files will be removed after completion of a job, but enough space should be provisioned for the intermediate time.
  • maxThreadCount – set this for the maximum number of AI jobs that can be executed in parallel.
  • maxRetryCount- set this for the maximum number of times a job should be retried on failure before marking it as failed.
  • previewServerUrl – this tag should point to your Preview Server’s URL. Without populating this tag, the Hub will not be able to download the proxies for AI analysis.
  • bucketName inside veritoneSpec – you need to specify the name of the S3 bucket for storing proxies for AI analysis. The hub needs a temporary staging S3 bucket to upload and store proxies. This is only required when Veritone is used as an AI connector.
  • awsAccessKeyId inside veritoneSpec – specify the Access Key ID for the S3 bucket specified above.
  • secretAccessKey inside veritoneSpec – specify the Secret Access Key for accessing the S3 bucket.
  • region – set the AWS region where the S3 bucket is hosted.
  • defaultPublicLinkExpiryInDays – set this as the number of days after which data won’t be available in the bucket. This is the same as the number of days configured for the expiration rule on this bucket inside lifecycle rules.
  • veritoneAiProviderLimit – set this property to limit the total number of jobs that the Hub will send to Veritone for processing. The Hub will stop processing more files after reaching this limit. This is used to prevent going over the allotted number of jobs for billing purposes.
  • maximumVeritoneJobs – controls the maximum number of Veritone AI jobs to run at one time. If these many jobs are running on Veritone, no further jobs will be scheduled.

 

Save and close the file.

Restart your Hub by running Hub restart from the bin folder inside Hub’s installation directory.

 

After setting up Hub Server and configuring Veritone settings on it, we also need to configure detailed Veritone parameters in Zoom using the Web Management Console. We need to set Veritone as the AI Provider, provide configurations for AI engines, and then map AI rules to AI engines. Follow these steps:

 

AI Providers

You need to add one or more AI providers to work with Zoom. Follow these steps to add a provider:

  1. Log in to the Web Management Console and click AI Providers under the AI Configuration node in the Admin Menu sidebar. You need to add an AI Provider here.

  2. Inside the AI Providers panel, click Add and choose Veritone from the list of AI Providers.

  3. Select the newly added row for Veritone. Its details will be loaded in the right side Information panel.

  4. Add/update these values in the Information Panel:
    1. url – enter the URL to access Veritone services. It should be the exact value given by Veritone. For example, https://uk.api.veritone.com/v3/graphql  OR  https://api.veritone.com/v3/graphql 
    2. token – specify the API token which will be used for authentication with Veritone. Without the token, Zoom can not authenticate to access AI services. The Veritone API token is generated from Veritone’s user interface. For more details, check Veritone’s documentation at https://docs.veritone.com/#/apis/authentication.
    3. enable – this is true by default to enable this provider. Setting this value to false will disable this provider in Zoom, so it will not be used further. If this value is missing in the configuration files, it will still be treated as true.
      The values specified here are all case-sensitive.
      For other providers, different sets of values may be needed.
  5. After updating the provider values, move to configure this provider on the AI Configurations page in the next section.

 

AI Configurations

You need to add one configuration per AI Engine provided by your AI Provider. You would get the Engine ID and type from them for each AI engine. Follow these steps to add the necessary configurations:

  1. In the Web Management Console, click Configurations under the AI Configuration node in the Admin Menu sidebar. You need to add AI configurations here.

  2. Inside the Configurations panel, click Add to add a new configuration. Provide a name for the new configuration.

  3. Select the newly added row for the new configuration. Its details will be loaded in the right side Properties panel.

  4. Choose an AI provider from the AI Providers dropdown box above the Properties panel. Choose the provider, Veritone, that we had added in the previous section.
  5. Click Add in the Properties panel for each property.

  6. You need to fill Label and Value for each property at the minimum. The Description is optional.
    For example, for the AI Type property, you should add the Label as type and Value as its AI type. So, Label will be type and Value may be transcription.
  7. Each configuration that you add should have exactly one AI Engine and one AI Type property in it. Some configurations will also need additional properties depending on the AI Engine that is being configured. Each group of AI Engine and AI Type together should be exclusively configured in one configuration only, and these cannot be repeated. Add these properties for the new configuration:
    1. type – add the AI type that should be associated with this configuration.
    2. veritoneEngineId – for Veritone provider, add the engine ID that will be associated with this configuration. It should be the exact same value as provided by Veritone.
    3. payload:libraryId – the Library ID can be specified if it is required by the engine. The given library will be passed as an argument to the engine. Libraries are used at Veritone to train engines to look for specific faces, celebrities, logos, or objects to identify in the footage.
    4. minConfidence – specify the confidence level in the range from 0.00 to 1.00. Only data that has a confidence level equal or higher will be added to the Zoom database. The rest will be discarded. The default value of 0 is used when this property is not specified.
    5. minimumValueLength – specify the minimum length of value to save with Zoom. Any value which has a lesser number of characters than this value will be ignored. The default value is 0, meaning all values from Veritone AI engines are pushed and saved with the Zoom server. This default is used when no value is set for this property.
    6. enable – setting this property to false here will disable this AI Engine and no further jobs will be created using it.
      Setting enable as false for an AI Provider will disable the provider and all the engines associated with it. This was covered in the previous section about AI Providers.
    7. payload:<key> – specify this property if there is a need to pass extra arguments to the engine. To create an argument for an engine, prefix it with payload: (payload + colon).
      For example, to pass a debug=true argument to the engine, the Label should be entered as payload:debug. The value should be true.
    8. maxJoinWords – specify this property to join AI data sent by the AI Engine. Sometimes, AI data is too granular, such as with transcription engines returning one word every few milliseconds. Using this property, we can join the AI data every few words. This value has to be greater than one to have any effect.
    9. maxJoinWordsRange – similar to maxJoinWords, specify this property to join AI data based on the number of seconds elapsed. Its value has to be greater than one to have any effect.
      The properties minimumValueLength, maxJoinWords, and maxJoinWordsRange are optional.
      All values specified here are case-sensitive.
      maxJoinWords and maxJoinWordsRange, if set greater than one, will affect any AI engine data, not just transcription.
  8. After updating the configurations for engines inside your AI providers, move to map the engines with configurations on the AI-Mappings page in the next section. 

 

AI-Mapping

Finally, you need to map AI engines with Zoom, to control what assets will be analyzed by each engine. Follow these steps to map asset rules to an engine:

  1. In the Web Management Console, click Mapping under the AI Configuration node in the Admin Menu sidebar.

  2. Click Add to add a new mapping rule.

  3. Specify these values:
    1. Rule Name: specify a name for the mapping rule.
    2. File Format: specify a comma-separated list of file extensions associated with this mapping. For e.g.: MP4, MXF. Only files matching the given extension(s) will be sent for analysis to the AI Engine.
    3. Engine: choose an AI engine for this mapping. Each mapping needs an engine. The associated engine will be used to extract the AI data as per the rules specified in this mapping.
    4. Metadata Rule: specify metadata key-value pairs to be matched for selecting a file. The metadata properties can be specified in this field, and only those assets that have these properties will be sent for AI analysis.
      Properties can be specified like:

      1. IPTC_City=Milan (Assets with city value Milan are only selected).
      2. IPTC_COUNTRY=Italy,AND,IPTC_Season=2019 (Assets with the country Italy and Season value 2019 will be selected for this engine)
      Multiple metadata values should be separated by commas. Only AND/OR combinations are supported.
      Value comparison for metadata properties is case insensitive, which means Zoom Server will match Italy with italy.
    5. Paths: specify the path to match. Only files present in this path are selected for AI analysis.
    6. Run Once Per Asset: if this is enabled then any asset is analyzed with AI only once throughout its lifetime, even if it is moved, renamed, or a new revision is checked in.
      Rule name and Engine cannot be empty. Any other field left blank will match everything. For example, leaving Paths blank will match all paths on the Zoom Server.
  4. Click Save to save the mapping rule.

The AI configuration in Zoom is now complete.

 

AI Types supported in Zoom

 Zoom supports a fixed number of AI types. AI type is the way data is matched by the AI engine. These AI types are provided by your AI provider. These are later configured on Zoom as shown in the AI Configurations section above. The following AI types are supported in Zoom:

  1. face – denotes and categorizes AI data related to human faces. This data has various information related to faces, such as age, emotions, race, etc. 
  2. object – identifies and names objects and items in the real world, like glasses, bicycles, crayons, ties, suits, etc.
  3. text – denotes extracted text (like OCR) from images or videos.
  4. celeb – denotes well-known celebrities and is attached to their name.
  5. transcription – this is used when speech (audio) is transcribed from an audio or video file.

You should check with your AI provider about the AI types that they will provide with their AI engines.