Evolphin Response to CVE-2021-44228 (Log4j)

By /

At Evolphin, our engineering team has conducted preliminary investigations and have concluded that this vulnerability does not DIRECTLY affect our flagship product Zoom. However, like many other software products, Zoom uses several third-party software like Jetty, Solr etc. which use Log4j internally, and may have exposure to this vulnerability. Therefore as a matter of caution, we are assuming that our product deployments, as of now, are at risk of this vulnerability.

Mitigation Measures

  • All cloud services of customers hosted by Evolphin at AWS have been updated with a modified version of the log4j-core-*.jar from which the problem class has been removed. This is as per the recommendations outlined in this article: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228
  • All customers who have on-premises deployment, are being notified about the issue along with recommendations to address this issue. Check the section below for detailed instructions.  
  • All customers (cloud as well as on-premise) are being advised to apply the mitigation steps at their users’ client machines as well. Even though the client machines may not have a high level of security risk in this case, there is still a potential threat and therefore must be addressed. 
  • Additionally, as we are using an up-to-date Java 8 version, we are at a reduced level of risk as the recent versions of Java disable execution of potentially malicious code in some of the scenarios related to this vulnerability.

Steps for Mitigation

Ensure that you carry out the steps on all machines where the various Zoom-related services may be running. For example:

  • Zoom MAM server
  • Preview servers
  • Curator servers
  • Evolphin Job Hubs 
  • Zoom Client Proxy

Remember to execute these steps on a staging or test/development setup if you have one.

For Zoom desktop client installations, only the Zoom 7.6.X, 8.1.x and 8.2.0 releases are affected.

For Zoom servers

Please use any one of the following 2 approaches to apply risk mitigation measures on your Zoom server installations.

Approach 1: Directly remove the problem class file from the library

Approach 1-A: [Linux]

On a Linux server deployment, you could directly patch the log4j library at your end, using the command given below. This is the same command that was recommended by Apache (and others) to mitigate the risk, and what Evolphin followed to resolve the problem on our managed cloud services.

  1. Open a shell on the linux server where the Zoom services are running
  2. Stop all Zoom-related processes. 
  3. Run the following command on the shell:
    • find . -name "log4j-core-*.jar" -print -exec zip -q -d {} org/apache/logging/log4j/core/lookup/JndiLookup.class \;
  4. Restart the various Zoom services

Approach 1-B: [Windows]

For deployments running on Windows machines, please use the Winzip or Winrar tool to remove the problem class file.

  1. Stop all Zoom-related processes. 
  2. Search and locate the problem log4j-core-*.jar file in the Zoom installation directory. 
  3. Open the jar file using Winzip / Winrar utility. 
  4. Navigate the folder structure to org/apache/logging/log4j/core/lookup/
  5. Delete the JndiLookup.class 
  6. Save the jar file. 
  7. Restart the processes.

Approach 2: Apply the library patch provided by Evolphin [Linux, Windows]

Note: The patched library files are only available for the latest Zoom versions [8.1 and above].
In the Zoom installation, there are two versions of the problem log4j-core-*.jar file that are packaged. One is used by the Curator service, and, the other by the Job Hub. Both these files must be replaced.

  1. Download the patch zip from here: log4j-mitigation.zip
  2. Unzip the patch zip file into a temporary directory
  3. Stop all Zoom-related processes
  4. Navigate to the installation path. 
    • From the patch zip file, copy curator/log4j-core-2.11.2.jar to [INSTALLATION_FOLDER]/lib/curator/log4j-core-2.11.2.jar # replacing the file already present there. 
    • From the patch zip file, copy hub/log4j-core-2.14.0.jar to [INSTALLATION_FOLDER]/ejh/lib/log4j-core-2.14.0.jar # replacing the file already present there.
  5. Restart the processes

NOTE: 
DO NOT KEEP THE OLD JAR IN THIS PATH BY RENAMING IT TO -OLD.JAR OR -BKP.JAR ETC. REPLACE THE OLD JAR WITH THE NEW JAR FROM THE PATCH ZIP FILE.

NOTE: 

The default INSTALLATION_FOLDER for various Operating Systems are given below for quick reference. 

Linux

Zoom – <user.home>/zoom/

Windows 

Zoom – C:\Program Files (x86)\Evolphin\DAM\

Hub – C:\Program Files (x86)\Evolphin\DAM\ejh\

For Zoom desktop clients

Please use any one of the following 3 approaches, depending on the applicable OS and Zoom version, to apply risk mitigation measures on your Zoom client installations. This is necessary only for Zoom client versions 7.6.x, 8.1.x and 8.2.0.

Approach 1: Directly remove the problem class file from the library [Windows]

Approach 1-A: [MacOS with Zoom 7.6.x, 8.1.x only]

On a MacOS desktop client with Zoom version 7.6.x and 8.1.x deployed, you could directly patch the log4j library at your end, using the command given below. This is the same command that was recommended by Apache (and others) to mitigate the risk, and what Evolphin followed to resolve the problem on our managed cloud services.

  1. Open the Terminal app on the MacOS client where the Zoom desktop client is installed.
  2. Run the following command on the Terminal to change the directory to the Zoom [INSTALLATION_FOLDER]# which contains the problem log4j-core-*.jar file.
    • cd /Applications/Evolphin/zoom/Resources/ejh/lib/
  3. Run the following command on the Terminal:
    • find . -name "log4j-core-*.jar" -print -exec zip -q -d {} org/apache/logging/log4j/core/lookup/JndiLookup.class \;
  4. Restart all the Zoom client processes, including the Client Proxy and C3 Hub.

NOTE:
After this is done on one machine, copy the same modified jar file to all your MacOS Zoom client machines, and, restart all the Zoom client processes on each.

Approach 1-B: [Windows]

For clients running on Windows machines, please use the Winzip or Winrar tool to remove the problem class file.

  1. Search and locate the problem log4j-core-*.jar file in the Zoom installation directory.
    • Typically found in C:\Program Files (x86)\Evolphin\DAM\ejh\lib\
  2. Open the jar file using Winzip / Winrar utility. 
  3. Navigate the folder structure to org/apache/logging/log4j/core/lookup/
  4. Delete the JndiLookup.class 
  5. Save the jar file. 
  6. Restart all the Zoom client processes, including the Client Proxy and C3 Hub.

NOTE:
After this is done on one machine, copy the same modified jar file to all your Windows Zoom client machines, and, restart all the Zoom client processes on each.

Approach 2: Apply the library patch provided by Evolphin [Windows with Zoom 8.1.x, 8.2.0; MacOS with Zoom 8.1.x only]

Note: The patched library files are only available for the latest Zoom versions [8.1 and above].
On a Zoom client installation only one log4j-core-*.jar file is packaged. This is the one used by the C3 Hub. Only this needs to be replaced with the one present in the patched zip provided below.

  1. Download the patch zip from here: log4j-mitigation.zip
  2. Unzip the patch zip file into a temporary directory
  3. Navigate to the Zoom client [INSTALLATION_FOLDER] on Windows^ or MacOS#.
  4. From the patch zip file, copy the file hub/log4j-core-2.14.0.jar to [INSTALLATION_FOLDER]/ejh/lib/log4j-core-2.14.0.jar, replacing the file already present there.
  5. Restart all the Zoom client processes, including Client Proxy and the C3 Hub.

NOTE: 
DO NOT KEEP THE OLD JAR IN THIS PATH BY RENAMING IT TO -OLD.JAR OR -BKP.JAR ETC. REPLACE THE OLD JAR WITH THE NEW JAR FROM THE PATCH ZIP FILE.

Approach 3: Add the configuration to disable lookups that expose the vulnerability [MacOS with Zoom 8.2.0]

This approach has been deprecated since it was originally published because it does not completely address the vulnerability. Therefore this is to be used only on the client side, and only when the above two approaches are not possible (as in the case of Zoom 8.0.1 and Zoom 8.2 versions of Mac clients where due to notarization, it is not possible to modify the library files.)

As per the mitigation advice from Apache, the vulnerability can be removed by adding a specific configuration directive in all the processes. On the Zoom client there are multiple processes running, including the Client Proxy and the C3 Hub. The configuration change must be made for all the processes.

Part 1

  1. Open [CONF_FOLDER_FOR_C3_HUB]/hub-vmoptions.conf #
    • If this file does not exist:
      • Copy the file [INSTALLATION_FOLDER]/ejh/conf/hub-vmoptions.conf.sample # to the configuration location above.
      • Thus creating the file without the .sample extension.
  2. Search for the last occurrence of the tag “wrapper.java.additional.<n>=” which has a value on the right-hand-side of the “=” sign. 
    • Note the number <n> present just before the “=” sign.  
  3. Insert the following on the next line. Here, the <nn> value must be one more than the <n> value found in the previous line. 
    • wrapper.java.additional.<nn>=-Dlog4j2.formatMsgNoLookups=true
  4. Save the file.

Part 2

  1. Open [CONF_FOLDER]/zm.vmoptions #
    • If this file does not exist:
      • Copy the file [INSTALLATION_FOLDER]/conf/zm.vmoptions # to the configuration location above.
  2. Insert the following on a new line
    • -Dlog4j2.formatMsgNoLookups=true
  3. Save the file.

After making all the changes in the configuration files, restart all the Zoom processes, including the Client Proxy and C3 Hub.

NOTE: 

The configuration and typical installation folders for various Operating Systems are given below for quick reference. 

MacOS

  • CONF_FOLDER_FOR_C3_HUB –
    • <user.home>/Library/Application Support/Evolphin/ejh/client/conf/
  • CONF_FOLDER –
    • Zoom 8.0.1 and 8.2.0:
      • <user.home>/Library/Application Support/com.evolphin.Zoom/conf/
    • Zoom 7.6.x, 8.0.0 and 8.1.x:
      • /Applications/Evolphin/zoom/Resources/conf/
  • INSTALLATION_FOLDER –
    • Zoom 8.0.1 and 8.2.0:
      • /Applications/Evolphin/zoom/Zoom.framework/Versions/Current/
      • Note: Do NOT modify the files in the Zoom installation folder.
    • Zoom 7.6.x, 8.0.0 and 8.1.x:
      • /Applications/Evolphin/zoom/Resources/

Windows

  • CONF_FOLDER_FOR_C3_HUB –
    • %APPDATA%\Evolphin\ejh\client\conf\
  • CONF_FOLDER –
    • C:\Program Files (x86)\Evolphin\DAM\conf\
  • INSTALLATION_FOLDER –
    • C:\Program Files (x86)\Evolphin\DAM\

References