ESXiArgs Ransomware: never too early to jump the gun

VMware ESXi is an enterprise-class, type-1 hypervisor that allows organizations to deploy and host several of their virtualized systems running multiple operating systems on a single physical server. If ransomware successfully infected a Windows computer, a single machine is affected but if it infects a VMware ESXi, it hits a whole group of servers at once rendering all the virtual machines (VMs) unusable. Thus, from an attacker’s perspective, infesting this hypervisor is like “hitting n birds with one stone”. It can wreak havoc in any enterprise environment.

Introduction

On February 3, 2023, the CERT-FR and the web hosting provider OVHcloud issued warnings regarding a global ransomware campaign targeting VMware ESXi servers. This ransomware is named “ESXiArgs”. It's been confirmed that the ransomware attack is not exploiting any zero days but rather the “two-year-old” CVE-2021-21974.

OVHcloud has stated that the vulnerability in OpenSLP was exploited as an initial compromise vector but the involvement of CVE-2021-21974 (a heap-overflow vulnerability where a malicious actor residing within the same network segment can trigger this issue in OpenSLP service resulting in remote code execution) has not been confirmed. The involvement of CVE-2022-31696, CVE-2022-31697, CVE-2022-31698, CVE-2022-31699, CVE-2021-21995, CVE-2021-21974, CVE-2020-3992, and CVE-2019-5544 has also been speculated by the media.

In October 2022, Juniper detected a custom python backdoor “vmtools.py" implanted on a VMware ESXi virtualization server. It was a cross-platform python script that got executed on startup and launched a simple web server accepting password-protected POST requests which could either run arbitrary remote commands with results as a webpage or launch a reverse shell on any port as per the need. There is mention of “vmtools.py" in the “encrypt.sh” script so this backdoor might also have a certain role in the infection routine.

On the first day of detection

Approximately 120 ESXi servers were encrypted. But, more than 4000 hosts worldwide have been affected as of now. On 8th Feb, Ransomwhere mentioned, “4 total payments have been made to date for a total of $88k.

Instructions were left by the ransomware on how to restore the files which also contains a bitcoin address where the victims should make the payments. Ransomwhere created a github page containing a list of ESXi ransomware payment addresses.

ESXiArgs ransom note (Source: BleepingComputer)

Here is a sample image of encrypted files we obtained from the bleeping computer forum.

Encrypted files on the victim's machine

VMware released a patch in their advisory on 23rd Feb 2021 regarding multiple CVEs including this one. Most organizations should have already patched their ESXi servers but, patching ESXi can be challenging and requires downtime. So, some organizations may not have updated to a fixed version.

As of 8th February, the second ESXiArgs ransomware wave started. It is now encrypting more extensive amounts of data, making it much harder, if not impossible, to recover encrypted VMware ESXi virtual machines. Preliminary reports indicated OpenSLP vulnerability to be the initial compromise vector but a comment that was posted in the forum stated that the SLP service was disabled on their devices and was still breached and encrypted. Also, the ransom note has been changed:

New ESXiArgs ransom note (Source: BleepingComputer)

In this new ransom note, the bitcoin address has been removed, and now advises the victims to contact the adversaries on TOX with the provided ID.

ESXi hypervisor is too sensitive to be exposed to the internet and can pose significant security risks. But still “85,546” ESXi servers are exposed to the internet as of 20th Feb 2023.

Here is an overview of the data depicting the overall infections so far as of Feb 20, 2023:

ESXiArgs Ransomware Tracker

Affected Products:

1. ESXi versions 7.x prior to ESXi70U1c-17325551

2. ESXi versions 6.7.x prior to ESXi670-202102401-SG

3. ESXi versions 6.5.x prior to ESXi650-202102101-SG

Attack Timeline:

Attack Behaviour

• Initial compromise vector: CVE-2021-21974 (not confirmed)

• The public key used for encryption along with other necessary artifacts are dropped in /tmp/ directory.

• The encryption process targets files with “.vmdk”, “.vmx”, “.vmxf”, “.vmsd”, “.vmsn”, “.vswp”, “.vmss”, “.nvram”, ”*.vmem” extension and leaves the .args extension after encryption.

• The malware tries shutting down VMs by killing the VMX process to unlock the files.

• The malware creates “argsfile” to store arguments passed to the “encrypt” binary (number of MB to skip, number of MB in encryption block, file size)

• No data exfiltration observed so far.

• Some cases show partially failed encryptions, allowing the victim to recover data with ease.

Quick Response

BleepingComputer has created a dedicated forum where people are reporting their experiences and receiving help to recover their machines. So, you can also find some relevant information here. Stay updated with this page.

For the first wave, security researcher Enes Sonmez created a recovery guide that allows admins to rebuild their virtual machines and recover their data for free.

Also, CISA has released a recovery script to allow organizations to attempt recovery of virtual machines affected by the ESXiArgs ransomware attacks.

Quick Mitigations for ESXiArgs ransomware

1. Check your version of ESXi and stay updated with the main VMSA page to apply the latest security patch. Here are the latest available supported releases of vSphere components. You can look at some tips regarding patching here.

2. Ensure your data is backed up and only necessary services are active. Make sure they are filtered with ACL to only trusted IP addresses.

3. Keep monitoring your system for abnormal behaviors and keep necessary defenses in place. One of our security researchers Bhabesh Raj had also published a sigma rule for detecting exploitation of VSphere Remote Code Execution vulnerability as described in CVE-2021-21972 so it can also be used.

4. Disable the OpenSLP service on the server if not used, you can follow this link.

5. Do not expose your ESXi instances directly to the internet if at all possible, place filters and additional security controls in front of them and review those controls for effectiveness.

Technical Analysis

Moving on to the technical analysis part, this ransomware drops “encrypt” an encryption program, and “encrypt.sh” the associated shell script and other artifacts in the victim machine. Upon inspecting the shell script we have observed the following.

First, it creates a staging directory and places its artifacts in that directory. It then lists out the virtual machine's configuration files and modifies the path to the virtual disk and swap files appending “1” before the file extensions and force-terminates all running virtual machines.

Then it moves towards encryption. It searches for the file with the specific extensions i.e: “.vmdk”, “.vmx”, “.vmxf”, “.vmsd”, “.vmsn”, “.vswp”, “.vmss”, “.nvram”, and “.vmem” and then calculates their size so that the files smaller than 128MB are encrypted in a single step and those larger than 128MB are encrypted in multiple steps using the binary executable “encrypt” that the ransomware dropped with an argument file “public.pem” in “/tmp/” directory.

Encryption

I‍n the new variant everything is the same, however only the block for 'size_step=0' in “encrypt.sh” has been removed and size_step is set to 1 so that the encryptor alternates between encrypting and skipping 1 MB of data.

Encryption v2

After the encryption process, “index.html” and “message of the day(motd)” are replaced by the ransom note, and all the files with “.log” extension are removed.

Then, It lists and checks the installed versions of VMware. If the number of builds with version “7.x” not equal to 0, it removes the last and the first 8 lines of this “/var/spool/cron/crontabs/root” and sets its modification time same as "/usr/lib/vmware/busybox/bin/busybox" to remove its tracks. However, if it is equal to 0, it removes only the first line of the “/sbin/hostd-probe” .

In the final step, It modifies and removes multiple files and their artifacts to evade detection. Among those files ”/store/packages/vmtools.py,” is the python backdoor, mentioned in this blog earlier.

After the clean-up operations, It makes sure the changes in the configuration files will be intact and deletes itself. Finally, it starts the SSH service, possibly in the event that the threat actor wants to return to the server.

Removing artifacts from the victim machine

Hunting ESXiArg with Logpoint

There has been almost no talks on its detection so far. While detecting is a first step, it is also a somewhat passive approach. Organizations need to adopt a more proactive posture in cyber threat hunting. For which Logpoint comes into play. We have tried to simulate the attack using “VMware ESXi 6.7.0” and used its log files for hunting this ransomware.

Log sources required:

• Webserver
• Web Proxy
• VMware ESXi

The initial access vector seems to be the exploitation of a heap-overflow vulnerability resulting in remote code execution but the specific CVE is not confirmed yet. So, Analysts should be on the lookout for the exploitation of potential attack vectors. The following query can help defenders to identify the exploitation of a VSphere Remote Code Execution vulnerability.

OVA uploads to the VSphere appliance might trigger a false positive.

The list of malicious IPs involved in the ransomware activity can be listed in the “ESXIARGS_MALICIOUS_IPS” list and the list can be further used for detecting the events where the suspicious IP addresses are involved.

Also, you can collect malicious hashes from sources like varonis or fastfire, which can be used to create a list say “ESXIARGS_MALICIOUS_HASH_LIST”. Now you can also look out for malicious hashes.

In a report published by Juniper, they observed the following one-liner reverse shell used by the adversaries:

Analysts can look out for this particular case where this specific one-liner is used,

Apart from Detection through IoCs, We can look for other suspicious activities that can indicate the compromise. VMware recommends disabling SSH on vSphere and it is also disabled by default. So, analysts can look for events where SSH is enabled. It can indicate the attacker trying to access the ESXi shell through SSH. However, legitimate enabling of SSH might also be listed in this result. ESXi 5.0 and higher maintains a history of all commands entered in the ESXi Shell and also logs the enable and disable of SSH. All of them are maintained in the shell.log file.

Adversaries might try brute-forcing or password-spraying against ESXi. Look for multiple failed SSH logins followed by a successful one. Use logs stored in “/var/log/vobd.log” with the following query to detect whether adversaries succeeded in a password-spraying attack on SSH.

Also, look out for suspicious failed remote login attempts. Account locking is supported for access through SSH and through the vSphere Web Services SDK. By default, a maximum of five failed attempts is allowed before the account is locked. The account is unlocked after 15 minutes by default. However, an attacker might persistently try multiple times to log in, so analysts should monitor this as well.

ESXi provides a web interface for admins to log in and manage their system. However, adversaries again might attempt brute-forcing the web login page as well. Analysts should look out for events with multiple failed authentications followed by a successful one. The logs are stored in “/var/log/hostd.log”

This can be further broken down and viewed more clearly with the following query. If we see multiple login failures from the same source address followed by a successful login, it can indicate a suspicious action. Whether wrong password or username, both are flagged authentication fails with the message “Rejected password for user …” and successful authentication with the message “Accepted password for user …” So analysts need to carefully monitor for such actions.

“/var/log/auth.log” also stores logs regarding the authentication events. From this also, defenders can lookout for suspicious logins. Multiple failed logins with different usernames followed by a successful login with the same source address might also indicate malicious behavior.

For further detail, you can view the attempted usernames followed by a successful ones using the following query:

As previously mentioned, after gaining access to the system, adversaries are seen dropping their artifacts and running malicious scripts.

On Thursday, September 29th, Mandiant published information on malware they discovered in the wild that leverages unsigned VIBs to install backdoors on a compromised ESXi host. Analysts are suggested to look for the command line used to force the installation of the malicious VIB and the corresponding command line used to verify the VIB signature in ESXi.

Among their multiple artifacts, “encrypt.sh” is a shell script that does the main work. One of the actions taken place is the forceful shutdown of multiple Virtual Machines. A forceful shutdown of multiple VMs in a short span of time indicates a malicious activity whose log is stored in “/var/log/hostd.log". So Analysts are suggested to look for this particular pattern as well.

You can view which machines were shut down with this query:

Responding to ESXiArgs Ransomware using Logpoint SOAR

Logpoint SOAR is by far one of the most useful tools for organizations to investigate and respond to ransomware attacks. With available playbooks, we can respond to the situation and prevent further damage to the system. Among many, the following playbook can be useful in instant response to this ransomware.

Block indicators
This playbook checks if any IP, domain, URL, or host exists in a list of IoCs, blocks them, and adds them to the blocked list preventing damage to the system.

Block Indicators

Tips for Ransomware Resilience with VMware Cloud Infrastructure

1. Always be prepared. Assuming that you will be attacked ensures they're covered in your organization’s disaster recovery & business continuity planning.

2. Use centralized logging of ESXi environments. Ensure all ESXi host and vCenter Server logs are being forwarded to the organization’s SIEM solution like Logpoint to provide visibility into security events.

3. Classify your systems according to their business criticality. It will help you assign correct security controls to systems, assess dependencies, and prioritize work during an incident.

4. Simulate these kinds of incidents every now and then, and make your staff aware of such situations.

5. Patching the vCenter server might affect the management interface. So, systems that connect to vSphere should be set to automatically reconnect and resume operations when vCenter Server becomes available again.

6. Manage the infrastructures with a separate dedicated admin account not used for regular tasks so that, if the attackers use stolen credentials, it shows up in the failed login attempt logs.

7. Avoid using authorization groups because attackers can add themselves to the group after they compromise the system and log in to systems they should not otherwise have access to.

8. Ensure that only specific authorized devices can access the management interfaces and systems. Also, avoid/restrict causal or malicious access to it and monitor it as well.

9. Manage vSphere through vCenter Server with Role-Based Access Control (RBAC) permission model and by default disable SSH and enable normal Lockdown mode in ESXi.

10. Use multifactor authentication and time-based access where possible.

11. Keep your backup systems separate from corporate identity systems. It should be an island, extremely isolated, managed by few staff, monitored heavily, but also protected against outside influence from centralized monitoring and management systems.

12. Use immutable backups and/or backup “air gaps” where possible.

13. Enable the Secureboot feature in ESXi to mitigate the risk of malicious actors persisting on a compromised ESXi host via malicious VIB installation.

In addition:

You can also follow these guidelines from VMware to harden your system.

1. Designing Infrastructure to Defeat Ransomware.

2. Ransomware: Defense in Depth with VMware (white paper).

3. VMware Cloud DR Ransomware Recovery Guide.

4. Cloud Infrastructure Security Configuration Guides.

5. Protecting vSphere From Specialized Malware Guide.

Conclusion

Always keep your systems up to date and apply security patches ASAP. However, if the patch cannot be applied immediately, make sure to apply temporary mitigations.

For the present issue: Disable SLP service, ensure the ESXi hypervisor is not exposed to the public internet if possible, or make sure that it is behind VPN. A recent report suggests the new variant of the Royal ransomware is also targeting ESXi servers so, stay extra cautious. If you have been infected by the first wave, solutions are available which are mentioned above. But for the second wave, there is a minimal chance that you can recover from it.

However, If you follow the hunting and responding portion mentioned above, you can almost prevent the attack or detect it in time. It is important to adopt a multi-layered defense approach for your system. Logpoint’s Converged SIEM is a key solution to monitor suspicious activities in your environment and detect any chained events of an ongoing attack through out-of-the-box alert rules. Further, the SOAR component can be actively leveraged for automated responses and to prevent any extensive damage to the system environment.

Be secure, be safe.