Log Jam: Lesson Learned from the Log4Shell Vulnerability

This past December, as if the world had nothing else to worry about, we were confronted with what some consider one of—if not the—most impactful software vulnerabilities ever discovered.¹ Specifically, it was a vulnerability found in the widely used Java-based logging utility called Log4j, a tool that was initially published by the Apache Software Foundation back in 2001.

The Log4Shell vulnerability has been described as “the single biggest, most critical vulnerability of the last decade”² and “a design failure of catastrophic proportions”³ that will affect us for years to come.⁴ Such words are not often used, even in cybersecurity, but are appropriate here because this vulnerability is relatively easy to exploit, opens up a variety of different ways for the attacker, and forms a huge attack surface due to the extraordinarily large number systems using Log4j. The “blast radius” of Log4j is seemingly unprecedented.⁵

The vulnerability was published by Alibaba Cloud on Dec. 9, 2021, and appears in the National Institute of Standards and Technology's National Vulnerability Database (ID CVE-2021-44228) with a maximum Common Vulnerability Scoring System severity score of 10.0 (critical).⁶ It affected a wide range of products, including Cloudflare, iCloud, Minecraft, and Twitter. The number of affected servers and systems probably will reach into the millions. And to add insult to injury, the initial mitigation released by Apache was insufficient and required multiple updates⁷ while, in parallel, security researchers uncovered additional vulnerabilities, though none with a 10.0 rating.

The response was swift, and companies started the mitigation and patch process in an all-hands-on-deck approach.⁸ In parallel, cyber adversaries ramped up their attacks and developed increasingly sophisticated exploits to take advantage of vulnerable systems, as well as were able to circumvent external protections such as firewalls.⁹

A month prior (on Nov. 3), the Cybersecurity and Infrastructure Security Agency (CISA) had issued Binding Operational Directive 22-01: Reducing the Significant Risk of Known Exploited Vulnerabilities.¹⁰ CISA applied this directive to the Log4Shell scenario by mandating that all federal civilian executive branch agencies must mitigate Log4Shell by Dec. 24, 2021.

In this article, I don't want to focus on specific guidances and best practices on how to mitigate Log4Shell; others already have done this quite thoroughly and in a timely manner (e.g., CISA,¹¹ GitHub,¹² TechSolvency¹³). Also, a number of tools are available, including those from CISA¹⁴ and Google.¹⁵ Instead, I want to take a step back and look at the bigger picture and the lessons we can learn from this scenario.

Lessons To Be Learned

A former boss of mine had a saying: “Nobody is completely healthy; it is just that some people have not been examined thoroughly enough.” In a similar vein: Is software free of vulnerabilities, or has it not been looked at yet? Considering the continual increase of published vulnerabilities,¹⁶ we can safely conclude it is the latter.

But even more concerning is that the number of deeply embedded, impactful, and pervasive vulnerabilities has been growing. Thinking back, it seems that for the past year or two, we have been moving from one software crisis to the next (e.g., Urgent/11,¹⁷ Amnesia:33,¹⁸ Ripple20,¹⁹ BrakTooth²⁰). Granted, Log4Shell trumped them all and put a noteworthy finish on the year.

Clearly, however, we need to recognize that not only is the number of vulnerabilities rising but also that their pervasiveness is growing, as they appear in lower levels of our software builds (and hence making it more difficult to determine if they are present). This affects software components that are widely used—and have been for many years—across extensive ranges of our infrastructure.

It appears that we have entered the Omicron phase of cybersecurity.

As more and more security researchers uncover an increasing number of vulnerabilities, and as more and more cyber adversaries (including nation states) engage in their own research to discover valuable zero-days, we need to ask ourselves if our current approach can succeed or, at a minimum, is sustainable?

Questions To Be Asked

First, the use of open-source software historically has been heralded as being safe and secure, as it is scrutinized as part of the larger community effort of generating and using these software components. But if we look at Log4j, a software tool widely used for decades, we learn that it has been maintained by only a handful of unpaid volunteers.²¹ And Log4j is not alone.²² For example, the widely used OpenSSL library was supported by two volunteers, eventually increased to four developers, and is funded through donations and a few support contracts. Considering the wide use and resulting risk, existing mechanisms for support seem to fall short.

Talking about risk: With Log4j being deployed in millions of systems and supporting billions of dollars of business as well as critical infrastructure, we do need to look at this as a wake-up call and reevaluate the risk equation and maintenance methodology of open-source software.

Risk management in open-source software, unfortunately, is not happening organically and automatically through a “many eyes” approach. It does raise valid concerns about responsibility versus accountability: It appears that at the lowest level—the level of the open-source component—risk management efforts are not scaled according to its financial or operational impact potential.

It further appears that with the current approach, we apply risk mitigation techniques (risk assessment, threat modelling, and various forms of software testing) at the level of the complete product and assume that this would uncover any supply chain vulnerabilities (be it open-source or commercial software).

Two supporting examples:

1. The Log4Shell vulnerability has been around for years and is being used in hundreds of products, yet it has only now been discovered.

2. On the commercial software supply chain side, the compromise of the SolarWinds IT management tool led to hacks of U.S. government agencies²³—it was, in a sense, another security failure of our software supply chain.

Rightfully, the advisory firm Forester is contemplating a more methodical, metrics-based approach to manage widely used software packages.²⁴ But in addition, it is raising the question of whether this would force us to confront the reality of open-source use that is “too widespread to fail.”

Source: https://xkcd.com/2347. Licensed under a Creative Commons attribution/noncommercial 2.5 license.

We live in a world of growing dependency on and increasing risk through software-based products. Unfortunately, it is not just a matter of well-resourced cybercriminals but also one of complex nation-state interests. Remember, the WannaCry and NotPetya attacks were using an exploit tool (EternalBlue²⁵) that had been developed by the U.S. National Security Agency—but was leaked and disclosed by the Shadow Brokers²⁶ hacking group and promptly weaponized by numerous cyber adversaries and nation states.

And government interests seem to take on some curious forms. The researchers that discovered Log4Shell were reprimanded by the Chinese government for not reporting the vulnerability to the Ministry of Industry and Information Technology first.²⁷

Clearly, we need to improve our software quality, as we can't just continue to move from one software crisis to the next. This appears to be an unsustainable model, and we need to look for improvements—some of which will probably need to be quite radical in nature.

One such improvement is software transparency—in other words, providing better visibility through software bills of materials (SBOMs).²⁸ I believe that Log4Shell has answered many open questions on what the future of SBOMs and SBOM-related processes needs to look like.

• How deep does an SBOM have to be? As deep as possible, so that low-level components like Log4j can be reliably discovered.

• What needs to be included in the SBOM? Everything, because any software component can lead to an exploitable vulnerability. For example, Log4j is not a functional program; it provides technical logging and debugging features.

• How do SBOMs need to be distributed and managed? With as much automation as possible, which requires a degree of standardization of data formats and processes so that operators of systems can easily ingest SBOM information and can reliably match newly identified vulnerabilities against their asset inventory.

None of this will be easy, and it's asking a lot, but only with radical change can we meaningfully improve our cyber-defensive posture.

In Closing

One thing is becoming clear, software risk and vulnerability management is coming to a reckoning, and between the increasing sophistication of cyber adversaries and the growing number and pervasiveness of published software vulnerabilities, we need to look at fundamental change and radical improvement.

We need to not only improve our processes (and security posture) across the supply chain, but we also need to understand the chain of dependencies better—meaning the developer of a software component needs to be more aware of its use and resulting risks, and the developer of the final product needs to be more conscious about decisions to include supply chain software components.

In the end, the open question is: What else is out there? We don't know what we don't know, but if past trends are any indication, we're in for a rough ride.²⁹ In cybersecurity, things usually don't get better on their own—and it appears that 2022 is already off to a wild start (e.g., Researchers find bugs in over a dozen widely used URL parser libraries³⁰).