Avoiding the Next Log4Shell: Learning from the Log4j Event, One Year Later

By Brian Behlendorf, OpenSSF

Log4Shell, a vulnerability in the widely-used open source Java logging library Log4j, was disclosed in December 2021, roughly two months after I took the helm of the Open Source Security Foundation (OpenSSF). As I said back then, open source software (OSS) foundations must work together to prevent the next Log4Shell scramble, and the same remains true today. One year later, let’s take stock of what else we’ve learned: the core issues around software supply chain security and vulnerability disclosure, the unique nature of securing OSS, and the best techniques for improving OSS security moving forward. 

Different kinds of weaknesses combine to create deep threats

Log4Shell is not exclusively a supply chain security story or a vulnerability disclosure story, but rather a hybrid of both. This reflects the ways different kinds of weaknesses combine to create deep threats. The driver for Log4Shell was a series of vulnerabilities (issued as a set of CVEs over a couple of weeks) that, had they been discovered and remediated in a more orderly, coordinated way, would have avoided the chaos that disrupted people’s holidays last year. But it became a supply chain story because of how difficult it was to detect and report on the presence of unpatched Log4j in the wild. The supply chain issue has given further momentum to the push for Software Bill of Materials (SBOM) adoption for the end-user community (governments, financial services, others), which is an important step in order to better understand when end-users’ software contains vulnerable components – both open source and closed source. But the vulnerability disclosure issue – ensuring that when vulnerabilities are found, the appropriate patches are distributed in a timely manner to minimize exploitation – is also critical to address in order to prevent Log4Shell-like events in the future.

Log4Shell has also shown how deeply closed source software has become dependent on open source software. According to Sonatype, the number of dependencies in the average application has nearly doubled from 298 in 2018 to 528 in 2020. Developer responsibilities managing third party dependencies continue to grow with the average Java application containing 20 more than last year at 148 dependencies, and the average Java project updates ten times a year which leads to developers needing to track nearly 1,500 potential dependency changes per year – per application they work on. Furthermore, Synopsys has found in its 2021 Open Source Security and Risk Analysis Report that 98% of its dataset of commercial codebases contain open source code, with 84% of codebases containing at least one OSS vulnerability. That same study found that typical closed source software applications are between 78% and 90% OSS. 

Log4j has been around for 20 years; it’s become embedded into nearly every meaningful Java application; and the Log4Shell event led to compromises in everything from iCloud to physical security systems. Moreover, malware groups are continuing to exploit unpatched Log4j instances. We are likely to see additional Log4Shell-like events unless we address its root issues.

Security imbalances in open source

Log4Shell has also revealed unique characteristics about open source software communities. While open source software communities include both paid and unpaid contributors, and certain projects do require financial support to be sustainable, it is unfair to characterize the contributors to software like Log4j as entirely unpaid and uncredited developers who work pro bono and outside of their immediate self-interest. Maintainers and contributors of large open source projects are often employees of technology firms and may be paid to work on open source software with the support of their employer, who may make use of Log4j in their own products.

The core issue is a gap in the mapping of motivations: developers are often incidentally working on open source software such as Log4j in the pursuit of fixing bugs or adding features, which rarely leads them to invest extra time on functions that would reduce the risk of security issues in their code. Just like closed source developers, OSS developers often need to fight with their managers for the right to work on things that can make a material difference in application security, from adding extra tests and adopting other security best practices, to simply paying down technical debt and removing underused features. Case in point: the JNDI-LDAP code in Log4j, which led to the Log4Shell vulnerability, was not a required or even popular feature, and so it went under-maintained.

Good security work is hard, and until recently we have lacked great tools or other measures to demonstrate the value of that work, other than “we avoided a CVE for another week.” Now, thanks partly to the work of the OpenSSF and others, we have tools that can help inform end-users as to the relative risk of one open source package over another: the risk scores from OpenSSF Security Scorecard; the presence of the OpenSSF Best Practices Badge; and looking for the last time an independent third party audit was performed on the code. It can also help core developers learn what kinds of practices they might undertake to improve security for their end users, and get credit for doing so. For example the Cloud Native community’s “Security Slam” event encouraged project maintainers to get their security scores up to a higher Security Scorecard baseline.

Avoiding the next Log4Shell

A valid, open question is: “was Log4J a ‘100-year flood’ kind of event, something we can’t plan entirely to prevent? Should we just accept infrastructure-threatening bugs as a part of the cost of innovation?” We don’t think this is the case. There are plenty of high-impact, low-cost, and leverageable investments that can be made to systematically drain risk from the OSS underpinning our critical digital infrastructure, as we detailed in our Open Source Software Security Mobilization Plan, and which the Cyber Safety Review Board (CSRB) report on Log4j extensively cited as containing credible approaches to addressing the root causes of Log4Shell. I believe an independent third party code review likely would have caught the vulnerabilities that led to Log4Shell, and such a review likely would not have cost more than $100k. Such a code review could have been done any time; however no single company or individual stood up to fund that work. So there was a “market failure” here, and this is where additional funding could step in to help. 

Areas of Focus

I can not tell you what the next software package will be that causes a Log4Shell-like crisis. However, we have ongoing work to identify the most critical OSS projects, including a first-draft list of over 100 projects, as well as efforts to measure security risks of OSS projects. Let’s imagine that we selected 100 projects, did a security review (at an average of $100K), and spent the same amount on fixing any issues. That would cost 100 x ($100K + $100K), and let’s say that process was repeated annually. Would a $20M annual investment eliminate all possibility of another such event? No, but it would substantially lower the risk of that for that package and everything else that depends upon it..

A second area of focus is the lack of a trusted, expert-driven, confidentiality-sensitive approach to assisting under-resourced open source projects (perhaps even those 100% volunteer-driven) when they hit a major security incident. If instead there had been a tiger team, on constant standby the way an emergency fire brigade is always at the ready, who developers could trust to keep undisclosed vulnerabilities confidential even from their employer, that could have reduced the thrash substantially and led to one set of fixes, tightly choreographed the way that we often see today. We have published a guide to coordinated vulnerability disclosure for OSS developers (and bug-finders), and we are in the process of defining and setting up a Security Incident Response Team (SIRT) who can be that trusted set of firefighters.

Some other initiatives we have been working on at OpenSSF include: Sigstore, a key tool for verifying and protecting software to prevent software supply chain attacks; defining processes to manage attestations and deliver more secure software such as SLSA, S2C2F, and FRSCA; and working with major OSS foundations and package managers to build greater capacity for security work and adoption of security tooling through initiatives such as the Alpha-Omega Project. We are also heartened to see efforts such as MavenCentral moving to use Sigstore as a key security tool. All of these efforts should help the average software developer with creating more secure code by default, without having to make substantial (or perhaps even any) changes to their own tools or workflow. But all this takes focused time and effort on the part of the toolmakers, both OSS and the vendors around them. That’s what the OpenSSF is working to help coordinate. Additionally, OpenSSF’s free Secure Software Development Fundamentals Courses teaches developers security basics to develop software that is hardened against attacks and how to reduce the damage when a vulnerability is exploited. That course specifically recommended avoiding the constructs that led to the Log4Shell vulnerability, before the vulnerability was discovered. If more developers are educated and trained to develop secure software, and are supported by good tools that help them do so, then the risks of such vulnerabilities drop dramatically.

Those of us at the OpenSSF and many others are working toward addressing the systemic issues that led to the Log4Shell vulnerability discovered one year ago. Such efforts across the board require investment both from the private sector and from governments in order to keep our software safe and secure. If we don’t work together and invest in proper security efforts, major OSS cybersecurity events will continue to disrupt our world.