Behind Enemy Lines: Understanding the Threat of the XZ Backdoor

The following is an excerpt from our new module on the recent XZ Utils backdoor, CVE-2024-3094.

On Mar 29, 2024, at 12:00PM ET, Andres Freund posted on the Openwall mailing list about a backdoor he discovered in the XZ Utils package. The backdoor targeted the OpenSSH binary, allowing remote code execution on impacted machines. This backdoor was not located in the GitHub repository, but only in release versions of the package, which hid its presence.

Given that XZ Utils had been installed (directly or indirectly) on billions of Linux systems worldwide, this finding stunned the international Linux and infosec communities.

Understanding the Timeline of the Attack

In late 2021, “Jia Tan”, an online identity, began a long and careful supply-chain attack with the goal of inserting a backdoor into Linux systems. Jia Tan submitted multiple contribution requests to several projects, including the XZ Utils project, and began bullying Lasse Collin and other open source maintainers through various sock puppet accounts to get their XZ contributions accepted and ultimately, by late 2022, Jia Tan became a maintainer of the XZ project.

During 2023 and early 2024, Jia Tan started making multiple changes to various open-source projects. This included submitting a merge request to google/oss-fuzz, which developers use to validate their code. Apparently, the goal of this update was to take ownership of the XZ Utils project and disable specific features in order to hide certain errors that could reveal the planned backdoor.

Finally, in early 2024, Jia Tan released XZ Utils 5.6.0 as a tarball, which is the most common method of releasing software on GitHub. Specifically, they performed two commits that added “test” files that included the backdoor in the released version of the software. Since these were binary files, and not plain text, the “test” files were more obfuscated from the community.

Following the release of version 5.6.0, Jia Tan rushed out a 5.6.1 release in an attempt to fix some failures that were occurring with the backdoor, with the hope of fixing them before they were spotted.

In total, Jia Tan made at least 450 commits to the XZ GitHub, beginning on June 10, 2022.

Eventually, on March 29, 2024, at 12:00PM ET, Andres Freund noticed a delay in the operation of SSH when running an unstable version of Debian. Without the discovery of this SSH delay and various valgrind test errors, the community may not have discovered this vulnerability until it had run its course, infecting untold thousands or millions of machines.

Despite the discovery of this vulnerability, the community is still wrestling with the potential scale and ramifications of the attack.

Russ Cox presented an excellent detailed timeline of this attack, and @rheaeve posted an informative timezone investigation, both of which served as outstanding community resources.

Grasping the Technical Overview of the Attack

In summary, the widely-used XZ Utils package, which contains xz and the liblzma library, contained obfuscated code that created a backdoor in the OpenSSH service on many Linux systems using XZ Utils (5.6.0 -> 5.6.1). This supply-chain attack was executed over a trust-building campaign spanning two years.

From a technical standpoint, the backdoor was created when the tarball was generated for the release tag. In other words, the backdoor was not created if the user built the tool manually through the git repo.

Given this, it is safe to assume the attacker was targeting Debian and RPM-based systems which depend on this tarball for their package installers. Conversely, systems that do not rely on this type of packaging, such as Arch Linux, were not impacted by this vulnerability.

Assessing the Potential Impact of the Attack

XZ Utils is widely used on over 3 billion machines. The included xz compression tool is lightweight and easy to use, making it commonplace on nearly all Linux devices.

Given the widespread usage of this package, this attack could have impacted the security of Linux systems worldwide, specifically any system running OpenSSH. The impact of this would have been catastrophic. Put simply, it’s possible that this vulnerability could have granted Jia Tan and his allies or sponsors open access to every affected internet-facing SSH server, globally, including those run by governments and high-profile global companies.

This attack had the potential to weaken the global Linux security posture and could have triggered untold millions of dollars in damages, or worse, it could have affected global infrastructure or even the health and safety of countless citizens.

Learning from the Incident

Let’s review some of the potential lessons that we can learn from this event.

First, this high-profile attack has raised questions about the security and trustworthiness of open-source projects. At the same time, the open-source process of testing and collective effort worked in this case, as the vulnerability was discovered before the change was pushed from unstable to stable distributions.

In addition, the discovery of this type of sophisticated attack raises concern that other tools or libraries may have been similarly affected.

This incident has highlighted the fact that solo open-source developers of high-profile projects are often overwhelmed. We simply can’t rely on one person to do everything. We also need to ensure that the developers and contributors are trustworthy.

We also need to develop tools, techniques, and procedures that ensure the validity and safety of our code, especially binary objects and components.

Finally, we must develop and implement tools, techniques, and procedures that help detect not only supply-chain attacks but also the systems affected by these attacks in the event of a breach.

Learn More About the XZ Backdoor

Learning the complexities of the XZ backdoor is essential for cybersecurity professionals to understand not only due to its sophisticated mechanisms and potential impact on security but also because of precisely how it was introduced. Modern systems run on a chilling amount of open source code that is taken for granted as secure because it is open source. XZ shows us that we can not (and indeed, never should) rest on our laurels as a security community.   

With a Learn subscription, you gain access to detailed insights into this cyber threat, along with an exclusive hands-on lab. This lab allows you to use a scanner tool we developed that will detect vulnerable instances of the vulnerable library.

For those looking to delve even deeper, the OffSec Cyber Range (OCR) chain is available to subscribers of OCR and Learn Enterprise. We’ve added a chain of machines into OCR that are not only vulnerable to the bypass but also exploitable. This way learners can apply what they have learned in the module and practice in a new environment.