During my recent interview with Chainguard, a company that's working to mitigate supply chain attacks by managing and distributing hardened container images (among other awesome stuff), I was asked to name the tool associated with the recent SSH vulnerability. Since this was an interview, my brain blanked, of course. I talked about the vulnerability itself, including how it worked and what was impacted, but I couldn't remember the name of the tool, XZ Utils. My head went back to glibc multiple times, probably because I had recently been mitigating a vulnerability for that at work recently. (Side note: glibc and its IFUNC support are actually components of the XZ Utils vulnerability criteria, but that aspect didn't come to mind during the interview.) Slightly embarrassed, I admitted to the interviewer that I'd have to look up the name of the tool. We both shrugged it off since I clearly knew about the vulnerability, but it was a misstep I wish I didn't make.
Afterwards, I spent some time reviewing the questions and the overall experience with the recruiter during the interview. The XZ Utils question continued to annoy me. Why couldn't I remember the name? I eventually settled on a somewhat "meh" reason for why I likely couldn't remember.
Like many others, we weren't vulnerable to this particular vulnerability. There are certain high-severity vulnerabilities that my team and I don't investigate simply because we don't use any systems that are affected by them. This makes sense. It isn't realistic for a small team to take a deep dive into every single vulnerability, regardless of severity level, unless there's a chance we're vulnerable. But this one was a little different.
My knowledge of the ZX Utils vulnerability was originally limited to what was widely reported in the more mainstream news channels, but it was big enough that our Executive Leadership Team also heard about it. I was asked to draft a communication for them, explaining the situation and how badly we were hit. I investigated and reported back how we weren't vulnerable, but I also covered what could have happened if we were. I also emphasized that everyone around the world dodged a major bullet; the vulnerability could have been much worse if it was distributed to more LTS distros, rather than nightly/development builds, and picked up as part of a broader set of updates. A scary, but relatively small blip among all the other things I had to tackle at that time. Then I moved on, mitigating other vulnerabilities and working with auditors on SOX and SOC 2 work. I had no reason to remember XZ Utils... until last week.
Out of sheer annoyance with myself, I'm going to revisit the XZ Utils vulnerability (CVE-2024-3094) in an attempt to never forget its damn name again. Let's dig in.
Who is this XZ Utils Anyway?
Rather than who, we're going to look at what XZ Utils is and how it was exploited. Sorry, Brooke.
XZ Utils is used in almost all unix-like operating systems (Linux and MacOS being the most popular). More specifically, it's used during operations where compression and decompression is necessary. Many folks might be surprised just how often compression is needed when performing even the most basic of tasks. XZ is common because it's very efficient and provides lossless compression, most notably using the LZMA2 algorithm. Various tools within the operating system can also leverage XZ Utils, including SSH.
What Was Vulnerable
Nefarious code was added to the codebase of XZ Utils in versions 5.6.0 and 5.6.1, which allowed the tool to modify how sshd made SSH connections. This allowed XZ Utils to be used in conjunction with sshd to establish a pre-shared key. Anyone with knowledge of the pre-shared key could gain access using a backdoor system. The vulnerable version was then packaged with OS distributions with the hope of being integrated by a large number of unknowing victims. Fortunately, there wasn't widespread adoption of the vulnerable version.
Vulnerability Criteria and Affected Operating Systems
For a system to be vulnerable to CVE-2024-3094, several conditions must be met:
The system must use glibc, specifically for IFUNC support.
XZ Utils version 5.6.0 or 5.6.1, or the corresponding versions of liblzma, must be installed.
Systems utilizing systemd and a patched version of OpenSSH are known to be vulnerable, though the risk may extend to other configurations pending further analysis.
OS | Affected | Comments | Reference |
Debian (testing, unstable and experimental distributions) | Yes | Vulnerable Versions are 5.5.1alpha-0.1 to 5.6.1-1 | |
Debian (Stable Version) | No | No Debian stable versions are known to be affected. | |
Fedora 41, Fedora Rawhide | Yes | Vulnerable versions are xz-5.6.0-* and xz-5.6.1-* | |
Fedora 40 | No | RedHat recommends that users downgrade to a 5.4 build of XZ as a precaution | |
Alpine Edge | Yes | Vulnerable versions are 5.6.0 to 5.6.1 | |
Kali Linux | Yes | Backdoored version of xz was included in Kali Linux (xz-utils 5.6.0-0.2) between March 26 and March 28 | |
Arch Linux | Yes | Vulnerable versions are 5.6.0-1 to 5.6.1-1 | |
openSUSE Tumbleweed and openSUSE MicroOS | Yes | Backdoored version of xz was included in Tumbleweed and MicroOS between March 7 and March 28 | |
SUSE Linux Enterprise and Leap | No | Both Enterprise and Leap are isolated from OpenSUSE and are unaffected. | |
RedHat | No | No versions of Red Hat Enterprise Linux (RHEL) are affected. | |
Ubuntu | No | No stable ubuntu versions are affected | |
Amazon Linux | No | Amazon Linux customers are not affected by this issue, and no action is required |
How This All Worked in Practice
The researchers at Akamai provided a nice and succinct explanation. Rather than regurgitating it in my own words, here's what they said.
The backdoor is quite complex. For starters, you won’t find it in the xz GitHub repository (which is currently disabled, but that’s besides the point). In what seems like an attempt to avoid detection, instead of pushing parts of the backdoor to the public git repository, the malicious maintainer only included it in source code tarball releases. This caused parts of the backdoor to remain relatively hidden, while still being used during the build process of dependent projects. The backdoor is composed of many parts introduced over multiple commits: 1. Using IFUNCs in the build process, which will be used to hijack the symbol resolve functions by the malware 2. Including an obfuscated shared object hidden in test files 3. Running a script set during the build process of the library that extracts the shared object (not included in the repository, only in releases, but added to .gitignore) 4. Disabling landlocking, which is a security feature to restrict process privileges The execution chain also consists of multiple stages: 1. The malicious script build-to-host.m4 is run during the library’s build process and decodes the “test” file bad-3-corrupt_lzma2.xz into a bash script 2. The bash script then performs a more complicated decode process on another “test” file, good-large_compressed.lzma, decoding it into another script 3. That script then extracts a shared object liblzma_la-crc64-fast.o, which is added to the compilation process of liblzma This process is admittedly hard to follow. We recommend Thomas Roccia’s infographic for a great visual reference and in-depth analysis. The shared object itself is compiled into liblzma, and replaces the regular function name resolution process. During (any) process loading, function names are resolved into actual pointers to the process memory, pointing at the binary code. The malicious library interferes with the function resolving process, so it could replace the function pointer for the OpenSSH function RSA_public_decrypt (Figure 1). It then points that function to a malicious one of its own, which according to research published by Filippo Valsorda, extracts a command from the authenticating client’s certificate (after verifying that it is the threat actor) and passes it on to the system() function for execution, thereby achieving RCE prior to authentication.
Another resource I highly recommend looking into is Russ Cox's post. He goes into greater detail about the script side of the attack, including how to actually configure the attack. Very cool if you want to gain a better understanding of the underlying Linux build process and resolvers. Also, shutout to Thomas Roccia over at Microsoft for publishing this infographic that helps summarize the vulnerability.
The Human Vulnerabilities That Were Exploited
One major aspect that was glossed over by many during the initial release of the vulnerability was the wild levels of social engineering that occurred for this incident to even happen.
I pulled from Evan Boehs' post when I originally began gathering resources for my XZ Utils report for the Executive Leadership Team. I certainly recommend reading through his post, but I'm also going to summarize much of it here.
The story of the 2024 vulnerability actually started back in 2021, when Jia Tan created their GitHub account with the username JiaT75. During this time, Jia opened a PR in libarchive to replace safe_print with an unsafe variant. Pretty suspicious.
In 2022 Jia continued submitting PRs and patch requests. This is also when a second actor, Jigar Kumar, began pressuring the ZX Util code maintainer, Lasse Collin, to merge the suspicious PRs from Jia. It was also around this time that Jigar began pressuring Lasse into adding another maintainer to the XZ project. Soon after Jigar's initial pressures on Lasse, JiaT75 made their first commit to XZ. Another account from a Dennis Ens, who also began pressuring Lasse, also popped up briefly around this time.
Moving to 2023, Jia merges their first meaningful commit, which seems to be an indicator of Lasse beginning to trust them at some level. Lasse's email was then replaced with Jia's in the project.yaml file for Google oss-fuzz. Commits were made to the repository that would later be used as part of the testing infrastructure required for the exploit. These commits were attributed to Lasse as the author, but were committed by Jia. Interestingly, the code was originally authored by Hans Jansen, another account working in conjunction with Jia (Likely just Jia using another account). As one of the final steps, an oss-fuzz PR was opened to disable IFUNC for fuzzing builds. This would mask the malicious code changes.
Finally, in 2024, Jia makes a PR for Google's oss-fuzz that changes the project from tkaani.org/xz to xz.tukaani.org/zx-utils/. This changed the the domain from a hosting company in Finland to one that points back to GitHub, thus giving Jia even more control. Eventually, the final backdoor was committed, which added the necessary changes to the test files. Shortly after, Hans reappears to request the vulnerable ZX Utils version be included in Debian. Further attempts were made to have PRs merged to 1password, Fedora, and Ubuntu around that same time.
The bullying and pressure placed on Lasse was clearly the last straw, especially for someone who was already struggling with mental health issues. Despite the impact (and potential larger impact) of this vulnerability, I find it abhorrent for anyone to be overly critical of Lasse succumbing to the continuous pressures. Rather, it's the entire system of leveraging open-source code developers that's at fault.
I really hope that this causes an industry-wide reckoning with the common practice of letting your entire goddamn product rest on the shoulders of one overworked person having a slow mental health crisis without financially or operationally supporting them whatsoever. I want everyone who has an open source dependency to read this message https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html -@glyph@mastadon.social
We've already seen more open-source project takeover attempts after the XZ Utils incident became widely known. We can expect these types of attacks to continue. I doubt things will change anytime soon in the industry - large corporations, governments, and other organizations will continue to build off the backs of the open-source community without adequately giving back - but I hope that something eventually changes in a way that provides the developers and project maintainers with some reasonable benefit for their efforts. We'll see.
Why is this important?
Mundane but important tools, libraries, or other pieces of code are shipped with the technology that we rely on throughout many aspects of our lives. Understanding how fragile these systems are can help us better mitigate against similar vulnerabilities. It's a reminder that we need to remain vigilant in our efforts. We need to know as much as possible about the software and tools that we use, reducing the chance of a supply-chain attack. Moreover, we need to continue mitigating the blast radius for WHEN (not if) we get hit by a more successful XZ Utils-style of attack. We need to continue implementing defense-in-depth that covers the technical, procedural, and basic human aspects of our organizations' day-to-day operations. We need to advocate for the thankless role of the open-source developer that supports our efforts. A healthy open-source community builds stronger defenses for everyone.
And with that, I think XZ Utils is in my brain forever.
Keywords: XZ Utilz Vulnerability, CVE-2024-3094, cybersecurity, data breach, software updates, security audits, access controls, cyber threats, mitigation.
Comments