Malicious xz backdoor reveals fragility of open source

Analysis The discovery last week of a backdoor in a widely used open source compression library called xz could have been a security disaster had it not been caught by luck and atypical curiosity about latency from a Microsoft engineer.

Yet the fortunate find has led industry observers to conclude not much will change to prevent this threat scenario from reoccurring, and that similar, ongoing efforts to compromise software infrastructure may have been missed.

"This might be the best executed supply chain attack we've seen described in the open, and it's a nightmare scenario: malicious, competent, authorized upstream in a widely used library," wrote open source cryptographer Filippo Valsorda. Meanwhile, Michał Zalewski, an ex-Googler now veep of security engineering at Snap, described the affair as "one of the most daring infosec capers" ever witnessed.

On Friday, Andres Freund, a principal software engineer at Microsoft and one of the developers of PostgreSQL, posted to the Openwall security mailing list that he had found a backdoor in liblzma, which is part of the xz package.

The xz software is used in many Linux distributions and in macOS for tasks like compressing release tarballs, kernel images, and the like.

But the backdoor was caught early – the malicious code only made it into a few bleeding-edge Linux distributions, such as the upcoming Fedora Linux 40; Fedora Rawhide developer distribution; Debian Unstable; and Kali Linux. Vulnerable distributions require glibc (for IFUNC, a way to make indirection function calls into OpenSSH authentication), and xz-5.6.0 or xz-5.6.1

It could have been much worse. According to Valsorda, the backdoor code enabled full remote code execution.

What appears to happen, simply put, is that on a machine that deploys a backdoored xz, the SSH daemon ends up loading the poisoned library during startup, via systemd, which alters the operation of the daemon. That meddling injects code that inspects incoming SSH connections for a specific set of circumstances that if met executes shell commands hidden in a payload supplied by the attacker during the connection initialization, giving the spy remote code execution on the machine.

This means that any machine with the vulnerable package that exposes SSH to the internet is potentially vulnerable

The backdoor is complicated and doesn't even fully appear in the xz repo, which now has an explanatory note from the maintainer. Instead, the malicious code spans multiple commits, some of which only appear in source code tarball releases as a presumed means of concealment.

"Currently, it appears as though the backdoor is added to the SSH daemon on the vulnerable machine, enabling a remote attacker to execute arbitrary code," said the Akamai Security Intelligence Group in a report on Monday. "This means that any machine with the vulnerable package that exposes SSH to the internet is potentially vulnerable.

"This backdoor almost became one of the most significant intrusion enablers ever — one that would’ve dwarfed the SolarWinds backdoor. The attackers were almost able to gain immediate access to any Linux machine running an infected distro, which includes Fedora, Ubuntu, and Debian. Almost."

According to Sam James, a Gentoo developer, who has assembled a summary of what's known so far, several malicious commits were made by someone called "Jia Tan" who first submitted a patch to the xz project in April 2022. That was followed by what appears to be a pressure campaign directed at xz maintainer Lasse Collin from a presumed sockpuppet account using the name "Jigar Kumar."

The incident attracted attention, in part, because it validates years of concern about the burden of maintaining software with little recognition or support. Responding to Kumar's provocation that xz needs a new maintainer because the current one has lost interest, Collin said in a mailing list post that his inaction reflected mental health concerns and other things. And he added, "It's also good to keep in mind that this is an unpaid hobby project."

• When software depends on a project thanklessly maintained by a random guy in Nebraska, is open source sustainable?
• Securing open-source code isn't going to be cheap
• Securing open source software: Whose job is it, anyway?
• Open source community split over offer of 'corporate' welfare for critical dev tools
• We need to talk about mathematical backdoors in encryption algorithms
• Anatomy of suspected top-tier decade-hidden NSA backdoor

A common interpretation among members of the security community is that this pressure campaign was the start of a long con – "Jia Tan" built up trust with apparently innocuous code submissions for about two years before submitting the backdoor code. One "Jia Tan" submission was a pull request to OSS-Fuzz, a security tool, to support disabling IFUNC, the mechanism that the malicious xz code uses to hook into SSH.

Microsoft security researcher Thomas Roccia’s diagram of the xz affair offers a succinct summary of the timeline of events.

The identity of "Jia Tan" (JiaT75) is not presently known. The name appears to be of Asian origin but there's also speculation that the name may represent an attempt to create an identity that people will assume is Chinese as a means of misdirection. An analysis of code commit times argues that the individual or group responsible operates out of Eastern Europe rather than Asia.

Sophisticated campaigns designed to compromise software projects over a long period of time suggest the involvement with a well-funded adversary, possibly with ties to an nation-state intelligence agency. However, there's no conclusive evidence to that effect at this point.

Gynvael Coldwind, MD of HexArcana Cybersecurity GmbH, said in an analysis of the malicious code, "I can't help but wonder (as I'm sure is the rest of our security community) – if this was found by accident, how many things still remain undiscovered." ®