You may have heard that a vulnerability was found in the xz tool and liblzma library on Friday, 29th of March 2024 which specifically targeted sshd being managed by systemd on Debian and RedHat Linux systems.
What happened
- The xz backdoor was very sophisticated and carried out by a professional, this is no work of an amateur
- The attack was carried our over years, Russ Cox has put together a timeline which shows the attack started around 2021
- Andres Freund reports on the oss-security mailing list about an upstream backdoor in xz, he previously notified linux distro maintainers about the backdoor, giving them time do downgrade the tool
The catastrophic scope of the attack
At first the backdoor seems very odd. xz is a compression tool and library which is not used by ssh directly, so how can it attack ssh? By using systemd! systemd uses liblzma, the xz library, and this is where sshd gets compromised: During loading of the liblzma library by systemd, the malicious code gets executed and patches the backdoor into sshd before the ssh daemon is started. The compromised daemon runs just like a normal ssh daemon, but when a connection is initiated with a specific key it allows remote code execution – so the compromise is incredibly hard to detect.
Wait, what? Imagine sending a ssh-login request to any server with a specifically forged key and the remote ssh-server executes code without any actual login happening1 – quoting from Filippo on bluesky:
The hooked RSA_public_decrypt verifies a signature on the server’s host key by a fixed Ed448 key, and then passes a payload to system(). It’s RCE2, not auth bypass, and gated/unreplayable.
Filippo Valsorda
The payload is extracted from the N value (the public key) passed to RSA_public_decrypt, checked against a simple fingerprint, and decrypted with a fixed ChaCha20 key before the Ed448 signature verification.
systemd was about to refactor its usage of liblzma, making the exploit impossible (or harder), which is probably why they pushed for a release so fast in the last weeks instead of keeping a low profile.
If this code would’ve shipped to major distributions, all linux servers running systemd and sshd would’ve been compromised, and the attackers would be able to execute code on any of them.
As far as I understand, the only visible log one could see is failed logging attempts in sshd logs, which is not unusual if you run a public ssh server (which a lot of people do, me included).
As I see the exploit the code would be executed in the context of sshd, which usually runs as root so the attackers would have a way of executing code on any compromised server using root rights. An absolute nightmare.
The same old Open Source problem
We all need to be super careful when we use Open Source – and we all do use it, either by choice or because literally every computer on this planet runs on some sort of Open Source software. Open Source is the base of all of computing, it’s the software which runs public clouds (AWS, Azure, Google, …), its the code used in Apps and Websites, it’s the editors we use (Code, vim, …), in the apps on our phones — we really dodged a huge bullet here, and who knows how many are hidden. It’s unlikely this is the only such attack going on, and we all need to be on alert.
At the core of this issue we once again have a tool maintained by a single person, which is used in production at every company (due to it being a dependency in tools such as systemd).
As always, there’s a relevant xkcd comic.
Final notes
systemd is not at fault, neither is the maintainer of xz
It’s easy to point fingers, and it’s not my intend to do. Neither systemd nor the Lasse Collin (the original xz maintainer) are at fault.
The attack targeted xz for its usage in systemd, and probably for its small maintainer base (1 person), which made getting in on the project easier than getting into a big community.
This is no easy attack
The amount of planning and time required to pull of such an attack is massive. It requires deep knowledge of linux systems and the way various system parts interact with each other during system initialisation and scheduling.
Links and sources
- Russ Cox xz Timeline https://research.swtch.com/xz-timeline
- Exploit demo https://github.com/amlweems/xzbot
- Techies vs. spies https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-backdoor
- Original report on openwall.com https://www.openwall.com/lists/oss-security/2024/03/29/4
- xz Developer notes on the incident https://tukaani.org/xz-backdoor/
Footnotes
- Filippo Valsorda explains the exploit in this bluesky thread ↩︎
RCE= Remote Code Execution ↩︎