"What we should learn from the xz backdoor" was originally published on kevingimbel.de.

What we should learn from the xz backdoor

You may have heard that a vulnerability was found in the xz tool and liblzma library on Friday, 29th of March 2024 which specifically targeted sshd being managed by systemd on Debian and RedHat Linux systems.

What happened

The catastrophic scope of the attack

At first the backdoor seems very odd. xz is a compression tool and library which is not used by ssh directly, so how can it attack ssh? By using systemd! systemd uses liblzma, the xz library, and this is where sshd gets compromised: During loading of the liblzma library by systemd, the malicious code gets executed and patches the backdoor into sshd before the ssh daemon is started. The compromised daemon runs just like a normal ssh daemon, but when a connection is initiated with a specific key it allows remote code execution – so the compromise is incredibly hard to detect.

Wait, what? Imagine sending a ssh-login request to any server with a specifically forged key and the remote ssh-server executes code without any actual login happening1 – quoting from Filippo on bluesky:

The hooked RSA_public_decrypt verifies a signature on the server’s host key by a fixed Ed448 key, and then passes a payload to system(). It’s RCE2, not auth bypass, and gated/unreplayable.

The payload is extracted from the N value (the public key) passed to RSA_public_decrypt, checked against a simple fingerprint, and decrypted with a fixed ChaCha20 key before the Ed448 signature verification.

Filippo Valsorda

systemd was about to refactor its usage of liblzma, making the exploit impossible (or harder), which is probably why they pushed for a release so fast in the last weeks instead of keeping a low profile.

If this code would’ve shipped to major distributions, all linux servers running systemd and sshd would’ve been compromised, and the attackers would be able to execute code on any of them.

As far as I understand, the only visible log one could see is failed logging attempts in sshd logs, which is not unusual if you run a public ssh server (which a lot of people do, me included).

As I see the exploit the code would be executed in the context of sshd, which usually runs as root so the attackers would have a way of executing code on any compromised server using root rights. An absolute nightmare.

The same old Open Source problem

We all need to be super careful when we use Open Source – and we all do use it, either by choice or because literally every computer on this planet runs on some sort of Open Source software. Open Source is the base of all of computing, it’s the software which runs public clouds (AWS, Azure, Google, …), its the code used in Apps and Websites, it’s the editors we use (Code, vim, …), in the apps on our phones — we really dodged a huge bullet here, and who knows how many are hidden. It’s unlikely this is the only such attack going on, and we all need to be on alert.

At the core of this issue we once again have a tool maintained by a single person, which is used in production at every company (due to it being a dependency in tools such as systemd).

As always, there’s a relevant xkcd comic.

A xkcd comic depicting a lot of tiny wooden blocks standing on each other, with one small block holding a lot of the weight. Text on top reads "All modern digital infrastructure", a arrow pointing to the small wooden block holding everything reads "A project some random person in Nebraska has been thanklessly maintaining since 2003"

Final notes

systemd is not at fault, neither is the maintainer of xz

It’s easy to point fingers, and it’s not my intend to do. Neither systemd nor the Lasse Collin (the original xz maintainer) are at fault.

The attack targeted xz for its usage in systemd, and probably for its small maintainer base (1 person), which made getting in on the project easier than getting into a big community.

This is no easy attack

The amount of planning and time required to pull of such an attack is massive. It requires deep knowledge of linux systems and the way various system parts interact with each other during system initialisation and scheduling.

Links and sources


  1. Filippo Valsorda explains the exploit in this bluesky thread ↩︎
  2. RCE = Remote Code Execution ↩︎