Backdoor XZ: IT's Hidden Easter Drama

Largely unnoticed by the general public, security professionals pieced together pieces of a story over the Easter weekend that have the makings of great drama: Secret Service agents infiltrate hidden back doors, and a likable geek thwarts a nefarious quest to control… The world is just in time. The tragic role is played by an exhausted open source developer with psychological problems, who is mercilessly exploited by malicious attackers. Everything is going well again – but now everyone is confused about what to do next. But first things first:

advertisement

preparation

Ask an IT professional you trust about the most important management tools they rely on most. Chances are good that SSH will appear at the top of this list. Administrators use this service for secure, low-level encrypted access to all types of servers. If any important service encounters a serious problem, sooner or later the administrator will log in to the relevant server via SSH at the command line.

You can access and manage servers across the network via SSH; The security gap in this service would be fatal. The Shodan search engine lists nearly 20 million SSH entries worldwide; There are still more than 2 million in Germany. A backdoor via SSH that allows arbitrary commands to be executed there is every secret service's wet dream.

But the popularity of SSH stems, among other things, from the fact that it is famous for its security. It was written by professional paranoiacs based on the latest security concepts and has been continuously developed over decades for optimal security. The mouse simply won't bite any string.

Supply chain weakness

But SSH does not operate in a vacuum, it depends on platform resources. Among other things, it loads more than 20 dynamic libraries on a typical Linux system, incl liblzma.so, which provides functions to compress (decompress) the xz format. This XZ Tools is a small open source project that has been maintained as a hobby by one volunteer for many years.

He suffered from severe fatigue and psychological problems, which is why he was no longer able to truly achieve this mission. That's why he gratefully accepted the help of an initially unknown person named “Jia Tan”. This was encouraged by several suspected accomplices who constantly increased the pressure on the supervisor with complaints and demands. Jia Tan eventually gained the trust of the xz moderator over the course of two years and soon became the official co-moderator who was able to independently contribute new code to the project.

the back door

He used this to build a very advanced backdoor in version 5.6.0 at the beginning of 2024. He did not put this backdoor into the publicly visible source code of the XZ project. Instead, make sure that the build process extracts the code from some supposed test files and merges it into the final library binary. The backdoor code can only be found in the project version's tarballs.

This process alone has so much ingenuity that it would provide enough material for an article. Suffice it to say that final 5.6.x versions of liblzma had a very special backdoor that only became active when an SSH process was started /usr/sbin/sshd user. Since this backdoor is only available in binary form and contains several anti-debugging measures, there is no final analysis yet.

According to current knowledge, it redirects the function of the recording process to itself. Specifically, every login attempt using the RSA public key ends via RSA_public_decrypt() In the back door code. It analyzes the public key used for this purpose, extracts command line commands from it, and executes them on the server.

To ensure that not everyone can use this, the sequence of commands sent must also have a specific digital signature. Only the attackers behind Jia Tan could create such a signature using their secret key. This is the typical “nobody but us” backdoor – NOBUS for short, which secret services in particular favor.

Publishing

This backdoor is found in current XZ tools xz-5.6.0 and xz-5.6.1 and their associated liblzma libraries. They have already made their way into some unstable and testing branches of distributions, but none of Debian, Ubuntu or Red Hat have included them in their stable branch.

However, as current analyzes show, this is exactly what a number of people have been very actively pushing for; Like Jia Tan, these are likely made-up personas of the attackers. If they had succeeded, it would have been a disaster with millions of systems that attackers could operate on as they pleased.

Enter: The Great Hero

We owe the fact that it didn't come to this thanks to the curiosity of software developer Anders Freund, who likes to get to the bottom of things. As he himself explained, a strange load on the CPU was interfering with his measurements on the test system. Further investigation revealed that on systems with the backdoor, a failed login attempt via SSH took approximately 500 milliseconds longer than on systems with older versions of liblzma.

With this evidence, he then tracked down the obfuscated text in the source code's tar file and alerted the open source community on Friday, March 29. Media outlets like Heise Security reported on the mysterious XZ backdoor shortly after, Red Hat gave the vulnerability the ID CVE-2024-3094 and warned organizations like BSI as well.

So the nerd who didn't put up with a slightly weird load on the CPU and half a second of unexplained lag made little difference. As it turned out, the warning came at the right time; The number of affected systems was manageable, and no visible damage has been found so far. But the potential of this NOBUS tailgate is huge; It is unparalleled in the history of the Internet.

Many questions, few answers

Ultimately, the Internet and much of global information technology came so close to disaster that everyone wondered: What would have happened if Freund had simply ignored this phenomenon like almost everyone else? How many of these backdoors have gone unnoticed? And above all: how can this be avoided in the future?

The search for these answers is currently well underway. Naturally, almost immediately there were voices declaring that this process was conclusive proof of the unsuitability of the open source development model, where Hinz&Kunz could inject backdoors into important projects. On the other hand, of course, there were those who celebrated that Freund's discovery was proof that control by many eyes works.

I think it is too early to assume the final results before the full scope of these measures is clarified. But what you can do now is draw some practical conclusions and draw preliminary ideas from them. All projects with widespread distribution that can be accessed directly from the Internet should now be systematically examined to ensure that there are no comparable backdoors. This includes not only your code, but also everything you upload. Of course, this does not only apply to open source projects such as Apache or nginx. Commercial software also loads external libraries in batches.

This is where the social element inevitably comes into play: an incredible number of important projects fall on the shoulders of individuals who end up being hopelessly overworked. The xkcd comic above explained this perfectly. It was identified as an issue during the Heartbleed debacle in Openssl and was discussed in detail during the Log4j hiatus follow-up. I'm curious to see if more Magic Security Dust will come out of these discussions this time around.

(Yes)