Xz Utils Supply-Chain Attack and What it Means for You

The following are notes for a lightning talk I gave at the New York Amateur Computer Club's monthly meeting on 11 April 2024.

The short URL for this page is https://go.glump.net/xz .

Xz Utils is a part of the Linux operating system and ecosystem, and Microsoft researcher discovered at the end of March 2024 that Xz Utils has been comprormised with a backdoor feature designed to allow an attacker with a certain secret key gain full access to any system where the code got installed.

This type of compromise is known as a "supply-chain attack". A spply-chain attack is one where someone installs malicious code onto a system by placing the malicious code somewhere controlled by the attacker where the target system will fetch and install the malicious code. The two most common forms of supply-chain attacks are:

• Compromising the software's official update channel and forcing the channel to include the malicious code.
• Typo-squatting: setting up an alternate channel for the software at a different URL that's close to the official url, hoping that someone will pick the wrong url or type it in incorrectly. For example, a typo-squatter might try hosting a compromised copy of Visual Studio Code at https://github.com/mircosoft/vscode instead of the official url https://github.com/microsoft/vscode .

The attack we found in Xz Utils is more insidious: an attacker picked a component of Linux that they thought they could influence, and they spent years building up trust as an actual authorized editor of the target project. They then inserted obfuscated attack code into the testing code of Xz, which when run inserts another layer of the attack code into the software builder's workspace containing Xz's real source code; that gets compiled, and then delivered as part of an operating system's software library. The attack code actually depends on SSH and Systemd — other components of Linux that most servers have installed — to achieve its goal.

The goal was to get installed everywhere on as many Linux systems as possible, and when the attacker knocks on the door of one of the infected systems, the attacker is let in and given full access. The attack is setup is such a way that no one else can take advantage of it; if you don't have the attacker's secret key, you can't get it.

How the Xz Comprormise was Discovered

Andres Freund, a researcher working for Microsoft, was testing software with the "latest" version of a Linux OS. He discovered an unexpected high CPU load at the start of every SSH remote login session, and eventually figured out that Xz was causing the problem, and then looked at Xz's source code, and had to eventually look at changes in Xz's test scripts to finally find the malicious code that explained the performance problem he found.

He quickly published what he found, and the malicious versions of Xz were removed from current builds of Linux operating systems.

The attack was discovered months before the malicious code was propagated to mainstream "long-term support" releases of popular Linux OSes, and billions of dollars in economic damage were averted — damage from the attack itself and the eventual scramble to fix it after it would have maybe been found in the wild with basically the whole world already compromised.

What Does this Mean for Me?

The Xz Utils supply chain attack is a rare and extremely dangerous attack where someone with a lot of resources and time spent years building up trust and getting their attack payload installed through the proper channels in the proper way, acting as a real author and maintainer in the project they chose as their vector of the attack.

It was only pure luck that allowed a researcher to find the problem when they were testing something completely unrelated, and that it happened months before being included in major operating system releases that get installed and used for years on systems you depend on all over the world for web services that you use every day.

We can't stop this from happening again. In fact, similar attacks are probly going on right now, perpetrated by different people, using software packages other than Xz as attack vectors. And they haven't been discovered yet and may never be discovered.

Nothing can be completely secured. The best we can do is have layers of security that make it harder for any one comrpromise to drastically affect our lives. Here are some general layers of security you can add to your life to help you get through incidents like this when they become a global disaster:

• NEVER use the same password for more than one account.
  • ESPECIALLY for your email address. Your account on a casual online game may not be well secured. If someone steals your password from the game and finds out this is the same as your email password, they now have the keys to your life!
• Never create passwords from data in your own personal life. Your favorite things, names in your family, and places you've been can all be harvested from social media. Using these things as a password is the same as not having a password at all. Instead all of your passwords must be truly RANDOM words and strings of text.
• A password manager tool takes care of creating and remembering passwords for you. Pick one and learn to use it, and move all of your passwords into it.
  • KeePassXC works on one computer at a time, and doesn't care care of syncing for you.
  • BitWarden is trusted by the industry and runs a free secure syncing server for you.
• Never store data online that doesn't need to be online.
• Don't have private conversations in public online. If the conversation is for you and one other person, use a tool or a system that guarantees privacy. There is no reason it should be archived forever on Facebook where everyone can use it against you later. EFF has a page about how to protect your privacy.
• If you must store data online, encrypting it before uploading it helps a lot in case the service gets compromised.
• Use full disk encryption on all of your desktop computers, laptops, tablets and phones. (Look up how to do that in the OSes you use.) If you lose a laptop on the train, and if it was encrypted, you can at least not have to worry about what was stored on it that someone might read if they found it.

Links to More Details About the Xz Utils Compromise

Andres Freund published his findings on 29 March 2024 in the OSS Security mailing list.

Bleeping Computer reported on the story, and then posted a follow-up about a scanner to help find infected systems.

Fireship did an excellent 4 minute technical summary video about what happened.

Low Level Learning reported on some follow-up analysis done by the community last week.

Comments