Specifically, SSH logins were consuming too many CPU cycles and were generating errors with valgrind, a utility for monitoring computer memory.
Through sheer luck and Freund’s careful eye, he eventually discovered the problems were the result of updates that had been made to xz Utils. On Friday, Freund took to the Open Source Security List to disclose the updates were the result of someone intentionally planting a backdoor in the compression software.
It is lucky that Andres Freund checked and found the issue in valgrind that was maybe intentionally or maybe unintentionally.
I'm interested in figuring out what happened and more information on the contributor behind the attack.
Everybody keeps saying it was found by luck, but this seems like it was found by a guy who maintains his repository properly and monitors his CPU cycles diligently just to prevent this sort of thing. I guess I would call it lucky he found it so quickly, but it was definitely not found by luck.
The fact that that person happened to be looking on a system downstream to this one, while also having the context needed to pin it back to xz in particular is the lucky part. The same attack in any of countless other places wouldn't have gotten spotted the same way, or as quickly. That's not to say diligence on Freund's part wasn't a big factor here, but it's important to identify that luck was a big factor.
It's more so lucky that there was someone diligently doing that.
It could've easily gone unnoticed had there not been someone like him.
Yep. It could also be found by luck a bit later, or caught in a strict process sooner or later that we won't know.
No process is perfect, but the fact this attack is super complex, with many levels of indirection on run time, detecting if it's in debug mode, forcing maintainers to disable Oss fuzzer (a tool that scans source code for bugs and vulnerabilities) and also involved social engineering and took years to get close to widespread release is to show what it takes to plant a back door on 100% Foss systems.
As opposed to closed source, where all it takes is a conversation between govt and a CEO.
to me: the most chilling thing is that someone involved in the open source ecosystem introduced this vulnerability and, if it was intentional, what else did they do?
That's what I'm most curious about. Was it government? Was it Microsoft, Apple or Google? Was it some lone hacker or group looking for money? Was it just an OSS developer that wanted revenge? It would make for a spicy story.
It looks like a state actor, but hard to tell who. The way they did it was by bullying the original overworked maintainer into making someone else a co-maintainer, and that new co-maintainer introduced the backdoor. The accounts that pressured him into adding another maintainer all appear to have been sock puppets.
It just shows how little support there is for the lone maintainers of basic utilities that we all use, and it's really something we need to do something about.
It isn't if it was intentional. It was intentional. Otherwise the exploit chain wouldn't be so convoluted.
At the moment, it’s unknown if there was ever a real-world person behind this username or if Jia Tan is a completely fabricated individual.
Well, I'm pretty sure it wasn't a platypus. There was definitely one or more real-world people behind this. I doubt anyone thinks that their name is actually Jia Tan, though.
The industry has too long neglected the true threat posed to information security by the platypi.
Incredible, just incredible. I am looking forward to the upcoming time when we know more about exactly how it works.
Yeah it sounds pretty wild already with some kind of, like, door knock mechanism using certificates? So you can't scan for it. And some reverse engineering countermeasures.
Like everyone else, I have to wonder what libraries have been compromised in a way that nobody has noticed yet.
Yeah, that's the scariest part. This was caught, but are there other projects out there that have been attacked with similar methods that no one knows about?
I have a feeling there are a lot of busy people trying to answer that question, now. Yikes.
There's also the ones known by very few people. There's companies and unofficial groups of people who collect and sell this information. Usually to state governments, off records.
I don't think it's the case for Linux, but I occasionally follow the state of things for bounties offered by Google and Apple to white hat hackers. Though this case is clearly malicious, I understand most vulnerabilities can easily pass as a bug/mistake.
From what I've read is not authentication bypass, it's a RCE using certificates to deliver the payload. If a specific signature is found it runs the code that was sent in place of the signing public key. It also means that only someone who has the ability to generate that specific key signature could use the RCE.
There were some other bits that looked like they could have been placed to enable compromising other build systems in the future when they checked for xz support.
It's crazy how they pressured/manipulated the maintainer. Especially fucked up considering he wasn't in a good mental state and was still helping the community by maintaining FOSS software.
This is the best summary I could come up with:
They were likely very close to seeing the backdoor update merged into Debian and Red Hat, the two biggest distributions of Linux, when an eagle-eyed software developer spotted something fishy.
Andres Freund, a developer and engineer working on Microsoft’s PostgreSQL offerings, was recently troubleshooting performance problems a Debian system was experiencing with SSH, the most widely used protocol for remotely logging in to devices over the Internet.
Through a combination of sheer luck and Freund’s careful eye, he eventually discovered the problems were the result of updates that had been made to xz Utils.
On Friday, Freund took to the Open Source Security List to disclose the updates were the result of someone intentionally planting a backdoor in the compression software.
Malicious code added to xz Utils versions 5.6.0 and 5.6.1 modified the way the software functions when performing operations related to lzma compression or decompression.
For instance, Tan replaced Collins' contact information with their own on oss-fuzz, a project that scans open source software for vulnerabilities that can be exploited.
The original article contains 640 words, the summary contains 171 words. Saved 73%. I'm a bot and I'm open source!
So if I don't use SSH am I fine? Because my distro doesn't let me remove the package. Other packages depend on it.
Unless you're running Debian testing you're safe. If ssh isn't open to the internet you're safe. Just make sure everything is up-to-date.
Debian testing has 'updated' to 5.6.1+really5.4.5-1 anyway, so as long as you've updated within the past few days it will have been downgraded to 5.4.5.
You can't remove it but you can downgrade.
How quickly was it caught?
After a month from what I read, most distro hadn't included it yet
Edit: it had been prepped for 2 years or so and the malicious code was introduced a while ago, but from release it took about a month to detection
It is lucky that Andres Freund checked and found the issue in valgrind that was maybe intentionally or maybe unintentionally.
https://www.openwall.com/lists/oss-security/2024/03/29/4
I'm interested in figuring out what happened and more information on the contributor behind the attack.
Everybody keeps saying it was found by luck, but this seems like it was found by a guy who maintains his repository properly and monitors his CPU cycles diligently just to prevent this sort of thing. I guess I would call it lucky he found it so quickly, but it was definitely not found by luck.
The fact that that person happened to be looking on a system downstream to this one, while also having the context needed to pin it back to xz in particular is the lucky part. The same attack in any of countless other places wouldn't have gotten spotted the same way, or as quickly. That's not to say diligence on Freund's part wasn't a big factor here, but it's important to identify that luck was a big factor.
It's more so lucky that there was someone diligently doing that. It could've easily gone unnoticed had there not been someone like him.
Yep. It could also be found by luck a bit later, or caught in a strict process sooner or later that we won't know.
No process is perfect, but the fact this attack is super complex, with many levels of indirection on run time, detecting if it's in debug mode, forcing maintainers to disable Oss fuzzer (a tool that scans source code for bugs and vulnerabilities) and also involved social engineering and took years to get close to widespread release is to show what it takes to plant a back door on 100% Foss systems.
As opposed to closed source, where all it takes is a conversation between govt and a CEO.
to me: the most chilling thing is that someone involved in the open source ecosystem introduced this vulnerability and, if it was intentional, what else did they do?
That's what I'm most curious about. Was it government? Was it Microsoft, Apple or Google? Was it some lone hacker or group looking for money? Was it just an OSS developer that wanted revenge? It would make for a spicy story.
It looks like a state actor, but hard to tell who. The way they did it was by bullying the original overworked maintainer into making someone else a co-maintainer, and that new co-maintainer introduced the backdoor. The accounts that pressured him into adding another maintainer all appear to have been sock puppets.
It just shows how little support there is for the lone maintainers of basic utilities that we all use, and it's really something we need to do something about.
It isn't if it was intentional. It was intentional. Otherwise the exploit chain wouldn't be so convoluted.
Well, I'm pretty sure it wasn't a platypus. There was definitely one or more real-world people behind this. I doubt anyone thinks that their name is actually Jia Tan, though.
The industry has too long neglected the true threat posed to information security by the platypi.
Platypodes
Platypoxen
Platypoodles
Dammit, this attacker is sounding more and more adorable!
I saw someone trying attribute this to Hong Kong based on the possible origins of the name.
Seems like a stretch to me
Failing that hard to not know what is, and isn't knowable, is a good reason to not give a single f about what they think
Incredible, just incredible. I am looking forward to the upcoming time when we know more about exactly how it works.
Yeah it sounds pretty wild already with some kind of, like, door knock mechanism using certificates? So you can't scan for it. And some reverse engineering countermeasures.
Like everyone else, I have to wonder what libraries have been compromised in a way that nobody has noticed yet.
Yeah, that's the scariest part. This was caught, but are there other projects out there that have been attacked with similar methods that no one knows about?
I have a feeling there are a lot of busy people trying to answer that question, now. Yikes.
There's also the ones known by very few people. There's companies and unofficial groups of people who collect and sell this information. Usually to state governments, off records.
I don't think it's the case for Linux, but I occasionally follow the state of things for bounties offered by Google and Apple to white hat hackers. Though this case is clearly malicious, I understand most vulnerabilities can easily pass as a bug/mistake.
From what I've read is not authentication bypass, it's a RCE using certificates to deliver the payload. If a specific signature is found it runs the code that was sent in place of the signing public key. It also means that only someone who has the ability to generate that specific key signature could use the RCE.
There were some other bits that looked like they could have been placed to enable compromising other build systems in the future when they checked for xz support.
It's crazy how they pressured/manipulated the maintainer. Especially fucked up considering he wasn't in a good mental state and was still helping the community by maintaining FOSS software.
This is the best summary I could come up with:
They were likely very close to seeing the backdoor update merged into Debian and Red Hat, the two biggest distributions of Linux, when an eagle-eyed software developer spotted something fishy.
Andres Freund, a developer and engineer working on Microsoft’s PostgreSQL offerings, was recently troubleshooting performance problems a Debian system was experiencing with SSH, the most widely used protocol for remotely logging in to devices over the Internet.
Through a combination of sheer luck and Freund’s careful eye, he eventually discovered the problems were the result of updates that had been made to xz Utils.
On Friday, Freund took to the Open Source Security List to disclose the updates were the result of someone intentionally planting a backdoor in the compression software.
Malicious code added to xz Utils versions 5.6.0 and 5.6.1 modified the way the software functions when performing operations related to lzma compression or decompression.
For instance, Tan replaced Collins' contact information with their own on oss-fuzz, a project that scans open source software for vulnerabilities that can be exploited.
The original article contains 640 words, the summary contains 171 words. Saved 73%. I'm a bot and I'm open source!
So if I don't use SSH am I fine? Because my distro doesn't let me remove the package. Other packages depend on it.
Unless you're running Debian testing you're safe. If ssh isn't open to the internet you're safe. Just make sure everything is up-to-date.
Debian testing has 'updated' to
5.6.1+really5.4.5-1anyway, so as long as you've updated within the past few days it will have been downgraded to 5.4.5.You can't remove it but you can downgrade.
How quickly was it caught?
After a month from what I read, most distro hadn't included it yet
Edit: it had been prepped for 2 years or so and the malicious code was introduced a while ago, but from release it took about a month to detection