And keep in mind, the falcon sensor exists for Linux. All those big companies largely use it.
Essentially we just got lucky that their buggy patch only affected the windows version of the sensor in a showstopping way. Could have been all major OS.
I don't think the Linux culture is very similar to the windows culture. At least for me personally, I wouldn't use crowdstrike and let them install whatever they want into my environment.
Maybe it's just me.
It's not your machine, your choice of distro, or your choice of specific packages to use or not use. It's a work tool you get handed as part of a job. So whether CrowdStrike runs on it or not is not your decision and you aren't allowed (and usually not capable) to change that.
That's an entirely different situation from one where you get a PC to do with as you please and set up yourself, or a private machine.
Plus we're mostly talking endpoint devices for non-technical users with many of these difficult-to-fix devices as techs have to drive out to them. The users expect a tool, and they get a tool. A Linux would be customized and utterly locked down, and part of that would be the endpoint protection software.
We tried to fight against having to install Crowstrike on our Linux servers but got overruled by upper management without discussion. I assume we are not the only ones with that experience in the world due to the need to check a checkbox for some flimsy audit.
I bet you could bring it up with them nowโฆ
You're actually confirming their point about culture though. The fact that you couldn't stop them doesn't mean that it also happened to everybody else: some management may have listened. Linux users abhor adding weird shit to their OS, Windows users do it all the time.
Essentially no one has crowdstrike on their personal machines. Not Windows users, Mac users, or Linux users. So it's corporate/large organization culture that matters. And they absolutely use it.
Are you an admin in a corporate data center? If not, you're not in the target audience for that product.
Yup. And I think that says more about the corporate culture than the company that caters to them.
Welcome to the world of big retailers! They would rather run Linux with crowdstrike than make their own system.
That's only true if you run falcon-sensor in ebpf and not kmod mode.
Yep I know
The issuw didn't affect Linux and macOS systems with Crowdstrike Falcon installed, though, only Windows systems.
On Windows, booting into Safe Mode and removing C:\Windows\System32\Drivers het bestand C-00000291*.sys temporarily solves the BSOD issue, as well.
The point is that it could have. Or maybe some unknown 0-day gets used by someone out to cause chaos instead of collect random.
That's true
On one hand I hope people are smart enough to run updates to critical systems on a test environment, first. On the other hand I've learned that that is not at all the case yesterday.
Many security products have no test option. One Iโm using has a best practice of a 15 minute delay between test and prod and no automation to suspend besides relying on the vendor to pull the update it within 15 mins if it were to go full crowdstrike.
The problem her was that this wasn't a traditional update. It was delivered automatically as a "content" update (like how old av would have definition update). We were given no room to test.
Then the internet would blame it all on Linux.
However, the recovery process would be much faster. The Linux kernel would try to load the kernel module and if it fails it would skip it.
Don't forget that ftp.cdrom.com , the biggest server on the Internet at it's peak, was running on FreeBSD.
I have no idea what the hell that is...
But Netflix runs on some BSD too
It's where you would download your anime and Quake 2 installer twenty years ago.
Interesting!
Freebsd, as does whatsapp and PlayStation
There's a free blue screen of death? All of these people paying for Windows for no reason /s
2038 is the next big thing to hit older *nix based OS. It will be Y2K all over again.
Maybe on my 32-bit ARM server with ancient kernel it will. Any 64-bit machine is immune.
...unless it's running software that uses signed 32-bit timestamps, or stores data using that format.
The point about the "millennium bug" was that it was a category of problems that required (hundreds of) thousands of fixes. It didn't matter if your OS was immune, because the OS isn't where the value is.
...timestamp is signed? Why?
Edit: Oh damn, I never noticed that the timestamp is indeed signed. For anyone curious, it is mostly historical as early C didn't really have a concept of unsigned
It also allows users to store dates back to ~1902.
It'll be 911 times 1000.
It'll be 911,000? As long as it's stored with 32 bits that should be fine ๐คท
I agree. We've been able to do 6 digit math for decades now
Probably not. Most Linux admins know their systems and are able to navigate out of the situation with ease. But also most people don't use any corporate off-the-shelf software, because there are better options that are freely available.
Furthermore a Linux installation is dedicated and slim for one single purpose. The flexibility creates diversity.
Are you implying that Windows server admins don't know their shit?
I've scene some supposedly 20 year veterans who don't know the architecture of AD
Not to say that is all of them but I've scene some who really can't do anything outside of click some buttons.
No. They don't. They always need Microsoft support to solve situations and upgrades. You can also ask simple questions that they cannot answer. Try Active Directory: how to run AD in a secure fashion? Or: What services do rely on DCs in our company?
My guy, I work cloud support for both Linux and Windows VMs.
I get dumbass cases from both all the time.
As a Windows engineer, the number of times I've seen other "engineers" open a case with Microsoft is insane. It seems to be a lot of their first reactions. No logs, no trying anything, just "this broke, why no work". I think it's that the Linux guys are mostly self taught, and the windows guys aren't.
I think it's more of "we pay Microsoft (or any company) for this. Make them handle it."
It's that kind of thinking that makes shit like the crowd strike problem possible.
Windows server admins: "We pay Microsoft for the service, damn right we'll use it!"
Linux server admins: "We don't pay anyone for the service, hopefully someone else had the same issue and posted about it somewhere..."
Interestingly, the latter ends up with better stability and security!
I think the shower thought is centered around IF a ubiquitous bug that required physical access to the machine to resolve occurred simultaneously across all Linux machines.
If you couldn't remotely resolve the issues, regardless of your competence, simply the WALK to each machine and hooking up a KVM to each one would take a long time.
Connect to your hypervisor remotely, pray it uses something like bhyve/FreeBSD
There won't be such case is my argument. No one patches a system "for fun" and automatically there except they really set it up like that. It would be only one kind of a case in one company.
Furthermore, you cannot compare Linux systems. A modem firmware with busybox is not the same as a Debian PC desktop. It works differently and has only the kernel in common. And in both cases they aren't patched at the same time. They are not even the same version, hell not even the same platform.
E.a. nothing will ever break like this. If it does, it will be one single case of a single IT department.
This combination of arrogance and complacency sort of thinking is how it does happen on Linux one day.
Linux also isn't as popular on the desktop or end user devices
Doubtful. By far, most servers responsible for Internet traffic are not running crowdstrike software.
This incident was a bunch of fortune 500 companies caught with their pants down.
Who do you think runs those servers? What do you think those companies run on their Linux servers?
Those companies aren't "the Internet." They're products connected to the Internet.
The OP argument is like saying the Internet is dead because Netflix is down.
A lot of people would say the internet was down if a large number of those products werenโt available. Also companies like Google do own parts of the physical Internet infrastructure.
If all of the parts of the internet that the average person finds useful goes down, then it matters little that technically "the internet" is not down. If it can't be useful then it is as good as "down".
And keep in mind, the falcon sensor exists for Linux. All those big companies largely use it.
Essentially we just got lucky that their buggy patch only affected the windows version of the sensor in a showstopping way. Could have been all major OS.
I don't think the Linux culture is very similar to the windows culture. At least for me personally, I wouldn't use crowdstrike and let them install whatever they want into my environment.
Maybe it's just me.
It's not your machine, your choice of distro, or your choice of specific packages to use or not use. It's a work tool you get handed as part of a job. So whether CrowdStrike runs on it or not is not your decision and you aren't allowed (and usually not capable) to change that.
That's an entirely different situation from one where you get a PC to do with as you please and set up yourself, or a private machine.
Plus we're mostly talking endpoint devices for non-technical users with many of these difficult-to-fix devices as techs have to drive out to them. The users expect a tool, and they get a tool. A Linux would be customized and utterly locked down, and part of that would be the endpoint protection software.
We tried to fight against having to install Crowstrike on our Linux servers but got overruled by upper management without discussion. I assume we are not the only ones with that experience in the world due to the need to check a checkbox for some flimsy audit.
I bet you could bring it up with them nowโฆ
You're actually confirming their point about culture though. The fact that you couldn't stop them doesn't mean that it also happened to everybody else: some management may have listened. Linux users abhor adding weird shit to their OS, Windows users do it all the time.
Essentially no one has crowdstrike on their personal machines. Not Windows users, Mac users, or Linux users. So it's corporate/large organization culture that matters. And they absolutely use it.
Are you an admin in a corporate data center? If not, you're not in the target audience for that product.
Yup. And I think that says more about the corporate culture than the company that caters to them.
Welcome to the world of big retailers! They would rather run Linux with crowdstrike than make their own system.
This mastodon user claims otherwise:
https://nondeterministic.computer/@mjg59/112816011370924959
That's only true if you run falcon-sensor in ebpf and not kmod mode.
Yep I know
The issuw didn't affect Linux and macOS systems with Crowdstrike Falcon installed, though, only Windows systems.
On Windows, booting into Safe Mode and removing
C:\Windows\System32\Drivers het bestand C-00000291*.systemporarily solves the BSOD issue, as well.The point is that it could have. Or maybe some unknown 0-day gets used by someone out to cause chaos instead of collect random.
That's true
On one hand I hope people are smart enough to run updates to critical systems on a test environment, first. On the other hand I've learned that that is not at all the case yesterday.
Many security products have no test option. One Iโm using has a best practice of a 15 minute delay between test and prod and no automation to suspend besides relying on the vendor to pull the update it within 15 mins if it were to go full crowdstrike.
The problem her was that this wasn't a traditional update. It was delivered automatically as a "content" update (like how old av would have definition update). We were given no room to test.
Then the internet would blame it all on Linux.
However, the recovery process would be much faster. The Linux kernel would try to load the kernel module and if it fails it would skip it.
Don't forget that ftp.cdrom.com , the biggest server on the Internet at it's peak, was running on FreeBSD.
I have no idea what the hell that is...
But Netflix runs on some BSD too
It's where you would download your anime and Quake 2 installer twenty years ago.
Interesting!
Freebsd, as does whatsapp and PlayStation
There's a free blue screen of death? All of these people paying for Windows for no reason /s
This already happens any time the domain name servers go down: https://techcrunch.com/2022/06/20/cloudflare-outage-knocks-popular-services-offline/
2038 is the next big thing to hit older *nix based OS. It will be Y2K all over again.
Maybe on my 32-bit ARM server with ancient kernel it will. Any 64-bit machine is immune.
...unless it's running software that uses signed 32-bit timestamps, or stores data using that format.
The point about the "millennium bug" was that it was a category of problems that required (hundreds of) thousands of fixes. It didn't matter if your OS was immune, because the OS isn't where the value is.
...timestamp is signed? Why?
Edit: Oh damn, I never noticed that the timestamp is indeed signed. For anyone curious, it is mostly historical as early C didn't really have a concept of unsigned
It also allows users to store dates back to ~1902.
It'll be 911 times 1000.
It'll be 911,000? As long as it's stored with 32 bits that should be fine ๐คท
I agree. We've been able to do 6 digit math for decades now
Probably not. Most Linux admins know their systems and are able to navigate out of the situation with ease. But also most people don't use any corporate off-the-shelf software, because there are better options that are freely available.
Furthermore a Linux installation is dedicated and slim for one single purpose. The flexibility creates diversity.
Are you implying that Windows server admins don't know their shit?
I've scene some supposedly 20 year veterans who don't know the architecture of AD
Not to say that is all of them but I've scene some who really can't do anything outside of click some buttons.
No. They don't. They always need Microsoft support to solve situations and upgrades. You can also ask simple questions that they cannot answer. Try Active Directory: how to run AD in a secure fashion? Or: What services do rely on DCs in our company?
My guy, I work cloud support for both Linux and Windows VMs.
I get dumbass cases from both all the time.
As a Windows engineer, the number of times I've seen other "engineers" open a case with Microsoft is insane. It seems to be a lot of their first reactions. No logs, no trying anything, just "this broke, why no work". I think it's that the Linux guys are mostly self taught, and the windows guys aren't.
I think it's more of "we pay Microsoft (or any company) for this. Make them handle it."
It's that kind of thinking that makes shit like the crowd strike problem possible.
Windows server admins: "We pay Microsoft for the service, damn right we'll use it!"
Linux server admins: "We don't pay anyone for the service, hopefully someone else had the same issue and posted about it somewhere..."
Interestingly, the latter ends up with better stability and security!
I think the shower thought is centered around IF a ubiquitous bug that required physical access to the machine to resolve occurred simultaneously across all Linux machines.
If you couldn't remotely resolve the issues, regardless of your competence, simply the WALK to each machine and hooking up a KVM to each one would take a long time.
Connect to your hypervisor remotely, pray it uses something like bhyve/FreeBSD
There won't be such case is my argument. No one patches a system "for fun" and automatically there except they really set it up like that. It would be only one kind of a case in one company.
Furthermore, you cannot compare Linux systems. A modem firmware with busybox is not the same as a Debian PC desktop. It works differently and has only the kernel in common. And in both cases they aren't patched at the same time. They are not even the same version, hell not even the same platform.
E.a. nothing will ever break like this. If it does, it will be one single case of a single IT department.
This combination of arrogance and complacency sort of thinking is how it does happen on Linux one day.
Linux also isn't as popular on the desktop or end user devices
Doubtful. By far, most servers responsible for Internet traffic are not running crowdstrike software.
This incident was a bunch of fortune 500 companies caught with their pants down.
Who do you think runs those servers? What do you think those companies run on their Linux servers?
Those companies aren't "the Internet." They're products connected to the Internet.
The OP argument is like saying the Internet is dead because Netflix is down.
A lot of people would say the internet was down if a large number of those products werenโt available. Also companies like Google do own parts of the physical Internet infrastructure.
If all of the parts of the internet that the average person finds useful goes down, then it matters little that technically "the internet" is not down. If it can't be useful then it is as good as "down".
The end is nigh, I tell you!
Y2k38