Some bad code just broke a billion Windows machines
TimeSquirrel@kbin.melroy.org to 
youtube.com
Cybersecurity firm Crowdstrike pushed an update that caused millions of Windows computers to enter recovery mode, triggering the blue screen of death. Learn ...
You are viewing a single comment
Whoda thunk automatic updates to critical infrastructure was a good idea? Just hope healthcare life support was not affected.
Many compliance frameworks require security utilities to receive automatic updates. It's pretty essential for effective endpoint protection considering how fast new threats spread.
The problem is not the automated update, it's why it wasn't caught in testing and how the update managed to break the entire OS.
It is pretty easy to imagine separate streams of updates that affect each other negatively.
CrowdStrike does its own 0-day updates, Microsoft does its own 0-day updates. There is probably limited if any testing at that critical intersection.
If Microsoft 100% controlled the release stream, otoh, there'd be a much better chance to have caught it. The responsibility would probably lie with MS in such a case.
(edit: not saying that this is what happened, hence the conditionals)
I don't think that is what happened here in this situation though, I think the issue was caused exclusively by a Crowdstrike update but I haven't read anything official that really breaks this down.
Some comments yesterday were claiming the offending file was several kb of just 0s. All signs are pointing to a massive fuckup from an individual company.
Which makes me wonder, did the company even test it at all on their own machines first?
Nah EDR is pointless like all of cybersecurity. All these compliance frameworks are just a further grift to get a slice of B2B procurement budgets. The practice of cybersecurity has caused a more severe widespread outage than any malware ever could.
lol, ok
OP is not entirely wrong. At least in Linux land you can now implement EDR like functionality entirely with EBPF without installing a fucking rootkit. So traditional EDR products are a grift if you are on the bleeding edge.
Ok Russian comrade. Security in companies is terrible. You’re right. It’s just a giant grift.
Now, go buy some limited time offer fight fight fight shoes from agent orange.
Genuinely, what? What is "fight fight fight shoes" and "agent orange" like the chemical? What does me being Russian have to do with it? Is this some kind of twitter lingo I've touched grass too much to understand?
EDIT: Figured out it's probably a trump reference. Idk I'm not a trump fan so idunno.
Hospital stuff was affected. Most engineers are smart enough to not connect critical equipment to the Internet, though.
I’m not in the US, but my other medical peers who are mentioned that EPIC (the software most hospitals use to manage patient records) was not affected, but Dragon (the software by Nuance that we doctors use for dictation so we don’t have to type notes) was down. Someone I know complained that they had to “type notes like a medieval peasant.” But I’m glad that the critical infrastructure was up and running. At my former hospital, we used to always maintain physical records simultaneously for all our current inpatients that only the medical team responsible for those specific patients had access to just to be on the safe side.
That's actually a very smart idea, keeping physical records of every inpatient. Wonder why the ai companies don't do transcription of medical notes, instead of trying to add ai features to my washer/ dryer combo. Just seems like a very practical use of the tech
They do, one of the things my hospital is working on implementing, much to my chagrin, is an AI thing where the doctor leaves their phone out during the visit. It listens to the patient and the doctor and generates a note. I think it’s a Nuance product, I’m not directly involved with the implementation.
For me, as soon as I see a doctor have his phone out I’m telling him to put that shit away and I don’t consent to some app listening to what I’ve got to say.
I meant to transcribe handwritten patient notes to dogital after the patient is released. That way op can retain the physical records for inpatients to mitigate another crowdstrike situation, and still have searchable records long term. Recording a patient consultation sounds like a bad idea all round. They'd have to figure out how to read doctors handwriting though,bbut I gather most of the difficulty is due to the cryptic names they have for things.
I doubt you can decline, any more than you can tell them not to use a notebook.
It is subject to actually serious HIPAA regulations though.
I’m aware of the regulations, I’m an Epic analyst and been in the world of healthcare since 2009 (used to be an MLS). Watch me decline right out the door and immediately contact patient relations. Put your fucking phone away. Healthcare has managed for quite a while without them.
Saying a personal cell phone is equivalent to a written notebook is wild. Can you compromise a notebook with malware?
The doctor will very likely have no choice but to send you to complain to someone else. Not recording will violate policy if that's what their system is, and it's an entirely reasonable policy to have. Medical interactions are heavily documented for a reason, and it's because mistakes can literally kill people.
All your records are already electronic, and a notebook is a far bigger security risk than a cell phone is. It's not encrypted. Anyone can walk away with it.
I disagree. I don’t consent to a doctor having his personal device out recording. Doesn’t seem like either of us will be changing our minds, so I’ll bid you adieu.
I’m an Epic analyst - while Epic was fine, many of our third party integrations shit the bed. Cardiology (where I work) was mostly unaffected aside from Omnicell being down, but the laboratory was massively fucked due to all the integrations they have. Multiple teams were quite busy, I just got to talk to them about it eventually.
This is pretty much correct. I work in an Epic shop and we had about 150 servers to remediate and some number of workstations (I’m not sure how many). While Epic make not have been impacted, it is a highly integrated system and when things are failing around it then it can have an impact on care delivery. For example if a provider places a stat lab order in Epic, that lab order gets transmitted to an integration middleware which then routes it to the lab system. If the integration middleware or the lab system are down, then the provider has no idea the stat order went into a black hole.
Our lab was absolutely fucked from multiple integrations going down. I’m a Cupid analyst and we weren’t really affected. What app do you work on?
I’m an integration guy at my roots but I lead a variety of different teams at the moment. We use Corepoint as one of our interface engines and it shat the bed big time. We had to restore it from backup, which was nuts in my opinion. We had a variety of apps impacted.
That’s cool. I was going to move over to our integration team but I’m looking into Epic consulting instead. Our integration team was very busy on Friday along with our clinical apps team. We use Cloverleaf for our interface engine, I’ve got a bit of experience poking around in there. HL7 is interesting, but I’d like to learn FHIR. Do you have a Bridges cert?
I’m Bridges certified as well as in Cloverleaf, which we also use. FHIR is great but it doesn’t require much in the way of integration engineers.
Huh. I thought medieval peasants were usually illiterate? Even less computer literate?
I work healthcare adjacent and some providers were affected as expected. Hoping as well that those critical systems were not, but that chance is non zero.