The Batshit Crazy Story Of The Day Elon Musk Decided To Personally Rip Servers Out Of A Sacramento Data Center

Technology@lemmy.world – 1030 points – 1 years ago

Back on Christmas Eve of last year there were some reports that Elon Musk was in the process of shutting down Twitter’s Sacramento data center. In that article, a number of ex-Twitter employees wer…

You are viewing a single comment

View all comments

I guess I'm also batshit crazy.

It's Twitter. Who cares if people can't tweet for an hour.

I'm with Elon on this, don't overcomplicate the closing of a data center.

That manager, when asked to do it in 90 days, if s/he was competent should have said: I'll do it, but you'll have to accept a downtime risk.

(But aside from this entertaining story, I do think Elon lost his shine. Wasting $40B on twitter and sabotaging Ukraine while simping for Putin and Trump and not paying taxes... yeah, get rekt Elon).

It's effectively a case of "I left my house unlocked and unarmed while I went on vacation. No one broke in, so I don't see the point in door locks and alarm systems."

Twitter got very VERY lucky that the worst that happened was some outages.

They moved hyper sensitive user data in a moving truck. If anything had gone wrong they would've exposed millions of peoples sensitive data.

You are supposed to wipe the servers before you move them, you shouldn't be driving servers around on the highway while they are still chock full of peoples credit card info and shit.

What sensitive data does Twitter hold? Genuinely curious

We don't know what was on those servers, but it was apparently sensitive enough that the government redacted descriptions of the data in court filings.

The US government brief said the relocated servers were not wiped before being moved to a new data center. The type of data on the relocated servers was apparently so sensitive that it could not be described in the US court filing, which redacts the sentence that describes what the servers contained.

https://arstechnica.com/tech-policy/2023/09/us-government-slams-musk-in-court-filing-describing-chaotic-environment-at-x/

Isn’t all of it encrypted though? Like I understand physical access to servers is generally bad, but you’d think once the the things are unplugged it would be difficult to access the data again without bypassing encryption. I’m not a software engineer though

I'm a security engineer, and encryption is great, but can be bypassed. Relying on encryption assumes it was implemented properly, that the system was shut down properly so all keys were flushed correctly, and the encryption algorithm doesn't have weaknesses.

Generally if somebody dedicated enough can acquire physical access to a system, they can probably find a way into it given the right resources. Did that happen here? Probably not. Could it have? Absolutely. That's why most enterprises or government hard drives are shredded rather than just relying on them being wiped or encrypted.

Encryption is part of the solution, but it's not automatically the complete solution.

Probably because the government is still illegally spying on citizens and they don't want the specifics to leak out.

You don't consider credit card info sensitive? May I have yours?

BS, I don't know if Twitter holds credit card data, but if they did, they would have needed to abide by PCI DSS rules, which requires encrypting the data in special hardware security modules.

So no, moving those servers wouldn't put the data at risk.

encrypting the data in special hardware security modules

Tell me you don't understand how PCI works without saying you don't understand how PCI works.

Those systems can very much store PCI data and it's very much possible that those were the systems that contain information as most of the times it's on general servers.

Personally identifiable information (PII) is any set of data that has a chance to uniquely identify a person, including name, address, credit card info, social security, etc. It can also include things like birthdate, city, IP address, and so on, depending on how the combination of data works. The general rule of thumb is that you want to aggregate out to the city level at least, or completely anonymize the data. These, I’m supposing, we’re raw records that contained account info.

While the engineers could have said that, equally Elon could have asked what the problems/downsides were to doing it faster.

You think?

I'm 100% sure!

Edit: you may want to reconsider you being "on the side of musk on this one." From the article:

" And, of course, it didn’t really “work.” As we detailed, Twitter toppled over a few days later, and this excerpt admits it was because of the “server move.” The article does note that Musk himself eventually said he shouldn’t have done this and it did cause a fair bit of problems for the site, including the disastrous “Twitter Spaces” "

That manager, when asked to do it in 90 days, if s/he was competent should have said: I'll do it, but you'll have to accept a downtime risk.

That is the correct take in general, but I've worked for managers a bit like Elon before, and that never would have worked. It would have been the equivalent of tendering that resignation, because he sees any pushback at all as insubordination, and not to be tolerated.

The only way out of this is to suggest an alternative course of action, but make it seem like it was his idea all along. My favorite method was to tie it to some bullshit metric he previously set for no reason. "Yes, sir, we could do it right away, but it may have an impact on our ad throughput, and I know ad revenue is a key prioriry you have personally set for the company. If we take the time to transition our ad platform first we can keep our revenue consistent". Then that time ends up identical to the amount of time needed to move things correctly.

It still might not work, but at least you've stroked the Boss's ego, which can keep you employed for a few more months while trying to find the exit ramp.

Musk quite literally said it was equivalent of resigning:

Over the years, Musk had been faced many times with a choice between what he thought was necessary and what others told him was possible. The result was almost always the same. He paused in silence for a few moments, then announced, “You have 90 days to do it. If you can’t make that work, your resignation is accepted.”

Exactly, and that's why turning it into a risk assessment, which would work with competent management, would never work with him. The only way to make it work would be to find a way to stroke his ego.

I know the type. Elon is different though.

Yeah, he is overrated by many. But he's not the typical stupid middle manager, who get brain freeze when presented a simple dilemma.

Fuck, Elon would fail hard as a middle manager. This clock ticks different. I have worked with a lot of people, but I don't think he fits in one of the common types.

I think “asshole” is a pretty common type.

I mean it was more of a data loss and data security risk but okay

I can understand the pushback. Twitter employees work for years to transform a site that went down a lot (remember fail whale?) Into reliable well oiled machinery. Having new owner suddenly tell you to dismantle it all must've been awful.

That, I can understand.

But reforming a company is always ugly business. Best to get it done with quickly and efficiently.

A slow painful death like Yahoo also isn't in anyone's interest.

1 more...