The Batshit Crazy Story Of The Day Elon Musk Decided To Personally Rip Servers Out Of A Sacramento Data Center
stopthatgirl7@kbin.social to 
techdirt.com
Back on Christmas Eve of last year there were some reports that Elon Musk was in the process of shutting down Twitter’s Sacramento data center. In that article, a number of ex-Twitter employees wer…
This article is fucking hilarious top to bottom and if you came here to comment without reading, I highly suggest you read it.
Absolutely worth the time.
Pocket-knife-prying-open-floorboards crazy. Biggest asshole I've ever read about crazy.
Jesus, I thought you were just using that as a figure of speech so that we could all understand that Space Daddy Musk was exhibiting meth-head-like tendencies, but no, he literally diverted a flight from Austin to Sac at the suggestion of his cousin, drove in a Corolla to the data center (edit: at 2 in the morning on Dec 24), and used his pocket knife to pry up the floorboards.
Fuck, how much cocaine has he been doing? He's about to hit John McAfee levels of bad decision making.
Also, Elon, I was just kidding about the "Space Daddy" stuff. If you want to send a few pounds of blow my way, HMU.
Do you really want to ask anything out of someone who diverts a flight and drives to a random data center at 2 am on Dec 24 to take it apart with a pocket knife?
I mean, I've gotten drugs from sketchier people. And if we're talking about stimulants, that's almost an advertising point. "So good, it got Elon pulling up floorboards and crashing servers!" is a decent testament to how strong it is.
Plus, if this was a serious conversation, the first thing I would do would be to test for purity and contaminants. Test kits are relatively cheap, and even if you're open to getting baby powder spiked with bath salts in place of cocaine, it's better to know what you're getting into. Test your drugs, people!
Some poor engineers getting paged late at night:
"WTF?! Why does the servers shutting down one after another? Do we have a rodent on the loose in Sacramento?"
*Check CCTV*
"What is that thing crawling under the floorboard?"
It was a Muskrat, very damaging to your companies infrastructure. Some even deem them a plague.
"Rodents. Release the halon gas."
"Wait no, thats Elon Musk!"
"..."
"Release the gas."
It's the corolla bit that makes this. I don't know why, but he needed to list the detail of the car and that had to eat Musk up a little bit.
just imagine the face of "Alex the Uzbek" while he is watching the manChild pry open the floorboards, knowing that he should stop the maniac and calculating that he shouldn't stop the second richest man in the world 🤣
Alex didn't want to go back to Uzbekistan, or be threatened with it regardless of citizenship by ol' tusky Musk.
Yeah, there were many rumours that Musk's actions have caused waves and waves of people to quit Twitter or do something they know will get them fired. The majority of the ones who are left are the ones on H-1B programs who get kicked out of the country if they lose their job and can't find a new one within something like a month.
Très cool comme pseudo :p
;)
You were right, I wasn't gonna read it because I knew he was an idiot but the article is hilarious and everyone should read it.
Good article. Nobody else would get away with being this risk taking and careless, and the only reason he does get away wirh it is because he is the boss.
Money is one hell of a drug.
Techdirt is the shit tbh
"What I wasn’t told was that we had 70,000 hard-coded references to Sacramento. And there’s still shit that’s broken because of it.”" Im pretty sure he was told but was either not really listening or comprehending.
Yeah "I wasn't told" is really "I wasn't listening" in narcissist language.
"I wasn't notified in triplicate, in writing, with the relevant sections underlined and requiring my signature for acknowledgment"
Even that wouldn't stop a stubborn narcissist.
Well, no, but it's good to have such documentation for legal reasons when said narcissist loses their shit and attempts to scapegoat you
"I wasn't notified in triplicate, sent in, sent back, queried, lost, found, subjected to public inquiry, lost again, and finally buried in soft peat for three months and recycled as firelighters".
Spot on
I’m sure he was told… But brain exploding emoji
Sounds like he did his best to make sure no one could tell him
Well, there's your first problem. Musk doesn't give a shit about what women think. The man is an S tier misogynist. He probably zoned out listening to this woman and wondered how many horses it would cost to get a handy out of her.
or he just didnt bother asking anyone before he started
What a fucking moron. This is the guy whose self-driving car you trust?
Nope. Once upon a time long ago I thought I wanted a Tesla. Now? I'd rather drive a Ford Focus.
I'd rather drive a Ford pinto.
For real. I love those ugly things.
Ah man, I love the Ford [Dick in Portuguese]
I'd rather drive an old lada made out of depleted uranium than a Tesla.
I'd rather ride a bike
I feel personally attacked here.
Well, I drive a Kia Soul, so I'm probably not one to be throwing stones.
Luckily, he's not the one making them!
Nope, just the CEO of the company that has regularly overridden the decisions of his engineers, driving anyone with any actual experience away.
Autonomous vehicles built by hundreds of first-year software engineering graduates seems like a good idea though.
But but, it makes fart sounds!! /s
......what?
Teslas have an "Emissions Testing Mode" on the centre console that's just a fart sound generator.
Hs still impacts critical decisions. He needs to be kept very far away from money and business.
I’m shocked that the data center required retinal scans but that the employee with access could then just hold the door and let him and others in.
I used to work at a data center with lots of security. To get into the area with the servers you had to go through a man trap. It was a room a little larger than a telephone booth with automatic doors on both sides. To open the first door you needed a physical card key. Once inside the door closed, then to open the inner door you needed to both enter a PIN and have your hand scanned in a biometric scanner. Only after all that could you get inside. The booth also weighed you, and if your weight was off by a certain amount after your last pass through then it wouldn’t let you in. That was to prevent somebody from piggybacking with you.
lmao mental image of Daniel Craig riding someone piggyback in a tuxedo and holding a silenced pistol
Prop yourself from wall to wall so your weight doesn't count.
Spiderman be stealing yo bits.
It'll only work if you use three Bonds' in a trench coat!
How do you get big equipments (e.g.a pallet of server components, or a whole rack of new servers) into the area?
Separate double bay doors. They have a pair for each floor that opens to an outside wall. You use a forklift to get the pallets up. That or there is a big ass freight elevator, depending on the data center.
So it's basically this setup https://youtu.be/cP4d74Qk3ac?si=Fq_I12sU4uIAgm7w
likely admin override right? i think as a lowly employee this is what he had to do
There was third party security at this particular facility. You had to show your ID, have them confirm you’re authorized to bring equipment in, then they’d bring it through a locked freight door for you.
Oh, god, so that little death hallway to the red queen in resident evil was ACCURATE
Even the smaller data center I used to go to would have an alarm go off if the door was open for more than a few seconds. The first door opened with your hand being scanned and the cage to our racks could be opened with a key card.
How did they get racks in there? Or like big blade servers?
I'm sure there were other, larger, entry points that could be opened for moving equipment in and out. They would then be locked down during normal operation.
That's pretty cool, sounds like some MacGyver shit!
Fatphobic security smh.
My understanding is you can be as heavy as you want, you just can’t gain more than 20kg in a week
Is that a challenge?
- Niko Avocado
This is the same guy who wants to put implants into people's brains and send them to Mars. Let that sink in.
Yeah, I literally just sent this article to my partner with the commentary "...and this is why I no longer think we should ever buy a Tesla"
I don't want this egomaniac in charge of anything that has real-world safety implications for me and mine.
"Look, just hand me scalpel and I'll do it."
"look, just go to home Depot and get a carpet knife for 1/10th of the cost!"
You think Twitter is made of money?!? Grab some plastic cutlery from the cafeteria.
Lmao, Musk is the new Gary Young, the supplment king. Behind the Bastards did an episode on him, he really wanted to be a surgeon without that pesky medical license requirement.
https://www.listennotes.com/podcasts/behind-the-bastards/part-one-gary-young-the-fake-LlhDs_-FESt/
I am all for billionaires wanting go to Mars. They will just be on their own, getting back. Oopise daisy, the rocket only had enough fuel go to Mars and not back. If they want to come back, well they all that wealth to pay us and we'll get them, one day....soon...eventually.......maybe.....probably not.
If you're cold, they're cold. Let those sinks in!
LMAO who's this James guy and why does he understand Musk so thoroughly like his own spouse?
Plot twist: Elon and James are the same person. Like in Fight Club.
James is the personality that comes out during particularly extreme manic episodes.
So all the dumb shit that Musk has done over the past year is all just part of project mayhem. (Don't eat the soup, it has piss in it)
But that would make him cool somehow.
Then think of him like OJ Simpson and Charlie. Problem solved.
On CHRISTMAS FUCKING EVE! He has like 10 kids. He started a fire drill for employees on Christmas Eve, they have families too. What a cartoonishly villainous thing to do.
I think James is Elon's cousin.
https://www.businessinsider.com/elon-musk-has-hired-two-cousins-to-work-at-twitter-2022-12
Twitter employees must've dreaded seeing this guy. "Oh no, Elon is with James again. Shit is about to hit the fan"
I can't imagine how annoying that would be. Companies with family members as employees, especially startups, are the WORST and most toxic you can get.
A former company was full of nepotism. Chinese managed company with all the stereotypical inner circle politics.
The CEO's daughter was appointed "chief Green officer". We got metal bottle waters.
The daughters boyfriend was put in Sales, and never sold a thing.
To be fair, that seems to be the best case scenario for nepotism. Imagine if the daughter and her boyfriend were put into positions of real power without the expertise to back it up. The dad seems to understand and put them kids into harmless job role.
Chinese take nepotism to a new level
One day, one of these stunts he pulls is going to end up ruining whatever company he does it in, and I'm all here for it. Though we'll probably never know since he'll just blame it on something / someone else and his little muskettes will follow along.
It’s not one day. It’s happening in the headlines as we watch. Some estimates are that twitter has lost 90% of its value in the (a bit under a full) year since Elmo took over. Post-rebranding, some financial institutions and even one of Musk’s own dumb-ass shoot from the hip tweets puts twitter’s current value at around $4-5B.
Even if that’s low, I think the best case estimate, before rebranding, was sitting around $15B. That’s still a loss of 2/3 value in less than a year (that was in May) and it hasn’t gotten better since the attempted rebrand.
It’s happening, and his incompetence is on full display. He’s even reached the stage of megalomania where he’s blaming the Jews.
No, Elmos own estimates are that it lost 90% of its value, probably lost a lot more.
Hey now, you might be on to something here...
Shhh.... don't tell him how this year went. It'll ruin the story.
That’s company’s name? Twitter.
*X
Fucking Apple autocorrected me lol
Even Apple does not agree with their shitty new name. 😄
I respect it enough to call it Xitter. Pronunciation is up to your imagination.
Looks like he is literally asking for trouble with decisions like that. It’s just a matter of time until he manages to cause a major disaster.
It's a total lack of the concept of scale. I've moved servers like this, but when decommissioning them. Things are different when running a corporate data center than when moving a home lab. He doesn't grasp the difference, because he doesn't understand the scale.
Running and moving one computer is different from 150k of them. Hooking them back up the the network without a plan or documentation must have been a challenge.
I love how a data center is not considered reliable unless it has something like 99.9% uptime. It was costing him so much for a damn good reason.
And yet this idiot decided to start unplugging servers by himself at 2 in the morning.
Did he not realize that he’d have to pay the DC company for at least the quarter, regardless of if the servers are physically there or not? It’s not like you pay these places by the hour. They’re payed for by months or years, and there’s contracts involved.
He still had to pay the bill, I guarantee it. He also just wasted thousands of dollars paying random people from the street to move multi-million dollar servers, and opened himself up to millions of dollars in liability if any sensitive information was lost or stolen during this stunt.
Why do people give this man money again?
i mean, he may very well not have paid. this is the guy who refused to pay rent on headquarters, and two different people quit/were fired because they wouldn't just... not pay rent on his orders.
has he paid rent since? as far as i can tell, he just gets away with this shit. edit - looks like an eviction notice was issued this summer, so i'm guess he still hasn't paid.
If I miss my rent, it would take days for me to be fucked. How do the wealthy assholes of the world get away with this?
Shit, if I missed my rent by a day, my old landlord would have an eviction notice on my door by 6AM the next morning.
They have been evicted out of several offices for nonpayment.
"What do you mean "challenge", just do it" -Elon probably
“Just plug them in, Jesus fucking Christ.”
Whelp, this is just one of the reasons why Twitter employees sleep in their office.
You never know when your coked up employer might barge through the door.
Elon seems like the type of guy that would barge into someone's office when they're working unpaid overtime, offer them a line of coke, and then fire them for using drugs on the job if they accept (and take the cost of the line out of their last paycheck).
And if they don't accept... "You think you're better than me?!? You're not better than me! You're fired!"
Then snort the line after they walk out, followed by a short exclamation like “what an asshole that guy is.”
With servers or a sink.
Elon Musk is a privileged manchild who never grew out of his teenager phase, throwing around his inherited wealth like the kid from Blank Check and throwing temper tantrums anytime someone calls him out on his bullshit. Any claims to success he may have had been entirely in spite of him, not because of him. He doesn’t have any fucking idea what he’s doing and if any one of you or I failed even a fraction as much as Musk had, we’d have all been fired ten times over.
Sounds a lot like another conservative figurehead I could name...
Actually, it sounds a lot like almost all conservative figureheads in US politics.
They're completely idiotic and yet way more successful than I'll ever be. But then again I have a guilty conscience.
They're "successful" if you define success as being born into wealth and/or being willing to ignore the suffering of others. For you and me (and most right thinking people), that's no real measure of success. Unfortunately, capitalism and the US political systems gives power to the people least suited to wielding it.
And that's why their base votes for them. Because they can relate.
That is an insult to teenagers
So glad I ended up not working at Tesla
This is fucking insane and would drive me nuts.
This guy really is a Space Karen
This guy is considered to be a genius? This guy is a fucking billionaire?
I’m dead.
As a non-physicist, what is the technical reason Elon was wrong? I assume that when the CEO said 500 pounds, they meant 500 pounds of force relative to some surface area of the floor? I'm guessing that surface area was significantly larger than one wheel on the rack, so the combined force of all 4 wheels was still well over the limit. Maybe someone who knows physics could explain better.
Let's say you weigh 200 lbs. When you stand on a scale with two feet, that's 200lbs ÷ 2 feet. So the scale reads 100 lbs, right? Of course not. Increasing the number of touch points doesn't reduce the mass.
Now what if you stand on two scales side by side, one foot on each? Then they'll each read 100lbs. The load is distributed across the touch points, but the total mass when you add them back up remains the same.
So what does that mean for ol muskaroo? It's hard to say who's correct without knowing more about the floor. If it's server tiles that are hollow underneath and each tile can hold 500lbs individually, maybe it's ok if the cart was large enough that two wheels would never be on the same tile.
But the bottom line is that when the guy that runs the server room says not to do it, you don't fucking do it. Have a little respect. Sure, Musk is the owner so it's kind of technically his server room, but he's being a prick regardless.
I think he was renting the space, so he doesn't own the server room, just the servers in it.
I don’t know physics too well, but I’ll try to explain.
First of all, look out for pressure. Slamming your hand on a desk(lots of surface area) may not hurt much, but doing the same thing on a thumb tack(very low surface area) will suck, even though it’s the same amount of force. Pressure is just force/area (I’m probably oversimplifying).
So not only is there still 2000 pounds of force on the floor, it’s all concentrated on one(well, four) areas. Meaning that there’s a high chance the floor will break under those wheels. You’d actually have better luck just sliding the server across the floor.
Elons logic is also just stupid here. An elevator can’t lift a 1,000 pound box, but can it lift four 250 pound boxes? No! Even a child could answer that. The fact that he just assumed that adding four wheels magically distributed the weight is stupid. What if you had five wheels? Eleven? It’s not rocket science (which is quite ironic, given the company he owns).
So yeah. I’ve got no idea how he’s a billionaire. No fucking clue.
The simple reason is that it depends on what in the floor can't actually handle the 2000 lbs. If it's a floor 1'x1' floor tile that will break, then Elon is right. If the loade limit is a beam that spans a larger distance, then he's totally wrong.
In places like a server room, you typically have a raised floor that supports tiles in the neighborhood of 1.5'-3 feet square. (The raised floor allows for all the cabling and air con to be run around the systems.) If you say that the floor can't support more than 2000lbs that typically means they can't guarantee enough of a safety margin and you run the risk of the object breaking through the floor. Musk's wheel argument is crap unless he can be sure each wheel is not on the same floor support area. (Which obviously he can't.)
However floors the spec will typically have some safety margin and that probably kept him from going through the floor. His logic, while not 100% wrong in the basic statement, lacked a deeper understanding of what was going on and certainly doesn't help the idea that he's a Tony Stark genius. It was a Dunning-Kruger level dumb statement to make.
If this statement was made in isolation I would give him the benefit of the doubt and assume he realized it was a stupid statement once he said it but he just didn't bother to correct himself. However he's made so many dumb and arrogant statements over the past few years, I assume it was just a dumb unsophisticated statement from someone who isn't that bright.
Funny thing is this kind of behaviour isn't unique to Musk. A lot of entrepreneurs and CEOs seem to have similar kind of attitude. They want everything done cheaper faster and there's no 2 ways about it. It's their way or highway. If shit goes to hell it's other people's heads that roll.
It's frustrating they don't even know what to be angry about. Like instead of flying to Sacramento and ripping out 5000 servers why not flip out that the code has 70,000 different hard coded references to a single data center instead of one.
He did not know that at the time because his poor little brain started hurting.
This is exactly why our last governor here in Illinois, didn't CEO Governor Rauner, was probably the most ineffectual governor of our state ever had. He literally had no capacity for compromise, and was a Republican, in an Illinois where the legislature has long been solidly Democrat. I can't think of a single thing he actually got done. All because he was a CEO trying to run Illinois like a business.
I do not recommend working in Compliance then either lol!
Yeah, that's basically why I got out of IT. Too many managers/clients refusing to listen to warnings about what would happen when they did X, then blaming the techies when things went to shit.
Because they are the "boss", they have 0 accountability. Worst case for them is a golden handshake, and failing upwards where the cycle starts again.
I do not think it is a coincidence that Tesla has recently released the updated Model 3 to some decently positive reviews. I think that is in no small part to Musk being so distracted by Twitter that he hasn't been able to fuck up things over at Tesla in a while.
OMG OK that's it. Tesla cars are now out of the question for me and if I ever get the chance to ride on a SpaceX ship (not very likely) I think I'd decline. Totally different companies ofc but the same master "mind" behind.
This guy represents everything that you do not want to see in a CEO.
NO THANK YOU
It really seems like SpaceX worked for real. They now have the best safety record for any booster and most of the world’s space traffic. What’s their reusability record now, is it 16 flights on one rocket? You can’t argue with that result.
I don’t know what he did to get to the point of “fail fast” during development but they put their money where their mouth is. Multiple catastrophic test failures that would have been career ending anywhere else, seemingly weren’t, and they appeared to have a very fast (for rocketry) and very successful program
From what I understand, SpaceX made real effort to split the company into two operations. One uses the reliable Falcon 9 system launching from Cape Canaveral (and other established launch facilities) to put satellites and astronauts into space. The other operation is Elon Musk's playground in Boca Chica where he tries to build the biggest spaceship ever!
Don't get me wrong, there are some good engineers working at the Boca Chica operation, I've heard the Raptor engine is really good and there's probably some other things they've made there that will be useful for rocketry in general. And who knows, the really smart people may get the biggest rocket ever to actually work someday despite Musk's stupidity.
SpaceX is now old space. They are the launch capacity of the USA. It's ridiculous actually
Stockton Rush was an orphan in comparison
He got lucky nothing disappeared.
At a previous work place they rounded up a few employees to move stuff from one office to the new office. That ended up with a few monitors less than they started with. They couldn't ask who took it because they never wrote down who they rounded up for the move.
And that's how companies end up with a bunch of silly regulations how you're not allowed to move any hardware to the next room
Seriously. A crew with no IDs and some of them formerly homeless hauling around hundreds of thousands of dollars of servers all secured with "big" padlocks. What could go wrong? Not like the crew could get a bolt cutter to open the padlocks and then sell the servers. I doubt many people would have qualms with buying stolen servers from Twitter.
I didn't see anyone steal this server
Did anyone else reading this bit immediately think of that other rich idiot that died in his ridiculous submarine?
has anyone tried to get musk a submarine?
maybe we could crowdfund this
Call it "Xubmarine 420" and tell him he invented it. I think he'd take that bait.
One of the richest people in the world, everyone.
This is why I wouldn't trust a thing that comes out of his mouth. He lies, he says really stupid shit and then he gives people an ultimatum to turn his stupid shit into reality or get fired. Safety, security and reality be damned. If you've ever wondered why people end up dying in fiery crashes because of "autopilot", or "full self drive", this is why.
They were somewhere over Las Vegas when James made his suggestion that they were in batshitcrazy country.
Can't stop here, this is asshole country
How many assholes we got on this plane?
They were somewhere around Barstow, on the edge of the desert, when the drugs began to take hold.
That reads like something out of The Hitchhiker's Guide to the Galaxy.
Musk must have been the offspring of an unspeakable accident between Zaphod and one of those Sacramento racks.
I was thinking an episode of Silicon Valley but yeah, Hitchhiker's Guide fits too...
I really hope someone will just push musk out a window. He's so incredibly stupid he doesn't deserve his money or power.
Don't worry, he'll upset Putin soon enough.
What a giant twat muskrat is.
The dude is a war criminal
You mean how he undermined US and Ukraine intelligence and leaked information to Russia? Those war crimes?
No how he shutoff Starlink to Ukraine as they were mounting an offensive to reclaim their stolen land.
Sounds treasonous
Why is your user tagged as
banned?I posted a funny drawing that had a dick in it and got banned for posting "porn".
I've heard of chaos testing going as far as walking into a DC and pulling plugs until alarms start going off, but this is just awful all around.
Imagine one day, he walks into spacex, board on a rocket, and shoot to mars.. just like this!
Maybe someone should trigger him in twitter
"Elon musk is a drooling socialist cuck who is apparently far too cowardly to meet my challenge of a firm July 4th, 2024 launch date for his personal mission to mars. Jeff bezos could easily do this if he wasn't too busy actually running the company that he owns."
Maybe we could get him interested in submersibles...
Yeah, get him to try to debunk the English cave diver for real this time.
"Elon Musk couldn't possibly make a submarine better than OceanGate!" 🤭
Welcome to software engineering Elon.
I'm picturing Taika Waititi's character in Free Guy
He's pretty obviously a parody of Musk. Also Mark Rylance's character in Don't Look Up.
Free Guy is 2021.
Mark Beaks from Disney's 2017 Ducktales remake was ahead of its time with regards to asshole Silicon Valley type CEOs, and obviously has a few riffs on Elon Musk as well. https://www.youtube.com/watch?v=4iPhrnUtQ5U
Ducktales turned out really well. You've got the long-running protagonist Billionaire, Scrooge McDuck and his long-time rival Billionaire Glomgold. However, as a 2017 reimagining, they added the Silicon Valley type Mark Beaks and it was chef's kiss perfect. Old money vs new money is still a relevant subject a hundred-years after the "Gilded Age" of the 1920s / 1930s, that these characters were originally based on.
I think Pirates of the Silicon Valley did a good job of this many years ago
Hahahaha, absolutely in character for that fucking moron
Batshit crazy for a normal person, just another day for Elon.
This needs to be brought up everytime someone claims Elon is purposely sabotaging Twitter. Someone actively sabotaging would not get their hands dirty going in and doing this themselves. These are the actions of the physical embodiment of Dunning-Kruger.
Well that was a crazy read! I really feel for the workers who had to witness all that nonsense.
Stop making stupid people famous!
This behaviour isn't because he's famous it's because he's incredibly wealthy and no one has ever told him no.
While this is of course true, to me part of saying him no is not giving him media attention. If his stunts didn't have the publicity they do, he might not actually do them. If people stopped caring about whatever shit he writes on Twitter then he wouldn't have a power to destroy Starlink's government deal or affect stock values.
I sold my Tesla stock that I had held for many years not long after he took over Twitter.
...so you sold your Tesla stock before it was even a public company or sold a single car? Stock that you held for "years" when the company itself was less than a year old?
lol what bullshit. Elon Musk was literally one of the first and biggest investor in the company before they even began designing their first car. That's when he "took over" (and thus got to be named as a founder)
Okay.
Alright
Lol. No, you didn't.
Why even make that up. So obviously untrue.
I meant to write "after he took over Twitter". Fixed.
Now fix this one 😜
elonMusk is not the problem, people who continue to use twitter are!
Ok but also Elon Musk is a problem
As are any single persons with this much money and, in consequence, power. It's the old problem with monarchy all over again. Sure, you could get a benevolent leader that favoured the arts, but it was more likely that is was a spoiled inbred who wanted to be famous by starting a few wars with neighbors.
elonMusk is a problem because people let him be a problem.
sing this part with L7 in your head: the masses are asses
Oil companies are not the problem, people who still drive their car to work are!
Are you sure about that?
Asteroids are not the problem. Dinosaurs who live in their path are!
But it’s a problem people keep using Twitter because of what musk has done. Without his antics, using Twitter wouldn’t really be a bad thing, would it?
Well, arguably the microblogging format does have some intrinsic disadvantages.
Yes, but that wasn't the question, now was it?
I would argue that the format incentivizes short quips and discussions lacking nuance in favor of brevity, and yes, therefore it's "bad" (to use their term) to use Twitter even if musk wasn't turning it into Truth Social.
I would argue that a brief, broadly encompassing feed has its advantages when it's not being driven into lunacy. Discussions on Twitter are always terrible, but as a source for news headlines and media announcements it works pretty well.
Sure, but you can get that with something more long-form, too; it's not exclusive to Twitter/microblogging .
Conversely, if someone wants a summarized feed, having to browse past walls of text is an inconvenience.
Well, that's a good point but I still think there are better services than Twitter/microblogging for that. Like our old friend RSS
I like RSS in theory, I used it a lot, but these days websites don't maintain them as well as they used to.
until he bought the thing, it wasn't
hate posting racists used to be banned. They had to create a dozen accounts just to spread their scum
Every great genius inventor and businessman can be a little eccentric. Remember that before you decide to call them mean names like "abuser", "megalomaniac" etc. It's actually quirky and endearing once you factor in that he's a genius inventor and businessman (I'm also one of the misunderstood people in the same category).
At the risk of sounding like an apologist for this prat, the frustration at being told it will take months to move systems is understandable. Also, the idiot developers who hard-coded the data center location deserve to be fired. Data center floor tiles can be removed easily with a flat blade as a lever instead of using suction cups.
Obviously a coke-fuelled man-child doing it in the middle of the night is ridiculous but if you have unlimited resources you can move any number of servers in a few days, easily. In some ways it's impressive that he was able to pull this off on a whim without a catastrophe (the DeSantis fiasco notwithstanding). It definitely should not have been suggested by a competent data center engineer that it would take months to move anything if the CEO wants it in weeks. Even though he's an ass, I don't blame him for being annoyed about that.
Remember he just gutted Twitter before pulling this stunt, so the estimate of a few months to move the servers might be true if the entire department that handle infra has been gutted with only skeleton crew left.
Well, yes. If nobody left has a clue then it's going to take a little longer but you could physically move just about anything in a few weeks with the right crew, even if you had to bring them in cold. An open checkbook solves a lot of logistical problems.
The proof is more or less self evident. If this idiot and his cousin were able to pull it off without breaking anything critical, then it stands to reason that a properly managed team would have been able to do it in a more orderly way in a few days.
I get that everyone wants to paint this as completely irresponsible, but apart from the fact that it was done so haphazardly in the dead of night and gate crashing the data center security (nobody is going to refuse access to the CEO), there's really nothing here that's completely out of order. Locking the gear in the trucks is pretty standard for intact secure data transport. The real mistake is the infra manager sandbagging the move estimate - or not understanding how to plan and execute it.
Physically move them is one thing. Reassigning each server into the new data center network is a whole other thing. It won't be as simple as connecting the power and network cables. From the post, the rack density is different so you'll probably have to change the each server name to match the new rack position. Then the hostname and subnet probably changes in the new data center, so now you'll have to map everything again (the hard coded references to Sacramento mentioned by Musk). The 100MM contract means they have a lot of servers to account for. This is the real headache of the migration and probably the reason Twitter keep having random outages for months after this stunt. They probably took shortcut and can't bring all those servers online in time to handle traffic bursts which leads to another Musk's shenanigans (e.g. forbidding visitors from viewing tweets unless they're logged in to limit servers load, etc).
Edit: the more I think about this, the more my head hurt. If any infra people reading this, what are you going to do if you suddenly received truckloads of servers yanked from another data center location and told to bring them online again ASAP, while more than half of your team has been laid off? Seriously, what's the step you're gonna do to bring all these servers online again? Oh, and those servers probably not gracefully shut down and just have their power cable yoinked off.
Oh by no means am I suggesting it was reasonable to do this. Musk would be a fucking nightmare as an employer. As a customer probably not much better but you know what they say about a fool and his money. This fool would be a great customer as long as you had a good lawyer to write the contracts.
I do suspect that some of the details of this story are somewhat embellished though, if only for the sheer joy of it, which I'm all for. It's a great story. I don't believe, for instance, that they could possibly have moved 5000 racks - or even 5000 servers - as I think the story was intimating. It sounds like they filled a few semis, which would be a small fraction of the systems. Maybe this was just the last of it that was too hard to move earlier. As for the rack configs at the other end, they would need power and services and an empty space if they are just rolling the stuff in. That's only a few weeks of lead time in a properly run facility.
If they had their reservations set up correctly they wouldn't need to change hostnames or even addresses, just wheel in the racks, brace and connect them. Ideally stuff would be shut down gracefully, but it shouldn't really matter if they just pulled the plug. The software should be resilient enough to restart ok. Again, no idea if they had anything thought out, probably not, given the way it was done. But I have seen a big tech co move several rows this way when they basically couldn't be bothered figuring out how to logically migrate them. Of course they weren't doing it with a coked up CEO at 2am on Christmas Eve, but it wasn't as difficult as you might imagine. But yeah not 5000 racks at once. Not even close to possible.
I love this. I'm a former IT engineer/CTO turned renegade entrepreneur, so this story tickles both of my feet.
Yeah, any reasonable person would know this idea to move the servers without a plan was ridiculous.
Yet as a roll-up-the-sleeves entrepreneur your entire job is to fucking destroy the red tape that is put up in front of you constantly. Or else you and everyone who works for you is out of a job. Of course there will be problems, but that's why you have smart people who can sort it out afterwards faster than they can preplan for it.
And a lot of really smart people make "doing it the right way" a religion, so when the cash is going to run out shortly, well, sometimes the big guy needs to just roll the dice.
Honestly, outrage-bait / circlejerk articles like these is why I stopped using twitter and reddit to begin with. Hur dur, Elon bad, upvotes please. I don't disagree - I just don't want to see this kind of low-effort posts, which OP seems to excel at. Time for a mute.
The article is honestly really interesting for the details it gives. It's easy enough to dismiss anything as clickbait, but sometimes it just sounds like that old tired "Trump Derangement Syndrome" BS.
Don't get me wrong, I don't think it's clickbait, but personally I'm not that interested in a retelling of how he started gutting twitter shortly after he bought it last year. Maybe it's not this article per se, just the straw of musk spam that broke the camels back.
And you brought me into this why? You decided you needed to make that little swipe at me why, exactly?
Your words, my emphasis.
Thanks for emphasising I never called the article clickbait, just outrage-bait, circlejerk and low effort. And I do still think posting it here is just that.
Outrage-bait is clickbait. More specifically, it's a subset of clickbait.
Wikipedia:
My bad. I thought clickbait just referred to headlines that don't deliver. Today I learned.
"Don't get me wrong"
Then say different words that mean different things...
This one isn't click bait. It's not about him saying something dumb, it's about something really dumb he did and in depth. If any other CEO did what this article outlines it would still be worth reading after getting some popcorn ready.
For all the people complaining that "Elon news are not tech", this one does go into relevant technical details of the matter.
Which is precisely why I posted it - because it went into the details of he did.
I guess I'm also batshit crazy.
It's Twitter. Who cares if people can't tweet for an hour.
I'm with Elon on this, don't overcomplicate the closing of a data center.
That manager, when asked to do it in 90 days, if s/he was competent should have said: I'll do it, but you'll have to accept a downtime risk.
(But aside from this entertaining story, I do think Elon lost his shine. Wasting $40B on twitter and sabotaging Ukraine while simping for Putin and Trump and not paying taxes... yeah, get rekt Elon).
It's effectively a case of "I left my house unlocked and unarmed while I went on vacation. No one broke in, so I don't see the point in door locks and alarm systems."
Twitter got very VERY lucky that the worst that happened was some outages.
They moved hyper sensitive user data in a moving truck. If anything had gone wrong they would've exposed millions of peoples sensitive data.
You are supposed to wipe the servers before you move them, you shouldn't be driving servers around on the highway while they are still chock full of peoples credit card info and shit.
What sensitive data does Twitter hold? Genuinely curious
We don't know what was on those servers, but it was apparently sensitive enough that the government redacted descriptions of the data in court filings.
https://arstechnica.com/tech-policy/2023/09/us-government-slams-musk-in-court-filing-describing-chaotic-environment-at-x/
Isn’t all of it encrypted though? Like I understand physical access to servers is generally bad, but you’d think once the the things are unplugged it would be difficult to access the data again without bypassing encryption. I’m not a software engineer though
I'm a security engineer, and encryption is great, but can be bypassed. Relying on encryption assumes it was implemented properly, that the system was shut down properly so all keys were flushed correctly, and the encryption algorithm doesn't have weaknesses.
Generally if somebody dedicated enough can acquire physical access to a system, they can probably find a way into it given the right resources. Did that happen here? Probably not. Could it have? Absolutely. That's why most enterprises or government hard drives are shredded rather than just relying on them being wiped or encrypted.
Encryption is part of the solution, but it's not automatically the complete solution.
Probably because the government is still illegally spying on citizens and they don't want the specifics to leak out.
You don't consider credit card info sensitive? May I have yours?
BS, I don't know if Twitter holds credit card data, but if they did, they would have needed to abide by PCI DSS rules, which requires encrypting the data in special hardware security modules.
So no, moving those servers wouldn't put the data at risk.
Tell me you don't understand how PCI works without saying you don't understand how PCI works.
Those systems can very much store PCI data and it's very much possible that those were the systems that contain information as most of the times it's on general servers.
Personally identifiable information (PII) is any set of data that has a chance to uniquely identify a person, including name, address, credit card info, social security, etc. It can also include things like birthdate, city, IP address, and so on, depending on how the combination of data works. The general rule of thumb is that you want to aggregate out to the city level at least, or completely anonymize the data. These, I’m supposing, we’re raw records that contained account info.
Isn’t all of it encrypted though? Like I understand physical access to servers is generally bad, but you’d think once the the things are unplugged it would be difficult to access the data again without bypassing encryption. I’m not a software engineer though
While the engineers could have said that, equally Elon could have asked what the problems/downsides were to doing it faster.
You think?
I'm 100% sure!
Edit: you may want to reconsider you being "on the side of musk on this one." From the article:
" And, of course, it didn’t really “work.” As we detailed, Twitter toppled over a few days later, and this excerpt admits it was because of the “server move.” The article does note that Musk himself eventually said he shouldn’t have done this and it did cause a fair bit of problems for the site, including the disastrous “Twitter Spaces” "
That is the correct take in general, but I've worked for managers a bit like Elon before, and that never would have worked. It would have been the equivalent of tendering that resignation, because he sees any pushback at all as insubordination, and not to be tolerated.
The only way out of this is to suggest an alternative course of action, but make it seem like it was his idea all along. My favorite method was to tie it to some bullshit metric he previously set for no reason. "Yes, sir, we could do it right away, but it may have an impact on our ad throughput, and I know ad revenue is a key prioriry you have personally set for the company. If we take the time to transition our ad platform first we can keep our revenue consistent". Then that time ends up identical to the amount of time needed to move things correctly.
It still might not work, but at least you've stroked the Boss's ego, which can keep you employed for a few more months while trying to find the exit ramp.
Musk quite literally said it was equivalent of resigning:
Exactly, and that's why turning it into a risk assessment, which would work with competent management, would never work with him. The only way to make it work would be to find a way to stroke his ego.
I know the type. Elon is different though.
Yeah, he is overrated by many. But he's not the typical stupid middle manager, who get brain freeze when presented a simple dilemma.
Fuck, Elon would fail hard as a middle manager. This clock ticks different. I have worked with a lot of people, but I don't think he fits in one of the common types.
I think “asshole” is a pretty common type.
I mean it was more of a data loss and data security risk but okay
I can understand the pushback. Twitter employees work for years to transform a site that went down a lot (remember fail whale?) Into reliable well oiled machinery. Having new owner suddenly tell you to dismantle it all must've been awful.
That, I can understand.
But reforming a company is always ugly business. Best to get it done with quickly and efficiently.
A slow painful death like Yahoo also isn't in anyone's interest.
I really dislike Musk, but I find it hard to criticize this when it generally worked.
The platform formerly known as Twitter is still running, and there’s no more $100 million/year data center.
6-9 months would have meant $50-75 million dollars. I don’t know what the outages and re-engineering ended up costing them, but that’s a ton of money.
Looks like you’re the type the writer talks about at the end:
What risks, exactly? Twitter goes down? Proprietary Twitter data gets stolen in some server heist scenario?
Millions of people's personal data gets leaked, Musk's cowboy "pry open the floor and electrical panels with a knife" electrocutes him, or blows the power for the room/floor/building or starts a fire.
That one is a risk I'm willing to take. I had to stop reading the article for a moment to marvel at just how close we really were.
Elon nearly took out both himself and Peter Thiel by rolling his uninsured McClaren F1 trying to show off during the PayPal days.
What could have been.
EDIT:
https://www.thedrive.com/news/32191/did-you-know-elon-musk-wrecked-an-uninsured-mclaren-f1
God, it would've been some universal shit.
Isn’t all of it encrypted though? Like I understand physical access to servers is generally bad, but you’d think once the the things are unplugged it would be difficult to access the data again without bypassing encryption. I’m clearly not a software engineer though lol
The servers were not actually secured in the truck properly, so another scenario would have been the damage and destruction of some or all of them.
Plus, yes, theft. And it's not just proprietary data, it was also personal and financial data for users and advertisers.
I imagine thousands of pounds of unsecured load would be potentially dangerous for the driver and all other drivers on the road too.
The problem is risk. A lot of the bureaucracy that exists for any company is risk mitigation. The wiping of servers, or using suction cups, or any of that is a security against a large dollar amount to spend if something goes wrong. But that's just the cost of security, it's worthless if it isn't tested. If a locked door isn't rattled or deter someone, it might as well have been unlocked.
He took a gamble and the doors were not rattled and everything worked. The thing to criticize here is really the carelessness. What if one of those servers got out and somebody stole all of that data? What if while under those floorboards he got damaged, or something related did? And it's not just these two questions, there's stuff in that article that probably wasn't covered that we can question.
There may be things that are not in the book that we can question, and that is the problem with Elon. He needs a string of bad luck to show how truly dangerous he is.
That's like saying why wear PPE and follow safety protocols on a construction site, it'll save us money if we don't do any of that. Nobody died or got hurt? Perfect.
There's a reason things need to be done a certain way, if something had gone wrong what would've been the consequences? What if all those data racks full of personal user information were just straight up stolen by the random movers they hired off the street?? What if the floor had collapsed under the weight of the servers being moved, tipped the server over and crushed someone? Just because things worked out relatively fine doesn't mean no harm no foul.
Musk is an idiot. Deciding to do things his way to save money and time reflects poorly on all his other companies.
Are the severs running SSDs or hard disks?
I remember the story of the guy moving a shopping cart full of his company’s HDs across the street or something and destroying them all just rattling the fucking cart across the shitty surface.
No, they were just extremely lucky that nothing worse than twitter going down happened. There’s a reason protocols exist for data center moves. The infrastructure manager told Elon that the destination DC in Oregon had different rack and power setups and you just can’t plug and play a server you pulled out of Sacramento. Elon also went under the floorboards and disconnected the power cables and seismic detectors which could’ve caused electrocution, fire, earthquake false alarms, or compromising the detection system itself. Then they were moving equipment that weighed more than what the floor was rated for, which could’ve caused cave-ins or compromising the structural integrity of the floor. Not to mention the possible damage to the equipment by moving them the same way you’d move a couch. They also hired some random cheap moving company they found on yelp to move the servers because they charged 90% less than the existing contractor. No contracts and paid in cash.
Tons of things could’ve gone wrong. Just because downtime was the worst that happened, doesn’t mean it’s ok to do. It is also those same data center protocols that help prevent idiots like Musk from causing catastrophic issues when they pull off stunts like this.
First of all, Musk burdened twitter with a level of debt that cost (last estimate that I saw) $1B/year to service. This data center would not have been a problem if he had actually been a good businessman and, you know, didn’t massively overpay with a debt-funded takeover while waiving due diligence on a company he didn’t want in a market he completely doesn’t understand. He set fire to $44B. Twitter’s current valuation has been estimated to be as low as $4B. I personally think that’s low, but the May estimate was $15B (which didn’t include the loss of branding hit).
So his recklessness and complete lack of understanding combined with his overconfidence and incompetence made the $100M savings into peanuts compared to what he destroyed by pulling exactly the same kind of move throughout the business.
Now combine that with the very probable fact that this saved no where near $100M. Shitty shifting of servers breaks hardware. They weren’t prepped to receive them at the destination. They ended up with major drops in service, including Elmo having to shut twitter down for a weekend because they couldn’t handle the traffic. Now he’s whining about “scraping” and trying to squeeze blood from a stone in the face of advertisers abandoning him.
This in no way generally worked. Things are absolutely falling apart around their ears. I’ve stopped even trying to follow twitter links because they work less than half the time since I don’t have an account.
Elon is Dilbert’s pointy-haired boss with a lot of money and a PR firm.
Dilbert's boss is lazy, Musk is full of passionate intensity.
My first gig out of college in the Valley was working with (later for) a relatively charismatic "I know better" untreated bipolar guy. This dude actually had the chops, he was actually smarter than you. His demos and product ideas were amazing, legit visionary. Inspirational.
But gods it was soul-destroying to try to work for the guy, he kept pulling exactly this kind of "it's not that complicated!" stunt, changing plans on a whim, editing history to make himself consistent, hair-trigger switching between praise and abuse...
It got a lot of good work out of me, I learned a lot, I was well-compensated, but I now that I know the signs I'm never working for a person like that again.
(see also: the subject of another fawning Isaacson hagiography, Steve Jobs)