Not me personally, but one of my career mentor's friend's took down the entirety of Google Ads as an intern for like 10 minutes. Apparently it was a multi-million dollar mistake, but they fixed the issue so it couldn't happen again and all was well afterward.
In my first couple months, I broke Amazon so that no-one in Europe could buy video for a few hours. On a Friday, right before going on a week's vacation.
The way that the ensuing investigation and response was carried out - 100% blame-free, and focused on "how did these tools let him down? How can we make sure no-one ever makes that same mistake again?" - gave me a career-long interest in Software Resiliency and Incident Management.
Junior dev: "I fucked up bad, I'm so fired"
Senior dev: " I have 3 production outages named after me lol"
Yep. And every time there's a thread about an Internet service having an outage, there's some kid saying "oh, someone's getting so fired for this one!"
Yeah, the competent business folks know that if you fire people for outages, you lose everyone who even stands a chance of preventing outages. And you tell the rest of your staff to hide problems. Businesses that do that kind of thing tend to end up with a valuation in the single digits.
Not always.
Sometimes the internet service outage is due to a car taking out a green box or a pole.
If they fire the backhoe driver, then backhoe drivers will never learn.
Oh yeah, That too. Plus trenchers, and hole boring for like fences or billboards.
If an intern (or damn near any employee) can be in a position to single handedly take down that scale of system it’s not the intern that should be fired - it’s the architect that baked that kind of weakness in the first place.
You're not a real SRE until you've caused at least a $100K outage. You're not a good SRE until you've fixed it so nobody can ever make that particular one again.
Not me personally, but one of my career mentor's friend's took down the entirety of Google Ads as an intern for like 10 minutes. Apparently it was a multi-million dollar mistake, but they fixed the issue so it couldn't happen again and all was well afterward.
In my first couple months, I broke Amazon so that no-one in Europe could buy video for a few hours. On a Friday, right before going on a week's vacation.
The way that the ensuing investigation and response was carried out - 100% blame-free, and focused on "how did these tools let him down? How can we make sure no-one ever makes that same mistake again?" - gave me a career-long interest in Software Resiliency and Incident Management.
Source: https://twitter.com/CarlaNotarobot/status/1481458190722207747
Yep. And every time there's a thread about an Internet service having an outage, there's some kid saying "oh, someone's getting so fired for this one!"
Yeah, the competent business folks know that if you fire people for outages, you lose everyone who even stands a chance of preventing outages. And you tell the rest of your staff to hide problems. Businesses that do that kind of thing tend to end up with a valuation in the single digits.
Not always.
Sometimes the internet service outage is due to a car taking out a green box or a pole.
If they fire the backhoe driver, then backhoe drivers will never learn.
Oh yeah, That too. Plus trenchers, and hole boring for like fences or billboards.
If an intern (or damn near any employee) can be in a position to single handedly take down that scale of system it’s not the intern that should be fired - it’s the architect that baked that kind of weakness in the first place.
You're not a real SRE until you've caused at least a $100K outage. You're not a good SRE until you've fixed it so nobody can ever make that particular one again.