Software update crashes city of Seattle websites

TheOneWithTheHair@lemmy.world to Technology@lemmy.world – 135 points –
seattletimes.com
9

Looks like they need to learn a thing or two about integration tests, redundancy, and fail-overs.

A software update didn't take it down. Negligence did. You should wonder why it doesn't work before you ship to production.

Yep.

Where's the non-prod test environment? For major systems we always have a multi-tier setup, of Test (pretty open, wild-west, but completely isolated and only test data on it), then Non-Prod Test, which looks a lot like production, has production-like test data, with User Acceptance Testing capability, still isolated from both prod and test. (We also have other layers of test too, say for hardware integration to backend, etc).

Also, this example is why I constantly argue against auto updates for anything, even my phone apps.

Without validation testing, you don't know shit won't work and then disrupt your work flow. My phone is a tool that I rely on to work a certain way.

Integration testing is great for release validation.

But there's also development work that should have been done on the release process itself, like support for progressive rollouts and easy rollbacks.

Progressive rollout? Nah, these people are acting like it's 1992 and just cowboy that shit up!

Senior developer: first time?

Principal developer: that’s why we always hav failover, backups, and a rollback plan, and insist on release testing in a staging environment.

This is what happens when the city pays below average wages for developers in a city where developers have nearly unlimited job opportunities available.