All of Japan's Toyota Assembly Plants Shut Down for a Day Because Their Server Ran Out of Disk SpaceFlying Squid@lemmy.world to Technology@lemmy.world – 994 points – 10 months agoreuters.com104Post a CommentPreviewYou are viewing a single commentView all commentsShow the parent commentA system this critical is on a SAN, if you're properly alerting adding a bit more storage space is a 5 minute task. It should also have a DR solution, yes.A system this critical is on a hypervisor with tight storage “because deduplication” (I’m not making this up).This is literally what I do for a living. Yes deduplication and thin provisioning. This is still a failure of monitoring or slow response to it. You keep your extra capacity handy on the storage array, not with some junk files on the filesystem. You also need to know how over provisioned you are and when you're likely to run out of capacity... you know this from monitoring. Then when management fails to react promptly to your warnings. Shit like this happens. Then when management fails to react promptly to your warnings. Shit like this happens. The important part is that you have your warnings in writing, and BCC them to a personal email so you can cover your assExactly, I was being sarcastic about management’s “solution”
A system this critical is on a SAN, if you're properly alerting adding a bit more storage space is a 5 minute task. It should also have a DR solution, yes.A system this critical is on a hypervisor with tight storage “because deduplication” (I’m not making this up).This is literally what I do for a living. Yes deduplication and thin provisioning. This is still a failure of monitoring or slow response to it. You keep your extra capacity handy on the storage array, not with some junk files on the filesystem. You also need to know how over provisioned you are and when you're likely to run out of capacity... you know this from monitoring. Then when management fails to react promptly to your warnings. Shit like this happens. Then when management fails to react promptly to your warnings. Shit like this happens. The important part is that you have your warnings in writing, and BCC them to a personal email so you can cover your assExactly, I was being sarcastic about management’s “solution”
A system this critical is on a hypervisor with tight storage “because deduplication” (I’m not making this up).This is literally what I do for a living. Yes deduplication and thin provisioning. This is still a failure of monitoring or slow response to it. You keep your extra capacity handy on the storage array, not with some junk files on the filesystem. You also need to know how over provisioned you are and when you're likely to run out of capacity... you know this from monitoring. Then when management fails to react promptly to your warnings. Shit like this happens. Then when management fails to react promptly to your warnings. Shit like this happens. The important part is that you have your warnings in writing, and BCC them to a personal email so you can cover your assExactly, I was being sarcastic about management’s “solution”
This is literally what I do for a living. Yes deduplication and thin provisioning. This is still a failure of monitoring or slow response to it. You keep your extra capacity handy on the storage array, not with some junk files on the filesystem. You also need to know how over provisioned you are and when you're likely to run out of capacity... you know this from monitoring. Then when management fails to react promptly to your warnings. Shit like this happens. Then when management fails to react promptly to your warnings. Shit like this happens. The important part is that you have your warnings in writing, and BCC them to a personal email so you can cover your assExactly, I was being sarcastic about management’s “solution”
Then when management fails to react promptly to your warnings. Shit like this happens. The important part is that you have your warnings in writing, and BCC them to a personal email so you can cover your ass
A system this critical is on a SAN, if you're properly alerting adding a bit more storage space is a 5 minute task.
It should also have a DR solution, yes.
A system this critical is on a hypervisor with tight storage “because deduplication” (I’m not making this up).
This is literally what I do for a living. Yes deduplication and thin provisioning.
This is still a failure of monitoring or slow response to it.
You keep your extra capacity handy on the storage array, not with some junk files on the filesystem.
You also need to know how over provisioned you are and when you're likely to run out of capacity... you know this from monitoring.
Then when management fails to react promptly to your warnings. Shit like this happens.
The important part is that you have your warnings in writing, and BCC them to a personal email so you can cover your ass
Exactly, I was being sarcastic about management’s “solution”