Toyota says lack of disk space shut down all of its factories

L4sBot@lemmy.worldmod to Technology@lemmy.world – 172 points – Toyota says lack of disk space shut down all of its factories
asia.nikkei.com

Toyota says lack of disk space shut down all of its factories::TOKYO (Reuters) -- The malfunction that shut down all of Toyota Motor's assembly plants in Japan for about a day last week occurred because some serve

9

The issue occurred following regular maintenance work on the servers, the company said, adding that it would review its maintenance procedures.

Two people with knowledge of the matter had told Reuters the malfunction occurred during an update of the automaker's parts ordering system.

Uh-oh. Someone forgot to uncomment include /etc/logrotate.d and bounce the service, didn't they.

If only this bot would post the important bit of an article instead of cutting off in the middle of a

It’s always either /var/log, or my personal favorite some process spamming tiny files running the disk out of inodes.

Oh man, the first time I ever ran into the inodes issue I was still a squirrely, eager young admin and not the broken, cynical shell you see before you. That one fucked me up for a solid day and change.

Mine was "deleting" a LARGE log file at 3am when I was oncall to clear up some disk apace and not getting that space back.

At the time I has no idea why and went into panic mode because the disk was 98% full and growing. Damn software kept the file open and was spamming the same line at least millions of times. After that I spent the tine to learn about common issue, inodes were one of them.

Dont forget that other favorite "No you can't have more hardware, these were fine in 2006 and they're fine now! Just delete some old stuff like you IT people always do"

Good job setting up alerts for system resources.