How I accidentally wiped my server by having a typo in my Nextcloud docker config

Moonrise2473@feddit.it to Selfhosted@lemmy.world – 171 points –

So, I moved my nextcloud directory from a local SATA drive to a NFS mount from a nvme array on a 10G network

"I just need to change /docker/nextcloud to /mnt/nfs/nextcloud in the docker-compose.yml, what's the issue, i do it live" - i tell myself

So i stop the container, copy /docker/nextcloud to /mnt/nfs/nextcloud, then edit the docker-compose.yml... and.... because I'm doing it during a phone call without paying too much attention i change the main directory to /docker

I rebuild the container and I immediately hear a flood of telegram notifications from my uptime-kuma bot.... oh oh...

Looks like the nextcloud docker image has an initialization script that if it doesn't find the files in the directory, it will delete everything and install a fresh copy of nextcloud... so it deleted everything on my server

Luckily i had a very recent full borg backup and i'm restoring it (i kinda love-hate borg, i always forget the restore commands when in panic and the docs are a bit cryptic for me)

Lessons learned:

  1. always double check everything

  2. offsite backups are a must (if i accidentally wrote / as path, i would have lost also the borg backups!)

  3. offsite backups should not be permanently mounted, otherwise they would have been wiped as well

  4. learn how to use and schedule filesystem snapshots, so the recovery wouldn't take ages like it's taking right now (2+ hours and i'm not even half way...)

34

You are viewing a single comment
View all comments

Was it the official container image or 3rd party? Whichever it was, they should get notified so that init script can get fixed to prevent similar happening to others.

Official image

Edit: official but from docker, not official from nextcloud, because I don't like AIO images, I like having everything separate

Edit 2: the documentation says to use named docker volumes. I don't like using volumes because I feel they're harder to backup, I want individual file control so I used mount points. Because they're assuming that everyone is using named volumes, they assume they can wipe without problems. But they don't specify to avoid using mount points and/or that's dangerous because of those assumptions.