what if your cloud=provider gets hacked ?
Hi all,
As self-hosting is not just "home-hosting" I guess this post should also be on-topic here.
Beginning of the year, bleeping-computers published an interesting post on the biggest cybersecurity stories of 2023.
Item 13 is an interesing one. (see URL of this post). Summary in short A Danish cloud-provider gets hit by a ransomware attack, encrypting not only the clients data, but also the backups.
For a user, this means that a senario where, not only your VM becomes unusable (virtual disk-storage is encrypted), but also the daily backups you made to the cloud-provider S3-storage is useless, might be not as far-fetches then what your think.
So .. conclussion ??? If you have VMs at a cloud-provider and do daily backups, it might be usefull to actually get your storage for these backups from a different provider then the one where your house your VMs.
Anybody any ideas or remarks on this?
The real issue here is backups vs disaster recovery.
Backups can live on the same network. Backups are there for the day to day things that can go wrong. A server disk is corrupted, a user accidentally deletes a file, those kinds of things.
Disaster recovery is what happens when your primary platform is unavailable.
Your cloud provider getting taken down is a disaster recovery situation. The entire thing is unavailable. At this point you're accepting data loss and starting to spin up in your disaster recovery location.
The fact they were hit by crypto is irrelevant. It could have been an earthquake, flooding, terrorist attack, or anything, but your primary data center was destroyed.
Backups are not meant for that scenario. What you're looking for is disaster recovery.
Yes. Fair point.
On the other hand, most of the disaster senarios you mention are solved by geographic redundancy: set up your backup // DRS storage in a datacenter far away from the primary service. A scenario where all services,in all datacenters managed by a could-provider are impacted is probably new.
It is something that, considering the current geopolical situation we are now it, -and that I assume will only become worse- that we should better keep in the back of our mind.
It should be obvious from the context here, but you don't just need geographic separation, you need "everything" separation. If you have all your data in the cloud, and you want disaster recovery capability, then you need at least two independent cloud providers.
I've thought about how I could handle disaster recovery for my homelab environment, but I haven't come to any good solutions. For example, if my main concern was being hit by crypto. I can't just recover from a regular backup, since I'm not sure how I can make a backup without that backup just being encrypted along side everything else. Since I mainly just backup everything to my file server, which is then synced to the cloud. In that setup, my cloud backups would be lost as well.
Would you have some starting points on how others handle disaster recovery? I'd like to avoid manually making an offline backup, because inevitably I'd forget to do it, which would make it useless anyway.
What cloud backup solution are you using? A lot of them offer additional protection that would keep a history of your files. You can essentially say "once a week create a point in time recovery of all my files" and then you could recover your files from that point in time.
This usually costs extra, and it makes sense why. They're essentially keeping extra copies of your data for you.
How that is configured allows you to determine your RPO, or recovery point objective.
https://www.imperva.com/learn/availability/recovery-point-objective-rpo/
So you can decide how much data you're comfortable losing by determining how often those point in time recovery events happen.
Did that make sense?
It does make sense. Thank you. I appreciate the link!
However, my cloud usage is purely as a proxy/load balancer, as none of my cloud providers hold any actual data. They're just routing traffic, and all data/processing is on premises. What I'm interested in, is how to setup something like what you describe, but on premises also. From a design stand point, if I wanted to protect myself from a ransomware attack, obviously my cloud backups would be lost because they're a mounted filesystem during a backup eventually. So I don't know how to wrap my head around handling this, just storage design wise as specific tools I can figure out. How does one create a recovery point, and keep it safe from something like this? Just image the entire file system from a live booted offline environment? Feels like a chicken-egg problem to me.
By definition a disaster recovery solution needs to be geographically separate. You're protecting yourself from catastrophe, and some of those scenarios include your main location burning down, flooding, being hit by a tornado, etc etc.
So you either need to collocate systems with a friend who you trust, purchase colocation services from a provider, or use a cloud service to achieve what you're looking for to truly have a DR solution.
As far as how to do that, the main idea is to have that point in time available on a system that, even if you get compromised, the backups won't. The old school method here is to use an external hard drive or a tape device, and physically store that offsite. So like use your regular backup mechanism, and in addition to what it's doing now schedule a daily/weekly/monthly job that backs up to this other device, and then store that away from your main location.
That's essentially the idea though, and there are any number of solutions you can use to do it.