Funny how bittorrent solved this with a simple distributed hash algorithm...
I guess fuck using what works, amirite?
Pirates are unironically better digital stewards of content and history than media organizations.
Seriously. If these "media pros" are actually concerned, it appears my personal server adheres to higher standards than their industry.
I wish there was a good way to store a backup of my media. I recently suffered a terrible hard drive issue. I lost a terabyte of media. Fortunately, the pirate ship has saved me and has me rethinking some of my backup methodologies.
Outside of periodically backing up onto an external hard drive, I haven't been able to find a reasonably priced online backup solution that isn't going to fuck me when I have to pull data out. Egress fees are killer.
It's a lot of up front cost, but a NAS that is RAIDed with parity puts you in a pretty safe spot.
The short explanation is you have at minimum three drives, and you "stripe and span" them. This is a setup called RAID 5 where, if any one of the three drives fails, it can be replaced with a similar-sized drive and the "parity bits" from the other two drives can rebuild the data on the third drive. Yes, this means you only have the effective space of only two out of the three drives. So say you had 3x4TB drives, you'd have a total of 8TB to work with, and one drive is the "parity" drive (although this is actually split among the drives, so if any one fails, it can be revived by the other two).
However, in practice, the space lost is worth it for redundancy. It does mean an up-front cost in buying drives, a NAS enclosure (or using something like TrueNAS plus off the shelf parts to build your own), and includes the cost of physical maintenance and support (a Uninterruptible Power Supply to keep the hardware safe, for instance, on top of eventual maintenance of physical parts).
The offers the cloud solutions seem cheap up-front, but they don't buy you as much time as the one-time up-front cost of building your own NAS and maintaining it. I understand why people choose the cloud solutions, it's much easier. But if you're dedicated to this lifestyle, it's something worth looking into, at the very least.
Obligatory raid isn't backup.
While yes, this will protect you from a disk dieing if you monitor it enough to notice. But it doesn't save you from a nas dieing. Maybe you could rebuild the array with similar hardware but that's not a sure thing.
This exactly. NAS+RAID gives you a backup of your local media. It can account for one of your three copies and one of your storage mediums. But you still need something off site.
So assuming you had a copy on your computer proper, it could work. Better than no backups.
Not trying to be a dick just trying to help, but,
Dying*
Backblaze personal is $9 a month or $99 a year for unlimited backup. The first result on Amazon for a 4tb HDD is $85. Building a NAS costs the same as 2.5 years of this cloud backup for the drives alone, and doesn't actually give you a backup at all. The costs scale even more poorly if you need to store more than your 8tb.
raid cannot repair corruption/bitrot. zfs/btrfs with redundancy can.
Serverbuilds.net. its really not that much up front cost.
All you need is to use ZFS or BTRFS locally to prevent master version bitrot and provide failover/redundancy, manually sync that to a separate "offline" HDD periodically, then setup a simple pi with tailscale + HDD at a family member or friends house, and rclone all your data to it (encrypted) as a cron job every night or week. This performs the function of a cloud provider (offsite backup); alternately, just manually sync the offline HDD once a month.
With this approach you're covered for accidental deletion, hard drive failures, bitrot, ransomware, and fire; possibly many natural disasters, depending how far away the offsite is.
Then you can just keep your most important data E2E encrypted in 1 or 2 cloud storage providers.
Note: zfs/btrfs cannot repair bitrot without redundancy, only detect it. But if redudancy, is repaired automatically (self-healing).
Raid 1/5/6 cannot repair, only detect bitrot, cannot decide which copy good.
I have never seen an implementation of e.g. a mirror that gives up on disagreements of both disks. Repairing/redundnancy is what raid is there for.
What was it again? “Don’t attribute to malice when it can be attributed to stupidity” something something?
Hanlon’s Razor
A friend has a corollary for that: "Don't attribute to stupidity when it can be attributed to capitalism".
Huh, I assumed they were spending the money to archive digital content with redundancy just like they did celluloid.
They aren't spending the money to preserve film either. The best case is storing the film in salt mines, and that only slows the degradation. Film isn't being digitally scanned unless there's a uhd release to profit from it, and every week that it isn't scanned, it degrades a little more
That's not profitable in this quarter.
This
The current censorship of media companies and stream services is a much bigger threat to the preservation of media than digital decaying could ever be.
I don't understand the interest in DNA as a storage. It's only long-living as part of the evolutionary proces in a living organism (with no guarantee for the survival of the data), but otherwise really fragile. And hard to interface and with slow read/write on top of that.
Personally, I think it's okay for things to disappear sometimes. Nothing is permanent. I have no anxiety about this.
Am I the only one kind of relieved when I loose digital files in a drive failure? Its kind of like a chance to start fresh. Yes I liked having pictures of my dead pets and relatives from 10 years ago but I wasn't going to ever look at them again. I have my memories and that's enough. Yes I liked having a copy of all the shows I ever enjoyed but I lost interest in half of them over the years and was likely never going to rewatch them locally or not.
There's a time and a place for all things, digital files are no exception. When the time has passed long ago and the place no longer exist as it once did, its time to let go
Yes I liked having pictures of my dead pets and relatives from 10 years ago but I wasn’t going to ever look at them again. I have my memories and that’s enough.
I used to think like this but I've found that going through those old pictures is a good way to bring up memories you wouldn't otherwise think about again. I like to do that when I'm in a funk and it really helps to clear my head of nasty emotions.
Funny how bittorrent solved this with a simple distributed hash algorithm...
I guess fuck using what works, amirite?
Pirates are unironically better digital stewards of content and history than media organizations.
Seriously. If these "media pros" are actually concerned, it appears my personal server adheres to higher standards than their industry.
I wish there was a good way to store a backup of my media. I recently suffered a terrible hard drive issue. I lost a terabyte of media. Fortunately, the pirate ship has saved me and has me rethinking some of my backup methodologies.
Outside of periodically backing up onto an external hard drive, I haven't been able to find a reasonably priced online backup solution that isn't going to fuck me when I have to pull data out. Egress fees are killer.
It's a lot of up front cost, but a NAS that is RAIDed with parity puts you in a pretty safe spot.
The short explanation is you have at minimum three drives, and you "stripe and span" them. This is a setup called RAID 5 where, if any one of the three drives fails, it can be replaced with a similar-sized drive and the "parity bits" from the other two drives can rebuild the data on the third drive. Yes, this means you only have the effective space of only two out of the three drives. So say you had 3x4TB drives, you'd have a total of 8TB to work with, and one drive is the "parity" drive (although this is actually split among the drives, so if any one fails, it can be revived by the other two).
However, in practice, the space lost is worth it for redundancy. It does mean an up-front cost in buying drives, a NAS enclosure (or using something like TrueNAS plus off the shelf parts to build your own), and includes the cost of physical maintenance and support (a Uninterruptible Power Supply to keep the hardware safe, for instance, on top of eventual maintenance of physical parts).
The offers the cloud solutions seem cheap up-front, but they don't buy you as much time as the one-time up-front cost of building your own NAS and maintaining it. I understand why people choose the cloud solutions, it's much easier. But if you're dedicated to this lifestyle, it's something worth looking into, at the very least.
Obligatory raid isn't backup.
While yes, this will protect you from a disk dieing if you monitor it enough to notice. But it doesn't save you from a nas dieing. Maybe you could rebuild the array with similar hardware but that's not a sure thing.
This exactly. NAS+RAID gives you a backup of your local media. It can account for one of your three copies and one of your storage mediums. But you still need something off site.
So assuming you had a copy on your computer proper, it could work. Better than no backups.
Not trying to be a dick just trying to help, but,
Dying*
Backblaze personal is $9 a month or $99 a year for unlimited backup. The first result on Amazon for a 4tb HDD is $85. Building a NAS costs the same as 2.5 years of this cloud backup for the drives alone, and doesn't actually give you a backup at all. The costs scale even more poorly if you need to store more than your 8tb.
https://www.raidisnotabackup.com/
raid cannot repair corruption/bitrot. zfs/btrfs with redundancy can.
Serverbuilds.net. its really not that much up front cost.
All you need is to use ZFS or BTRFS locally to prevent master version bitrot and provide failover/redundancy, manually sync that to a separate "offline" HDD periodically, then setup a simple pi with tailscale + HDD at a family member or friends house, and rclone all your data to it (encrypted) as a cron job every night or week. This performs the function of a cloud provider (offsite backup); alternately, just manually sync the offline HDD once a month.
With this approach you're covered for accidental deletion, hard drive failures, bitrot, ransomware, and fire; possibly many natural disasters, depending how far away the offsite is.
Then you can just keep your most important data E2E encrypted in 1 or 2 cloud storage providers.
Note: zfs/btrfs cannot repair bitrot without redundancy, only detect it. But if redudancy, is repaired automatically (self-healing).
Raid 1/5/6 cannot repair, only detect bitrot, cannot decide which copy good.
I have never seen an implementation of e.g. a mirror that gives up on disagreements of both disks. Repairing/redundnancy is what raid is there for.
Edit: maybe old hardware raid does not check?
No, raid for disk failure: https://en.wikipedia.org/wiki/RAID
Raid has no checksum, see argument here: https://superuser.com/questions/769104/is-raid1-or-similar-needed-for-btrfs-zfs-to-protect-against-bit-rot
But look like some raid implementation (and lot of drive) have error correction now: https://en.wikipedia.org/wiki/Error_detection_and_correction#Data_storage https://serverfault.com/questions/77710/is-bit-rot-on-hard-drives-a-real-problem-what-can-be-done-about-it
Other people are your backups.
Sharing is caring.
Backup often and check the backups.
Backup often.
Backup often.
Use an open format. For extra sure, make sure it doesn't carry DRM.
Uuuuh don't be corrupt?
Like, really, it's not like one's asking too much.
Some executive somewhere:
That costs $
That costs $
That costs $
That costs $ (Probably, I'd ask IT but we laid them off as a cost reduction so meh )
That costs $
They can just use Hollywood Accounting, no problem!
Some of that could also be incompetence.
What was it again? “Don’t attribute to malice when it can be attributed to stupidity” something something?
Hanlon’s Razor
A friend has a corollary for that: "Don't attribute to stupidity when it can be attributed to capitalism".
Huh, I assumed they were spending the money to archive digital content with redundancy just like they did celluloid.
They aren't spending the money to preserve film either. The best case is storing the film in salt mines, and that only slows the degradation. Film isn't being digitally scanned unless there's a uhd release to profit from it, and every week that it isn't scanned, it degrades a little more
That's not profitable in this quarter.
This
The current censorship of media companies and stream services is a much bigger threat to the preservation of media than digital decaying could ever be.
zpool scrub movies
Maybe someday all this media will be archived as dna: https://www.scientificamerican.com/article/dna-the-ultimate-data-storage-solution/
I don't understand the interest in DNA as a storage. It's only long-living as part of the evolutionary proces in a living organism (with no guarantee for the survival of the data), but otherwise really fragile. And hard to interface and with slow read/write on top of that.
Personally, I think it's okay for things to disappear sometimes. Nothing is permanent. I have no anxiety about this.
Am I the only one kind of relieved when I loose digital files in a drive failure? Its kind of like a chance to start fresh. Yes I liked having pictures of my dead pets and relatives from 10 years ago but I wasn't going to ever look at them again. I have my memories and that's enough. Yes I liked having a copy of all the shows I ever enjoyed but I lost interest in half of them over the years and was likely never going to rewatch them locally or not.
There's a time and a place for all things, digital files are no exception. When the time has passed long ago and the place no longer exist as it once did, its time to let go
I used to think like this but I've found that going through those old pictures is a good way to bring up memories you wouldn't otherwise think about again. I like to do that when I'm in a funk and it really helps to clear my head of nasty emotions.