The Internet Is Failing The Website Preservation Test

This is a very good point and one that is not discussed enough. Archive.org is doing amazing work but there is absolutely not enough of that and they have very limited resources.

The whole internet is extremely ephemeral, more than people realize, and it's concerning in my opinion. Funny enough, I actually think that federation/decentralization might be the solution. A distributed system to back-up the internet that anyone can contribute storage and bandwidth to might be the only sustainable solution. I wonder.if anyone has thought about it already.

This is why stuff like the internet archive exist: to try and preserve this content. The problem is that governments are trying to shut down the internet archive...

Relevant xkcd

I worry about this too. I've always said and thought that I feel more like a citizen of the Internet then of my country, state, or town, so its history is important to me.

Yeah it’s funny how I always got warned about how “the internet is forever” when it comes to being care about what you post on social media, which isn’t bad advice and is kinda true, but also really kinda not true. So many things I’ve wanted to find on the internet that I experienced like 15 years ago are just gone without a trace.

Remember a few years ago when MySpace did a faceplant during a server migration, and lost literally every single piece of music that had ever been uploaded? It was one of the single-largest losses of Internet history and it's just... not talked about at all anymore.

Well stone tablets, writing, songs, culture can disappear with time, either naturally (such as erosion and weather) or through human action (such burning books, destructive investigation of ancient artifacts/ruins)

That's why we try to keep good records.

It’s important here to think about a few large issues with this data.

First Data Storage. Other people in here are talking about decentralizing and creating fully redundant arrays so multiple copies are always online and can be easily migrated from one storage tech to the next. There’s a lot of work here not just in getting all the data, but making sure it continues to move forward as we develop new technologies and new storage techniques. This won’t be a cheap endeavor, but it’s one we should try to keep up with. Hard drives die, bit rot happens. Even off, a spinning drive will fail, as will an SSD with time. CD’s I’ve written 15+ years ago aren’t 100% readable.

Second, there’s data organization. How can you find what you want later when all you have are images of systems, backups of databases, static flat files of websites? A lot of sites now require JavaScript and other browser operations to be able to view/use the site. You’ll just have a flat file with a bunch of rendered HTML, can you really still find the one you want? Search boxes wont work, API calls will fail without the real site up and running. Databases have to be restored to be queried and if they’re relational, who will know how to connect those dots?

Third, formats. Sort of like the previous, but what happens when JPG is deprecated in favor of something better? Can you currently open up that file you wrote in 1985? Will there still be a program available to decode it? We’ll have to back those up as well… along with the OSes that they run on. And if there’s no processors left that can run on, we’ll need emulators. Obviously standards are great here, we may not forget how to read a PCX or GIF or JPG file for a while, but more niche things will definitely fall by the wayside.

Fourth, Timescale. Can we keep this stuff for 50 yrs? 100 yrs? 1000 yrs? What happens when our great*30-grand-children want to find this info. We regularly find things from a few thousand years ago here on earth with archeological digsites and such. There’s a difference between backing something up for use in a few months, and for use in a few years, what about a few hundred or thousand? Data storage will be vastly different, as will processors and displays and such. … Or what happens in a Horizon Zero Dawn scenario where all the secrets are locked up in a vault of technology left to rot that no one knows how to use because we’ve nuked ourselves into regression.

I think preservation is happening, the issue lies in accessibility. Projects like Archive.org are the public ones, but it is certain that private organizations are doing the same, just not making it public.

This is also something that is my biggest worry about the Fediverse. It has tools to deal with it, but they are self-contained. No search engine is crawling the Fediverse as far as I've looked, and no initiative to archive, index and overall make the content of the Fediverse accessible is currently in place, and that's a big risk. I'm sure we will soon be seeing loss of information for this reason, if not already happened.

Isn't that like a lot of older television shows? Lots of shows are lost as no one wanted to pay for tape storage.

Capitalism has no interest in preservation except where it is profitable. Thinking about the long-term future, archaeologist's success and acting on it is not profitiable.

Another problem is that even if sites and their content stay up they often reorganize it for various reasons - often by importing old content into some new platform - and don't care about the URLs the content is available at. Which breaks all links to it.

Some pages at least try to show you a page with suggestions what you might've been going for, but I've also seen those less and less over the years.

For my stuff I've been making sure to keep links working for over two decades now - on my personal page you can still access everything similary to /cgi-bin/script.cgi?page even though that script and the cgi-bin directory as a whole has been gone for over a decade. But I seem to be pretty alone in efforts trying to keep things at stable locations.

edit: I just noticed matrix.org broke all links coming from google search at least for bridges. They should've known better.

A friend of mine talked about data preservation in the internet in a blog post, which I consider to be a good read. Sure, there's a lot lost, but as he sais in the blog post, that's mostly gonna be trash content, the good stuff is generally comparatively well archived as people care about it.

We need deliberate efforts to archive everything efficiently.

We also need a way to decouple everyone's personal info from publicly available information about them, keeping in mind that not all publicly available information is intended to be that way.

Storage ain't cheap and it definitely ain't infinite.

This is a way harder problem than "the internet" being a bit more mindful can solve easily.

Not to absolve any companies from responsibility or anything.

I've had some thoughts on, essentially, doing more of what historically worked; a mix of "archival quality materials" and "incentives for enthusiasts". If we only focus on accumulating data like IA does, it is valuable, but we soak up a lot of spam in the process, and that creates some overwhelming costs.

The materials aspect generally means pushing for lower fidelity, uncomplicated formats, but this runs up against what I call the "terrarium problem": to preserve a precious rare flower exactly as is, you can't just take a picture, you have to package up the entire jungle. Like, we have emulators for old computing platforms, and they work, but someone has to maintain them, and if you wanted to write something new for those platforms, you are most likely dealing with a "rest of the software ecosystem" that is decades out of date. So I believe there's an element to that of encoding valuable information in such a way that it can be meaningful without requiring the jungle - e.g. viewing text outside of its original presentation. That tracks with humanity's oldest stories and how they contain some facts that survived generations of retellings.

The incentives part is tricky. I am crypto and NFT adjacent, and use this identity to participate in that unabashedly. But my view on what it's good for has shifted from the market framing towards examination of historical art markets, curation and communal memory. Having a story be retold is our primary way of preserving it - and putting information on-chain(like, actually on-chain. The state of the art in this can secure a few megabytes) creates a long-term incentive for the chain to "retell its stories" as a way of justifying its valuation. It's the same reason as why museums are more than "boring old stuff".

When you go to a museum you're experiencing a combination of incentives: the circumstances that built the collection, the business behind exhibiting it to the public, and the careers of the staff and curators. A blockchain's data is a huge collection - essentially a museum in the making, with the market element as a social construct that incentivizes preservation. So I believe archival is a thing blockchains could be very good at, given the right framing. If you like something and want it to stay around, that's a medium that will be happy to take payment to do so.

Long ago the saying was that "be careful - anything you post on the internet is forever". Well, time has certainly proven that to be false.

There's things like /r/datahoarder (not sure if they have a new community here) that run their own petabyte storage archiving projects, some people are doing their part.

during the twitter exodus my friend was fretting over not being able to access a beloved twitter account's tweets and wanting to save them somehow. I told her if she printed them all on acid free paper she had a better chance of being able to access them in the future than trying to save them digitally

We need deliberate efforts to archive everything efficiently.

We also need a way to decouple everyone's personal info from publicly available information about them, keeping in mind that not all publicly available information is intended to be that way.

Storage ain't cheap and it definitely ain't infinite.

This is a way harder problem than "the internet" being a bit more mindful can solve easily.

Not to absolve any companies from responsibility or anything.

In this aspect, the internet is closer to spoken language, than any written media. Even if you use a service to archive the things you find, it's still possible, that they shut down, too.

For my own stuff I try really hard to host it myself and the oldest still surviving thing is from 2003 and it's still online https://paradies.jeena.net/artikel/webdesign

I don't think it's a problem. If everything or most of internet would be somehow preserved, future antropologists would have explonentially more material to go through, which will be impossible. Unless the number of antropologists grows exponentially, similarily how internet does. But then there's a problem, if the amount of antropologists grow exponentially, it's beceause the overall human population grows exponentially. If human population grows exponentially, then also its produced content on internet grows even more exponentialier.

You see, the content on the internet will always grow faster than the discipline of antropology. And it's nothing new - think about all the lost "history" that was not preserved and we don't know about. The good news is that the most important things will be preserved naturally.

thankfully we do have people trying to archive things. sadly not everything will make it into that. just to much new stuff all the time to keep up with. but if we can keep the important and mostly important stuff

It sucks that we already have internet lost media

To be realistic we need to pick and choose what to keep and expend effort/resources on those chosen things.

Without a technological breakthrough in data storage at some point there's got to be some kind of triage done. We all generate more information now than ever before, and this trend just keeps increasing. With things like A.I, XR, the metaverse or other similar concepts it'll also get exponentially more insane how much data we generate. It's not realistic at the moment, technologically or financially, to keep all of it in multiple geographically distributed copies, in a format that will last forever. For a lot of people or organizations it's not even feasible to keep one copy in some cases due to costs.

To do otherwise we would need a breakthrough that enables insanely cheap, infinitely scalable storage, that is immune to corruption (physical or digital) and optionally immutable to prevent modification. It would have to function in such a way that any reasonably advanced civilization can use the basic laws of physics to figure out how it works and consume the contents without any context of what the devices are. It would also have to work regardless of how fragmented it is, to use terms of today's technology if they only find one hard drive out of what used to be a pool of 100, it still needs to work on some level.

It's an interesting thought experiment and hopefully there's some ridiculously smart people working on it.

One of the most interesting aspects of historic preservation of anything is that it's an extremely new concept. The modern view of it is about a single lifetime old, dating back to the early 20th century. Historic structures were nothing but old buildings and would be torn down with the materials repurposed as soon as there was a better use for the land or materials. Most historic buildings that date to the 19th century and earlier are standing not because people invested significant time and money into maintaining a historic structure as it originally was but because people were continuing to live, work, socialize or worship in the structure.

Preservation is entering a very interesting new phase right now particularly in transportation preservation as many of the vehicles in preservation have now spent significantly longer in preservation than they did in active service. There are locomotives that were preserved in the 50s and 60s who's early days of preservation are themselves a matter of their history. There are new-built replicas of locomotives from a hundred years earlier that are now a hundred years old. In railroad preservation there's also now the challenge of steam locomotives being so old and so costly to maintain that some museums are turning to building brand new locomotives based on original blueprints

Other historical artefacts like pottery, vellum writing, or stone tablets

I mean I could just smash or burn those things, and lots of important physical artifacts were smashed and burned over the years. I don't think that easy destructability is unique to data. As far as archaeology is concerned (and I'm no expert on the matter!), the fact that the artefacts are fragile is not an unprecedented challenge. What's scary IMO is the public perception that data, especially data on the cloud, is somehow immune from eventual destruction. This is the impulse that guides people (myself included) to be sloppy with archiving our data, specifically by placing trust in the corporations that administer cloud services to keep our data as if our of the kindness of their hearts.