Is there a collection of all human knowledge ever created ?

saltynuts420@lemm.ee to Asklemmy@lemmy.ml – 120 points –

Recently I was wandering if there is someone or some group preserving , collecting , organizing and publishing all the knowledge of mankind ever created throughout its existence so that if ever mankind faces the 6th mass extinction we don't have to reinvent the wheel and can have a kick start to our new post apocalyptic civilization .

73

Wikipedia is a great start. You can download its entirety, roughly 100gb. Most of the basic and advanced human knowledge.

Check out kiwix to get it offline

Seconded Wikipedia. The amount of knowledge that can be gleaned in mere minutes from Wikipedia is insane. It contains enough information to do most stuff, aside from blatantly illegal things.

Luckily it isnt that easy to burn down every server as burning carpets in south america or books in other places

Wikipedia is basically useless without the sources, though. It's just a TL;DR summary of things.

Wikipedia is absolutely not useless without the sources. I don’t even know what to say, that statement is so deranged.

WP as it is is of course not useless. But don't confuse it with a real library. Then, imagine in the apocalyptical worst case, having archived only that summary of humankind's knowledge. There's a vast amount of detail that WP is just not the right place for.

Well, sure, of course it leaves a lot of material out. But you'd start with Wikipedia surely, then move onto the source material. If Wikipedia leaves out a lot of material, any one of its sources leaves out a lot more.

Then you download the source material too, shouldn't be more than a few Yottabytes 🤷

Yes!

https://libraryofbabel.info/About.html

But there's a catch.

Quite a lot to read in that library.

Such an insightful commentary on the importance of the social contract and the irreplacibility of the individual. The only way forward is to share our personal experiences and strive for understanding. Once we know each other's value, we will never surrender our common bonds, disappoint one another, go behind each other's backs, nor do each other harm.

I feel like you're stretching the definition of "knowledge" and definitely "human knowledge" a bit there.

Maybe so — I think that's kind of the fun of it though 🙂

There isn’t. Yes Wikipedia has a lot of info but think of all the information that is in the hands of governments and corporations that are closely held secrets. Or that’s only in the minds of a few experts on the planet.

Like sure Wikipedia can tell you what a CPU is. But to build one from scratch, from the silica to building the machines and factories, that information is spread across multiple companies and never shared with the public. And only a few experts truly know how to do every step in the process, they have vital knowledge of that process that they feel is common sense and is not written down, which they pass on to the people they mentor. If those few people die at the same time in a catastrophe the knowledge that isn’t written down dies with them.

We already lost a lot of information of old tech from not that long ago because the companies went bankrupt or the people involved all died. Like we don’t even have all the knowledge to rebuild the Saturn V rockets, because the people involved, who hold vital knowledge, are dead and the supporting infrastructure, like the sub contractors (who also had vital knowledge), is gone as well.

Perhaps https://archive.org/ is the closest you could get? With nearly a trillion web pages in its archive, I don't think I've ever come across a database of knowledge that comes close to it's collection. What's interesting is that archive.org preserves not only web pages, but several pieces of binary content such as music, movies, art and even software applications and entire operating systems. Not sure if it would be enough to rebuild our society, but it would be a great starting point for most of our essentials.

Specifically their OpenLibrary division. They had a mission to make as many books as possible digitally available and free for everyone to borrow but unfortunately they keep getting hit with lawsuits and slowly take down more and more of their collection.

It’s “its”, not “it’s”, unless you mean “it is”, in which case it is “it’s “.

your bein pedantic. its okieish. Kaithx for grammly lesson. your the bestest.

yes archive.org is pretty good in finding some very obscure and rare movie from 30s or 40s or some old books or niche software or music but most of the things on it are webpages and 100s of copies of the same webpage in different times (not very useful in a post apocalyptic world tho as most of the things on the archived websites you can't even click because its only a snapshot )

All of Wikipedia is <256 gb.

All of Wikipedia in English <64 gb.

Then archive.org for multimedia, ~10 peta bytes. Yipes.

1 more...

That is pretty much exactly the goal of the Wikimedia Foundation which runs Wikipedia and its sister projects.

But by now we figured out what wikis can do well and what not. Wikis are suitable for crowdsourcing objective facts about the world (all it takes is one person to add any given fact), they are not a universal remedy for everything, especially not contentious issues or useful instructional materials.

I have made more than 100000 edits to their projects. I don't participate there anymore. The time when they were a force for good in the world is long past.

Why aren't they a force for good anymore?

Many reasons most of which you'll only understand if you pay some attention to what's going on behind their scenes.

There are reasons why nowadays pretty much everywhere else on the Internet more content is created all the time than on the Wikimedia projects.

The Wikipedias' "neutral point of view" policy used to mean "we try to treat all sides fairly", now it means "we are writing an unconditional propaganda organ for the status quo". The mainstream media that is accepted as "reliable" as Wikipedia sources just isn't that credible anymore.

Also, when I started editing there, the individual projects were mostly left alone by the WMF. Nowadays the WMF issues intransparent sanctions, up to lifetime bans from all projects, left and right.

I wish someone started an organization with the same goals as the WMF with an actually working system where people could actually enjoy participating.

Be the change you want to see in the world...

Heck most of the hard work is already done for you, since the software that runs Wikipedia is open source.

Many people have tried that before. Wikis just aren't that appealing anymore. Today's internet is all about social media.

If there's anything that is absolutely atrocious as a searchable repository of knowledge, it's social media.

A Library. Or if digital, Wikipedia and Archive.org.

most libraries lack 90% of collective human knowledge and most libraries today (in asia particularly) are pretty shit in number and quality of books .. Wikipedia yes but still it can be manipulated by rich people or government for their own interests

Not just can be. Wikipedia has been proven to be manipulated by political interests.

I don't personally know of any but in a similar vein there are some stone monuments intended to convey information after an apocalypse like the Georgia Guidestones or the nuclear waste site warning stones. GitHub put a snapshot of all active code repositories from 2020 in arctic permafrost, and there is the arctic seed vault for preserving plant species.

Man those nuclear waste messages makes it sound like it's a cursed land

That's their intention.

The problem with such approaches will be human curiosity. Imagine today's scientists find such a site from the late paleolithic which has messages like "This site is cursed; we buried here what causes death and pestilence to us; go no further or it will do the same to you!" -- You bet they will want to see what is inside the "buried temple of death".

Yes but they would only send one person or a small group, carefully and with protection. Without any sign people would just walk in and slowly suffer the consequences.

"On July 6, 2022, an explosive device was detonated at the site, destroying the Swahili/Hindi language slab and causing significant damage to the capstone. Nearby residents reportedly heard and felt explosions at around 4:00 a.m" the rocks got destroyed by a mere explosive and they thought it could survive a nuclear war lol

Well, if a nuke actually hits anything built to withstand nuclear war, it will break. There is nothing really that can withstand direct exposure to powerful explosives.

I assume the plan was Georgia since there's nothing worth nukeing there.

Check out this book: https://en.m.wikipedia.org/wiki/The_Knowledge:_How_to_Rebuild_Our_World_from_Scratch. It analyses that precise question in the first chapter. The author argues that even though Wikipedia is probably the closest thing there is, there is a clear lack of practical knowledge that will be essential in the situation that you are describing. Science progress heavily relies on industrial progress, and even if you know how to build something that doesn't mean that you can do it, as there are other things that are required first.

Damn that sounds like interesting book to read. Got to get a copy

I think the internet as a whole is going to be the closest we'll ever come. Capitalism will make sure it's never even close to complete so it always has something to monetize.

I wouldn't say "complete" can even be sufficiently defined in this case. Every functional definition I can think of has a limiting factor.

Let's try to define knowledge. What kind of information qualifies? We can usually think of important, useful info like physics and medicine. But what about other data, like sports game stats, atmospheric sensor readings, or even something more esoteric, like the location data of every object on earth.

And even if we could have the information of every single thing at any particular time, what about when things change in the next second? And the one afterwards?

Essentially, nothing will ever be "complete". Thanks for listening to my rant on semantics.

you can read pretty much (except the lost media like those lost in library burnings , film destruction and wars) read any book written by humans since 2500 bce (example Rig Veda the first ved of Hinduism was written even before 2500 and is today said to be 98% at its original state thanks to Indian gurus and saints who passed it on orally and was made into a book only after 8th century) , watch any movie ever released , hear any music ever made after recording was invented .

ofcourse there is a catch that these medias are not freely and publicly available and most you have to pirate in order to consume it thus we need to have a centralised database of these things safely kept somewhere so that we don't have to reinvent the wheel in case of a catastrophic event .

Besides what other commenters already said, archive.org does a great job.

I'm surprised no one mentioned projects like libgen and scihub. They are much better than Wikipedia imo.

imo zlib is much better but they keep changing their domain ... also sci hub is only for research papers which most people can understand

It's never going to be all knowledge, since a lot of stuff is just lost or never recorded. A ton of stuff (like this thread) are probably low on the priority list for recording as well. But the closest you'd probably get to a full catalog of human knowledge (at last text based) are the huge data sets of nearly all text data on the internet used for training LLMs. I wouldn't be surprised if there are ones soon that include video and pictures as well, since newer AI models are starting to be able to interpret those too.

I believe this is one of those data sets: https://github.com/yaodongC/awesome-instruction-dataset

Edit: here's a big data set used for a lot of gpt3 https://commoncrawl.org/

Related monument / installation:

https://www.historyingranite.org/

While its not all of Human knowledge it's a very important part of it, our history.

Ultimately if the goal is for it to survive societal collapse to the point where knowledge gets lost then books are a better bet than any digital storage. As such most large libraries hold the knowledge needed to take us up to the industrial revolution at least. Semiconductors is hard to teach in only text, by that point you really need the ability to produce the machines needed etc. But the underlying theory will be in a library. Also a lot gets proprietary and trade secrets when we get within the last 30 years and that knowledge won't ever get written down in a central repository until it's aged out of being protected.

Knowledge is not what make people distroy this planet though, or what helps against it. We know enough already

read the query again

I read it, what I meant is: is knowledge the thing that makes our society nice? Or would it lead to the same shit we have right now, if its not accompanied by something else?