Can the fediverse really scale?
Reddit Migration@kbin.social – 9 points – Can the fediverse really scale?
I'm thinking about kbin/lemmy here, but my understanding is that every instance has to maintain a copy not just of the magazines/communities it hosts but also every other one its users subscribe to? So in a worst case scenario, the storage requirements would grow in sort of an N-squared fashion, would they not? If all of reddit wound up here, that's a lot of storage.
I guess the idea is that a smaller instance would only have to manage some subset of magazines of interest to its membership. But all it would take would be some bots sneaking in there and subscribing to everything, sort of like the kid who hits every button on the elevator.
I'm not sure what the solution is? I think it's good to have more than one copy of any given magazine/community kicking around in case its host instance goes dark/defederates/whatever. But maybe there is some sort of middle ground? Like perhaps a somewhat torrent-like scheme for backup where instances can contribute as much storage/bandwidth as is realistic for them? I'm not sure how that would work, but you would somehow want to ensure that there are a few redundant copies of every community distributed across all instances. And of course an instance could still go on caching the more active communities for practical reasons.
i guess at some point they could Introduce timed caching.
Like a Post from lets say 2021 could get removed at a foreign instance till someone wants to access it again on Instance X while its original from Y.
This is the solution Usenet uses. I'd expect it to make it's way to the fediverse if it gets big enough.
I'm still learning but share your concern.
I also think there's different dimensions to the growth too. A lemmy server such as programming.dev may have many communities which become popular and it's primary task is to be the home to those communities and federate that out to the wider community.
At the same time it has to pull in any random community that even a single user on that server wants to look at and store it.
The server that is home to programming discussion could buckle under the load of too many posts to /c/funny. It doesn't seem right. They are different responsiblities.
Does lemmy federate images as well or only text? I think it only does text and loads images from the original instance. In that case, its not a lot of storage.