Do you think lemmy will sustain scalability?

Cloudkid@lemmus.org to Asklemmy@lemmy.ml – 30 points –

Not just speaking technically but economically. As user number mount up their post is just going to add up. I know instaces may distribute user, but server space isn't going to go down for anyone. It just seems like built to fall.

25

@Cloudkid It's something that we almost had a soft test when Reddit went downhill a few months back, if users do scale across different instances the load is lifted the main problem you would have is if Reddit goes down and all the users go to one instance such as Lemmy.world or Lemmy.ml as it would certainly make the Fediverse look bad and crash the servers.

The Server space is already a problem for platforms such as YouTube, Facebook and Reddit but they control it by ads and payments from users such as YouTube Premium on YouTube. Main Instances of each platform may have to take up the Ads system if donations don't come in or it may have to become easier for users to create their own website. I read a good article earlier today about how there are issues for the Fediverse in the short term but it seems promising if they get solved.

https://www.theverge.com/2023/10/23/23928550/posse-posting-activitypub-standard-twitter-tumblr-mastodon

Great read, my only problem with publishing content on own domain is it gives power to those who can afford to pay for website and hosting. Internet should be democratic, and only way to do that is to make it accessible, but I also want to see more people owning their creation.

I'd love to see cooperatives where people pay some monthly amount to support a server that's democratically administered. To benefit from economies of scale while still having a say in how things are run.

@Cloudkid Yes it's something that I have thought about as well, it's good that people have created instances for the Fediverse as it means that you only access it from someone you trust, this will mean that most likely you would access it from a website/instance that you, your family or friends have created allowing an easier cost effect hopefully for the users. I know if I had enough money and time to work out how instances worked I would be creating my own instance locked to me, close family and friends especially as I trust someone I've met up with a few times over these billionaires that are making money from our data.

Lemmy is setup quite ok to scale users and traffic, since the users are (theoretically) distributed over many instances. In reality it doesn't work out perfectly, since people generally are more likely to join the biggest instances, so there's quite an imbalance there.

But what's worse is content replication. As soon as an user requests to look at a community for the first time, the whole community will get completely replicated on that user's instance. So any decently sized instance will pretty much replicate almost all communities, at least all that have content.

There is no scaling or storage balancing mechanic here. Even if no user ever touches that replicated community again, it will continue to be replicated, will fetch all new posts/comments and store them in the instance.

There is also currently no workaround to this (like there is for users/traffic, which you can just tell to join a different instance).

So if Lemmy ever gets to the point where gigabytes of data gets posted every day (which is only about 1000 pictures a day) storage demand will get so high, that hosting an instance will be seriously costly, which will probably lead to instances without any kind of cash flow shutting down, which will in turn lead to more users and thus traffic on the remaining instances.

I guess, that's one of the biggest technical (and conceptual) roadblocks that need to be addressed if Lemmy ever grows that big.

As the project evolves, there will be fixes for that kind of problem, it is not an immediate problem.

Hopefully. Because right now this is not a simple bug but engrained in the core concept of how Lemmy works.

Is there a feature to migrate users from one instance to another ?

I think it will be important when some instances will shut down. As long as people can move, then I think we can spread out the load.

I know my instance is handling the load very well and I'm really grateful for it.

Account migration is planned, at least partially, for the next release IIRC

i think the resources youre referring to are easily, fluidly handled and there should be no long term concern as strategies are developed to handle those resources. server space can be recycled, caches cleared, etc.

youre not wrong in that storage is a concern, but it is one of the cheaper components from my POV running a full instance.

Without ad revenue, I think server bill will pille up quick. I wish we could have insight from someone running instance.

I'm running an instance, on local home fiber that I'm already paying for (30€/m around 700Mb up for real, 2x2.5Gb down theoretical) so no problem today, at all.

Don't make the error thinking today's tech will linger, we will most probably have better and cheaper hardware in the future, but also smarter algorithms dealing with it all, remember it's a first try and can only get better.

i think the only sustainable model is having a donation process built into the site. its very successful in other platforms.

its that or ads. no one wants that.

my storage costs are currently < 1/10th of the total o f my other costs. sitting at $1.00/day on one of the most expensive providers. my server has 12 users

running a full cdn, cloudfront, smtp, etc.

Just purge stuff older than X if storage becomes an issue and make people use image hosts outside of lemmy

I'm more of an archivist, belive that will cause loss of many meaningfull conversation and information.

@Cloudkid This is where hosting your own instance is better as even things on Facebook, YouTube and Twitter won't last forever as it only really stays till they can't be bothered to keep the older stuff no one looks at intact, The only way to make sure things stay is by hosting it yourself and I really do hope it gets easier over time to host your own Fediverse Instance.

i would like to see a drop in word-press like (fuck wordpress) install for a fully-federatable instance.

im sure there are many products there or nearly there. It smells like opportunity for the right, talented folks.

im having a blast with mbin on aws

@originalucifer Glad you're having fun with Mbin, I know I recently moved across to Mbin on Kbin.run as of the bug fixes and stuff. Sadly don't know how to create my own instance as would love to create it on a sub-domain of the website I run for business as I'm sure connecting to the Fediverse from my own instance would be really cool and would help me control what I see, so much more also would make it easier when it comes to getting Family and Friends Involved.

theres a great crew in the devs/other admins of mbin on matrix, and i am always happy to help if you ever decide to give it a go.

you can probably do it very cheaply. i went the most expensive route, and im payin ~$1.00/day. not using aws, you could maybe that down to $0.25
one thing to keep in mind, once your instance is up with a domain name (myinstance.mydomain.com) it cant really be changed.

im an old sysadmin, i hate social media... i just like looking at the numbers..keeping things running. my hate for the new reddit has overtaken my hate for social media.

@originalucifer Thank you, it's something that I am really interested in, just need to be in the right mindset to be ready to get that going. I knew about the domain not being able to change and have already thought about the subdomain I would use as I think just calling it something simple like Social.mydomain.com is something that would work and make people understand why it's called that.

Economically, I'd say yes. You can get a vServer for like 10€ per month and it'll have something like 200GB of storage and more than enough computing power to run lemmy. Text is small so only posting many pictures/videos could be a problem. But 10€/$ is kinda affordable even for a single person and the storage should last you quite a bit. Even if you share the instance with a few of your friends.

And you could even have a community or non-profit pay for a larger vServer and ask everyone to contribute a bit of money. I'd be surprised if you needed more than say $3 per month and person. (probably less?)

So economically, I'd say it's easily doable.

Technically, depends on what the developers do. There are still quite some issues to fix. And I'd consider caching and how easy it is to set up an instance for your community more technical problems.