Lemmy.world status update 2023-07-04

Ruud@lemmy.worldmod to Lemmy.World Announcements@lemmy.world – 1979 points –

Status update July 4th

Just wanted to let you know where we are with Lemmy.world.

Issues

As you might have noticed, things still won't work as desired.. we see several issues:

Performance

  • Loading is mostly OK, but sometimes things take forever
  • We (and you) see many 502 errors, resulting in empty pages etc.
  • System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

Bugs

  • Replying to a DM doesn't seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
  • 2FA seems to be a problem for many people. It doesn't always work as expected.

Troubleshooting

We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @nutomic@lemmy.ml is also helping with current issues.

So, all is not yet running smoothly as we hoped, but with all this help we'll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!

304

I'm just excited to be back in the Wild West again -- all of the big players had bumps, at least this one is working to fix them.

And giving updates!

I’d rather have to deal with hiccups and bumps along the way, because the community only grows more each time.

That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%

Lemmy has a memory leak? Or, should I say, a "lemmory leak"?

Lemmy has a memory leak? Or, should I say, a “lemmory leak”?

A lemmory meek, obviously!

On some of the latest release candidates I think so.

Wait isn't lemmy written in rust how do you create a memory leak in rust? Unsafe mode?

let mut memory_leak = Vec::<u8>::new();
loop {
  memory_leak.push(0);
}

That's not a memory leak though. That's just hording memory. Leaked memory is inaccessible.

In the example it’s inaccessible.

It's not. The vec is still accessible and if it goes out of scope rust automatically clears the memory.

But it will never fall out of scope because of the loop

Exactly but it's still accessible since it's in scope.

Right but where does it say that something has to be inaccessible to qualify as a memory leak?

"A memory leak is a process in which a program or application persistently retains a computer’s primary memory. It occurs when the resident memory program does not return or release allocated memory space, even after execution, resulting in slower or unresponsive system behavior." Source

That’s not a serious source. Any unbound allocation is a memory leak if it serves no useful purpose.

Really appreciate all the time and effort you all put in especially while Lemmy is growing so fast. Couldn't happen without you!

Thank you so much for your hard work and for fixing everything tirelessly, so that we can waste some time with posting beans and stuff lol.

Seriously, you're doing a great job <3

As somebody who flocked to Voat during the height of the Ellen Pao controversy and remembered the site being rendered unusable for whole days at a time from the Reddit Hug of Death, I'm remarkably surprised at how well Lemmy.world has held up. I thought the fediverse would have truly crumbled from this exodus.

I remember when Voat came out and the slight exedous that brought. I made an account and everything but it never properly took off. I checked on it two or three years later and it was just filled with alt-right/racist/transphobic garbage. Sad it never took off as a reddit alternative, reddit likely would have greatly benefited from a proper alternative, not sad it closed down after I saw what it ended up.

So far the fediverse feels really different tho, very explicitly anti that type of shit. I'm sure it will pop up, they always do, but maybe now people know how to deal with it. Block it, defederate, deplatform.

FYI, it has popped up explodingheads is a great example but many servers including lemmy.world became proactive in defederating from the instance

Yeah, but as soon as instances defederated from there, those bigots started making alts on those instances.

Because it's not enough for them to have their own safe space for bigotry, they want to argue that bigotry is ok so they get their rage fix from arguing online.

If Lemmy can instances can't IP ban like reddit does, they're just going to keep making accounts here every time they're banned. Then go to exploding heads to circle jerk about how they got banned again.

One can also increase the cost of registration by requiring some questions and a waiting period. Adding friction makes this attack a lot less fun for the attacker.

It's not a perfect solution, but perfect must not be the enemy of good.

At the same time, it's also not suitable for a server the size of lemmy.world. We just have too many users, and enough new people joining that we'd need several people whose singular task was to process registrations in that case.

One of the only servers I've defederated my personal instance from.

Lemmy is a distributed system that can't quite be perma-killed, kind of like ye olde usenet. Anyone can create a new instance at will.

As a game dev for bigwigs I know all too well about memory leaks, and so very much appreciate your patch notes, updates, and transparency. You're doing great with such fast exponential growth

💙 Thanks for your hard work!

Amen. So happy to be done with Reddit.

I want this to succeed so badly. I truly feel like it’s going to be sink or swim and will reflect how all enshitification efforts will play out.

Band together now and people see there’s a chance. Fail and we are doomed to corporate greed in every facet of our lives.

Wow. Drama! Chill bruh. They'll get there. If it takes too long for some, they can switch to other instances.

I'm here for the drama too!

Fuck the cleanliness of the corporate web.

This is the level of transparency that most companies should strive for. Ironic that in terms of fixing things, volunteer and passion projects seem to be more on top of issues compared to big companies with hundreds of employees.

You said it: passion projects. While being paid is surely a motivator, seeing your pet project take off the way Lemmy is can be so intoxicating and rewarding! I plan to donate as soon as I get paid on Friday! I want to see this succeed, even if it is just to spite Reddit, and I am willing to pay for the pleasure.

Could I get a discord invite? I’m an ex sysadmin with a. Lot of free time

What was that? We're going to need more and better hardware soon, and you have a Patreon and a paypal on the sidebar?

Yeah, that sounds pretty reasonable, we can work with that.

Just signed up for the 8€/month tier, hopefully my blue checkmark will arrive soon!

It’s here!

✅

(Please do note that the blue checkmark is for our 1€ “Bean” tier subscribers.)

We’re very pleased to provide you with our premium deluxe green checkmark for your generous 8€ subscription to our “Bacon” tier!

Feel free to reach out to our Lemmy Cares team @ 1-800-FUC-SPEZ for any questions or concerns that you may have.

There is a sidebar?

On web browser, yep.

I'm thoroughly confused by most of what is going on. I'm using kbin on mobile. The interface isn't terrible and it fills the random surfing void that I am used to.

Just a heads up, you're using kbin which is a different "website" than lemmy. You are also seeing a thread that originated in lemmy. That means both kbin and lemmy can see each other's content, but the website UI is different between them so if you read about some UI element and can't find it in kbin that might be one of the reasons.

That being said, kbin also has a sidebar with useful information. In the mobile site you can find it at the very bottom on the page, since it doesn't fit to the side in such a small screen.

Omg I never scrolled that far down lmao, now I can go block 196 and the other shit posting magazines

Yesss! Lemmy was difficult to use until I did that. Then eventually I had a long enough list of communities/magazines that I could browse subscribed only and that really supposed the issue. I'm busy not a fan of random porn and bigoted memes y'know?

I mean if a random nude crosses my feed I'm not bothered by it but I really just want to read some cool stuff maybe news etc for like 15 mins at a time throughout the day and this is scratching that itch for me.

If you're on Kbin mobile, you can also tap the hamburger bar on the top left to bring up that menu directly, instead of having to scroll all the way down.

Also great for if you want to fiddle with dark mode and all of that.

Every community has a sidebar. If you’re at the page for the community it should be at the top of the website. “sidebar+”

If you’re using a 3rd party app it’s different for each one in terms of how to see it.

i just wanted to thank you for doing your best to fix lemmy.world as soon as possible.

but please, don't feel forced to overwork yourselves. i understand you want to do it soon so more people can move from Reddit, but i wouldn't like that Lemmy software and community developers overwork and feel miserable, as those things are some of the very motives you escaped from Reddit in first place.

in my opinion, it would be nice that we users understand this situation and, if we want lemmy so bad, we actively help with it.

this applies to all lemmy instances and communities, ofc. have a nice day you all! ^^

Plus, slow steady growth means eventual success. Burnout is very real if you never take a break.

so true, pal! slowly, with patience, no rushing, putting love into it, organizing ourselves, working smart is better than working hard and fast.

because of the federated nature of fediverses like Lemmy, it is very possible that many people are doing the very same task without even knowing they are duping each other's efforts.

and that's sad because if they knew, they could be teaming up, or splitting the task in two, in order to avoid wasting different efforts into dupe results.

i have learnt a thing or two about burnout, it's better for me to make 40% planning, and 40 % self-care and so the 20 % of execution becomes piece of cake.

but this is just my opinion. anyway, please take care, pals <3

@ruud > That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%

Hmm, makes me curious if there is a Lemmy memory leak, or simply that the load wants to stabilize above of the RAM you have? I hope contributions can help you with another 32 GB RAM? Thank you for your work! 🍻

We have 128GB of RAM. It just skyrockets after a while!

@ruud Oh damn. This spontaneously sounds crazy but I’m admittedly a novice at servers on this scale.

I haven't checked into the code yet, but I imagine you can map out what all is in memory and force more aggressive garbage collection to find some middle ground.

Rust is not a GC language.

I read down the list afterwards and found it was using Rust. I skimmed through the source and it is well organized, but would still take quite a while to get up to speed on.

I saw unit and integration tests. It might be beneficial to generate or capture some data to replay to simulate the load and add debugging. I don't know much about the abstraction layers. I did see opentelemetry, which is a project I got frustrated with on the lack of stability (fast changes on api).

I have only dabbled with Rust to test the waters. The largest thing I've made was a GUI snake game, and made it portable so it could be compiled for cross platform.

The work you're doing is greatly appreciated! It's like you invited half the internet into your house. I feel like I should've brought a cake or something

You should consider setting up a small donation to the site, if you can afford it. If you throw $5 a month at them, you're not only covering your own hosting fees, but covering for some of the folks who can't afford to as well. And maybe Ruud gets to buy some beans.

Thanks for all of your effort. Even though we are on different instances, it’s important for the Fediverse community that you succeed. You are doing valuable work, and I appreciate it.

Not just that, but the code contributed to Lemmy by this debugging will make Lemmy run faster for everyone on every instance, which is makes the ecosystem that much better.

Huge respect for what you've built here, but it might be worth reaching out to the lemm.ee admin. I only know enough DevOps and cloud hosting to be dangerous, not helpful. But his instance seems stable and scalable. He might be able to offer some insight into the issues here

Yes he's one of the other admins in our Discord, he's very helpful!

Of course these performance issues are a bit annoying, but I gotta say that I love these updates and explanations here. Great communication, keep it up, please!

That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%

who'd have thought memory leaks would be possible in Rust 🤯

(sorry not sorry Rust devs)

Ackchually, leaking memory is totally possible in Rust! But so is hogging way too much of it because you're hanging on to too many things in memory at once🤷

Keep up the good work!

I created an account in lemm.ee until the issues are fixed. Then I will happily go back to my lemmy.world account.

Lemm.ee is also a good choice!

If I have an account at lemmy.world, but have another instance selected, does that take some of the burden off the admins or do I need to do something more?

.world is definitely running smoother than when I joined 3 days ago, back then it was impossible to comment and the lag was immense, right now I just have to occasionally reload the page, but that's nothing in comparison.

You guys are doing an amazing work! I'm broke, so here are some coins 🪙🪙🪙🪙 beans 🫘🫘🫘🫘

good news, a fix might be in the works: https://github.com/LemmyNet/lemmy/pull/3482

Yes phiresky is working with us improving performance, has helped a lot so far! (We're now running a custom build with some of his improvements)

I've noticed lemmy.world has been much more responsive today so something seems to be working!

Thanks for sharing. It was very interesting to see the graphs showing the before and after.

I am very forgiving of the bugs I encounter on Lemmy instances because Lemmy is still growing and it's essentially still in beta. I am totally unforgiving of Reddit crashing virtually every day after almost two decades.

The need to restart server every so often to avoid excessive ram usage bit is very interesting to me. This sounds like some issue with memory management. Not necessarily a leak, but maybe something like server keeping unnecessary references so the object cannot be dropped.

Anyway, from my experience Rust developers love debugging such kind of problems. Are Lemmy Devs aware of this issue? And do you publish server usage logs somewhere to look deeper into that?

server keeping unnecessary references so the object cannot be dropped

You indeed just described a memory leak :D

System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

Shouldn't we be discussing closing registrations?

There's a lot of momentum to move away from reddit right now, and closing registrations would be a wet blanket. Personally, I'll take the performance issues and transparency in the process over closing registrations.

This. Don't stop the train. People need to be able to come over freely.

Does Lemmy have the ability to replace default links?

Basically, replace signup link with one that redirects to a page that gives a very simple as possible explanation what's going on, what fediverse is and gives s list of other instances to try.

Reinforce "All are viable and can browse lemmy.world subs"... Or communities or whatever term we use here for lemmy equivalent of subreddits.

I was personally thinking more along the lines of if we could have a load balancer whose sole job is to route users to a random set of possible instances (which can all be administered by the same person, so that you're still joining the instance "group" that you want). The load balancer would route someone the first time they land on the page and also handle logins. That's it.

I'm assuming that the servers we're talking about are single servers, because that's how things sound. I'm personally used to only developing servers that use the "many servers behind a load balancer" approach. While distributed databases can certainly make those easier, in the absence of support for that, you could always run the backends as entirely separate servers, with the load balancer just serving to pick the backend. So you'd have a lemmy1.world and so on.

Of course, for all I know, maybe this is already the architecture of a Lemmy instance. I've never checked. Even with a good architecture, scaling can be difficult. An unnoticeable performance issue in a dev environment can be a massive bottleneck when you have tens of thousands of concurrent users.

I'm not sure I'm following.

Wouldn't this load balancer be swapping the user's current instance user420@lemmy.world may suddenly become user420@lemm.ee?

Or more like multiple servers within the same umbrella instance? User420@lemmy.world, User420@lemmy.world1, User420@lemmy.world2, User420@lemmy.world3.

Apologies, while I think myself fairly tech savvy, development and networking is still a bit out of reach.

Or more like multiple servers within the same umbrella instance? User420@lemmy.world, User420@lemmy.world1, User420@lemmy.world2, User420@lemmy.world3.

This is what I was originally picturing. So that logged in users would be browsing on pretty much entirely separate instances (avoiding them having to reuse as much).

I hadn't really decided on how I best liked the idea of handling logins, since there's so many possible options. It could be that users would just have to either know their server (so you'd have to sign in as User420@lemmy1.world). Or the load balancer could maintain a simple store of users/emails to instances to avoid that. Or at the cost of extra complexity (yay), you could replicate the user across all the instances but only make a single instance active for that user at a time (that's a pretty common technique with systems I work with, with servers being strongly coupled to some range of resources to maximize efficiency).

I noticed you talked about the load balancer being a person. Sounds like it'd be better if it was a bot. They just see which pool is currently the emptiest and put them there, right?

Although you seem to be suggesting live instance swapping. Which might be possible in the future. Right now appears to be tied to registration.

Federation-wise it would be better if new users spread out. Between clueless redditors and impossible ideal, I prefer if they at least made an account and check out what Lemmy has to offer. The curious ones will eventually settle down and even redistribute into smaller instances.

The curious ones will eventually settle down and even redistribute into smaller instances

Absolutely. I migrated from lemmy.ml when that was having too many sign-ups, and I'm not opposed to migrating from lemmy.world to help with their load. I'm sure I'm one of many

It'd help a lot if we had a convenient way to export a user profile to a different instance. It doesn't even need to attribute posts/comments in any way. Just saving settings, subscriptions, and blocks would be plenty for most people.

How can I donate to lemmy.world?

Not an expert, but here is where I set up a recurring donation. Apparently Mastadon.world and Lemmy.world are run by the same admins. Took me a while to understand this, please correct me if I'm wrong.

Cloud architect here— I’m sure someone’s probably already brought it up, but I’m curious if any cloud native services have been considered to take the place of what I’m sure are wildly expensive server machines. E.g. serve frontends from cloudfront, host the read-side API on Lambda@Edge so you can aggressively and regionally cache API responses, anything other than an SQL for the database — model it in DynamoDB for dirt cheap wicked speed, or Neptune for a graph database that’s more expensive but more featureful. Drop sync jobs for federated connections into SQS, have a lambda process that too, and it will scale as horizontally as you need to clear the queue in reasonable time.

It’s not quite as simple to develop and deploy as docker containers you can throw anywhere, but the massive scale you can achieve with that for fractions of the cost of servers or fargate with that much RAM is pretty great.

Or maybe you already tried/modeled this and discovered it’s terrible for you use case, in which case ignore me ;-)

You were so close until you mentioned trying to ditch SQL. Lemmy is 100% tied hard to it, and trying to replicate what it does without ACID and Joins is going to require a massive rewrite. More importantly - Lemmy's docs suggest a docker-compose stack, not even k8s for now, it's trying really hard not to tie into a single cloud provider and avoid having three cloud deployment scripts. Which means SQS, lambdas and cloudfront out in the short term. Quick question, are there any STOMP compliant vendors for SQS and lambda equivalent yet?

Also, the growth lemmy.world has seen has been far outside what any team could handle ime. Most products would have closed signups to handle current load and scale, well done to all involved!

If Postgres becomes the bottleneck I wonder whether something like Citus could work to shard the data (relatively) transparently?

One could also move to having multiple read-only PostgreSQL replica instances used when generating the site and a single read-write instance that you'd use whenever anything changes (which is comparatively rare).

cloudfront helps a lot with the client and is absolutely compatible with lemmy if you set it up correctly. possibly it could also help cache api responses, i haven't looked into that part yet.

the database, on the other hand, would need a nearly full rewrite. lemmy uses postgres and dumping it for something else would be a huge pain for the entire federated community. it could probably tear it in half.

there's also the issue of pictrs, which uses a stateful container and isn't yet able to use an external database which would allow you to scale it horizontally. resolving that one is on the roadmap though, and for the most part you can aggressively cache the pictrs get requests to alleviate the read-side load.

but whatever the solution is, it kinda needs to be as simple as developing and deploying docker containers you can throw anywhere. the vendor-agnostic setup is a very important part of the open-source setup of lemmy. it's fine to build on top of that, but currently anyone with docker-compose installed can run the service and that really should be retained.

Staying cloud agnostic is very important and CDN services like cloudflare/cloudfront have inherrent privacy issues. IMO the stack should remain hostable on anyones home server environment.

People of lemmy.world should encourage the new users joining up on lemmy.world to other create accounts on other instances instead

Thanks team, love your work! Appreciate the updates and improvements but happy to have this place despite the bugs and quirks :)

It has become way smoother the last few hours. The work you guys from .world and Lemmy's devs have been doing is amazing!

Would donations help in the long run? How can we donate?

Donations invested into REITs would fund Lemmy indefinitely, has anyone pondered this idea?

Donations definitely help. They are a real motivator and allow people to make more time for contributing. For instances it is also a way to cover hosting costs.

You can donate directly to the Lemmy development or you can see if your instance accepts donations. Personally I just signed up to the Lemmy Patreon.

Thanks again for all the hard work on Lemmy World. It feels fast today

Its amazing this site hasn't completely melted down yes it sucks trying to use it and having nothing load but the fact that the site has mostly worked when I want to use it and the fact you guys have less than 1% of 1% of reddits resources is mind blowing

I feel that lemmy runs smoothly the past few hours with very few hiccups and mostly on the upvoting and commenting side. I encountered no issues yet on the loading of the post.

Thank you so much for all your hard work.

Agreed, it seems fine now, makes such a difference to whether I want to keep interacting with Lemmy.

I'm very much so okay putting up with hiccups right now. I know things will improve in the future.

Thank you and everyone for all your hard work!

I can't imagine how hard it is to rua server that has influx of large users. I Thank you for your hard work to run and maintain this instance. Hope it works out well and future will be a smooth sail.

Thanks for the update, you're leading the way for Lemmy.

I was prepared for servers crashing and lagging, but you're keeping things moving thus far, despite receiving 85k new users in a month. Heroic stuff.

Thank you for all of your effort on this. Adjusting to rapid growth is a huge challenge, and we all appreciate the effort you and everyone else is putting in to trying to make this instance run smoothly.

Can anyone give me a link to donate to help out? Ever since discovering lemmy I hardly use reddit and would like to contribute to this community more!

Yep! Donating to both their Open Collective page for Mastodon.World (didn't grab that link by accident), as well as their Patreon page for Mastodon.World works!

They have a note up on Patreon that donations there also contribute to World.Lemmy, as well as another Lemmy instance!

Here's the link to their Patreon; https://patreon.com/mastodonworld

1 more...

Thank you for the transparency @ruud@lemmy.world It is rare for platforms to scale this quickly so issues like this are inevitable. Good luck for the troubleshooting!

Thanks for all the work you, and all the others do to welcome so many new users, and thank you for keeping us posted on your efforts to smooth things over.

From what I see the general sentiment is very positive and understanding, and it feels special to be part of a community that's more upbeat than what you see in most online places.

Thanks u/ruud!

How are you managing all of this in addition to having a life outside of Lemmy? This kind of went from 0-60 in a blink.

Same happened with mastodon.world in November. Family goes first, then work, and then all of my hobbies, of which this is one. (But the one taking up most time at the moment...)

You’re a legend! Thanks so much Ruud!

Glad to hear that your priorities are in the right order. Family comes before helping random strangers posts pictures of beans on the internet.

Small remark instead of /u/.... you should use the at-sign @ to mention somebody. For example: Hey @Ghostalmedia@lemmy.world.

How do you know which community the user is from?

It should tell you on mousehover if not straight up listed. If there's no community, it's because they come from lemmy.world.

TBH its all confusing and all names should be full names like email adress.

In the browser when you hover over the display name, you should see the full username. I don't know how it's shown in the various apps but it should appear if you view the user's profile.

The redirects to baseball games glitch is gone. I am unsure if this is because the glitch is fixed or if baseball season is over.

😆 sorry about that! Baseball season is still very much on.

This issue was caused by the websockets replacing the current thread you were on with newly updated or created posts. Our game threads get updated frequently so they were constantly popping up in people's faces. We decreased the frequency of the updates pretty substantially in order to be less annoying to unrelated instances but it was still happening occasionally.

Now that we're all through with websockets it won't be an issue anymore!

I am sure the stress is unreal, but It is appreciated. You all stepped up when everyone was stepping down (or trying to sellout). I am hopeful this will turn into something great.

The good thing about the fediverse is we can sign up on federated servers and still interact with everything on lemmy.world without the errors, then come back to our lemmy.world accounts when everything is smoother

Yeah, when I understood this I joined another small local instance, it does improve performance by a lot!

I totally appreciate all of your efforts! Thank you for being a pioneer.

Thanks for the transparency, and communication. I think it's always better when the userbase is able to understand what is going on rather than being left in the dark wondering what is going wrong. Keep up the good work, but also be sure you guys get enough rest, and take care of yourselves too.

I expect it is federation outbound activity growing in memory usage, it's all held in RAM for every post/comment/like going out to subscribed servers. !lemmyperformance@lemmy.ml is a community for the focus on scaling Lemmy.

Lemmy also buffers failed posts/votes in memory, such as if there's server issues like Lemmy.world and Lemmy.ml are having, to retry later.

Couple that with the server having to deal with an exponentially increasing amount of servers to federate with (since our Federation model currently links every server with every other server in a worst-case scenario), and that's probably doing bad things to the performance.

Thank you to you and everyone behind the scenes working hard to make Lemmy a great place. I am turkey enjoying the experience even with the bugs. Brings back a ton of nostalgia with modern mixed in.

I love the community here and am excited to see it grow. Thanks for giving a report, it awesome how invested y'all are and makes me want to stick with this instance even through the hardships. Keep up the good stuff!

around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

25 GB of RAM????? Looks like your instance is the edge case with Lemmy's scalability, when there's really big amount of active users. Looks like some problems won't fade with really high-grade hardware. But I have to appreciate, that's one gargantuan job!

I think that’s a premature conclusion. Sounds like a memory leak of some sort.

Fixed in software.

I’ve had instances with only a dozen or so Wordpress sites that will chew through 8gb and go into swap when certain tasks were poorly scheduled (like a couple of sites clearing their cache at the same time).

So I’m not surprised they are hitting 25gb but the gradual nature (from the sounds of it) off the memory build up does sound like a leak.

Don’t envy the devs but I’m sure some of them are enjoying it 😂

This has been such a refreshing change from reddit! I'm willing to deal with occasional hiccups, and I know it will only get better with time. Keep it up!

Amazing work team. I am already seeing improvements. Hope you are not killing yourselves though, I'm sure everyone realises how difficult it is and that it will take time to fix. We're here for the long haul! Thanks again

Yall are navigating waters that you probably have little professional experience with. Keep at it and don't get discouraged.

Its nice to have a place where the admins and developers aren't trying to scheme and manipulate the community to get a big payout. Thanks for all that you guys do. Curating this community is no doubt selfless hard work but we need a place like this free from the influence or potential take over by bad actors and malignant narcissists.

I can come whine in the discord if that'll help.

To all the folks that are worried, don't be. Let me tell you, Mastodon was a wreck when Musk took over Twitter and that all got sorted out within a week or so. The mods and sysadmin are obviously working hard to get things up and running, but growing pains are growing pains. To paraphrase an old adage: "Facebook wasn't built in a day." In the beginning, Zuck and Co literally limited signups to only people with college email accounts and only added universities a few domains at a time... scaling is very difficult, but it's not impossible. The way things are going, Lemmy is going to thrive!

Thanks for your work, you are making the Internet a better place!

If you need any Python QA test automation help I’m happy to invest. Thanks for the new home.

Thanks for putting in all the hard work. Especially on a holiday. We appreciate you guys!

It’s not a holiday here… just a normal work day

Encouraging to hear about how much help you're getting. A happy lemmyverse is a happy fediverse?

Sometimes when I post a comment (without any image) it takes forever to post and sometimes it keeps spinning and I need to cancel and post again. My internet connection is fine (100 Mbps) and it works flawlessly on other websites. Is it happening due to overloading on the lemmy server?

I wonder if it's possible you're getting unlucky and are posting when the server is being rebooted

I can confirm it blackholes on text only posts more frequently than 30 minutes, so this could be part of the problem it isn't the complete problem.

I've had the same issue. A poster in another thread suggested selecting "English" in the Select Language drop-down menu before hitting the Reply button. It may be a coincidence, but that has worked for me to prevent the eternal spinner.

That DM sending bug was already fixed in a later rc version. Although rc.9 and I think 10 have a weird bug where you can't sign out. Fun stuff.

🎵99 little bugs on the wall🎶

Do you need any spare server hardware? @ruud@lemmy.world I would be happy to donate some!

No thanks, it's all at Hetzner, and thanks to all donations we can extend when needed!

Thank you for everything you do. With any luck, you'll get all the support you need, and have it running smooth as silk.

Thanks a LOT for your work. It's amazing seeing what people can do when they work together for a good cause.

Are there any plans to make user up and down votes not viewable publicly? I know for a lot of new adopters this can be a deal breaker.

If someone cares about their privacy that badly why vote at all? I'm sure votes are tracked by the sites on other websites?

On reddit only reddit knows your up and down votes which are never made public unless you check an option in settings. The fediverse is already a target for brigading due to the decentralized nature, allowing bad actors to figure out who to target seems like a terrible idea. Imagine what people like /r/againsthatesubreddits would do with that info.

On the flip side, it also makes brigading easier to detect, so instances can defederate communities that routinely vote in bad faith.

As a former programmer, I appreciate the hard work you folks are putting in so the rest of us can putter away our free time. You're doing great. Thank you.

What sort of container orchestration and database tools are you using? Maybe you need some sys admin help?

Can we configure the website to not automatically collapse deep replies? Or at least to collapse them only past six or seven layers deep?

All these updates are really much appreciated. I would love to see the conversations happening in this discord but I understand that you would want to limit that to only a small group: do you have someone making sure this is archived somewhere, and available in a few years? That would be a nice read then.

Again, thanks for all your hard work, and your transparency about the whole ordeal. Hopefully, the userbase in general is being patient and understanding given the situation.

Thank you for everything! Can we donate to cover the costs? If more people throw in $5, we all will benefit. Can Lemmy server scale up without needing RAM exponentially growing with the user number? I hope the system will get better optimized for bigger user base as the time goes on..

Keep up the good work! Things have been really snappy the past few hours 😎

Thanks for the hard work and transparency. It gives confidence that you guys will get through this.

It is hard not to root for your success in this. I'm temporarily off .world, but would love to come back when you guys get this under control more.

Y’all doing amazing things keeping us going 🔥

Keep up the great work! We believe in you! Just a reminder to everyone to help sustain lemmy.world by donating in the sidebar! If 1000 people all donated $2/month then that's 24K a year to keep the hardware running! Reddit is what happens when you rely on Corporate Overlords to provide the services you love.

I switched to a regional instance to take the load off for everyone, Lemmy is even better!

More props for you and the other admins as you ride the waves. It’s for a great cause.

Thank you. Y'all are doing great.

I'm sure it's stressful but I know I'm here to stay. It's been a blast so far

Thank you for everything you’ve done and continue to do!

Is there anything that people can do to help if they’re unable assist in troubleshooting?

Get me coffee. I prefer cappuccino.

I’ll bring you a couple. I don’t think one is gonna cut it for as busy as you’ve been the past few days.

I recall that previously we couldn't update jerboa due to it not being compatible with the last server version. Is that updated now, can we update Jerboa?

Yes, Jerboa now works with lemmy.world

I installed Jerboa, but I can't login with my account on there.

I am using Jerboa right now and I installed just a couple days ago. Must've been silently fixed.

lemmy.world is on 0.18.1 since a couple days ago, that was the problem.

0.17->0.18 changed some APIs, and lemmy.world was sticking with 0.17 for a while due to broken stuff in 0.18, which broke Jerboa which just expects 0.18.

Huge heartfelt thank you to everyone involved - given the huge growth you're doing awesome!

It would be great to see plans on how to grow in a controlled and safe way and plans for improving the performance. Really hope you have the funds needed and continue to have the income to support everything needed.

Thank you again, it's wonderful to get an update!

Using the connect for Lemmy app produces constant Under Maintenance or html code or 504 errors. It seems to load better on Chrome.

Keep up the good work @Ruud and you all wonderful admins, mods and lemmings for making this possible! 💙💚

I appreciate the transparency. Keep it up! Everyone is loving it here (⁠◍⁠•⁠ᴗ⁠•⁠◍⁠)⁠❤!

The biggest frustration is 502 errors on submit. Sometimes th8ngs go through, sometimes not

My workaround is to submit/browse from federated servers until lemmy.world is running well

Appreciate the clarity and hope things keep moving in the right direction

Thanks for all of the updates and work on this instance! As a fellow sys admin, I sympathize with the issues trying to get this update going.

Thanks for working so hard on this and keeping lemmy going. This instance seems to have become synonymous with lemmy as a whole for some of the new comers so keeping it up and running is really helping to keep the new comers on board, this can only be good for all of lemmy in the long run!

Really appreciate the transparency about what's going on behind the scenes too. It's nice to know what my donation is doing and I hope you have time for cappuccinos.

Thanks! Love updates. I moved to lemm.ee for now, I'll come back to your world when you got room for my fat arse.

Can’t go wrong with either, both admins are very communicative.

Keep up the good work. It's nice to feel like we are back on the bleeding edge.

It’s really good to know that there are more people helping sorting everything out.

I’d just like to suggest publishing the workarounds and fixes public so that other small instances can get it sorted faster when they grow.

Don't know about anyone else but I keep getting a "cannot load comments"error and the whole post goes blank.

Still happening? What browser/app and OS?

Glad to see these progress updates. Thanks for the transparency.

Thanks for all your efforts.

I'm sorry to hear you're using Discord, for me it fits right in with the likes of Reddit, Twitter and Facebook. Why not use something independant and federated like Matrix or XMPP?

I think I use all chat software there is. I'm in hundreds of Matrix rooms. But I think one of the team at least didn't like or use Matrix. Don't remember. And I have Discord anyway for the Mastodon channels..

I really appreciate the updates. As someone in the tech field, it's been fascinating getting the inside scoop on how it's all coming together.

Edit: I made another account off of your instance to ease up on the pressure, but so many communities and users I care about are on your instance, so I'm using it too!

Yup, I'm having the 2FA problem. I can't log into mobile, because no code gets generated. Luckily I'll just stay logged into a browser for now.

But thanks for all the hard work! :) You're probably getting a flux of Reddit migrants, such as myself now

Thanks for the update, glad you guys are being transparent about the issues everyone seems to be facing. Looking forward to the coming improvements. 😁

Hi, am I correct that typing on iPad currently doesn’t work? As in the keypad won’t show up. I have to type in google or notepad and paste (the paste popup still works) to lemmy.world lol! Anyone else?

As always, thanks for the hard work everybody puts into this!

Had that happen earlier this week. Update your Lemmy client, then update your iPad, then restart your iPad. That fixed it for me with WefWef.

I hope hosting all these massive communities is enough a portfolio to land Ruud a crazy job offer one day that pays him stupid levels of money while allowing him plenty of time off to maintain these massive hobby projects.

You lot are internet heroes. Thank you for putting in the work and creating this community!

Thanks for the update!

So seems that Reddit refugees influx indeed hit this instance.

Thanks again for the hard work that you and the others are putting in Ruud. It can't be easy trying to patch the boat while the boat keeps growing ridiculously fast at the same time. You are very much appreciated!

Thanks for your hard work. Noticing janky bits getting ironed out everyday. Has anyone had any luck making Lemmy.world work in Qutebrowser? It just hangs on loading comments/posts..Any work arounds? I'd love that..working quite well in Firefox and mobile though.

I think most of the users can understand the current situation. Take care guys and girls!

Thanks a lot for addressing these issues. It's really nice to see how this platform keeps growing and growing.

Thanks for the update and keep up the great work! :)

Appreciate all the work. I’ve only been on Lemmy a few days but the openness is refreshing and I think this place has been pretty cool.

Curious about the server info. Is there more info available somewhere else?

Is lemmy.world running on a single server? What's the CPU and memory capacity? You mentioned a percentage of CPU usage and an amount for ram, so that made me curious about what's available for each.

Does restarting take down all services or is the application clustered or not monolithic somehow?

It's a single server with 32core/64 thread AMD EPYC and 128GB RAM. At the moment we run multiple containers for lemmy so restarting doesn't mean outage.

Out of interest what's the network 95 percentile like over a day? Keep up the great work :-). Thanks for your hard work.

Are you running the database on the same server as the backend?

Thanks u/ruud!

How are you managing all of this in addition to having a life outside of Lemmy? This kind of went from 0-60 in a blink.

Thanks for all your hard work. Wow that is a lot of CPU usage. I hope it can be tuned more. Is it mostly postgres causing the load?

As much as I hope that Lemmy grows, I wouldn't want lemmy.world to suddenly get 1 million signups and multiply your headaches.

Perhaps we need more well run servers by other people to take the load.