00Lemming

PSA from Admin Team: The update completed roughly two hours ago. Since that time, the Admin team (and other site admins) have been working on the noted performance issues. We believe we have found a solution, but we still need time to test this out. You may still see brief outages and differences in performance as we are testing different configurations. We are trying to prevent rolling back.

While I know this can be frustrating - especially today - please keep in mind we have a team of volunteer techies (from around the globe!) collaborating on this issue. It is an inspiring situation. Also keep in mind that lemmy.world is quite a bit larger (and more active than any other instance). As such, we are a bit of a ‘test instance’ in regards to high volume requests. This is just part of the growing pain. We appreciate your understanding.

@ruud@lemmy.world will provide a debrief once we have completed testing.

This may be the best highlight of spez incompetence 🍿 For everyone talking smack on the moderators, this is exactly what they should be doing. Malicious compliance. Make it hard for Reddit to know what is going on. Love to see it.

Thanks for posting this over here. I otherwise wouldn't have known this was going on. As I used a mostly lurker account for game day threads I wasn't going to bother with a data request. It's a whole different story if we can inundate them though. Request submitted. o7

Great work, team. Happy Halloween! 👻

Technically speaking, yes, a portion of our issues are due to the highest user base of an Lemmy instance. So in theory, if half of our users dispersed to other instances, we would likely see some performance improvement here. However, lemmy.world is intended to be an accessible instance for the general population. The server itself that is running lemmy.world is beyond spec'd to handle much more than this user load. We are running up against code-level issues that we may or may not be able to get around with our internal configurations. This is just part of developing software in an environment were you go from a few thousand users total to hundreds of thousands in the space of a few weeks. There is no directive to have users create accounts on new instances, though if you are looking for an immediate performance improvement, that may be your best option currently. That is up to you to decide :)

Yeah this is accurate. We wanted to get away from websockets ASAP while also maintaining Captcha functionality.

Thanks for the kind words! Yeah, there are definite growing pains, and likely will be for some time (just do to the codebase we are working with, understandably). We have a really solid group though heading up lemmy.world though so we will be just fine ;)

The login form should be related to the overall spiking we were seeing, though I can't say we had conclusive evidence of that. I have been able to get it to load properly with a few refreshes. Maybe try clearing your cache? I had to do that several times during testing.

Amen! This movement is far more about he actual implications of federation, not that it isn't there yet. If we have any chance to live in a world not controlled by government/corporate greed, then we have to start trusting in one another and helping to support an internet that we all control and can take part in.

One of the best things about open source software - there is no need to hide useful information from your user base. A concept largely forgotten in today corporate hemisphere.

As you mentioned, I think the mods have done a fantastic job given the learning curve and lack of mod tools. This is just a good tool for accountability and transparency. Keep it up!

🙌 Great work team!!!

Yup you nailed it. For additional context, Ruud is running an almost identical server for his Mastadon.world server which has 160k users. Relatively speaking, these are large, performant, and expensive servers. They can absolutely handle the current user influx we are getting from the Reddit exodus. Are hands are tied by software limitations unfortunately. I can confidently tell you were are constantly in communication about ways we can amplify user experience with the tools that we do have access to. For instance, this status page was recently spun up which you can access anytime you think there might be server issues to help confirm that what you are seeing is recognized at a server level. Things like that.

All that being said, for users who are looking for a smoother experience right now, I can recommend lemm.ee as a solid home as well. Their admin Sunaurus has been very active and helpful throughout this process and handles his instance very professionally. He is essentially another Ruud (though Ruud is the best! ;)). Just something to keep in mind going forward as I can't make any promises about the time frames for these issues being resolved. Hopefully once we get contact back from the Lemmy devs we can start expediting a resolution. They have a lot on their plates right now though, haha, so we will see. Cheers!

Bravo to you. This was absolutely the right call and I hope you let it rot if there isn't a massive change of direction. Frankly, for me, even if they do roll things back I am still done with them. P2P is how the internet should be (funny considering torrenting has been a thing for how long now?) and thankfully we have guys like you to step up and help get this off the ground. Thanks!

Ruud is the MVP of the Fediverse 🥳

stares intensely

This is being reviewed by the Admin team. Are you logging in via browser or app?

Honestly, while I could care less about karma, this is likely just due to the infancy of the software. You had 2 people working on this for ~3 years, and in three days their user base increased 10x. It’s not like they were given time to prepare (thanks spez). These things will be worked out, we just have to be patient. It is worth the wait for federation.

Frankly one of my favorite parts about the blackout - and opinions like this - is that Lemmy is going to end up being populated by people who have the capacity to think about others and form intelligent opinions. All of the people with this attitude will stay on reddit, which is what will ultimately kill it. I hope he stays.

Godspeed Christian. Personally, I am very much looking forward to the impact you will have on the fediverse. There's not many whom I would trust more with it.

That is good to hear :) we continue to analyze the config to see where this failure is potentially happening.