Lemmy.world status update 2023-07-05

Ruud@lemmy.worldmod to Lemmy.World Announcements@lemmy.world – 2714 points –

Another day, another update.

More troubleshooting was done today. What did we do:

  • Yesterday evening @phiresky@phiresky@lemmy.world did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
  • @cetra3@lemmy.ml created a docker image containing 3PR's: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
  • We started using this image, and saw a big drop in CPU usage and disk load.
  • We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws.
  • We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
  • We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set proxy_next_upstream timeout; max_fails=5 in nginx.

Currently we're running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the proxy_next_upstream timeout; max_fails=5 workaround but for now it seems to hold with 1.

Thanks to @phiresky@lemmy.world , @cetra3@lemmy.ml , @stanford@discuss.as200950.com, @db0@lemmy.dbzer0.com , @jelloeater85@lemmy.world , @TragicNotCute@lemmy.world for their help!

And not to forget, thanks to @nutomic@lemmy.ml and @dessalines@lemmy.ml for their continuing hard work on Lemmy!

And thank you all for your patience, we'll keep working on it!

Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.

Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that's now started, and I noticed the proxy_next_upstream timeout setting didn't work (or I didn't set it properly) so I used max_fails=5 for each upstream, that does actually work.

370

server load is too low, everyone upvote more stuff so i can optimize more

edit: guess there is some more work to be done ๐Ÿ˜

I don't understand your graph. It says you are measuring gigabit/sec but shouldn't the true performance rating be gigabeans/sec for a Lemmy instance?

And where's the statistics for days between each core dump? A healthy instance should have at least three days between each one

1 more...
2 more...

Double the image upload size and you will see more shitposts

I was gonna argue that you'd see more bean posts, but at this point they're the same thing, both in the pun sense and the literal sense

Web-ui is very smooth rn.. is this .world?
๐Ÿ˜…
Joke aside, the improvement is like heaven and earth. Love it!. Good work teams!

I'm on another instance, but here's some federated activity for you.

I was just going to post a meme about choosing either creating activity or spare the server from overloading. Now the joke won't stick.

4 more...

Test:

Upvote if you can see this comment. ๐Ÿ‘

Looking good from here.

Edit: And comment rapidly going through. :)

Shame you're getting no karma. Take my upvote.

I think i was on r for a year or two before I learned what karma was. I still don't understand it's value.

2 more...

The change is noticeable. Good job guys.

Thanks for the updates.

I agree. Felt it immediately when I started browsing. Everything is faster and more responsive, on top of the error messages disappearing

Yup I can even post comments first try, without getting an error! Things are working well!

Really noticeable. Cool update. Thank you, guys! โค๏ธ

This is why having a big popular instance isn't all bad. It helps detect and fix the scaling problems and inefficiencies for all the other 1000s of instances out there!

This, if everyone kept just spreading out to smaller instances as suggested in the beginning, while still a sensible thing to do, no one would have noticed these performance issues. We need to think a few years out, assuming Lemmy succeeds and Reddit dies, and expect that "small instance" will mean 50k users.

I sincerely doubt reddit will die anytime soon, it'll just exist as its own thing that it's new target audience gets bored with and moves on from in a few years when something new and flashy catches their eye in the app store. Just like they do all the other apps designed in exactly the same fashion that reddit is currently morphing into.

Meanwhile Lemmy will be slowly building it's communities up to be what reddit used to be.

I'm actually kinda waiting a few releases to start promoting my instance anywhere, letting some other brave instance admins work the kinks out a bit first.

Agreed. I decided to keep my community on lemmy.world specifically because of the community investment I see being put into it.

If this project is to stay for the long haul, we gotta load test it and stabilize it. These folks are doing the important work here. Large instances are more or less inevitable if Lemmy sticks.

I love the smell of updates in the morning.

Thank you guys for your awesome work!

Also to other people: DONATE TO FOSS PROJECTS. If 50.000 people donate only 0.5โ‚ฌ, we have 25.000โ‚ฌ for funding the servers, coding, motivating/ people etc. Just don't take a cup of coffee for 1 day. We are already 2 millions in Lemmy instances. We can build a decentralized world together!!

You can pry my cup of coffee from my my cold, dead hands.

Will donate anyway, I really want this project to keep going.

you inspired me to serve the greater good!

For example, if you speak a second language, you can even help with translation in projects. Its very easy. E.g. I translated the Jerboa (Lemmy client for Android) in Greek 2-3 days ago. I needed only 1 hour to finish and special 15-20 minutes for fixes that I missed, yesterday.

Is there a good link? One for USD?

If there's a thing you really need to pay in a foreign currency, look into Revolut or Wise. Since I occasionally have to pay stuff in Turkish lira, GBP and donated to USD-only Receivers I like to keep Revolut as my secondary bank account since exchanging one currency to another is completely free!

Boy does it feel good to have those reports and understand the work you guys do. It's really inspiring! Thanks for your hard work, everything has been silk smooth! This instance is really great, Lemmy and its devs are really amazing and I feel at home in a nice, cozy community.

Submitting PRs is literally the most effective response that helps everyone who uses Lemmy. Thanks to you all.

So that's why it was so smooth today... Great work!

upvoting posts is so much more stable now, we might actually see more bean posts as a result

Upvotes are still getting rejected. Replies hang so I cancel out and it turns out they did post.

That said, browsing is pretty snappy and smooth. I know the kinks will get worked out eventually. Thanks for the update.

Edit: This now appears resolved minutes later. All smooth on my end.

Hmm. Seems to work for me.. (Yes this is a test reply)

Everyone itโ€™s a test reply, deploy the upvotes

I'm going to take the opportunity to test a reply as well.

It's a good thing the points mean even less on Lemmy.

Iโ€™ve since successfully upvoted some comments and made replies without it hanging with the spinning circle. Not sure what the issue was but it all seems to be running smoothly now. Thanks.

I was getting upvote errors earlier, too, but now it seems smoothed out.

Iโ€™ve also had errors posting but the reply still goes through.

This is why I love open source. The fact that a community can directly debug the code that's it's being hosted on and directly contribute the improvements back is just wild. Thanks for all the hard work @ruud@lemmy.world and the rest of the lemmy.world team! The site already feels much more responsive.

This is better optimization than most enterprise devs will see in their lifetimes.

Some managers of the devs are not that interested in significant optimizations... Depends on what incentives and company culture drives them

Some company would rather throw more hardware at the problem and make the devs work on another useless feature no one use

Yes! And that's a very short seighted solution. And it feels so good to improve performance in code! That's extra performance for "free" ๐Ÿ˜Ž

Am I getting this correct: the whole lemmy.world instance run in one single container on one single host?

You'd be surprised at how much performance this kind of setup can squeeze off. Often the limitation is more on the DB/storage than network handling and processing power.

This. Most of the time, the bottleneck will be the database backend.

Curious if lemmy.world uses separate reader/writer instances.

if it runs this well on a single container, considering the amount of users it has, I fear the power it'd hold with more

Lemmy is built on an async work stealing runtime and you can get very large instances from Amazon and the like.

I'm sure the instance isn't massive but number of containers and a single "host" are not great indicators of efficiency.

The server is absofuckinglutely flying today! It feels smooth and bug free!!! You guys are legends.

As a data engineer, I'd be interested in hearing more about the SQL troubleshooting.

EDIT: It looks like !lemmyperformance@lemmy.ml is a good place to subscribe to for more technical info on some of these performance improvements.

Also the Lemmy GitHub of course contains more information on bugs/enhancements/etc.

Same, my job is like 80% SQL, so it'd be cool to see what is used in the background and maybe help improve things.

2 more...

It now feels pretty good to browse and it now makes the experience of using Lemmy much more enjoyable. Having to spam the vote buttons was really annoying.

Appreciate that these updates use the yyyy-mm-dd format :D

Hell yeah. I use YYYY.MM.DD_HH:MM:SS in my filesystem, for screenshots etc, so alphabetical is chronological. :)

Literally a night and day difference in performance and stability! Thank you all for the hard work. To other users like me, consider reducing or replacing one of your lesser used subscriptions and directing that money to Lemmy. Itโ€™s much better served here if you ask me.

Thanks to all involved across the board. Great work all around ๐Ÿ‘๐Ÿ‘

Even though i'm not from this instance, this is such a nice way of keeping the users posted about changes. I wish more companies (I know this is not a company) went straight to the point, instead of using vague terms like "improved stability, fixed few issues with an update" when things are changed. I hope all instance owners follow this trend.

The owner of your instance has been a big help. You've also chosen a good instance!

@sunaurus@lemm.ee is awesome. He keep us aware of what's happening, planned maintenance hours, etc. His commits on making lemmy scale horizontally is what kept lemm.ee snappy even when we had a huge influx of users. I hope Lemmy as a whole continues this ethos of collaboration.

As an IT support person, I've learnt to dumb things down for management. They don't want to hear stuff like "Increased the SGA, changed the buffer size and added a function based index...etc". Sometimes I'll do a short and a long version something like "the issue was around memory settings which have been increased", plus the detailed info.

Huge props to everyone working on the project. It's awesome seeing everyone work together and resolving issues so quickly!

Can we have an update on the status of Lemmy.world and how close ties we are going to have with Meta's threads? Threads is going to support ActivityPub, but time has shown that this is an attempt to try to kill this open platform and eventually replace it with theirs once they get everyone in their ecosystem. (Embrace, Extend...extinguish) Mastodon has said today that they don't mind sleeping with vipers when their demise / dissolution is in Meta's best interest.

Please tell me we are defederating from Meta....or let us know what to expect

EDIT: I originally stated that Mastodon told them to fuck off, but I got confused with Fosstodon (who did that). Mastodon doesn't mind being in bed with Meta

I don't support defederation in general but damn. That's just cancerous.

You may be right and as sad as it is...Meta wanting to work with the federated network should be seen as two things... an attack on the ActivityPub standard via infiltration...and that Meta sees the Fediverse as a threat to its own closed ecosystem.

We need to be on the defense and protect our platform and standard from corporate meddling and fuckery which Meta will absolutely not hesitate to do.

I think we as a community should take this very seriously. It's going to be more than a matter of moderation/defederation. Soon enough majority of the Fediverse platform will be populated with Meta users and they'll be the one who drive the whole platform.

I'm very concerned about this approach meta is taking. They are creme de la creme of cancer like companies. We need some kind of class action or else we'll lose this community in a matter of few months.

Yeah, even if meta is acting in good faith, looking at what it's like here now vs Reddit last week vs Reddit a decade ago has me thinking that I don't really want to be part of a site with such a large community. It's hard to put into words exactly what it is about this place that is nicer than Reddit, maybe it's the difference between the kind of person that left and the kind of person that stayed, maybe this place just isn't popular enough to be targetted by a certain kind of shittiness.

Whatever it is, I'm worried that meta will bring a bunch of users that have that thing that made fb and Reddit less good over time.

And then there's also the angle of anyone can create an instance and suddenly gains a lot of visibility into user activity. Especially a company like meta, though any company could quietly just start sucking up that data. Maybe the fediverse needs different levels of trust between nodes instead of an all or nothing approach. Though tbh I'm still not very familiar with the implementation details or how much sense that even makes for it.

Where have you seen Mastodon formally state they have no interest in working with them?

I'm genuinely asking because I'm relatively new to Mastodon and Lemmy and want to be as informed as possible with this whole Meta situation. And just to be transparent, I'm not a Meta fan at all, to the extent I've never had an account with any of their products

I did read this official Mastodon blog post today...

https://blog.joinmastodon.org/2023/07/what-to-know-about-threads/

and my question to you is because I'm not seeing as quite aggressive a stance as you mentioned.

1 more...

Thanks for the detailed update and all the hard work you guys are doing!

I don't understand anything other than you worked diligently to make things smoother. Thanks to everyone for their wonderful work!

Same! My first thought was โ€œthatโ€™s an impressing looking graph. I have absolutely no idea what it means.โ€ The proof is in the pudding though - lemmy.world is much improved!

The site is running so much better now, thanks to all. BTW: Love these updates!

Lemmy's devs and the .world admins have done in a month what Reddit hasn't done in it's whole existence: having a smooth and almost bug-free experience.

Jerboa feels so damn FRESH to use now!

Not to undervalue the efforts going into this, because I appreciate the new community and the transparency, but I believe we have wildly different definitions of 'almost bug free'

Which, is also something to consider about user experience consistency. Will be a challenge with growth. Fortunately, plugged in admins and devs will help.

By almost bug free I was speaking only about my own experience. There are probably loads of things under the hood I'm not noticing, but it's been hours since I last noticed any issue.

I agree that we've still got a long way to go though: both Lemmy and Jerboa are far from their 1.0 release yet.

You guys had better quit it with all this amazing transparency or it's going to completely ruin every other service for me. Seriously though amazing work and amazing communication.

Is it safe to use 2FA yet?

It doesn't really work I think. Havent tested yet.

It's always been safe to use 2FA if your authenticator app supports SHA256. Unfortunately, it turns out that a lot don't. The only solutions are going to be Lemmy switching to SHA1 or users switching to auth apps that support SHA256. I think the first is more likely to happen than the second.

2 more...
2 more...

I just love the transparancy you guys are coming forward with. It's absolutely awesome! Thank you for that and for all the work you put in. It means a lot to me that you folks are taking the time to keep us updated. Much love!

It blows my mind with the amount of traffic you guys must be getting that you are only running one container and not running in a k8s cluster with multiple pods (or similar container orchestration system)

Edit: misread that a second was coming up, but still crazy that this doesnโ€™t take some multi node cluster with multiple pods. Fucking awesome

4 more...

I'm not sure wtf you just said, but lemmy.world feels very smooth today, so thank you for your continued hard work!

Whilst I'm aware that too many users on one instance can be a bad thing for the wider Fediverse, I think it is a great thing at the moment in terms of how well people are banding together to fix the issues being encountered from such a surge in users.

The issues being found on lemmy.world results in better lemmy instances for everyone and improves the whole Fediverse of lemmy instances.

I'm very impressed with how well things are being debugged under pressure, well done to all those involved ๐Ÿ‘

I agree, it's great to see Lemmy being battle-tested in large instance scenarios.

Is it weird that Iโ€™m always excited to read the update posts?

Not at all. The admins here are doing great work and their updates are often informative and helpful, it makes sense you'd look forward to them.

Awesome work - things seem to be running much more smoothly today.

Do you have anything behind CDN by chance? Looking at the lemmy.world IPs, the server appears to be hosted in Europe and web traffic goes directly there? IPv4 apparently seems to be resolving to a Finland-based address, and IPv6 apparently seems to be resolving to a Germany-based address.

If you put the site behind a CDN, it should significantly reduce your bandwidth requirements and greatly drop the number of requests that need to hit the origin server. CDNs would also make content load faster for people in other parts of the world. I'm in New Zealand, for example, and I'm seeing 300-350 ms latency to lemmy.world currently. If static content such as images could be served via CDN, that would make for a much snappier browsing experience.

1 more...

Things have been super smooth lately, thanks for all the work!

Yeah this morning everything has loaded so much quicker, Iโ€™ve been able to post and vote on comments no problem! Lemmy is really starting to take form. I fucking love this whole thing.

It felt like Iโ€™d jinx us all if I commented but THANK YOU! This has been a wonderful experience today. Absolutely loving it and knew you just needed some time to work out the kinks that happen with fast growth.

Shouldn't the correct HTTP status code for a removed API be 410? 404 indicates the domain wasn't found or doesn't exist, 410 indicates a resource being removed

Or 418 for the wrong API being used :^)

Gadzooks! These are huge fixes. Compliments to the team, you guys pulled off a small miracle today.

Awesome work. Any way other devs can contribute?

1 more...

I took a SM break for a few days, and it's running noticeably better today...I think. (:

Thanks a bunch for floating us degenerates.

The transparency here is awesome, thanks for the update and all the hard work; it's running so fast and smoothly this morning.

I like that the post goes in detail and allows us tech nerds to get hard watching this stuff instead of the regular corpo jumbo change log that consists of:

  1. we uhh fixed some stuff so yeah good?

Really great job, guys! I know from my experience in SRE that these types of debugs, monitoring and fixes can be much pain, so you have all my appreciation. I'm even determined to donate on Patreon if it's available

Donation links are on the frontpage. Thanks!

I love reading these! Thanks for all the work

I hope to start on some small contributions sometime next week. Stability has been noticeably better the last few days and I imagine itโ€™s only going to get better.

Thanks for the updates! Seeing the details of how you work through these early issues is valuable to those of us thinking of starting an instance.

Love the transparency. Thanks to the entire team!

Compared to days prior, things are running much better today. Page load speed, reliable post/reply/upvoting. I dunno if it's just happenstance but whatever knobs, levers and keystrokes you're manipulating, keep going the things! Thanks so much for a home absent of corporate BS.

Installed Jerboa again and it feels smoother than Reddit itself, great job!

It reminds me somehow thw old Google+ in the mid 2010s

Sweet. Yeah it is definitely snappier today. Feels great.

Great job. Everything seems to be working smoothly for me now. The past several days have been a bit rough but now it's all working.

Definitely noticeable!

Also, the amount of communication here is AWESOME! Thank you for keeping us updated.

That's so awesome! Look at that GRAPH!

I'd volunteer to be a technical troubleshooter - very familiar with docker/javascript/SQL, not super familiar with rust - but I'm sure yall also have an abundance of nerds to lend a hand.

1 more...

Thanks for the update and thanks for the amazing collaboration youโ€™re fostering!

I feel like Lemmy.World is leading the way for other instances to follow, and thatโ€™s a good thing to do. Even if itโ€™s painful for everyone involved.

How great is it to be a part of history in the making -

This is Web 3 in its fomenting -

Headlines ~5yrs:

The ending of Web 2 was unceremonious and just ugly. u/spez and moron@musk watched as their social media networks signaled the end of Web 2 and slowly dissolved. Blu birdโ€™s value disintegrated and Redditโ€™s hopes for IPO did likewise. Twitter and Reddit dissolved into odorous flatulence as centralization fell apart to the worldโ€™s benefit. Decentralized/federated social media such as Mastodon and Lemmy made their convoluted progress and led Web 3โ€™s development and growthโ€ฆ

This is how history is made, itโ€™s ugly and convoluted but comes out sweeetโ€ฆ

Things have been very noticably improving every day since the big update! I've had very, very few problems today browsing on Jerboa.

Responsiveness is significantly better than it has been at any point since Friday's Reddit exodus.

Well done...

I agree that everything is a lot more stable!

I'm very curious: does single Lemmy instance have the ability to horizontally scale to multiple machines? You can only get so big of a machine. You did mention a second container, so that would suggest that the Lemmy software is able to do so, but I'm curious if I'm reading that right.

A single instance, no. You run multiple instances on multiple machines, then put a frontend (nginx in this case) to distribute the traffic among them.

5 more...
5 more...

Was just noticing how much smoother it is this morning. Great work!

ROFL Back in the day, when AOL was a thing - I used to think it stood for 'Assholes Online'.

Good luck (",) appreciate the efforts you make fighting the crazy deluge.

@ruud@lemmy.world is this docker container y'all are using available on a registry? We'd like to use it. And do you have a load balancer in front of your lemmy-ui image to allow two containers to run? or is that built in and I just never noticed it?

Well we use the cetra3/lemmy:the-phiresky-cut image but you can also wait for the next RC or release which will have the PRs. We load balance with nginx which works if you use max_fails=5 for each upstream.

It's really fun to see these updates, and get insight into what you're putting into this instance. It's real obvious that it's a labor of love.

The responsiveness of the website has felt drastically better today.

Great job guys! It really feels more responsive today

Ever since I joined Lemmy I had problems posting replies but that kink seems to have been ironed out. Thanks!

It's so smooth now; the speed difference is insane! You all are doing excellent work!

Nothing feels more exciting than constant update, communication and the potential of lemmy!

Wow I applied these PRs on my server as well, running waaay lighter now. And it seems the federation misses have cleared up! Bravo Lemmy.world team!

I'll be honest, I think I understood three words of your OP. Sounds like the general gist though is Woo so great ๐Ÿ˜ƒ

Thanks for all your efforts, glad someone knows what they're doing

Thank you guys, you are doing an amazing job. These are great times for the internet and you are a huge part of it!

As a Performance Engineer myself, these are the kind of performance improvements I like to see. Those graphs look wonderful. Nice job to all.

Thank you to everyone that helped.

You guys are absolutely amazing. So many thanks to you @Ruud and the entire admin/troubleshooting team! Thank you.

Was waiting for this update. Woke up a couple hours ago and Lemmy was working great so I knew yโ€™all put in some time. Thanks for your efforts!

Thank you for all your great work, and I really appreciate and enjoy the updates about the gritty details!

I'll be honest I don't know what any of this means but what I can say is I absolutely love the transparency of all of this. It's so refreshing and maybe I'll start learning more about what I'm looking at because I'll keep seeing it. Great work!

Good job on troubleshooting!

Have you looked into possibly migrating to kubernetes or some other form of docker container management/orchestration system to help with automatic scaling and load balancing?

Don't quote me, but I recall reading on GitHub that there are a few things to be refactored before Lemmy can support horizontal scaling approaches.

Itโ€™s very much helping the third party apps as well. Memmy is running way smoother now

You all are champions today. When I loaded this up today, I was very happy and impressed. I knew tweaks were made.

Things are looking a lot more stable, thank you (all) for the amazing work!

Fantastic job in troubleshooting and submitting PRs! Improvements are very noticeable already :)

Things feel a ton better today. Thank you for all the hard work!

Awesome update, great to see and feel the progress from all the hard work!

The instance seems to be much better. Posting and commenting is not taking as long and loading times are way better. I hope things can stay this good or even get better.

Thanks for your efforts and improvements. Keep it going โ™ฅ๏ธ๐Ÿ™

Memmy is great! Been using it since about 2 weeks now. Really feel at home.

Can confirm, having way less problem browsing lemmy right now. Thank you admins!

Upvotes appear to be working but I still can't post images. The post displays the infinite spinning loading circle.

How large is the image (both in dimensions and filesize) ? What image type is it? Is it in a post, comment?

209KB, 720x891px, jpg. It was a post.

Do you see the separate line for uploading the image? If not, can you refresh the browser?

Man I thought I noticed something different. For the past week or so I've gotten nothing but network error and Java errors in Jerboa which are completely gone now. Posts load almost instantly too. Appreciate the effort guys, was going insane.

damn bro, yโ€™all coming in clutch to improve stability of this lemmy instance.

Good shit bros. Hope to contribute upstream and find more performance related bugs. I browsed the code for lemmy, and could not find any performance tests.

I can probably start there.

Thanks for the update! Things seem way speedier now ^^

Great work! Awesome to see how fast the technical side of the Lemmyverse is evolving and improving!

Please recommend people to update their app in a topic title. Connect couldn't even load a topic without failing out today. An update fixed it, but I had to manually force it because it didn't apply automatically.

This will drive people away. Literally none of the communities I subscribe to on world even seemed to have a new comment.

Amezing improvement! Lemmy felt so much more responsive suddenly today, now it makes sense why :3

Everything is much faster and smoother today. Thanks devs!

The changes are very much noticeable, thanks as always everyone!

Thank you to all the code monkeys for getting things straightened out!

System appears to be working as intended

So much smoother! Thank you for all your charity and hard work!

Not that it really matters, but did anyone else notice their comment/post points get deleted after this update? Seems odd.

What points?

Its not on the default lemmy-ui on the web, but many third party apps for lemmy are showing total comment and post โ€œscoresโ€ on userโ€™s profile pages. Not exactly sure how they are calculating that, but ive noticed that those scores were wiped for at least my account today. Again, not a big deal, just thought it was strange how that would happen. It is showing like this in memmy as well as wefwef so itโ€™s definitely some data out in the lemmy verse that went bad. it is starting to โ€œcountโ€ stuff from comments ive made today though so, who knows.

Hmm that's odd. @aeharding@lemmy.world do you know why score counts are wiped? I have no clue how they are calculated.

It looks like on the person_view counts object API response coming back from Lemmy.world, the amounts for scores are not right. Hmmm

This is a lot of work, I genuinely feel honored by the service of everyone involved. Thank you!

It's definitely snappier than before. Constantly having to refresh got old quick.

Are you guys able to create reports? I am not.. It keeps spinning.

The new updates made a ton of improvements. Thank you very much. It is near ideal now.

That is so hot!

Thanks for sharing :).

Loving the transparency.

Awesome work! I'm seeing a lot less errors and I can actively scroll through posts quickly now.

I'm in awe of how quickly these issues are being resolved.

Thank you for all the hard work.

Looks really good, and have definitely noticed less issues just over the last couple of hours. Great job folks!

I can't imagine the amount of work Lemmy's devs and ITs are under since few days, but those are important for the future of Lemmy. Keep the good work! You're awesome!

Would HAProxy work better as a load balancer? For work we switched due to some issues with NGINX; so far, the service has been much more consistent with pretty much no downtime, even when restarting server hosts.

Can be considered, for now it's working ...

Woo hoo, nice graphs! Monitoring data that shows big improvements is always fun to see.

Absolutely great job! Everything feels buttery smooth today, and Iโ€™m yet to see any errors.

Thank you for your work people. You are doing great.

I read this post after browsing for an hour and Iโ€™ve definitely felt a change. Great work!

You know there's something about dealing with the lagginess in the past few days makes me appreciate the fast and responsive of the update. It nice to see the community grows and makes the experience at Lemmy feels authentic.

Thank you so much for all the efforts! So happy to be part of this community.

Thank you so much! I will be donating a few cappuccinos your way when my next check arrives. I really appreciate how awesome of a community youโ€™ve brought together & all of the transparency with the updates (and the frequency) is astounding! Keep up the great work but donโ€™t forget to take breaks :)

Really appreciate all the hard work going on behind the scenes! Feels night and day different after the changes. Also appreciate the transparency. Nice to see in this day and age.

Everything is feeling great so far. The only bug I'm encountering is that when opening a thread (in Firefox on desktop) it auto-scrolls down past the content to the replies.

Awesome, no more errors on my end! Thanks for the hard work, donated as a small token of thanks for the team here.

Building history one update at a time ๐Ÿ˜Ž

Can you describe your setup? Single instance or multi-node setup behind load balancer?

1 big server with nginx load balancing to the docker containers running Lemmy

You guys are doing god's work. Thank you thank you thank you...