Version 0.19.X Deployment

lwadmin@lemmy.worldmod to Lemmy.World Announcements@lemmy.world – 976 points –

Version 0.19.X Deployment

Hello world!

I'm sure many of you fine folks have been wondering why we have not upgraded to version 0.19.X yet.

The whole team here has been getting asked this question quite a bit, from both members of the community and other instance admins. We want everyone to know, YES, we will upgrade to version 0.19.

We do not have a hard date set for when we will be upgrading as of yet, as we have about 7 associated task items that go along with this upgrade for things we need to double-check and confirm there will be no breakage.

Another big reason why we have not upgraded is due to reports from other instance admins that the front-end GUI can have performance issues. While a large portion of folks access our site using apps on their phones, we would rather not degrade the user experience for the rest of everyone else.

Worst case in terms of UI slowness, we would ask everyone to use either Alexandrite () or Photon () to mitigate the potential performance issues.

One final point to mention to address the delay is that we usually wait 1-2 weeks after a new release before applying it to production. This was complicated by the fact that there were multiple releases right after each other, with each new release having issues we hoped would be patched in the NEXT version. A new version would come up, and we would have to evaluate the stability and performance of it, which would take about 2 weeks, and then ANOTHER version would come out, which we would need to test. This was further complicated by the fact this was during the holiday season.

Version Release Date
0.19.0 2023-12-15
0.19.1 2023-12-20
0.19.2 2024-01-10
0.19.3 2024-01-24

For us, stability is paramount, as we care very much about keeping everyone happy here.

While we can appreciate the rapid release of patches on 0.19, I hope everyone can now understand the delay. If all goes well, we will set a release day in the next week or two, most likely on a Saturday around 1800 UTC.

-jelloeater85

103

I just love how professional the Lemmy instances have become. They went from having stability issues to being extremely stable. I love our Lemmy community.

Keep up the good work!

I just love how professional the Lemmy instances have become.

Too bad upstream Lemmy development didn't adapt. The early 0.19 releases left several big instances hanging with severe bugs leaving them practically unusable.

In the developers defense, there are some bugs that don't appear until a a certain workload threshold is crossed. It's entirely possible for code to pass all tests and yet break when under heavy load in the real world (which can be difficult to simulate).

Yeah many bugs are difficult to catch, but the federation and timestamp issue, was something different and almost like a joke that got out of hand.

the federation and timestamp issue, was something different and almost like a joke that got out of hand.

YOLO-driven development

The testing in production development

Everyone has a test enviroment, some are lucky to have a separate production enviroment

The testing in production development

That's what Mastodon developers seem to do but ahead of formally releasing new versions and on their own instance only:

Thanks for all this info, I knew running the largest Lemmy instance would be challenging in term of version upgrade, and I'm happy you keep us posted on all your plans! Keep up the good work!

Yeah, I think the transparency and inclusivity of this instance serves as a model. I love knowing that places like this can exist.

Thank you!

You're welcome, and thank you too! šŸ˜Š

No, YOU are welcome here ;) You guys make this instance and the fediverse the best ;D

Speaking as someone who is on an instance (lemmy.today) that ran into a bunch of breakage from the 0.19.X releases, and which still isn't fully resolved, and where the instance admin said that he wished that he could downgrade to 0.18.X but couldn't due to schema changes, I strongly endorse a conservative approach. The releases have not really met the bar that one might want for stability.

That's especially true for lemmy.world, since it hosts a large chunk of the Fediverse communities, and if it has serious problems, there are gonna be spillover effects even on users elsewhere. I'd wait until less-critical instances have been the guinea pig for a bit on releases.

Seriously. Less is more.

There's no need to keep up with the meme of releasing updates as quickly as possible.

Well said. I will happily forgo some new features for a while if it means that the overall experience is more consistent.

I've talked a lot of shit about the whole fediverse before, but I appreciate your stance towards stability. It's what users crave! Thank you!

Thanks for the update. I am looking forward to instance blocking!

Just don't be too excited since it only blocks communities on instances, not the users on it.

Excerpt from Join-Lemmy.org

Users can nowĀ block instances. Similar to community blocks, it means that any posts from communities which are hosted on that instance are hidden. However the block doesnā€™t affect users from the blocked instance, their posts and comments can still be seen normally in other communities.

It's still definitely useful, but it's not as great as it's been hyped, and won't solve the spam problems from specific instances.

I am more interested in cleaning up All than the user brigade at this point.

I turned off nsfw due to all the porn and yiff, and I really would like to filter out instances that are in a language I donā€™t speak.

Yeah I just want to make sure people know so they don't get their hopes up and get disappointed or frustrated that it doesn't act like personal defederation.

That is very likely what I'll use it for as well, since I basically stopped browsing all due to the NSFW communities who's posts would pop-up randomly in it, sometimes even unflaired šŸ˜¬.

filter out instances that are in a language I donā€™t speak.

I'd avoid that as it's a bit overkill and many of those instances may also have English-speaking communities. A better solution is to exclude the languages you don't speak from your settings so the posts and comments won't appear for you.

exclude the languages you don't speak from your settings

Thanks for the tip. This option is not surfaced in voyager, so I had no idea of its existence.

There is a warning that deselecting ā€œunspecifiedā€ will remove most posts, so it remains to be seen how much this will help.

Yeah it would be good if more communities and instances chose to enforce setting a language, they can do that by excluding unspecified from their community which will disallow people to make posts there that don't have the language set. Obviously nothing can be really done for posts that don't set the language, as well as for most apps which don't set the language whatsoever, or have really weird ways of doing it.

Good luck with the update! One great thing about 0.19 is that it allows users to check federation status between instances, will be awesome to get that for lemmy.world as well.

And one thing bad is that logged in users are logged out and can't log back in, getting errors when loggin in or worse, the site showing them a server error and they think the site is down! All fixed by clearing cookies. Or in apps, logging out then back in.

It looks like 0.19.3 may have a fix for this, which is great. IMO it's good that the largest lemmy instance hasn't updated to 0.19 yet, because a bunch of users may have disappeared thinking the site was dead. And you'd never know, because 0.19 counts active users in a different way.

Hopefully this is the one, though!

Just so you know, there are some major security issues on 0.18. Thatā€™s why programming.dev immediately updated. The later bugs on 0.19 we were easily able to work around, while the security issues are major and should be resolved as soon as you can, as it affects every user.

We already patched it. No worries, we got already a tip for weeks. Sad that it got through the tests of the lemmy devs. Such issues should never go live.

Manually patched by pulling the commit and building your own source?

No, we just block the affected api endpoint

No need to rush it if the software is in heavy churn. Just pick a date when the dust settles.

Hey just FYI, I've been experiencing a postgres memory leak issue on my instance since the upgrade to 0.19.2. Theres a GitHub issue open about it (#4406), not sure if it's something that will affect you but probably worth keeping in mind

Hi,

thanks for the tip, we will look into this issue and check if it affects anything on our scale.

We are investigating another possible bug, that sorta spams other instances full of requests ( we say 5k requests per 5 seconds ) kinda like a DOS attack. We only saw this phenomena on 0.19.3 instances, and they are hitting us currently hard with many requests. We will have to block them until it gets fixed on their side.

Hmmmm interesting, i hope im not on the ban list! I haven't noticed any weird traffic like that

Nope you are not ;)

We noticed it that it happens on smaller instances, that probably doesnt have the perfect hardware to run the 0.19 version and crashes => retries federation and they probably dont have a backoff strategy.

Because we noticed the site the domain was leading too wasnt responsive half of the time we tried to look if we can contact the admin.

LETSGO

Yeah my instance is small but my uptime is pretty good (except this db issue tbh) so that might be why

Thank you for looking out for us mobile/desktop web browser peasants. ā¤ļø

I for one am thankful for the admins' continued dedication to platform stability.

Thanks for the update. I appreciate the transparency and honesty.

No pressure from me, I know all too well about upgrading to the newest version and finding out it has a showstopper bug for the features i was planning to use.

Thankyou for running the server and helping build the community and fediverse.

Hey ever since the update boost won't show my subscriptions and my post/comment history, I can only browse in all

Is this a known issue that I should bring up with the boost people?

Edit: I logged in to the wrong instance lol

Edit: I logged in to the wrong instance lol

That'll do it.

Lebowski.Social has also held back with the 0.19 upgrade. We are monitoring lemmy.world for now and will make our upgrade after that most likely.

Thanks for the transparency. I'm firmly of the "if it ain't broke, don't fix it" school. Upgrading to 0.19 is important for security, but until it is stable, it is also a risk.

We do not have a hard date set for when we will be upgrading as of yet, as we have about 7 associated task items that go along with this upgrade for things we need to double-check and confirm there will be no breakage.

Are there tickets open for these tasks so we can see what you're waiting for? Genuinely wondering

We dont have a public Roadmap yet, but we will have to test it in multiple steps. And we dont want to get suprised with bugs ( like those in 0.19.0/1/2 )

I for one am glad you take your time. I jumped ship from another instance that upgraded to 0.19 back in december and was almost unusable far into january. It's now on 0.19.3 and while federation seems to work again, it's still quite slow and unstable.

0.19 breaks jerboa and idk if/ when that is getting updated

@dessalines@lemmy.ml released v0.0.51 in mid December, but I think it only reached Google Play on Dec 18th. I'm not sure when/if it was pushed up to f-Droid.

There are some minor issues, but largely it works fine for me on instances running 0.19

That's weird I've gotten no notification there was an update out

Did you install from Google Play?

Open the Play Store link on your phone - the automatic update process has been thoroughly broken for at least a year.

If you installed from f-Droid I have no clue - I use stock android without any alternate stores set up.

Play store updates being broken would make sense. I don't think I've had a single app updated automatically in that time frame.. Holy shit lol

Neat. Good to see constant improvement.

Unrelated: What ever happened to the weekly LW community spotlight? It hasnā€™t been updated in like a month

The admin that organized that has left the team. Iā€™m the new solo-lead for the community team, and will try to get that started again soon. Any suggestions for communities are more than welcome!

That makes sense. I was just wondering, since spotlight seemed like such a good idea on rollout.

As for communities, Iā€™m way deeper into the hobby side of lemmy. Showing off

!battletech@lemmy.world

!warhammer40k@lemmy.world

or

!star_wars@lemmy.world

would all personally be great to me.

Just posted the new spotlight, thank you for reminding me (:

Wow! Fast. Thanks so much for taking over and doing all the admin-y things!

If all goes well, we will set a release day in the next week or two, most likely on a Saturday around 1800 UTC.

Any chance we get can get a status update?

We're going on four weeks from the initial statement. The lack of status updates is what bothers me the most. I understand delays, but please let us know. It just looks like it's been forgotten about.

@Thekingoflorda@lemmy.world, we understand this is a volunteer effort, but as the community lead, would you be able to shed some light into the status of the upgrade? I don't think we are asking or pressuring for the upgrade to happen ASAP, just some transparency would be nice. We're two weeks past the proposed update, and almost a month beyond the last communication on the topic. Appreciate what you all do to provide the community for us, we ask because we care.

So ,if everything goes well the first update will be Saturday 10 februari around 18.00 UTC, and then every two weeks? Bonne Chance!

Take your time, stability and uptime is most important. Thank you :-)

Wondering if future plan of LW admins is to switch to Sublinks once it becomes stable?

The possibility is there, its something good to have. And with it we dont need to annoy the lemmy devs for implementing stuff for "just the big instances" and do it by with our team of java developers. And because it is written in java, its prefered and known by many devs, it accelerates the development of any feature a lot.

Sounds like the reasoning of someone who eats Jello for dessert...sensible and safe.

Hello,

Interesting, I didn't know about the UI performance issues, but I definitely felt it recently

I just have a question, when is this going to be fixed?

When I go to create a post/comment I'm always required to pick the language, makes no sense. https://github.com/LemmyNet/lemmy-ui/issues/2317

It would be nice to set that at the settings level and have it auto populated instead of asking each time on a post.

That's what I've been saying since there's lemmy...

"Iā€™m sure many of you fine folks have been wondering why we have not upgraded to version 0.19.X yet.

The whole team here has been getting asked this question quite a bit, from both members of the community and other instance admins. We want everyone to know, YES, we will upgrade to version 0.19."

The issue here was never that decision, which on the whole seems sensible. It was the lack of comms. If there had been an announcement of that plan or delayed after say a week or 2, you'd probably have less of those questions. Users are stakeholders, and talking only to devs and other instance runners while ignoring users shows the level of respect you have for the users of the instance...

The ironic thing, is the communication and openness at the start was the best thing about Lemmy.World, and then later down the line, it just became an after event to communicate to users. "Decided to defederate x, y and z on Discord.". Silence on attacks for months, and silence on upgrade plans.

There is many roles on Lemmy.World, and communications probably needs more focus...

We understand your frustration, but we had in the last few weeks a team restructuring and that was at that point out of our sight.

We had some mixed signals from the devs and admins at that point too, so we were unsure if we want to upgrade or wait for a another week, then those bugs struck, with federation, Timestamp incompatibilities and some other bugs. But earlier the week we had the courage to say, we could see 0.19.3 as a upgrade candidate, we did not want to announce "Hey, we upgrade to 0.19" then quickly go "Sorry no upgrade because of a critical bug" and then after new version "Hey guys upgrade time" and so on.

and announcing defederation is something to do adhoc most of the time, so it cant be like "Hey we will defederate x,y and z in a week" then, because then the reason why LW wanted to defederate that instnace floods the comments.

Please understand we all are just here on our freetime, no one here gets paid for this. So please consider this a hobby project. You can demand something like this on reddit or twitter, but not in a open source and volunteer driven Website.

I don't need to and I choose not to use reddit or twitter. lemm.ee are doing a great job. lemmy.ml is also a great option. It's a shame that they're nailing it better than world is.

Was a topic labelled 0.19 upgrade and something along the lines of "0.19 has been released by Lemmy. We are going to be waiting a few weeks to see the stability of this. Once we are satisfied, we will update here with more information."

And "due to certain federation bugs, we are going to hold back on upgrade to 0.19 until these are ironed out, we will update when we plan to do this".

It's not hard, and it seems like you're coming up with excuses rather than considering feedback...

Hexbear defederation happened before they even federated. No consultation, or discussion, just "oh, we decided to do this, and whether you care or not, ah well". lemm.ee had a post, reasons for, against and welcomed the communities view on this. Maybe you guys need to check them out and learn from them. I think it's only 1 guy too, so probably more than repeatable as a process...

Considering this is the most staffed and resourced instance, you'd expect a reasonable standard.

Ok next time we straight up swap to the beta release of lemmy as you wish. If there is a issue, will redirect them to you.

So I suggested how you could do comms better, and also a sensible approach to defederation decisions, and that is your response.

Rather than thinking, "oh, is there some way we can do this better?", just get defensive and blame the person giving feedback.

It's human to err, but to refuse to learn, well that's quiet something... I had expected a more mature response.

Considering my original point was about how users on Lemmy were an after thought. Not worth factoring in to decision or consulting, you sure aren't really do a good job refuting it...

Lemmy world was the instance I originally joined when the whole exodus kicked off and others closed their doors. I was very grateful and had enormous good will for it. I never expected in the space of several months to have 180'd on it...

Yeah my response, was a bit harsh! I will forward your feedback to the rest of the admins.

It wasnt my choice to defederate, but defederation is a non destructive thing and federation could have been restored if there was a discussion from users with evidence that they improved themselves. We would have thought about refederating with them after 0.19, because then every user can decide themselves if they want to see them.

I appreciate you taking feedback on board, and I apologise if I was harsh about this and I didn't mean for this to be aimed at you individually. It was a general frustration with general decision making and communication. It's not the first time I've raised this and I care about Lemmy, it just gets frustrating when it feels the feedback is getting stonewalled.

I appreciate the position on consideration of refederation. it does make sense to consider it post 0.19.