[Update: Failed again] Update to 0.18.1-rc.1 tried and rolled back

Ruud@lemmy.worldmod to Lemmy.World Announcements@lemmy.world – 306 points –

We've upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.

(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)

The main issue we saw is that users can't login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)

Also new account creation didn't work.

I'll create an issue for the devs and retry once it's fixed.

Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!

Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won't be on 0.18.1 any time soon I'm afraid..

211

Thank you for the transparancy!

I love post like this, another reason of why this instance is so good

is that users canโ€™t login anymore

Ouch, that's bad :(

Thank you for your work, as always :)

Both Dessalines and Nutomic have been working their butts off to get 0.18.x ready for the Reddit API changes. Huge hopes they can pull through!

Dessalines:

Nutomic:

Yeah they've been hard at work all month. But it's also okay if things aren't ready in time. Most of the people who matter are already here.

Maybe we will blow up soon, maybe later, but the quality of content here is sufficient to drive growth regardless of whether or not we get the prophesized huge migrations from reddit

Agreed. I'm here to stay with the content from the last couple weeks. 3 day poop embargoes alone is worth it. And recognizing users because the community is still small enough is SO different from they who shall not be named.

Oh god, so much coffee... it's also thrilling to grasp a sense of what's going on under the hood of such big social networks at a development level (not like I could understand it, but it was very interesting to see twitter's recommendation algorithm being open-sourced)

Just curious, why are we updating now instead of waiting for the proper 0.18.1 release?

Because of a few things, really anoying bugs and the Jerboa app not working properly with older versions

Reasons I can think of:

  1. The official Android client for lemmy, Jerboa, only supports 0.18 and later, unless users download older version from github and sideload manually.

  2. Sorting is broken pre 0.18, new posts keeps flowing in.

  3. Performance improvement by removing web socket from lemmy. (which fixes 2, which is why 1. happens)

Try wefwef.app on mobile

(no I'm not a bot account shilling I just really like the app)

If there was a way to adjust the font sizes it would be 100% perfect. Damn! I'm surprised I hadn't heard of it before. Thanks internet stranger!

I'm sold. ๐Ÿ˜Š I did need to try a couple of times to figure out I needed to swipe left to reply to a comment, though. But it looks good, works well, and I'll get used to the quirks.

Why the hell does Jerboa even limit support to 0.18? That's so stupid.

Likely because Jebora is written as a side project of the devs of lemmy so it's always going to be cutting edge

I understand that. I'm not complaining about the quality. I'm just confused from a technical perspective why it doesn't support rollback or older server versions in the event of....this.

Fixing the issues with posts appearing and scrolling the page the backend needed some pretty drastic rewrites. I'm not super familiar, but from what I've seen of the code, it would be a decent amount more work to support both versions.

Would it have been worth it? Yes, but it wasn't anticipated that devs would stay on 0.0.17 for more than a day or two. With the time it takes for app stores to update, servers would have been updated before lemmy 0.0.45 was updated for the vast majority of users. At most, it would be a day or two instead of a week or two.

Jerboa works fine with lemmy world, it just gives a warning and crashes on occasion. Not an issue to use it though.

That sounds like Bethesda's definition of "works fine"

Yes, if you are already logged in. If you are not logged in, or get logged out you will not be able to log back in until the server version is above 0.18

For me at least, I'm unable to change the community scope or sorting method. The only thing I can use Jerboa for is "local" and sorted by "active". It isn't totally unusable, but I would much rather be able to switch to subscribed and hot.

I had the same bug. A full restart of my phone seems to have fixed it.

Unfortunately that didn't work for me. Thanks for the idea, though. I'm sure it will work for some. I suspect that if I wiped the app and set everything back up it might work, but that's more than I'm willing to do at the moment and I can live with the inconvenience for now.

With the last couple updates jerboa has got to a point that I probably won't even move to sync when it comes. I'm pretty happy with it. But if Lemmy Is Fun ever happens, I'm out.

I'm of a similar mind: I'm happy with Jerboa, yet also looking forward to Slide.

Well you're a lucky one. Jerboa has been unusable for me and other folks post v0.34 since the first rc release of 0.18. I finally managed to downgrade to 0.34 and login after lots of black magic. It used to crash instantly, even on 0.34.

That being said, good luck to lemmy.world admins and thanks a lot for the open comms !

"crashes on occasion"

Unfortunately, the occasion seems to be at least once every five minutes for me. thisisfine.jpg This is not fine and definitely a major issue. Especially as we get closer to July 1. After this news from lemmy.world, I will have to switch, probably to Connect, until Sync arrives.

As a new user I tried logging in on Jerboa and it says the server must be on 0.18 and refuses the login.

The official Android client for lemmy, Jerboa, only supports 0.18 and later, unless users download older version from github and sideload manually.

Even that won't work, because Jerboa initially tries to connect to lemmy.ml which is running 0.18, which older versions of Jerboa aren't compatible with. The app just crashes instantly without giving you the opportunity to log in to your instance.

For example the frontpage is no longer being constantly filled with random posts when you set the filter to "active" or "top day".

I suspect there will be a large influx of new users in two days, and that having Jerboa not working on .world might cause a few issues

Federation is completely borked with .18 servers. Its very difficult for us to interact with https://lemmy.ca in any way: subscribing, upvoting, comments, posting... its all bugged.

Its maybe not that big a deal because Lemmy.world "has the most users", so in some regards its https://lemmy.ca's loss but... we need to restore reliable federation... especially before the July 1st rush IMO.

The .17 to .18 upgrade is basically a soft-defederation event, because of whatever this bug is between the two versions.

Ah that explains it. Someone posted a cool photo to my community from lemmy.ca but didn't interact further. Looks like my comment didn't even show up on their end.

Anyway, thanks to everyone working on the issue. I know these things aren't easy.

Just curious, why are we updating now instead of waiting for the proper 0.18.1 release?

A Release Candidate is supposed to be past beta testing already and in a state of no major bugs.

Cause if you are on the homepage for a little bit new post start pushing the old post down and it keeps doing it. something about websockets that 18 fixes it. im assuming the admins want to get it out now so people can stop complaining about it. i get it though just wait for the 18 instead of the rc-1 but users are impatient and can't wait for sh*t!

3 more...

UI: 0.18.0 BE: 0.18.1-rc.1 ๐Ÿ‘

seems to be working for me, I was already logged on

As any engineer who does ops can tell you - you did the right thing - the solution is always to roll back, never force a roll forward, ever.

We should totally do pre and post update parties though. Even if the update fails we can have an excuse for drinks and a fun thread.

Eveyrthing (UI) is extremely slow now. Not an improvement at all.

I am glad I didn't write my last post during the upgrade process.

Maybe next time, give a warning, or maintenance notice.

Thanks for trying though.

Thatโ€™s just a pain point of using modern asynchrony software and will never go away

Thanks for the info. You're doing a great job!

Linux/Firefox After upgrade many things didn't work, but they continued not to work after rollback.

Turns out I needed to clear the cache, then everything worked fine again.

@ruud@lemmy.world just wondering if you have considered setting up a second, beta, instance of lemmy.world open to the public?

With all the performance issues with 0.18.1, it's highlighted that there needs to be a way to stress test these updates before applying them to the main instance.

Yes, considering that. But we'll need people to use that when we will do testing...

Well, you have at least one person who would be willing to use it ;D

One more. You find a stable way to notify anout upgrades and get a test sheet to run through and we can generate posts and activity to help test with.

Light the beacons! lemmy.world calls for aid!

I'm not familiar with what the server architecture looks like, but is there a possibility of using a load balancer in front of the instance's server and swapping a "beta" server into the load balancer when you need to do testing? You could basically migrate your traffic with zero downtime, assuming Lemmy's architecture allows for it.

Well that doesn't differ much from what I do. I just copy the files to a second directory and test with that. Easy rollback. Downside is, that all data is lost between upgrade and rollback, which will be the same in the scenario you suggest.

Ah, that's fair. Best of luck either way. This is the tough part of admin work. FWIW, the stability we've had thus far has been pretty impressive!

Jerboa is almost unusable for me at the moment. A lot of crashes, pages not loading, messages not sending etc. But I know that's inevitable at times. Restarting my phone seemed to make it worse. But I know you guys are on it! ๐Ÿ‘

I recommend you this web app

https://wefwef.app

It is almost as functional as Jerboa and very impressive for a web app.

Currently rate-limited by Lemmy.world, per the dev. You might get a lot of red error boxes.

Why?

Too much traffic going through the app, as he explained it. I would try to link the post, but everything is so janky on both Lemmy.world and our wonderful Apollo clone that I canโ€™t be bothered.

Too much traffic because it is an outstanding app, gotcha.

I just switched to it, Had tried jerboa, connect and liftoff and this one is by far my favorite

Liftoff is a much better app. I used Jerboa as well but found an alternative cuz it was pretty much unusable. Just dowload Liftoff's latest apk and install

Seems 0.18 is up... and...

OH NO ! The interface changed ! Bring it back ! Bring it back... /s

Just kidding... It's looking good, so it's time to explore a little and find all the new stuff.

Nice work @Rudd, thanks for all the hard work.

Suddenly my Jerboa won't change from local and I can't change the sort. Not sure if this is related, but it was working fine yesterday after I updated the app. But no longer this morning.

What you're describing is an issue introduced with Jerboa 0.0.36-alpha and has been fixed in today released 0.0.37-alpha.

Mine randomly quit working a few days ago. All I get is a blank screen. I've switched to the website for now, though I need to look into the other app options. There are a couple more now.

My current session works now with 0.18.0 + 0.18.1.rc1

appreciate the transparency. how are things looking in the back in lemmy.world (server wise)? will we get to a point where it wont require complete rollbacks on the state when a botched update gets rolled out?

Jerboa crashes constantly for me now, doesn't even let me view anything. JSYK!

There's a fix the dev has just released. See reviews section on the app. He said it'll take a few hours to show in the playstore.

This is good news! I haven't been able to try Jerboa fully yet since I use lemmy.world. I'm not seeing this review section you're talking about but I'll be on the lookout for an update. Thanks for sharing!

Sorry, my comment wasn't very clear. On the app page within the playstore, if you scroll to the bottom and read the reviews you'll see the developer has replied to say a fix will be released today. I'm yet to see it though.

I am glad y'all rolled back. I had just enabled 2FA and could not login to my account. Rolling back removed 2FA so I was able to log back in. Thanks.

Sucks that lemmy.world won't be working with Jerboa when the reddit apps quit working in a couple days.

Jerboa still works fine with lemmy.world. It pops up a warning that the server version is unsupported, but it still seems to work fine.

After I get the warning I can tap the hamburger menu and it shows my account name up top but when I try to upvote a post it states I need to log in.

On the plus side, I'm about to open the beta that handles both just fine (plus a lot of features no one else is doing yet). The deadline I for myself is Saturday when the Reddit clients go down.

Unfortunately it was built by someone who lurks and comments, and I realized even though merging feeds from accounts on different servers is going to be awesome, other people like to post and I need to stop getting off track

The stress of the timeline is starting to get to me and I'm terrified no one is going to give it a chance, but I've fixed most of the things that drove me nuts in jerboa... That should count for something, right?

Hol up, are you soft announcing the development of a new Jerboa-based app? What's it called? Good luck to you! I hope you achieve your goals with the project and that is well received. โ˜บ๏ธ

Hey, so it's not jerboa based and I ended up pushing it back a few weeks, but I'm now looking for beta testers on Android if you're interested

As for the name, I think I'm going with Luna for several reasons, a big one being that someone suggested it and Flemmy was the only name I could think of beforehand

I'm posting updates to !flemmy@lemmy.world for now, if you're interested I'll be posting instructions for the beta soon.

Whether you're interested or not I'd love to get feedback. I'm good with data and UX flow, but I mostly just comment - I need to learn how other people want to use Lemmy

Thanks for the update! (I mean the communication regarding the update) 0.18 moved away web sockets right? Maybe that leads to different DB access patterns causing locks on such a big instance as lemmy.world.

PSA: If you've been using lemmy.world during the upgrades and rollbacks your browser might've cashed some stuff that quickly got out of date. If you have problems (especially JSON errors), try doing a hard refresh (usually Ctrl + F5) on the site. It might log you out.

I'm so new, I'm a baby, I have no idea what I'm doing. I did just create my account, I tried using the app but it wouldn't let me. Saying server version too low. I'm sorry for my ignorance, but is this what that means? And this means we can't use the app yet right?

Also, thank you for all of this. I will be here for it all, I'm staying forever! I already love this so much more than anything else ๐Ÿฅฐ

Until we can get the instance updated, yeah the jerboa app doesn't really work. You could try another app, I'm using liftoff but that's harder to install, you could Google "thunder Lemmy" and try that one, I haven't tested it much but I've heard good things

Great! I'm getting a bug, when I upvote a comment or post, the "loading animation" takes forever to complete. If I refresh the upvote is there. It reminds me to the bug we had with comments and new posts.

Edit: Is the same for comments.

Dude I posted the funniest comment in those 10 minutes but you all must have missed it. Oh well ๐Ÿ˜…

RC2 RC4 was just released, here's hoping it solves the issues you were experiencing with performance of RC1!

Thanks for the transparency. Maybe itโ€™s a good idea to have a test instance and some test cases/validation done there before updating the main instance. This is a regular process in any software/tech company/stack.

Testing should never be done directly on the prod instance.

i think they tested it which worked fine, but it is totally different when there are users online

Correct there was a test instance and that worked fine but once the live instance was upgraded everything became very slow. Can't really emulate that kind of load

In the meantime I've migrated over to Connect for Lemmy on Android, and it is working significantly better than Jerboa did.

The improvements are awesome, great to see it sorted out before Reddit cuts off API access.

I just occasionally see error message popups for JSON syntax errors.

Damn, Im glad you tried, and thanks for the update! I just really want the front page fixed without changing to page 0 lol. That and the 3/6/12 hour filter option.

Keep us updated. Im bouncing back and forth between LW, KBIN, and .ml for now though

any recommendations on other instances I should try outside the lemmy-verse and kbin-verse?

Iโ€™ve found from hopping around some other instances that have upgraded to 0.18 that it is still pretty buggy. It does seem to be giving more information about the errors, instead of just failing like in 0.17, but spend any time browsing on those instances and youโ€™re bound to be inundated with JSON and query errors. It also seems to get worse the longer you browse.

The UI changes are nice, and I do appreciate not having my feed auto-updating constantly, but I think youโ€™d be making the right choice to hold off on upgrading until they can iron 0.18.1 out all the way. Iโ€™m not super knowledgeable about TS and Rust, but as a user it seems that switching from WebSocket created/shined a light on Lemmyโ€™s issues with caching in general.

but as a user it seems that switching from WebSocket created/shined a light on Lemmyโ€™s issues with caching in general.

Or just adding actual error messages instead of ignoring them and throwing up a spinning wheel.

What changes does the new release candidate offer?

0.18 is where they have swapped WebSocket for HTTP, as well as tried to fix a load of bugs related to using WebSockets (like the new posts pushing the main feeds down).

0.18.1 is meant to re-enable CAPTCHA on signups, which was removed in 0.18.

I managed to visit when the server was 0.18.1-rc. I didnโ€™t see any posts and the selector for Subscribed, Local and All was โ€œemptyโ€, as in nothing was selected and I could not select anything.

Is it OK now? There's a bug in lemmy-ui 0.18.1-rc.1 which I think you've seen

Posted after Edit 2.

I would like to manage my expectations. Yesterday we were looking at 0.18.1 being sent out in about week from the devs. Was 0.18.1-rc.1 what we thought they'd have ready? Do we still think the 7-10 day window is likely?

rc.1 is the first Release Candidate of that version - not the final one, so between now and a week a lot can still happen.

Thanks. It's neat how active you and the other admins are. You guys dedicate a lot of time to individual comments, and not just on this thread.

No they said in a week they would have it final. But maybe it will take a bit longer....

Lemmy seems like a herculean undertaking. I can't imagine anyone doing the nuts and bolts for free or on any schedule, so it's quite amazing to me that we can talk about getting updates like this.

So how does this work..is there a beta server to try it on? Or it worked in resting and it only broke with the full amount of users?

Yes, there are three testing servers, but errors still do happen. especially when everyone is in a hurry to release

Running one of the largest and most vibrant Lemmy instances certainly makes this deployment a great test case for the devs and the system overall. @ruud@lemmy.world this community really appreciates you!

Sucks that you couldnt get it working yet, but there are bound to be problems at this scale.

Anyone have any resources for learning what running a lemmy server is like ? Seems pretty interesting

If you're running a small server it's pretty straightforward and hands-off. It's only when you get to the scale of these larger instances that you run into issues. I just rerun ansible every time there's an update and it otherwise just manages itself.

Looks like we're still having issues, more's the pity.

Do we know what's causing the issue, or is it still a bit of a mystery for the time being?

After the update I can no longer log in or interact with Jerboa. Hopefully y'all resolve this soon, as I am really digging Jerboa, but thanks for letting us know what's going on, and keeping up the good work.

I've been using Jerboa since I joined a few weeks ago, I haven't been liking any of the other clients so far, but today I tried thunder and I'm enjoying it more than Jerboa

There's a major breaking change in the 0.18 update (moving away from Websockets for posts and comments), that means that older Jerboa builds no longer work with instances on, or past, that version, and newer Jerboa builds won't work with servers older than 0.18.

Version fracturing is probably going to be a fact of life on a decentralized platform like lemmy, a "good" app should probably handle cases for the latest few revisions instead of just the latest.

I'm sure it will eventually, but early stage tech needs some allowances while they handle growth and scale. Better to break stuff now than be hamstrung by tech debt years later

It normally would, it's just that 0.18 is a breaking change, incorporating a major rewrite of how posts and comments work on Lemmy.

I really like Semantic versioning. This would be from 0.17.X to 1.0.0 as it's a breaking change to the public API.

https://semver.org/

Semver pre version 1 lets minor changes be breaking (points 4 & 5).

Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable.

1 more...

was there a test environment? why rc in production?

Could you upgrade to 0.18.0? The android app doesn't work with 0.17, but it seems to work with 0.18.0

Finally got signed up, had to change my username for some crazy reason.

So are we stable now? I lost some posts and comments this morning.

Yeah we won't try again until a stable version is released with the performance issues fixed..

Good luck next time. Thank you for your hard work & transparancy.

Dang, I updated Jerboa too hastily, time to figure out how to downgrade back to 0.34

I don't see version 0.0.34 unfortunately, but here are some options. I think apkmirror is a safe source. https://www.apkmirror.com/?post_type=app_release&searchtype=apk&s=jerboa

For the future; there are apps that can save your apk's, so you can always downgrade if necessary. Check out on f-droid for open source apps that you like. I have been doing this for years, because it happens quite often that an app version is buggy or they suddenly add annoying features and then you can instantly downgrade.

Is there any place that people can assist in trying to figure out issues?

Well you can either help the devs via their Github, or if you wanna help sysadmin the server I can send you a link to our Discord.

Thanks for the response, Going to look into the github/code side but I may reach out in the near future to help where needed. ๐Ÿ˜€

I want to upvote, but I get this instead.

Oh man, for a sec there you made me thing I was getting errors ๐Ÿ˜…

You're fighting the good fight on our behalf! ๐Ÿ’ช

This is exactly the kind of issue I was afraid of when switching to Lemmy and seeing its federated, decentralized organization

not really sure what a software bug has to do with a decentralized platform?

Even Big Tech has software bugs that fuck things up all the time. Hell, how many times has iOS updates had to be put on hold over a massive bug?

Lemmy.ml is on 0.18, you are more than welcome to sign up there, but personally I'm cool with a server that is in no rush and would rather wait for functioning softweare.