Lemmy.ml `v0.19` upgrade issues and downtime.

Dessalines@lemmy.mlmod to Announcements@lemmy.ml – 467 points –

In anticipation of Lemmy's upcoming 0.19 release, and to work out any final issues, we're going to deploy a test release on lemmy.ml within the next few days.

We're doing this testing on lemmy.ml only, so that we can encounter any issues before the release, and to make sure the upgrade process is smooth for other production servers.

Some of the following will happen during the process:

  • Apps will likely break (only for lemmy.ml)
  • Lemmy.ml may experience some downtime for the upgrade to complete (ideally no more than an hour).
  • If anything goes wrong, we may have to restore from a database backup, meaning content made in between backups may be lost.

If all goes well, we'll have an official announcement for the release after this testing period.

I apologize for the difficulties this might cause. At most this will be a week of hair-pulling, but its vital that we catch any issues before telling other servers to upgrade.

198

Proud to be on the official test server 🥲

haha, we appreciate the patience. We're not a massive company with teams of people, just a couple of FOSS programmers.

This is why this instance is so well. Thank you for all the hard work in reliberating the web!

Thanks everyone for the server and service. It's appreciated. If I wasn't broke and jobless at the moment, I'd be happy to kick over some monthly money on Patreon or Kofi or whatever people are using these days.

Start monetizing with features and donations. Being Foss doesn't exclude this

“Upcoming”? Are we talking hours, days, or weeks?

The release? 🤞 not more than a week, as long as we don't run into major issues.

Cheering you on from over here at lemm.ee! Thanks for making this place better for everyone!

can anyone tldr all big features coming in the new release?

19 more...

This is good to hear! Have you considered releasing more often with smaller sets of changes, for future releases? It seems like this will be quite the upgrade.

Yep, we'd def prefer to do smaller releases, and we'll try to make them more frequent and smaller in the future. The main reason this one was so delayed, is because of the large number of breaking changes, esp. w/ respect to authentication.

Great idea. Hope all is fairly smooth!

Can somebody explain what this scaling sort thing is to me? I've heard it a lot, but I don't know exactly what it does.

It's like hot sort, but gives a boost to inactive communities.

scaling sort thing? do you have a link describing what you mean?

One feature I'm hearing about it this algorithm called scaling sort and apparently it's supposed to boost smaller community posts up more. I was just curious as to how it works internally.

I had a little bit of a look at it a while ago.

From memory it’s not really sophisticated. I think it divides a post’s score by the number of users its community has. So it’s not so different from upvotes / user.

It may use monthly active users rather than absolute, I’m not sure.

It's like the other sort options, except it prioritizes communities with fewer subs, so that you'll see content from smaller communities more often.

The update has broken a bunch of things for me..
On my desktop using firefox, I cannot get images to display & everything is slow to load. On my android phone, boost was showing me a ton of stuff that I do not follow & wasn't showing me any info about my profile (profile tab would not load or something). Boost eventually logged me out & I have been unable to log back in....

Images were temporarily broken due to authentication problems, but thats fixed now. If you encounter problems in apps, please them to the developers so they can fix them.

I have experienced the same login bug on the Mlem app as well. One of my devices stayed logged in and is working perfectly; my other device logged me out of lemmy.ml and I have not been able to log back in on that device.

+1 for Connect app on Android. Got booted out and unable to log back in.

This is our fault, not the fault of app devs. The post text explains it, but we're working out a few bugs on a large instance, running an unreleased version of lemmy.

Again I'd just like to thank you for all the hard work you & all the other devs are putting into working on fixing everything that broke with this upgrade....

Thx! It turned out to be a lot more than we expected 😭

Well you all done an excellent job 👍👏

Good luck!! If we don’t hear any exploding noises than I guess that means it is going ok?

I was just wondering: Instead of testing these potentially disruptive changes on the main website, why wouldn't you make some testing instances where people can try everything out, like voyager, enterprise, and ds9 were back in the day?

I don’t think this is a bad suggestion, but it’s worth noting that lemmy.ml is explicitly intended not to be the “main” Lemmy, or even a major one: What is lemmy.ml?

Yep, I've been enraged by this decision from day one. This is depressingly amateur. Let people join, let other developers make very cool apps, and then introduce a deeply breaking change in a minor version, and deploy it on the most active instance, with mere days of warning. Or "How to destroy all the progress made by Lemmy, in one small change".

I get it if the devs and admins of Lemmy.ml are paying too much out of their own pocket, and if they want users to literally go away, to mitigate that cost.

But doing it in such an in such an insidious, demoralizing way, as opposed to being transparent with the costs and announcing (drastic) measures to mitigate that cost, is literally destroying most of the progress made so far, and driving most users back to reddit.

As of today, the list of most active servers of the fediverse has only one Lemmy server (Lemmy.world), in ninth position, and that is the only Lemmy server in that list, over four pages... The Lemmy instances used to be in the middle of the first first 10 instances, with Lemmy.ml leading the way.

Now, I guess the devs didn't want to take those drastic measures, and tell people to they would be closing down their accounts, ordered by creation date, until the costs become bearable again. Because that would mean "admitting the Lemmy.ml experiment to show the world that people are, when given the opportunity, rising to the challenge, and putting in the effort, in true communist fashion, is actually a failure". People aren't ready for communism. Communism requires education, intelligence, and empathy/compassion. Our western societies are fostering the opposite traits. When we become educated, intelligent, and empathic or compassionate, it is in spite of our societies, not thanks to them.

Now, a few people opened instances, but it wasn't enough, and fast enough, when the "reddit migration" happened, to absorb the insane influx of users to Lemmy.ml.

So I guess it is what it is, but it's still sad and depressing...

Is there an update on the state of the update?

Speedily trying to get things merged and more release candidates out... sry its taking longer than expected.

we love you dw!!

I can sign in with a web browser, but this broke Jerboa for me

That's expected. Jerboa has an open PR for v0.19 support, and it'll be deployed with the official release.

Same issue here.

I think it would be good practice, that verifying app support should take a front seat in the future before upgrading. The way it is currently handled, is not a good practice to retain users, but rather drive them away.

How much longer will it be before I can finally log in and use boost or liftoff. Is there a timeline for a fix? Both have been broken since the update.

Unfortunately that's an issue for boost and liftoff to fix on their end.

It should work in Boost. The new version fixes that. Give it a try.

Thank you for your hard work!!

I appreciate that you going through this test period. I hope it all goes smoothly and that at least a few hairs remain on your heads by the end of this week. Good luck!

Honestly, I have some problems with this update (and some things I'm super happy about too of course), I hope my feedback is seen so the problems can be made aware:

  1. I'm not a fan of new comments having yellow, the contrast is poor at least on darkly-compact and it just looks weird and out of place. I'd suggest maybe making them a slightly lighter or darker shade of the existing color to make them stand out, not an entirely different color.
  2. It seems the main page/posts pagination uses pageCursor instead of page number now, whereas profiles still use page number. Using pageCursor is an SEO downgrade and usability downgrade, it's a weird decision IMO if that was intentional and not a bug.
  3. I think I might have run into a bug, Lemmy no longer seems to save my upvotes or something. When I upvote some posts on the main page/posts list, even if I wait a few seconds, or 10 seconds or even 30 seconds, when I refresh my upvotes are gone. This one seems fixed now? Nevermind, it's broken again, very weird.
  4. With "Top Day" sorting, there used to be multiple pages of posts, now sometimes I don't even get a full page of posts: https://i.imgur.com/4LiZzJG.png (it looks like it's showing posts from only one community, I have it set to subscribed communities but I'm literally subscribed to like 50 different communities) but other times I get one full page of posts plus multiple other posts which is how it always behaved before 0.19. Since it seems to be inconsistent, I'm thinking that's a bug.

I would probably suggest on holding off on releasing 0.19 for now, it does seem like a really large release so it might be better to make sure it's fully polished and ready to go.

I'm super glad upvoting comments doesn't discard the comment you're writing, that's a huge improvement. I also see import/export settings, that's super awesome, I absolutely love features like this!

Thank you for your hard work, Lemmy is such a cool platform.

3 and 4 are most likely caused by problems with server-side rendering.

Awesome, glad to hear those issues are aware of! Best of luck with fixing them, it does sound like a challenging bug.

For the new comments having yellow being poor contrast, is that something that could still be fixed? It seems to be really poor contrast and hard on the eyes, a couple of my friends noticed this too.

For the pageCursor not using page number, is that intentional or could it be fixed to use page number again?

I got randomly signed out this morning, maybe this has something to do with that.

Voyager on Android seems to be working fine. 👍

Thank you for the heads up and for all the work you do. Im proud to be a part of the O.G. server!

Cheers. hopefully other apps will catch up a soon as the upgrade gets implemented across the lemmyverse

Is it normal that on an upgrade, users are forced to re-login?

They mentioned something about authentication changes. So I’m guessing it’s related to that.

The 2FA on Lemmy was broken in the sense that if you turned it on, it didn't test you to enter the correct token and then properly activate itself. Thus, it led to many people being locked out of their accounts(unless they were logged in somewhere else as well, in which case they could turn off 2FA). This led to the announcement that 2FA will be reset for users so that incorrectly logged out users can get their access back.

If you use apps, you can't even relogin. Expect this if you're using app like boost, sync etc. However you are okay if you're using Voyager because it is browser-based.

I've been having these issues at lemmy.sdf.org when they update to 0.19, for about two weeks now I think. I use Sync and the dev are aware of the issue. But there's still no fix even though the dev is very quick in fixing bugs before. So I think, this is a major issue for app and it will take quite a while before we will see any fix.

Nope, I only have a lemmy.ml account, and I only use librewolf on the desktop...

I had that issue of having to re-login, and then every now and then when I do reload, it posts not on my subscribed communities as if I were not logged in, although I'm logged in, but on a subsequent reload, then only my subscribed communities posts show up. I haven't had to re-login again, just after the upgrade, and then the other minor annoyance is that, but a subsequent reload is enough...

Pff, I just got logged out on a refresh, hehe. So what I said is not true, I'm getting logged out, somehow randomly (can't say I know under which conditions that happens). Not sure if there was a release candidate upgrade, since I can't remember the version before this new logout. Currently:

UI: 0.19.0-rc.11
BE: 0.19.0-rc.10

When I vote on a story while not logged in, it silently fails on 0.19-rc8

v0.19 still doesnt have the "subreddit" name in URLs?

also, I keep getting logged off

Screw community names, I want instance agnostic URLs for posts and comments.

Right now lemmy.ml/comment/123456 and lemmy.ml/comment/123456 are two different comments, and there is no simple way to find one comment on another instance (so you can interact with it from your logged in account). What we should have is lemmy.ml/comment/123456@lemmy.world to point to a comment made on another instance, then you can just change the instance name after the @ to find the comment (or post) on any other instance.

How can two urls that are exactly the same point to different content?

Comment IDs are not unique between instances so you simply cannot change just the part after the @ to link to the same comment but accessing it from another instance.

You can always take the comment’s full URL and put in in the search field of your instance, this will show that particular comment in your instance so you can interact with it from there.

I've saw a weird phenomenon where someone else user was written in place of mine for a second when I opened the web page. I was not connected, one a PC using Firefox.
Could it be related ? Is this the right place to signal it ?

I think someone we've seen this bug a long time ago, but its difficult to replicate.

This has happened to me a couple times before this.

This happened to me once, shortly after joining lemmy. I even took a screenshot as proof :) Glad somebody else can confirm it's actually a thing.

It had only lasted for a second. I hadn't time to take a screen shot but that's weird to feel like someone else has connected to your computer.

I was about the blame one of the android apps developers for breaking comments.

edit: This one was written from a browser. Apps don't like this new version.

Yeah, Infinity hates it. I've still got a Firefox browser app pinned that works fine though.

Nevermind, that forced me to look for other apps and after checking them all I've found one that fits me the best and mostly works.

Connect on Android doesn't work.

Connect dev here, I'm monitoring and releasing a fix (v152)

When I open lemmy, it automatically seems to go to the global screen, but still displays the "subscribed" tab as being selected.

I'm subscribed to neither the piracy, nor the world news nor the gay news communities.

1 more...

Boost for Lemmy (v1.0.3 Nov 16, 2023) seems to be having issues with this 0.19 release.

While using Boost, I stayed signed in, but my feed while set to Subscribed was showing posts not under my subscriptions. I blocked most memes communities for example yet they all showed back up, filtering by Subscribed or All they appear like identical feeds. After I signed out, to see if it made a difference, I am unable to sign back in. I'm posting this from Firefox/the web browser instead of the Boost app.

Same for me. Couldn't see my subs, now can't use the app anymore.

Is it normal that opening comments now completely refreshes the page instead of automatically loading the page semi-dynamically? if I click the comments button on beehaw the whole page doesn't have to reload, and it's actually faster. This seems like an annoying regression, is there a technical advantage to this?

edit: on further analysis, it doesn't do a full page load when you go to your username > profile

or when you click on comments from the link, so, I think this is actually a bug.

I seem to have been screwed over by TOTP.

Hearing that this update was supposed to make borking your account harder to do when setting it up, I enabled it. Put the secret in my authenticator app, got my six digit code, and away I went.

Now, a few days later, having changed nothing on my end, Lemmy.ml won't accept my TOTP code. My session token on desktop is expired so I can't remove it now.

Currently my only lifeline to this account is my logged in session in Voyager, which, as far as I can tell, cannot access the TOTP setting. (Or any profile setting, for that matter... am I just stupid?)

No email to recover from, either. That's on me, I guess. Ugh.

Not sure what my recourse is, if I even have any.

Have you tried logging in through other apps to see if they’ll take your TOTP?

Connect, Sync, and Boost all told me to go kick rocks.

Evidently, whatever happened, it doesn't seem to be an issue with your platform.

Ok, and you’re getting a new 6-digit code from your authenticator app every time you attempt to log in?

Yes.

I noticed my authenticator app (KeePassXC) offers the ability to customize the TOTP parameters (SHA function, time step, code size). But no combination of settings seems to produce a valid code.

I assume Lemmy uses the suggested defaults in the RFC 6238 standard?

I think using an authenticator app capable of generating codes using SHA256 might do the trick if you have any possibility to try that.

One problem is that a few communities, while they do show up in my subscriptions list, do not indicate that I am subscribed to them when I am on the community page (for example, !asklemmy@lemmy.ml is one of those communities where I noticed that issue, it mostly fluctuates between me being subscribed to the community or not). If it needs to be marked as an issue on GitHub, you can let me know.

Its probably due to a server-side-rendering related bug we're working through rn.

Thought I'd mention a problem (re: Apps will likely break (only for lemmy.ml)) I am developing an app and this one request stopped working:

https://lemmy.ml/api/v3/community/list?type_=Subscribed&auth=[MY_AUTH]&username=[MY_USERNAME]&limit=50

Any other type_ works. This one used to return only a users Subscribed communities but it just returns an empty array {"communities":[]}. When you load it up in a browser it works. I'm driving myself crazy

Was sort by Controversial supposed to show most Controversial of all time or what is currently most Controversial?

All time, since it doesn't take time into account.

I think it would be a good improvement to have 24 h / 1 week / all time

Is anyone else unable to see a list of their own posts in their profile area? For example, I have several posts in /c/caps@lemmy.world that aren't showing up in my profile.

My profile: https://lemmy.ml/u/yo_scottie_oh?page=1&sort=New&view=Posts

An example of a post that I'd expect to show up there: https://lemmy.ml/post/8549237

Which, oddly is the same as this post: https://lemmy.world/post/8833534 (but this might be off topic).

I can see posts in other people's profiles, e.g. https://lemmy.ml/u/KindnessInfinity?page=1&sort=New&view=Posts.

Anyone else not able to see this pinned post on every second refresh?

  • web interface
  • Subscribed, Hot

The site feels faster, the notifications are more reactive, and moderator view is a godsend.

As for now, the only new bug is fairly minor, markup-related. Examples here and here, screenshot:

Before the update, "há" "a" and "haver" would use a plain highlight colour (they were using the `code` markup); "dog**'s**" and "cat**'s**" had bold face.

I'm using the Liftoff app, which is broked.

On the main page it says "Something went wrong" and has a Try Again button which leads back to the same failure.

On my user profile under settings it says:

Error: type 'Null' is not a subtype of type 'int' in type cast

Please report such problems to the developers so they can fix them.

Has anyone experienced trouble logging in through Jerboa app? Jerboa crashes every time I try to log into this instance, but other instances log in fine.

Creators of Lemmy, owners of this instance, and creators of the Jerboa app are all the same people. As I understand it, Jerboa won't be getting an update until they're happy with the stability of the update to Lemmy itself. Lemmy is getting all the focus first. Also, no sense in pushing a client update for a server update that itself is not finished.

Getting really annoyed at that error toast in Jerboa, though, not gonna lie. And not being able to post on mobile. At least I can still read posts and comments, though. Hopefully the patch will release soon.

Yeay this was the update that killed Jerboa for me.

Now on Voyager which seems much nicer. And actually works.

Thank you for the suggestion of Voyager, I have installed it and it looks quite nice!

Yes, the exact same was happening for me. Jerboa would crash and android would show me the notification "this app has a bug" whenever I tried to login. Now using Eternity for Lemmy and that is working smoothly.

EDIT: all good, I worked it out, I found some other api docs.

Is the ModeratorView listing type in this release? If so what feed view is it available in? I can't see it in the lemmy.ml webUI at the moment, nor in the API docs, but the code is merged.

I have two accounts, one on feddit.de which runs stable all the time. But my older account here on lemmy.ml seems broken in the last week:

  • I can't see my own new posts in profile, comments work.
  • post and comment upvote-count is at zero.
  • log in with jerboa let the app crash.

Edit: last one got fixed with new jerboa update

I keep getting the "failed to verify JWT token" and am using a mobile browser to log in/post.

I was having this issue initially as well, but today I noticed my posts are back. Are yours back, too?

I read your comment, checked my profile, and you are right, posts are back.

lots of new bugs... including rss stopped working, getting logged off for no reason

The logout is normal, it's because there was an issue with the way logins worked before, that shouldn't happen on future releases, but that is to be expected this release.

Why does Lemmy randomly signs me out? This happens on Vivaldi 6.4.3160.44

There was a one-time sign-out with this release due to an issue with logins, see if the issue is still occurring for you, if it is, it's likely a browser-side configuration issue.

I'm having issues using the instance block, it just shows the first couple of instances in the list no matter what I type in.

You think it would be possible to add a "block instance" button from within the /instances menu?

Ever since the upgrade, lemmy feels dead

Its cause lemmy.ml doesnt show up on some of the apps now. I have to use my browser to use my account when i was using liftoff and boost before. Growing pains are gonna pain, but they are necessary.

1 more...