What happend to lemmy.ml? EDIT: Works with ipv6 disabled

Sören@iusearchlinux.fyi to Technology@beehaw.org – 26 points –

It seems like they are down for a longer time now. How will they recover? Does longer down mean they will have to do more catching up with other instances? Can I get updates somewhere?

55

Lemmy.ml did a server migration which was posted about 8 hours ago which was pinned on lemmy.ml meta. It appears in the comments that some users have experienced a 502 error, which was related to “iCloud’s Private Relay”. They were able to bypass the issue using a VPN, although I haven't had any issues accessing lemmy.ml on my pc. I also read something about AWS us-east-1 being down, so it could also be related to that.

@Sleeping I still wanted to follow a community there from my instance and I could not.

@soeren

yeah our instance is having problems too with lemmy.ml communities

Following communities could be another issue entirely, sometimes you have to subscribe then unsubscribe multiple times in order to get it to work.

@Sleeping I can't subscribe in the 1st place. Guess I'll just have to note down the community and try subscribing tomorrow again as it's late around here 😁

Oh! Hmm, I've never run into that issue before. Although I have noticed on some instances it works faster and more reliable than others.

@Sleeping I am on Friendica. It shows like this.

I've never used Friendica, so I'm honestly not sure. Maybe this could be related to the ipv6 issue that the instance hoster would need to fix.

It's weird being part of broad communities like technology or games again. On reddit I unsubbed all the default subs and subscribed to more niche things like 3D printing or weird games. It's quite a change to see people talking about things like mainstream AAA games. Intellectually I knew that a lot of people had to get hyped about the latest Bethesda game or Apple products or whatever, but I think part of me stopped believing it, because it still surprises me whenever I see it.

I also can't stand all these fuckin' normies. 🙃

I know, right? Next thing you know I'll end up touching grass or something.

This appears to have somehow ended up in the wrong spot. I hadn't even clicked on this thread before, and yet here's my reply to a different one. Ah well.

I've seen a couple of people mention this bug. I'll check if there's an issue in their bugtracker for it.

Retro games and patient games were pretty good subs.

Yeah, I'm a mod of retrogaming and a few few other subs in that network. Best group of users ever. 320k people in just the one sub and I think as a team we had to take down maybe 3-4 posts a day? Maybe? I banned 2-3 people in a month, that's absolutely unheard of in a sub that size. I think the worst argument I saw the whole time was a running "the playstation one was not called the PSX" "Yes it was" "NO IT WASN'T!" "HERE IS AN ARTICLE, YES IT WAS." thing that we had to specifically make a rule about. Odd little drama, but other than that? No fights at ALL. I'm talking a report maybe every few days for something being off topic if we didn't get to it quick enough.

The niche subs about fringe interests, even if they're popular ones, often bring out the very best people. And the patient/retro gamers are so very chill.

Not sure where they host but AWS is currently in the middle of a pretty big outage so it could be related.

It looks like the fix for this is to disable IPv6 - not all of us can do that (or want to do that).

Lemmy.ml will need to fix their AAAA record likely.

Used Reddit for 13 years, tried out Kbin and Lemmy yesterday and settled on Lemmy.

Long story short, I'm going back to Reddit.

  1. There needs to be ONE site, Lemmy.com, that people goto. This entire thing about having .whateveryouwant is VERY off putting. Most internet users have been trained to be extremely wary of odd or unusual things, so having anything besides .com/.net/.org will turn away a huge portion of users.

I initially setup an account on Lemmy.world, then realized that I couldn't migrate it to another server and that when I deleted that account on that server all my comments were deleted.

  1. Deciphering the distributed nature of it took me, a relatively tech-friendly person, almost the entire day and several 'What the fuck?' posts. I now understand it more. There are some very low-level guides that have been haphazardly put together, but there absolutely needs to be a MUCH smoother guide/explanation to this whole thing. That learning process will turn people away for sure.

  2. BECAUSE I understand it more now, I'm left feeling VERY uncomfortable about my data security. If this is going to become a mainstream thing, as it reaches and before it gets to that critical mass of users, there's going to be SO. MANY. SECURITY ISSUES. There's no 2fa at all, hacking and user-account hacking is just going to run rampant, and I'm left wondering 'Where is my username and password actually stored?'. The answer, sadly, is wherever the dude who's running the instance/server is. In the 'Fediverse' your server instance might be hosted in a US or EU data center with proper digital and physical security, or it could be Joe Blows basement in Iowa running off a NAS. The easy-to-see future here is that Lemmy will fail to attract a critical mass of people because they'll initially arrive, after a few months their instances will just cease to exist/get shut down/the hosts will decide its no longer a fun hobby to do.

With a large corporation, they have the staff and resources to secure and maintain the servers physically and digitally, and keep staff up-to-date on current infosec threats and get out in front of them. Beyond that, if there IS a breach, they have the ability to recognize it, understand the legalities and requirements of reporting it, and can be held accountable by regulatory bodies. Joe doesn't have the resources to really maintain and keep a server running, nor the knowledge of his responsibilities for keeping the data safe digitally or physically.

On top of that, if Joe's basement loses power/gets hacked/Joe decides he's moving to San Fransisco and can't bring his NAS with him and the server goes down, and that's where my instance is hosted well there goes my entire account/comments/data.

  1. Finding and subbing to communities is painfully difficult. It should be one-click, but somewhere I need to goto an external list, find what I want, and then copy/paste the URL into the search... and then 50% of the time, it doesn't work. This is an understandable growing pain and can likely be fixed by UI/UX upgrades, but for now it's a definite turn-off.

  2. There simply is no content. I'm not a creator, I want content aggregated for me, and I've gotten used to having a single place to get it from that floods me with thousands of different articles/memes/posts/etc every minute. Until the user base arrives in one single place and starts generating content, there's no reason for most people like me to be there as by far the larger number of users never create anything at all and only exist to consume the content generated.

Thats not the point of being federated. It is very simple to understand once you get over the learning curve. The analogies to countries/states communicating to each other but being seperate helped me

See, this is yet another issue with Lemmy. This is not the post I replied to. I replied to https://lemmy.ca/post/636721, "Redditors, how do you like Lemmy?", and yet it posted it in a completely different topic!

This is very insightful, even though it ended up here :) thank you!

Sites don't store passwords, they store password hashes. There is no reason to give any personal info you aren't comfortable giving. You can use the site just fine without posting any

Hacking an account is still a valid concern though for various reasons , and hashes can still be used against password lists. Additionally, Two factor authentication is a necessity for sure. Now don’t get me wrong, I completely understand this feature is coming and that this is a developing service but many of these concerns do seem valid to me.

Hacking an account is still a valid concern though for various reasons

Let's assume you're doing the best practice thing and using a long and unique password for each service you use.

What benefit does a hacker have hacking your lemmy-based account? Considering that everything you post is public... There's simply nothing of value that you would obtain by "hacking" an account here... The only thing I can think of is if your a moderator of a community or an admin of an instance.

I just don't see any value to it... But even then.. 2fa is slated for v0.18 which is probably coming out in the next few weeks.

2fa is slated for v0.18 which is probably coming out in the next few weeks.

Only basic TOTP 2FA though. Webauthn/FIDO2 should be coming in the future though.

Mostly thinking impersonation, spamming, deletion or modification of history…. Although I’m sure there are probably other reasons too.

Impersonation - Not sure this matters unless you're a mod or admin. Spamming - Just make the accounts yourself.... it's going to be infinitely easier to just make spam accounts from nothing (since it's free anyway) than to designate resources to cracking a password. Deletion/Modification of history - modlogs allow reverting ALL changes outside of full account deletion. Full account deletion - Well that's annoying at the very least. But not like it's the end of the world or has any actual cost associated with it.

It's a lot of work to do so little actual damage. It's not like twitter where hacking Elon's account can actually lead to monetary gain.

What makes you think large social media platforms don't have these exact problems? Because they definitely do.

This software being open source will hopefully make it more secure one day than even things like twitter. This is because everyone can see and inspect the source code and try to find vulnerabilities. When they are fixed then (hopefully) all of the instances get updated. It's what helps make Linux generally more secure than Windows.

It seems to me you don't see the value in open source platforms like this. If this is true then fair enough. Just don't come crying to me when reddit does something you don't like.

There needs to be ONE site, Lemmy.com, that people goto. This entire thing about having .whateveryouwant is VERY off putting.

The main feature of a federated system is that there's no one "owner" of the entire system; each admin uses their own servers and their own domain names. There's nothing stopping a server admin from using a .com domain name...

There’s no 2fa at all

There's basic TOTP 2FA now: https://github.com/LemmyNet/lemmy/issues/2363 but it hasn't been released yet (it's only available in the latest server beta). "Next-gen" TOTP (Webauthn/FIDO) is coming later: https://github.com/LemmyNet/lemmy/issues/3059

‘Where is my username and password actually stored?’. The answer, sadly, is wherever the dude who’s running the instance/server is. In the ‘Fediverse’ your server instance might be hosted in a US or EU data center with proper digital and physical security, or it could be Joe Blows basement in Iowa running off a NAS.

You should get to know your server admin, then. You have the freedom to pick any server you like :)

There simply is no content

Have you subscribed to many communities yet? You can browse other Lemmy servers to find interesting communities, and subscribe to them on your instance.

A large open source project is generally more secure than a closed source one as it has more developers working on it and more security experts reading the code or testing it. If you want an example look at how many developers are working on Linux vs Windows and there security history. Once lemmy is big enough it has the potential to be more secure than things like Facebook or Twitter even.

You apparently don't understand how password hashes work. Nobody is storing your password on any well implemented website. You can't just grab someone's password if there is a breach, provided the people who built the website aren't completely incompetent.

With these misunderstandings and the fact federation was confusing to you I am not sure I believe your claim that you are a "tech-friendly" person. You sound like someone who has above average exposure to technology rather than someone who is actually skilled in it. There is nothing wrong with this in itself but please don't claim to be proficient when you aren't.

I’m not sure how you’re being downvoted, you make some great criticisms of the fediverse. Maybe staying small is better for the communities than reaching a critical mass as that only attracts corporate sponsorship (VC/ads/etc.). Working out the security should be the initial goals for any interested programmers who venture over.

I noticed that there's one thread on lemmy.ca, made by an account on lemmy.ml, that I just absolutely cannot reply to. It hangs, and nothing gets posted. I'm able to reply to other threads in the same community.

How exactly are these things structured? Like, could that issue be related to lemmy.ml doing its migration, even if the topic was posted on another instance? It was very weird behavior.

In short, yes it will take longer to "catch up".

The longer answer, be patient. This is an issue that will be resolved. I'm sure that the admins are doing their best to get the instance up & running ASAP.

From my experience during the great Twitter migration to Mastodon, you saw the same thing. Especially the "main instance(s)", mastodon.online and mastodon.social in this case, were overloaded to the brim. After a day or two, the sysadmins and developers were able to stabilize the instance.

The sysadmins in this case were responsible to keep the servers up and running (and scale up where needed). The developers because their code hasn't been "battle tested" for this level of interaction. So changes were needed, to optimize and squash bugs.

I hope this helps 😊

Edit: fat fingers, ie fixed typo

This is a fantastic point. I watched several migration waves from Twitter to the Mastodon/*oma/*key network over the course of Nov-Feb and each became less impactful as instances and developers understood better how that traffic would flow. Consider also that those platforms had also been around for a few years already.

Lemmy and kbin are very, very young still, so it will be great to see them develop over the course of the coming months. I expect the next couple weeks leading to the June 30 3PA closures on Reddit will be spent preparing for another wave.

Oh definitely that the next couple of weeks will be spent preparing for a new wave. I wouldn't be surprised if that was the reason why Lemmy.ml was down. Yes I know that it was because of a server migration but, that server migration also didn't happen for no reason ;-)

The first migration wave to Mastodon was very bad. I still remember, that following a person was nearly impossible. Not to mention the fact that, even if you could follow someone, it didn't automatically meant their messages ended up in your feed.

I know for example that version 0.17.4 of Lemmy (that was released 2 days ago) already has some database optimizations.

I get 502 in germany with mobile data and wifi

Are you on an Apple product, since that could be related to the “iCloud’s Private Relay” issue, since you're getting a 502 error?

i tried with pixel 7 and macbook but both didnt work. Private Relay is disabled.

Working for me (West US). It is actually working really well. I think this is the fastest it has ever been.

Edit. Interesting I tried from another location and I am getting a 502 Bad Gateway error. I tried to do a ping and Lemmy.ml is resolving to a IPV6 address at that location. vs my current location is IPV4 address.

I disabled IPV6 and now it is working on the other location.

Not sure if it is an issue on my end or something on Lemmy.ml.

It's down for me too - southwestern U.S.

AWS Outage should be resolved now (it impacts my job, I would know).

IMO, looks like something else is up with this.

Apparently disabling ipv6 should make it appear again.

I haven’t yet looked at the Lemmy architecture of the server, but it should be fairly straightforward to have a redundant region on AWS and use Route53 for the failover.

Working for me (from UK).

EDIT: seems they migrated to a new server earlier today to handle their increased load. Maybe a hiccup related to that, or your DNS server is still pointing to the old instance or something (in which case it should eventually fix itself)