A reminder to move to smaller instances for a better experience

A reminder that this constant advice people blindly parrot to install and flock to smaller instance has now created something like 1000 new servers in 50 days that are poorly run and already going offline as quickly as they went online.

Github Issue 2910 is the kind of PostgreSQL problems that the developers ignored for months and people still defend the developer choices to have the code doing real-time counting of every single comment and post for numbers nobody needs to needs done in real-time.

PostgreSQL is voodoo to this project, they do everything they can to avoid going to !postgresql@lemmy.ml community and asking for help, learning 101 about how to fix their SQL TRIGGER logic like Github Issue 2910 spelled out June 4.

Up until now, social containers like groups, communities, or subreddits on all the largest social networks have existed as fundamentally separate locations on a single hierarchical level.

"Up until now"... Uh... no, Usenet... was the open standard for social media. Created in 1979. A foundation of the Internet. Just as much as e-mail was.

alt.tv.simpsons
alt.tv.futurama

the module can cause intermittent stuttering, depending on which Ryzen processor you're using. It appeared when the fTPM was in use, it would access its flash storage via a serial interface, and when doing so, held up activity by the rest of the system.

Great, thank you. The network is much more stable and working solid!

Query speed is Lemmy’s main performance bottleneck, so we really appreciate any help database experts can provide.

I have been pleading that Lemmy server operators install pg_stat_statements extension and share metrics from PostgreSQL. https://lemmy.ml/post/1361757 - a restart of PostgreSQL server is required for the extension to be installed. I suggest this be part of 0.18 upgrade. Thank you.

The project had gone on for 4 years without a lot of testing... old code like login form had all kinds of problems, etc. Lemmy-ui had almost no ability to cope with errors from the backend, and often error messages didn't even exist for the API calls. There was a huge rush to fix so many areas that were outright not working.

I turn the question around... people who are clearly liars, deceivers... politicians and businessmen that people line up to vote for with their money or public votes. You really wonder what people think an "asshole" is when you see the kind of politicians that get massive support in a population - to a point people have their photograph on the wall of their workplace or home, put stickers on their cars, etc. to support people that are clearly monstrous. A lot of people do not seem to like to study the crowds of Europe 1930's terrible leaders and just how many lined up to cheer on such persons.

The scientists a person believes also is a huge indicator of who they consider to be an 'asshole'. Just passively listening to people who support denial of climate change, denial of microscopic germs and virus, etc. The enthusiasm that followers to non-factual science seem to be very high, and they draw crowds in ways that fact-based science does not seem to do.

Why is there a lack of gifs/videos on Lemmy?

Lemmy's internal data performance is so horribly slow and crash-causing that I think the last thing they want is even more popular data.

Video is simply the most superior type of media there is, and I think that not having easy access to it on Lemmy is hurting it.

Video is more data, popularity is more data. For whatever reason, at every turn, I've seen developers turn away from scaling options like Memcache, Redis, or just abandoning ORM data management and rewriting the data interfaces by hand....

since the sites on which the videos are hosted can track you.

That's already true for images that are hot linked routinely, so I don't think video really changes it.

I've been baffled since June why data and fixing lemmy's data coding hasn't been front and center. It's pretty wild to witness so many come to Lemmy and then turn away... Elon Musk has been flocking people, Reddit, etc. It's as if the project wants to make code that won't work on any data. It's baffeling.

On the technical topic of renaming a domain of a Lemmy server... I think it is worth experimenting with the code. At minimum, I think it should be an option to try and keep the same login/passwords for users from the old install of Lemmy. But even that could prove tricky if a particular domain changed underllying ownership more than once - and user@domain became rewritten by an entirely different person. I guess in the real-world people do often get mail for previous residence of a house.

My biggest concern is legality because Lemmy claims to support privacy. I honestly think it's a bad idea to claim privacy because you run into so many problems. If the user never knows that their lemmy instance changed names and can't find it again, etc. Especially on technical topics, 15+ years of having Reddit keep messages from deleted user accounts offered a lot of great search engine hits. With Lemmy, a person moving to a different instance and deleting their account, so much content is going to get black-hole in favor of 50 instances having copies of a meme post or trivial website link - and solid original content (often in comment discussions) gets removed.

That feature you linked to is to flair users.... there is a different issue to flair posts: https://github.com/LemmyNet/lemmy/issues/317

There is no built-in “real-time” methods for admins via the UI to identify suspicious activity from their users, I am only able to fetch this data directly from the database. I don’t think it is even exposed through the rest api.

The people doing the development seem to have zero concern that their all the major servers are crashing with nginx 500 errors on their front page under routine moderate loads, nothing close to a major website. There is no concern to alert operators of internal federation failures, etc.

I am only able to fetch this data directly from the database.

I too had to resort to this, and published an open source tool - primitive and non-elegant, to try and get something out there for server operators: !lemmy_helper@lemmy.ml

I miss the days when the internet was populated largely by nerds aiming to make a better world

The BBS and early Internet days were dominated by people who read non-fiction books. RTFM was a common saying in those days.

does anyone else feel enslaved?

“Everything in our background has prepared us to know and resist a prison when the gates begin to close around us . . . But what if there are no cries of anguish to be heard? Who is prepared to take arms against a sea of amusements? To whom do we complain, and when, and in what tone of voice, when serious discourse dissolves into giggles? What is the antidote to a culture’s being drained by laughter?”
Neil Postman
Amusing Ourselves to Death

Up until early July, Lemmy was damned if you do, damned if you don't. Federation had massive performance overhead due to some bugs and each additional instance that went online and subscribed to the big 4 popular servers was causing an even worse load problem than if say 30 users had joined directly. Especially instances that wanted a fully populated All listing, that meant every single thing was being sent to the server even if nobody was really reading that stuff.

And things like searching for topic content are going to be pretty limited given these newer servers don't have much history.

The aftermath of this attempt to scale is that there is also likely a lot of duplicate data, conversations that are mostly repetitive and posts to the same topics. Let alone the bugs Lemmy has federating deletes and moderation removal that doesn't impact direct users on the main servers as much.

The "Hot" sort topic:

Let the servers keep crashing, tell everyone to add new instances to help with performance, which puts 1500 rows into the database tables that used to have 50 rows and invokes a massive federation 1-vote-1-https overhead... causing more crashing... all the while ignoring the SQL design of machine-generated ORM statements and counting logic hidden in the background triggers.

... keep users off your sever as a method of scaling by crashing. It's one of the more interesting experiences I've had this year! And I spent all of February and March with the release of GPT-4... which was also interesting!

This basically shuts my idea down

it's not very difficult to modify the code for something like this.... and closing off registration wont' let anyone else login and create new content form your istance.

Personally the load on the major servers by having one more instance that subscribes to everything is why I think people should back off from creating more than the 1500 instances Lemmy network already has. Delivery of every single vote, comment, post 24 hours a day just so one person can read content for an hour or two a day.

That makes sense for email systems where all that content doesn't have to be sent, but for Lemmy it's a huge amount of overhead.

Another instance was hacked too: https://lemmy.blahaj.zone/

The JWT are likely a hot issue, already some Issues on GitHub about them not being revoked properly.

Heads-up that a new issue on GitHub reporting this problem upgrading from 0.18.2 to 0.18.3:

thread 'main' panicked at 'Couldn't run DB Migrations: Failed to run 2023-07-08-101154_fix_soft_delete_aggregates with: syntax error at or near "trigger"', crates/db_schema/src/utils.rs:221:25 note: run with 'RUST_BACKTRACE=1' environment variable to display a backtrace

https://github.com/LemmyNet/lemmy/issues/3756

If you use "Lemmy from scratch" install,

you may be on an older PostgreSQL that Ubuntu distro provides, not the same as the Docker install method.

Things have been incredibly unstable there.

I wish lemmy.ml (also unstable) or lemmy.world would hand out a (nearly) full copy of the database so we can get more analysis done on PostgreSQL performance behaviors. Remove the private comments and password /2fa/user, or whitelist only comments/posts/communities/person tables - but most everything else should already be public information that's shared via the API or federation anyway. it's the quantity, grouping, and the age of the data that's hard to reproduce in testing. And knowledge of other federated servers, even data that may have been generated by older versions of Lemmy that new versions can't reproduce.

It's been over 60 days of constant PostgreSQL overload problems and last week Lemmy.ca made a clone of their database to study offline with AUTO_EXPLAIN which surfaced a major overload on new comments and posts related to site_aggregates counting (it was counting each new post/comment against every known server, not just the single database row for a server).

I have an account over on World too, and every major Lemmy server I use throws errors with casual usage. It's been discouraging, I haven't visited a website with this many errors in years. Today (Sunday) has actually been better than yesterday, but I do not see many new postings being created on lemmy.ml today.

Since Saturday: https://lemmy.world/post/2462635

It was a big deal when we got an archive we could search of all content...

"The Deja News Research Service was an archive of messages posted to Usenet discussion groups, started in March 1995 by Steve Madere in Austin, Texas. Its powerful search engine capabilities won the service acclaim, generated controversy, and significantly changed the perceived nature of online discussion. This archive was acquired by Google in 2001."

the people who run Lemmy don’t have the money to support a fleet of failover servers that take over when the main server goes offline.

That has nothing to do with the issue I'm talking about. Every server with the amount of data in them would fail. Doesn't matter if you had 100 servers on standby.

The Rust logic for database access and PostgreSQL logic in lemmy is unoptimized and there is a serious lack of Diesel programming skills. site_aggregates table had a mistake where 1500 rows were updated for every single new comment and post - and it only got noticed when lemmy.ca was crashing so hard they made a complete copy of the data and studied what was gong on.

Throwing hardware at it, as you describe, has been the other thing... massive numbers of CPU cores. What's needed is to learn what Reddit did before 2010 with PostgreSQL.... as Reddit also used PostgreSQL (and is open source).

That’s basically the only reason you don’t see lots of downtime from major corporations: investment in redundancy,

Downtime because you avoid using Redis or Memcached caching at all costs in your project isn't common to see in major corporations. But Lemmy avoids caching any data from PostgreSQL at all costs. Been that way for several years. May 17, 2010: "Lesson 5: Memcache;"

As I said in my very first comment, server crashing as a way to scale is a very interesting approach.

EDIT: Freudian slip, "memecached" instead of Memcached

I've largely given up on pull requests.... for sake of sanity. But I waded back in...

I made a pull request today... and I very strategically choose to do it with minimal of features so that it would just go through... and I got lectured that JOIN is never a concern and that filtering based on the core function of the site (presenting fresh meat to readers) was a bad use of the database. I've never seen hazing on a project like this. Memcached and Redis should be discussed every day as "why are we not doing what every website does?", but mum is the word.

Thank you for sharing.

Is your intention to have local copies of content from popular servers and read it locally? Major communities like news., memes, etc?

Many people seem to think this is offloading the major servers like lemmy.world - but I think the opposite is true in my measures of how lemmy_server performs. There is a lot of overhead to each additional instance in Lemmy 0.18.3 backend. Lemmy code does a lot of work to keep each of these subscribing servers updated with every post, comment, vote, person - attempted in real-time.

I already feel like I have to keep sticking my neck out to get them to question if using the ORM and a dozen JOIN statements isn't a problem.... but I guess I'll link it: https://github.com/LemmyNet/lemmy/pull/3900

As stated on my Lemmy user profile, I'm "RocketDerp" on GitHiub.

Honestly, the reason I keep making noise is because I'm sick of Lemmy crashing all the time when I come to use it... and I am on many servers that this happens. I really am not trying to piss off the developers, I even said I felt like I am being hazed, and I feel like hazing in general might explain what is going on with how much they are avoiding the elephant in the ROOM that ORM and a dozen JOIN might be the cause! Let alone the lack of Redis or Memcached addition being avoided, that's a second elephant on the second floor tap-dancing.... GitHub Issue 2910 was the straw that broke my back weeks ago, it took months for them to address it when it could be fixed in a couple hours (and it was weeks before the Reddti API deadline at the end of June.... and issue 2910 was neglected). The whole thing was a nightmare for me to watch...

Lemmy permission system is very limited, it's a boolean for admin

I’m not sure if there’s a minimum Postgres version specified anywhere in the docs, I took a quick look but couldn’t find it.

I do not find it in the "From Scratch Install" documentation. I will work to revise those documents this weekend to warn people that 0.18.3 and later is incompatible. Thank you again for sharing.

Lemmy is written in Rust, I don't think any other ActivePub platform uses Rust.

I'm seeing zero comments come out of Lemmy.world in the past 15 minutes, app users shouldn't have been redirected... and users commenting from other servers should be going to communities homed there. I wonder if they shut off federation. I normally see over 10 comments a minute: https://lemmyadmin.bulletintree.com/query/comments_ap_id_host_prev?output=table&timeperiod=15

It was cleaned up on the home page, but now back to being defaced as of this comment time.

Another user on the site confirmed this:

only on Kbin? Everything from there seems 1 or 2 hours ahead

I think lemmy.ml restarting the server helped the 'pending' subscribe problem, it started to come back for me once the server had been running an hour or more post-upgrade. It's better, but I still am having some get stuck.

Missing comments and postings are also not as glaring in the user interface as a 'pending' subscribe.

You’re thinking about it the wrong way.

I've had to go through a major change in thinking and adjust my interpretation in major ways.

Technical details, is it the sidebar: https://lemmy.ml/post/1896249

JWT issue opened 4 days ago: https://github.com/LemmyNet/lemmy/issues/3499

But each additional row in site_aggregates table was causing the instability itself. The SQL code had major flaws. Adding more servers actually made Lemmy crash more.