They've already replied with the reasons, but - for future reference - if you want to see specifics of things like this, a censure is often posted to https://fediseer.com. .world's censure of .nl is here

Hi

Pls check how much traffic you're now sending out for every activity - my server is recording that everything from lemmy.world is being 4 times (e.g. 1 Upvote is sent 4 times to every instance that has a subscriber. Those instances will reject 3 of them for being dupes, but it's still a lot to be sending out).

lemmy.ca had a problem where they were sending everything 3 times, and it was because they were running 3 containers, and they all had the same index number, so maybe it's that.

Thanks.

When I've mentioned this issue to admins at lemmy.ca and endlesstalk.org (relevant posts here and here), they've suggested it's a misconfiguration. When I said the same to lemmy.world admins (relevant comment here), they also suggested it was misconfig. I mentioned it again recently on the LW channel, and it was only then was Lemmy itself proposed as a problem. It happens on plenty of servers, but not all of them, so I don't know where the fault lies.

I can't re-produce anything, because I don't run Lemmy on my server. It's possible to infer that's it's related to the software (because LW didn't do this when it was on 0.18.5). However, it's not something that, for example, lemmy.ml does. An admin on LW matrix chat suggested that it's likely a combination of instance configuration and software changes, but a bug report from me (who has no idea how LW is set up) wouldn't be much use.

I'd gently suggest that, if LW admins think it's a configuration problem, they should talk to other Lemmy admins, and if they think Lemmy itself plays a role, they should talk to the devs. I could be wrong, but this has been happening for a while now, and I don't get the sense that anyone is talking to anyone about it.

They'll all POST requests. I trimmed it out of the log for space, but the first 6 requests on the video looked like (nginx shows the data amount for GET, but not POST):

ip.address - - [07/Apr/2024:23:18:44 +0000] "POST /inbox HTTP/1.1" 200 0 "-" "Lemmy/0.19.3; +https://lemmy.world"
ip.address- - [07/Apr/2024:23:18:44 +0000] "POST /inbox HTTP/1.1" 200 0 "-" "Lemmy/0.19.3; +https://lemmy.world"
ip.address - - [07/Apr/2024:23:19:14 +0000] "POST /inbox HTTP/1.1" 200 0 "-" "Lemmy/0.19.3; +https://lemmy.world"
ip.address - - [07/Apr/2024:23:19:14 +0000] "POST /inbox HTTP/1.1" 200 0 "-" "Lemmy/0.19.3; +https://lemmy.world"
ip.address - - [07/Apr/2024:23:19:44 +0000] "POST /inbox HTTP/1.1" 200 0 "-" "Lemmy/0.19.3; +https://lemmy.world"
ip.address - - [07/Apr/2024:23:19:44 +0000] "POST /inbox HTTP/1.1" 200 0 "-" "Lemmy/0.19.3; +https://lemmy.world"

If I was running Lemmy, every second line would say 400, from it rejecting it as a duplicate. In terms of bandwidth, every line represents a full JSON, so I guess it's about 2K minimum for the standard cruft, plus however much for the actual contents of comment (the comment replying to this would've been 8K)

My server just took the requests and dumped the bodies out to a file, and then a script was outputting the object.id, object.type and object.actor into /tmp/demo.txt (which is another confirmation that they were POST requests, of course)

A bug report for software I don't run, and so can't reproduce would be closed anyway. I think 'steps to reproduce' is pretty much the first line in a bug report.

If I ran a server that used someone else's software to allow users to download a file, and someone told me that every 2nd byte needed to be discarded, I like to think I'd investigate and contact the software vendors if required. I wouldn't tell the user that it's something they should be doing. I feel like I'm the user in this scenario.

Most likely reason is that you unticked 'English' as a language you understand when you were playing around.

I've since relented, and filed a bug

Just for clarity: it's not viewable through the API. As others have said, you need to spin up an instance. In contrast to the API, this means it's not free (due to server hosting and domain name costs), and it's not necessarily easy (for the non-techies).

I've not had any luck joining any of these rooms. I click the link, continue with Element (the web app), I can see that I'm logged in and can access a room I've already joined (for moderating another community), but for these rooms, I'm just getting:

#lwgeneral:h-y-p-e-r.space is not accessible at this time.
Try again later, or ask a room or space admin to check if you have access.
M_UNKNOWN was returned while trying to access the room or space.
If you think you're seeing this message in error, please submit a bug report.

EDIT: the updated links work for me now!

We were typing at the same time, it seems. I've included more info in a comment above, showing that they were POST requests.

Also, the green terminal is outputting part of the body of for each request, to demonstrate. If they weren't POST requests to /inbox, my server wouldn't have even picked up them.

EDIT: by 'server' I mean the back-end one, the one nginx is reverse-proxying to.

I'm only running one process, I'd assume the problem isn't happening for Feddit.dk.

Perhaps. The lemmy.ca post has a comment in from the mander.xyz admin who's only running one, and there's a new comment in this thread saying mander.xyx is one of the instances they see the most duplicates from.

Firstly, sorry for any potential derailment. This is a comment about the Markdown used in your post (I wouldn't normally mention it, but consider it fair game since this is a 'Fediverse' community).
The spec for lemmy's spoiler format is colon-colon-colon-space-spoiler. If you miss out the space, then whilst other Lemmy instances can reconstitute the Markdown to see this post as intended, Lemmy itself doesn't generate the correct HTML when sending it out over ActivityPub. This means that other Fediverse apps that just look at the HTML (e.g. Mastodon, KBIN) can't render it properly.
Screenshot from kbin:

Also, if you add a horizontal rule without a blank line above it, Markdown generally interprets this as meaning that you want the text above it to be a heading. So anything that doesn't have the full force of Lemmy's Markdown processor that is currently trying to re-make the HTML from Markdown now has to deal with the ending triple colons having 'h2' tags around it.
Screenshot from piefed:

(apologies again for being off-topic)

okay

Yeah, that's the conclusion I came away with from the lemmy.ca and endlesstalk.org chats. That's it due to multiple docker containers. In the LW Matrix room though, an admin said he saw one container send the same activity out 3 times. Also, LW were presumably running multiple containers with 0.18.5, when it didn't happen, so it maybe that multiple containers is only part of the problem.

Lemmy instances won't search outside of their own databases if you're not logged in.

But if you are, what it does can be recreated on a command-line by doing:
curl --header 'accept: application/json' https://nerdica.net/.well-known/webfinger?resource=acct:nate0@nerdica.net | jq .

This shows that your profile is at https://nerdica.net/profile/nate0. Lemmy puts all users at a /u/, but using webfinger means that other fediverse accounts don't have to follow the same structure. For lemmy.world, you're at https://lemmy.world/u/nate0@nerdica.net in the same way that a mastodon user is at e.g https://lemmy.world/u/MrLovenstein@mastodon.social.

edit: However if you webfinger your mostr.pub account, you get: {"error":"Invalid host"} so any ActivityPub instances will only ever be able to find you if you've interacted with them in some way to get a database entry. Edit: also, I tried to do this again, thought I'd try the npub1 account as well, but got Gateway Timeouts, so there's a bit of jankiness going on too.

Well, there's good news and bad news.

The good news is that Lemmy is now surrounding your spoilers with the expected Details and Summary tags, and moving the HR means PieFed is able to interpret the Markdown for both spoilers.

The bad news:
It turns out KBIN doesn't understand Details/Summary tags (even though a browser on it own does, so that's KBIN's problem).
Neither PieFed, or KBIN, or MS Edge looking at raw HTML can properly deal with a list that starts at '0'.
Lemmy is no longer putting List tags around anything inside the spoilers. (so this post now looks worse on KBIN. Sorry about that KBIN users)

(same person on different account, just so I can preview what the table will look like)

Hmmm. This might not help you much. That community was launched on 2024-01-22, with a page full of posts, so by the time the crawler picked up on it, it was already at 501 subs, 23 posts.
It lost 15 posts on 2024-02-12 (Stamets?).
There's data missing from 2024-02-26 - 2024-03-01 (this data is from the bot at !trendingcommunities@feddit.nl; I think those days are missing because I changed it from measuring Active Users Month (AUM) to Active Users Week (AUW)).
In terms of active users, the jump on 2024-03-04 is due to lemmy.world 'upgrading' to 0.19.3.

[data missing]

Ha, this is me. I did exactly that (with a community for the TV show Andor) and am guilty of the behaviour you describe.

I've probably been thinking along the same lines as you and OP though, 'cos I deleted the community a couple of days ago. I realized that if I had something more to say about that show, it doesn't belong in it's own niche community, or 'Star Wars TV', or 'Star Wars', or even 'Television'. Perhaps a 'Movies&TV' comm, although - at this rate - maybe even 'entertainment' would be best.

I'm starting to think that instances that limit community creation to admins have the right idea (e.g. Beehaw, or - to use a non-federated example - tildes).

Some instances have started 'Community Teams', but I sense that anytime they discover a dead community, their instinct is to find ways to promote it, get new mods, drive engagement etc, whereas I'm more of the opinion that they should be nuked and consolidated (along the lines of what the 'cooking' communities have tried to do, I suppose).

I've been coerced into reporting it as bug in Lemmy itself - perhaps you could add your own observations here so I seem like less of a crank. Thanks.

Update: for LW, this behaviour stopped around about Friday 12th April. Not sure what changed, but at least the biggest instance isn't doing it anymore.

Reading your link, a defederation is just blocking the traffic from my end, which I'm already doing by having nginx reject everything with a 403 code.

I'm hoping that I can stop the traffic from being sent in the first place.

I know for sure that Lemmy won't, it's likely the same for Mastodon.

I was wrong about not being able to WebFinger your account - I still had the @ at the beginning when I trying. Doing it properly:

curl --header 'accept: application/json' https://mostr.pub/.well-known/webfinger?resource=acct:910af9070dfd6beee63f0d4aaac354b5da164d6bb23c9c876cdf524c7204e66d@mostr.pub | jq .

gets the right response.

However, I'm logged into lemmy.world and it still couldn't get your account. At a guess, it's because there's a 20 character limit on usernames.

Yeah, I know what you mean. That note is misleading, and kinda redundant too - you can physically de-select Undetermined in the UI, but the change won't actually take if you press 'Save'.

I'm not sure they are supported.

Firstly, I should clarify that lemmon.website is mine, and I'm the only user that's ever been on there.

When I was running Mastodon, I noticed that, after I'd unsubscribed to the lemmy.ml communities, there was still entries for incoming activities in the systemd journal. Before I stopped running Mastodon, I unsubscribed from all lemmy.world communities. After I started up a new, basic server using Rust's hyper library, I saw that all activity from both lemmy.ml and lemmy.world was still POSTing to /inbox

I think what happens is that, when a Mastodon user unsubscribes, the software longer views the community's public key as valid, so it doesn't show the posts, etc, but you haven't 'really' unsubscribed because lemmy has errored. Here's how I've tested this.

1. Start a brand new lemmy instance, on a linux VM in Windows, using ngrok to tunnel though from a ephemeral URL. So, right now, there's lemmy service you can see running using:
   curl --header 'accept: application/activity+json' https://f2d0-77-100-144-83.ngrok-free.app/api/v3/site

2. Log in to a Mastodon server that's designed to let users experiement with ActivityPub (https://activitypub.academy/), to subscribe and unsubscribe to a new-created test community. It looks from there like:
   

But in lemmy, it shows the same 'WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: Unknown' that I quoted in my first post

I don't know what that error means. But I can see that lemmy still thinks it's federating with 1/1 instances, and if I create a post on the test community, I can see it do:

INFO send:send:send_lemmy_activity: lemmy_apub::activities: Saving outgoing activity to queue https://f2d0-77-100-144-83.ngrok-free.app/activities/announce/create/ea41951a-6430-4e53-a5c4-afd644bf0824
Jan 11 11:47:43 lemmyA lemmy_server[977]: 2024-01-11T11:47:43.343193Z  INFO send:send_lemmy_activity: lemmy_apub::activities: Saving outgoing activity to queue https://f2d0-77-100-144-83.ngrok-free.app/activities/create/bfcbf25e-37ab-4cfa-b44d-92cecacb1e56

3. To investigate the error, I tried sending JSONs in the same format that Mastodon uses, but utilising a command-line script. The Follow works, but the Undo-Follow doesn't. The header returned shows the same error as journalctl does: 400 Bad Request "{\"error\":\"unknown\",\"message\":\"\"}"

I then recrafted the JSONs to be more like lemmy, changing the 'id' of Undo to have 'activities/undo/UUID-string', and that works. No entries in journalctl, and a 200 OK Response header.

4. From lemmon.website, I can use the script to Follow and Unfollow communities on lemmy.ml

EDIT: I've had some success with this. Using the private key of the old Mastodon user I'm no longer getting stuff from 'memes' now, I think. (still stuff from 'asklemmy' though).

I'll keep tinkering with this. Maybe during testing I followed asklemmy from another user. In the meantime though, it'd help enormously if any Follows from anyone at lemmon.website could be undone at your end (like I say: it's all me)

(There is still the JSON problem though. It suggests that any 'Undo-Follows' you get from mastodon accounts aren't being processed properly, so you're maybe wasting subsequent federation activity. I realize that's more a lemmy software issue than lemmy.ml issue, but it's my understanding that you're the same people!)

I tend to use photon.lemmy.world, but it'd be really cool if I could log in to lemmy.world using tesseract. It's a fork of Photon - so I imagine it also has feature-parity with the official UI - but it's "designed for media-rich feeds and content". It's what they use at dubvee, and db0 also provides it as an alternative

I don't use the default one because it gets confused when lemmy decides to bring a gif in from an external site, and then convert it to a mp4 (it's a pretty hit-and-miss affair in the Gifs Community), though I realise it's ultimately lemmy itself that's causing the problem.

Yeah. It seems like the kind of community he'd post in. I think there was some drama between him and startrek.website, resulting in him or the admins deleting the account.

I'm not no, just the cheapest VPS I could find.

The requests never reach my backend, 'cos they're blocked by nginx rather than it reverse-proxying them like it does for the traffic I'm interested in. So, thankfully for my poorly-programmed backend, it doesn't have to process the deluge from lemmy.ml. It's a bit of blunt instrument though, 'cos it just blocks everything. In the future, there's likely to be some quieter communities on lemmy.ml I'd like, but right now, I can't follow them without also getting everything from 'memes' and 'asklemmy'.

I'm also thinking about how lemmy seems to get behind on it's federation activities, and it would maybe help if they didn't generate them in the first place for recipients who don't want them.

I'm essentially asking someone to stop coming to my house and knocking on my door. I could ignore the knocks, or put a sign up, or employ someone to stand in my garden and stop them. These are strategies to cope with the problem, but I think the real solution would be if they just stopped coming.

Oh, right. The chat on GitHub is over my head, but I would have thought that solving the problem of instances sending every activity 2 or 3 times would help with that, since even rejecting something as a duplicate must eat up some time.