WARNING: Lemmy Self-Hosters, There Have Been CSAM Attacks taking place against !lemmyshitpost@lemmy.worldlocked

Jamie@jamie.moe to Selfhosted@lemmy.world – 500 points –

cross-posted from: https://jamie.moe/post/113630

There have been users spamming CSAM content in !lemmyshitpost@lemmy.world causing it to federate to other instances. If your instance is subscribed to this community, you should take action to rectify it immediately. I recommend performing a hard delete via command line on the server.

I deleted every image from the past 24 hours personally, using the following command: sudo find /srv/lemmy/example.com/volumes/pictrs/files -type f -ctime -1 -exec shred {} \;

Note: Your local jurisdiction may impose a duty to report or other obligations. Check with these, but always prioritize ensuring that the content does not continue to be served.

Update

Apparently the Lemmy Shitpost community is shut down as of now.

155

Someone is trying really hard to hurt Lemmy by continually attacking the most popular instance. Is this all coming from right-wingers upset that their nazi instances were defederated across basically the whole fediverse?

The simplest explanation is 4chan types just doing it for the lulz.

My tin foil hat is telling me it’s one of the other social media companies funding a hacking group to do it. They stand to have the most to lose, and they’ve seemingly decided to enjoy changing the narrative regarding multiple topics. Lemmy stands directly against what the bigger social medias stand for.

I have no evidence to back this though. As a business owner I just know that things become very consistent when people are being paid, and very inconsistent when they aren’t. These attacks are seemingly very consistent/organized.

You think a company that is posed to go public is going to attack a competitor with a minuscule amount of traffic with extremely illegal material that could put them in prison for even having?

See, I don’t believe this was done by a large corp. But all the DDoSing that’s happened? I can see u/spez orchestrating that.

Lemmy isn’t a threat to Reddit. It’s the same old trolls doing it like every other time.

I don’t think they do see it as a threat, I just think spez is petty enough and juvenile enough to do it.

it’s this https://amp.knowyourmeme.com/memes/i-feel-bad-for-you-i-dont-think-about-you-at-all

Like, again, I pretty solely think it’s spez’s own personal ego shit. For example, he could have just shutdown the API. Instead, he had a weeks-long meltdown including committing libel against a developer. Someone like Zuckerberg doing this doesn’t make any sense to me, but I can totally see spez being exactly that kind of petty.

He’s have to acknowledge Lemmy being a threat, which it’s not, and which his ego won’t allow anyway. The simplest answer is the best.

Reddit? No. I was thinking moreso Meta. They have the deeper pockets and a proven track record of breaking privacy laws to their own benefit.

That’s even worse. Meta probably doesn’t even know what Lemmy is.

So then why was Meta trying to get Threads to be on the Fediverse? Of course they're aware of any potential threats, no matter how small.

Why reinvent the wheel if someone’s just going to hand you the backend? Lemmy is no threat to them.

The threat is a new sustainable community that's sheltered from advertising that people could leave Factbook/Instagram/whatever and go to.

Meta was talking about adding Mastodon federation to their Threads app. So I very much doubt it.

They'd probably take an Embrace, Expand, Extinguish approach.

You would pay a third party to do it. And keep details extremely vague so you have plausible deniability.

You have a massively inflated view of Lemmy's importance in the social media market.

There must be room under that tinfoil hat for the both of us, because this was my first thought too.

The longer it continues, the more likely that scenario is IMO. Bitter alt-right extremists would probably start losing interest after a short while, whereas social media competitors would stand to gain from long-term interference.

I'd go with state actors first.

When a particular social media platform is centralized, you can buy yourself a say percentage of stock and have sway over it (cough tencent), or have a useful idiot ruin the platform (cough musk), or another useful idiot to run propaganda you like anyway (cough truth social, cough fox news, cough newsmax...), or yet another that will sell out it's host country's citizens for cold hard cash (cough facebook).

But when that social media platform is decentralized? Well, then you'd need to figure out how to poison the well early on to stave off adoption. The Saudi Arabias, UAEs, Chinas definitely don't like the idea of lemmy, and it'll be way harder for them to control if critical mass is hit.

Yep, that’s a great point.

Add to that the fact that mainstream social media companies wouldn’t touch DDoS and CSAM attacks with a 100-foot pole, even if they contracted with a third party. Both of these attacks are highly illegal and would surely ruin a publicly traded company (or one that’s trying to go public, like Reddit).

And don’t forget Russia in your list of state actors who are threatened by the unrestricted flow of information. They definitely don’t want their citizenry to be informed of how disastrously their invasion of Ukraine is going, or what a murderous scumbag Putin is.

You don't get a lot of upvotes and sure we don't know but it isn't like the NSA infiltrated (in person) left wing groups and more.

It's definitely a possibility that someone doesn't like decentralised content enough to put some meager efforts against it.

This makes the most sense to me. It's a pretty vitriolic attack, therefore I don't think it's simply a troll while at the same time I don't believe it's any corporate social media.

Considering all the alt-right garbage that was popping up there the last couple of days this seems at least plausible. I sometimes envy their ability to utterly destroy anything they touch.

I'm sure you'd love to link to some examples

See people claim this constantly with no proof

You want me to link posts that the mods removed? That seems like an unrealistic expectation. You could always check the post pinned to the top of lemmyshitpost where they describe the recent problems, but I suspect you didn’t ask for proof in good faith

Ah that's actually my bad, I thought you were replying to a different comment in reference to hexbear

I wouldn't put it past the hexbear crazies throwing a tantrum. They claim to be left wing... Sure seem more like fascist trumper types though. Maybe it's just that they're all incels and incels all seem about the same.

Throwing a tantrum about what exactly? They're one of the oldest-running Lemmy instances. Until now they were running a fork based on a pre-Federation version of the codebase.

You believe they did a bunch of work migrating their database only to then negate that work by destroying the community they wanted to Federate with?

Well something to keep in mind is that hexbear isn't one person ... it's a whole community that's developed independently for a while. So it's reasonable to expect that there'd be variation in the behaviours of members in the same way there's variation on the rest of lemmy. From what I've gathered, not all hexbear members are keen on the re-federation, and some aren't too keen on being "well-behaved" around politically opposed users (ie "libs"), though hexbear admins and other users have promised moderation and that such isn't part of the core hexbear values.

It's social media, afterall ... and people can be rather shit and ruin it for the rest of us. In the end, the core service provided a social media platform isn't the hardware, sys-admin-work or software (however necessary they are) ... it's the moderation work.

The moderation keeps the place sanitary enough for people to actually want to be here ... however much we may have problems with particular actions of our moderators, we should really support and praise them at every turn.

At least a handful of users on hexbear had made their intention clear during the first week of re-federation, they were looking to cause chaos on Lemmy for there own pleasure. I don't know if they were banned and/or their comments deleted.

Big difference between a few users who did a bunch of work and the toxic goonsquad the majority of the userbase turned into.

they’re all incels and incels all seem about the same.

Downvote from me there. I’ve seen plenty of examples of hexbear people being nice, interesting and good sports. They definitely seem to have more of shitposting culture than is normal on mainstream lemmy. But all in all it’s seemed fun to me from what I’ve seen.

Beyond all that, this is just superficial and prejudicial. If you had some examples to link to or more substantial insights to share as to why it’d be “them”, that’d be worth reading.

Otherwise, they’re an instance. Not one person, I’m sure some on hexbear are assholes and some awesome.

So, so shocked someone it's from lemmygrad that is defending the notoriously toxic "communist" tanky trollfest instance.

Sorry, not from lemmygrad. And I'm on lemmy.ml because I joined before the Reddit migration and "Privacy and FOSS" (the focus of lemmy.ml) made a lot of sense for a lemmy instance/community.

Beyond that ... more superficial, prejudicial hate mongering without any description of why or for what purpose. Sorry, I don't think it's worth reading ... a downvote from me ... and, just being real for a moment ... at the moment it's more likely that you're a member of a "notoriously toxic ... trollfest".

Ironically, IME, I've seen significantly more troll-like tankie hate than I do tankie-trolling. I keep asking for receipts/links to tankie trolling here, as I'm genuinely curious to see it and understand what people are so upset about (please don't explain to me what's so upsetting unless it's culturally thorough or coupled with some links+descriptions) ... but no one has been able to do so.

Most people from hexbear provide sources, which is better than can be said for all the tankie hate.

1 more...
2 more...

So, from memory there has been:

  • This recent attack
  • Regular DDOS attacks
  • Frequent attempts to spam community creation
  • That one time the instance got hacked and set to redirect to shock sites

Am I missing anything?

This seems like more than just a few trolls. Maybe someone really doesn't want to see user-owned social media take off.

I see where you're going with this, but no, people really are just absolutely horrible. The fact is that with other social media they're just already very set up in managing this so we never see it. Lemmy wants to be open, this is the flipside of that openness.

It's generally easy to crap on what's 'bad' about big players, while underestimating or undervaluing what they are doing right for product market fit.

A company like Meta puts hundreds of people in foreign nations through PTSD causing hell in order to moderate and keep clean their own networks.

While I hope that's not the solution that a community driven effort ends up with, it shows the breadth of the problems that can crop up with the product as it grows.

I think the community will overcome these issues and grow beyond it, but jerks trying to ruin things for everyone will always exist, and will always need to be protected against.

To say nothing for the far worse sorts behind the production and more typical distribution of such material, whom Lemmy will also likely eventually need to deal with more and more as the platform grows.

It's going to take time, and I wouldn't be surprised if the only way a federated social network eventually can exist is within onion routing or something, as at a certain point the difference in resources to protect against content litigation between a Meta and someone hosting a Lemmy server is impossible to equalize, and the privacy of hosts may need to be front and center.

The solution in this case is absolutely AI filters. Unfortunately you won't find many people willing to build robust model for that. Because they'd be those getting the ptsd you mention.

Iirc, ptsd is something only certain characters get. We should probably focus on finding people who really have no problem watching rough content. I have ptsd so I probably am not the right person for the job.

I don't want to try. I have pretty low barrier. I set up NSFW filter on lemmy because I found disturbing the furry content that was common some time ago... I don't want even to try anything worst than that

Can absolutely relate. Just seeing nsfw if you‘re not anticipating it is very weird.

1 more...

big F in chat for those of you dealing with this. my #1 fear about setting upand instance.

It impacts everyone when this shit happens. It takes time for mods/admins to take down. And you can’t unsee it.

I hope nobody else has the misfortune of stumbling on that shit

There have been studies which found playing tetris for an hour or two after seeing something traumatic can prevent it taking root in our longterm memory.

I tried it once after accidentally clicking a link on reddit that turned out to be gore, I can't remember exactly what it was now (about 9 months later) so it must have worked

This advice is a few hours too late for me. Hope it helps others

Don't worry, life with hold many more traumas

Cue Homer Simpson: This is the worst thing you've seen in your life so far

I just posted an article explaining the study to the 'You Should Know' community, so hopefully some of the people who need to see it do so

That’s pretty genious. One way to work with trauma is moving the eyes from side to side along to a moving light. It’s basically to force the brain to work through something that used to stall it. So tetris, since it is highly reactive and logical, also needs spatial thinking, should very much force the brain to work instead of stalling.

Yeah you really can’t. I’m pretty desensitized from earlier internet with death and other shock gore content but had managed to avoid CSAM until today. It was a lot worse than I expected, felt my heart drop. Worse, my app autoplays gifs in thumbnail so it kept going while I was reporting it.

I’ve mostly forgotten and it wasn’t on my mind until I saw this thread (happened less than 24hr ago) but even the slightest reminder is oddly upsetting. Wish I’d thought of the Tetris thing.

Why do these deranged fucks do this

You really think the trolls would pass up on this golden opportunity?

This isn't trolling, this is just disgusting crime.

The crime happened in the past when the children were abused. This is some weird amalgam of criminal trolling.

Edit: yeah yeah I get that csam is criminal, that's why I called it an amalgam. It's both trolling and criminal.

spreading child pornography is in fact illegal in most of the world

It's still a crime. Taking the pictures is a crime. Sharing the pictures is also a crime.

Depending on jurisdiction, I am not a lawyer, etc etc, but I’d imagine with fairly high degree of probability that re-distribution of CSAM is also a crime.

The crime happened in the past when the children were abused.

That's true. You could look at it that way and stop right there and remain absolutely correct. Or, you could also look at it from the eventual viewpoint of that victim as a human being: as long as that picture exists, they are being victimized by every new use of it, even if the act itself was done decades ago.

Not trying to pile on, but anyone who has suffered that kind of violation as a child suffers for life to some extent. There are many who kill themselves, and even more that cannot escape addiction because the addiction is the only safe mental haven they have where life itself is bearable. Even more have PTSD and other mental difficulties that are beyond understanding for those who have not had their childhood development shattered by that, or worse, had that kind of abuse be a regular occurrence for them growing up.

So to me, adding a visual record of that original violating act to the public domain that anyone can find and use for sick pleasure is an extension of the original violation and not very different from it, IMO.

The visual records are kind of a sick gift that never stop giving, and worse still if the victim knows the pics or videos are out there somewhere.

I am well aware not everyone sees it this way, but an extra bit of understanding for the victims would not go amiss. Imagine being an adult and browsing the web, thinking it's all in the past and maybe you're safe now, and stumbling across a picture of yourself being raped at the age of five, or whatever, or worse still, having friends or family or spouse or children stumble across it.

So speaking only for myself, I think CSAM is a moral crime whenever it is accessed, one of the most hellish that can be committed against another human being, regardless of the specificities of the law.

I don't have a problem with much else that people share, but goddamn I do have a problem with that.

Archive.org gets crapped up this way, too. Repulsive people posting repulsive stuff, "troll" is too kind a word.

1 more...
1 more...

I literally am going to give up social media in general if this doesn't stop

Seen it last night late around 3am shit made me sick I honestly almost cried but I just closed the app and tried not to think about it

Whatever the goal is it's a stark reminder that there is monsters creeping in the shadows every where you go

Likely scum moves from reddit patriots to destroy or weaken the fediverse.

I remember when Murdoch hired that Israeli tech company in Haifa to find weaknesses is TV smart cards and then leaked it to destroy their market by flooding counterfit smart cards.

They are getting desperate along with those DDOS attacks.

Could be, but more likely it's just the result of having self hosted services, you have individuals exposing their own small servers to the wilderness of internet.

These trols also try constantly to post their crap to mainstream social media but they have it more difficult there. My guess is that they noticed lemmy is getting a big traction and has very poor media content control. Easy target.

Moderating media content is a difficult task and for sure centralized social media have better filters and actual humans in place to review content. Sadly, only big tech companies can pay for such infrastructure to moderate media content.

I don't see an easy way for federated servers to cope with this.

Yeah exactly. This is the main reason I decided not to attempt to self host a Lemmy instance. No way am I going to let anyone outside of my control have the ability to place a file of their choosing on my hardware. Big nope for me.

I got lucky. I am not subscribed to this community, and I am the only person on my instance. But what if I was subscribed and hadn't seen this post? This is too much responsibility for me.

I just shut down my instance until we can disable cached images. If that never happens, then I'm not bringing it back up.

Shout-out to https://github.com/wescode/lemmy_migrate. I moved my subscriptions over in a minute or two, and now, other than not having my post history, it's exactly the same.

Yup. Nope.

Pictrs is just completely disabled now. Rather be safe, then sorry.

Is this why I couldn't upload a meme to the Lemmy World servers earlier today?

Fuck...

Yeah... Just wow. I disabled pictrs and deleted all its images, which also means all my community images/uploaded images are gone, and it's more of a hassle to see other people's images, but in the end I think it's worth it.

Through caching every image pictrs was also taking up a massive amount of space on my Pi, which I also use for Nextcloud. So that's another plus!

Note, apparently, lemmy will get pretty pissy if pictrs isn't working..... and the "primary" lemmy GUI will straight-up stop working.

Although, https://old.lemmyonline.com/ will still work.

And- I am with you. My pictrs storage, has ended up taking up quite a bit of room.

There has to be a more elegant way of dealing with this in the future, like de-coupling between Lemmy-account hosting (which effectively means acitivypub-fediverse account) and Lemmy-communities hosting.

Does that disable image saving and processing for one's instance?

Yup.

So far, mostly everything appears to work still. But, trying to upload an image, just throws an error.

SyntaxError: Unexpected token 'R', "Request er"... is not valid JSON

I don't see a way to actually "gracefully" disable it, but, this works.

Edit- don't just stop pictrs.

Lemmy gets very pissy... and b reaks.

Is disabling Pictrs as simple as stopping the Docker container?

Yup.

I sent a step further, and commented out the pictrs related configuration from the lemmy.hjson too.

How desperate to destroy Lemmy must you be to spam CSAM on communities and potentially get innocent people into trouble?

Maybe you’re a dev on the Reddit team and own a lot of shares for what you know is about to go public?

i’d love for a good tech journalist to look into how and why this is happening and do a full write-up on it. come on ars, verge, vice

Self hoster here, im nuking all of pictrs. People are sick. Luckily I did not see anything, however I was subscribed to the community.

  • Did a shred on my entire pictrs volume (all images ever):

sudo find /srv/lemmy/example.com/volumes/pictrs -type f -exec shred {} \;

  • Removed the pictrs config in lemmy.hjson

  • removed pictrs container from docker compose

Anything else I should to protect my instance, besides shutting down completely?

I went ahead and just deleted my entire pictrs cache and will definitely disable caching other servers images when it becomes available.

Anyone know if this work is tracked anywhere? I’m suddenly really suspicious of continuing to run my own instance.

To be clear, if no one on a given instance sub to that particular /c, the content won't federate to said instance, correct?

At this point, the community is clean. So unless more is posted, then you should be good. If someone searched for the community and caused a preview to load while the content was active though, then it could be an issue.

Cool. Thanks. I cleaned up anything from the past 2 days, to be safe, and blocked that community.

blocked lemmyshitpost some time age because it is trash anyway

I was looking into self hosting. What can I do to avoid dealing with this? Can I not cache images? Would I get in legal trouble for being federated with an instance being spammed?

I checked and there shouldn't be any images stored on the server when running lemmy 1.18.4. The post was made in high emotional distress and shouldn't be taken at a face value. If the posts are bothering you I advise purging the posts in question. (I have already done that)

I'm on 1.18.4, once I deleted the most recent images, the former CSAM posts(among others) became broken images. So yes, it was pulling from local disk cache. Then I took care of the posts themselves after the content was invalidated.

How did you check this? From my understanding, images from external servers are copied (and transcoded) over locally. At least in my server (running 0.18.4), they do.

There is a possibility that my instance is buggy and it isn't caching images even though it should.

It's pretty inconsistent from my experience. Sometimes images do cache and sometimes they don't.

edit:

Here's an example from my instance:

https://ani.social/post/284147 - JPEG image that isn't copied/cached by my server.

https://ani.social/post/285861 - WEBP image copied/cached by my server.

Let me try to figure this out. The first is a photo uploaded to lemmy.world, the second is a photo originally uploaded to lemmy.nz, both posts are in a federated version of lemmy.world's shitpost community.

This is just a theory, but perhaps images hosted on the same server as the federated community will directly link, whereas images uploaded somewhere other than the federated community will be copied into cache, presumably in case the original host shuts down unexpectedly? See if this is the case?

images hosted on the same server as the federated community will directly link

https://ani.social/post/288601 - This image is uploaded from a user on the same instance as the federated community (lemmy.world) but the image is cached.

images uploaded somewhere other than the federated community will be copied into cache

https://ani.social/post/285354 - This image is uploaded from a user on a different instance (lemm.ee) from the federated community (lemmy.world) but the image is not cached.

The behaviour is pretty weird. Hopefully we can disable image caching/copying-over-locally so we don't have to deal with problematic images hosted by other instances.

It depends on how the image posted, the thumbnails might get federated. If the image is used in a post/comment body, usually the thumbnails are not federated.

You can refer to this post. The full image is copied to my instance (and transcoded). Not just the thumbnail.

I'm not subscribed to that community, but I guess I'm glad Pictrs doesn't work for me, since I am using the Yunohost version of Lemmy. The creators of the Yunohost package couldn't get it to work. I haven't really missed it honestly.

Can you run lemmy without pictrs? What behavior is different?

It just means that you can't upload pictures, including banners or avatars. However, when I want to create an image post, I just make the post on Pixelfed and then mention the Lemmy community I want to post to at the bottom of the post body. Supposedly there's a way to reference a remote image for a banner or an avatar, but I haven't figured that out yet.

Likely Spez’s personal jailbait collection

I am using the Lemmy easy deploy would this command works?

You'll need to find where the actual container files are being stored. I'm unfortunately not familiar with Lemmy Easy Deploy, but you should have a folder that has some files/folders like docker-compose.yml, volumes, lemmy.hjson.

The important one is the volumes/pictrs/files folder, take the full path of that folder and replace it with the /srv/lemmy/example.com... path from the original post, and then that command should work.

There was a weird JSON error I was getting in the last few minutes. I'm not sure if this is at all related.

Couldn't this be stopped with automatic filtering of bad content? There are open source tools and libraries that do this already

That's what we're pushing the lemmy devs to do. Honestly even if they want to use proprietary tools for this instance I'm okay, I'll happily go register an Azure account and plop an API key into the UI so it can start scanning. Lemmy should have the guardrails to prevent this from ever hitting our servers.

In the meantime, services like cloudflare will handle the recognizing and blocking access to images like that, but the problem still comes down to the federation of images. Most small hosters do not want the risk of hosting images from the whole of the internet, and it sounds like there is code in the works to disable that. Larger hosters who allow open registrations can do what they please and host what they please, but for us individual hosters we really need tools to block this.

Proprietary software isnt necessary there are plenty of project that detect scam

I'm saying when it comes to this I don't care if it is or isn't proprietary, frankly I'd be down if we used multiple ones. I'm all for my morals but when it comes to CSAM as long as it works. That's the most important, and yes I'd probably use multiples

Could someone please ELI5 that script. I'm all for keeping things clean, but old enough to remember the days of console based trolling.

sudo

As root

find /srv/lemmy/example.com/volumes/pictrs/files

Find files in /srv/lemmy... that:

-type f

Are plain files (not directories, symlinks, etc; includes images)

-ctime -1

And were created within an amount of time (probably last day, haven't used this flag in a while)

-exec rm {} \\;

For each matching file found execute rm on it (delete it).

Locking the thread. Information relevant to self-hosters has already been shared. Too many reports of off-topic comments to leave this open.

Is it possible to prune pictrs by community name?

Not really. You could technically locate the images and determine precisely which ones they are from their filenames, but that means you actually have to view the images long enough to pull the URL. I had no desire to view them for even a moment, and just universally removed them.

As mentioned in my edit above though, ensure you are in compliance with local regulations when dealing with the material in case you have to do any preservation for law enforcement or something.

There is a purge community option available for admins but that did nothing.

From what I was informed, purging a post doesn't remove the associated cached data. So I didn't take any chances.

That's it, I'm defederating from lemmy.world. the admins let their users make death threats against users of other instances on top of this.