I just developed and deployed the first real-time protection for lemmy against CSAM!

db0@lemmy.dbzer0.commod to /0@lemmy.dbzer0.com – 697 points –

In the past months, there's a been a issue in various instances where accounts would start uploading blatant CSAM to popular communities. First of all this traumatizes anyone who gets to see it before the admins get to it, including the admins who have to review to take it down. Second of all, even if the content is a link to an external site, lemmy sill caches the thumbnail and stores it in the local pict-rs, causing headaches for the admins who have to somehow clear that out. Finally, both image posts and problematic thumbnails are federated to other lemmy instances, and then likewise stored in their pict-rs, causing such content to be stored in their image storage.

This has caused multiple instances to take radical measures, from defederating liberaly, to stopping image uploads to even shutting down.

Today I'm happy to announce that I've spend multiple days developing a tool you can plug into your instance to stop this at the source: pictrs-safety

Using a new feature from pictr-rs 0.4.3 we can now cause pictrs to call an arbitary endpoint to validate the content of an image before uploading it. pictrs-safety builds that endpoint which uses an asynchronous approach to validate such images.

I had already developed fedi-safety which could be used to regularly go through your image storage and delete all potential CSAM. I have now extended fedi-safety to plug into pict-rs safety and scan images sent by pict-rs.

The end effect is that any images uploaded or federated into your instance will be scanned in advance and if fedi-safety thinks they're potential CSAM, they will not be uploaded to your image storage at all!

This covers three important vectors for abuse:

  • Malicious users cannot upload CSAM to for trolling communities. Even novel GenerativeAI CSAM.
  • Users cannot upload CSAM images and never submit a post or comment (making them invisible to admins). The images will be automatically rejected during upload
  • Deferated images and thumbnails of CSAM will be rejected by your pict-rs.

Now, that said, this tool is AI-driven and thus, not perfect. There will be false positives, especially around lewd images and images which contain children or child-topics (even if not lewd). This is the bargain we have to take to prevent the bigger problem above.

By my napkin calculations, false positive rates are below 1%, but certainly someone's innocent meme will eventually be affected. If this happen, I request to just move on as currently we don't have a way to whitelist specific images. Don't try to resize or modify the images to pass the filter. It won't help you.

For lemmy admins:

  • pictrs-safety contains a docker-compose sample you can add to your lemmy's docker-compose. You will need to your put the .env in the same folder, or adjust the provided variables. (All kudos to @Penguincoder@beehaw.org for the docker support).
  • You need to adjust your pict-rs ENVIRONMENT as well. Check the readme.
  • fedi-safety must run on a system with GPU. The reason for this is that lemmy provides just a 10-seconds grace period for each upload before it times out the upload regardless of the results. A CPU scan will not be fast enough. However my architecture allows the fedi-safety to run on a different place than pictrs-safety. I am currently running it from my desktop. In fact, if you have a lot of images to scan, you can connect multiple scanning workers to pictrs-safety!
  • For those who don't have access to a GPU, I am working on a NSFW-scanner which will use the AI-Horde directly instead and won't require using fedi-safety at all. Stay tuned.

For other fediverse software admins

fedi-safety can already be used to scan your image storage for CSAM, so you can also protect yourself and your users, even on mastodon or firefish or whatever.

I will try to provide real-time scanning in the future for each software as well and PRs are welcome.

Divisions by zero

This tool is already active now on divisions by zero. It's usage should be transparent to you, but do let me know if you notice anything wrong.

Support

If you appreciate the priority work that I've put in this tool, please consider supporting this and future development work on liberapay:

https://liberapay.com/db0/

All my work is and will always be FOSS and available for all who need it most.

114

Just want to add - i've been using this (via my desktop!) for my instance for a little while now and its great - While the evidence shows there are false positives, i've yet to see it affect anything in real time.

Beware your B2 transaction costs though! 😭 I'm sure there is a cheaper way to do it but backblaze costs went up quite a bit.

B2 cloud storage update says:

effective October 3, we’re making egress free (i.e. free download of data) for all B2 Cloud Storage customers—both pay-as-you-go and B2 Reserve—up to three times the amount of data you store with us, with any additional egress priced at just $0.01/GB. Because supporting an open cloud environment is central to our mission, expanding free egress to all customers so they can move data when and where they prefer is a key next step.

Yeah, I had the email yesterday, but they don't mention if this is specifically their download charge, or if the class b and class c transactions are included in this - I mean I'll be honest, I haven't had time to properly look into yet, but either way it should help.

How does one even test this ethically?

There are I think official "training kits"? I remember reading about this, there are sets of data you can get to train CSAM detection with.

Its a general image classifier. You don't need to train it specifically on CSAM if you are willing to accept a certain false positive rate.

I think "legally" and "without constantly wanting to vomit" is the trickier question. From a purely ethical standpoint I don't see a problem with taking CSAM that's apparently already flooding Lemmy and using it to test whether your filter works before nuking it. At least as long as you're making sure you're not exposing anyone else to it.

I'm curious how an AI like this is trained

it's my understanding that the csam datasets (once already labeled by people) are hashed to the point of being unrecognizable before being passed around.

I don't think training a model on hashes would be particularly useful - if the model were able to get any meaningful information out of it, that would mean the hash function itself is somehow leaking enough of the original contents to determine the image contents (which would essentially mean the hash function is broken beyond all repair)

you know what that makes sense so I looked into it, one method of detecting csam is hashing the image and comparing it to a database of hashes of known csam images. so I guess that method might not work for "original" csam images.

article about it

apple csam detection [download]

google talking about it

interestingly, Google says they use AI also but didn't really get into the details.

AFAIK it should work as long as the hashing function has some direct transformation property which can be extrapolated by the AI.

(Not an industry veteran, I dabble in AI mostly for hobby and sometimes work, but do have some accredited education on the subject. I may well be way off mark)

1 more...
1 more...
4 more...

Great work, this is the biggest issue that Lemmy has a the moment, I hope the admins will be able to set this up easily and start to take back all the preventative measures.

Has this been a problem since the initial spam wave? I wasn't aware the issue was ongoing. But the less pedos, the better.

Some major instances like .ee shut down image uploads immediately and only restored them in a truncated fashion, if at all.

I think people are going to be much more concerned about the false negative rate than the false positive rate.

Cool. After some testing Hexbear should run it. Not that the problem has ever been serious for Hex but still worthwhile and work that should absolutely be supported.

1 more...

Have you considered federating hashes of positive matches and working with the Lemmy team to not outward federate on a local positive match (and potentially have the hash go instead)?

The former can reduce overhead and electricity use, and the latter will stop more distribution and aid those sans-GPU who can't run it.

Over time, the hash DB will grow and get better. In addition, perhaps there is metadata that can be used to track image similarity to positive matches to reduce false-positives, but I imagine that algorithm would be much more complicated.

Hashes won't work for novel GenerativeAI images. For this kind of thing we need to be sharing tensors and comparing distances so that it catches format changes and compression artifacts. Theoretically possible. Practically, I don't know how feasible it is.

How large is each tensor? If it can be stored as JSON or Base64 and is of sufficiently small size, integration into ActivityPub wouldn't be all that bad. The time consuming part would likely be integration into Lemmy itself.

Another option would be a separate service, similar to how Lemmy Explorer works, where a list of the latest tensors can be downloaded. It's centralized vs distributed, but probably easier to implement. Just an API admins can register for to send and get tensors.I would be happy to assist with this if it is a route you would like to explore. Feel free to DM me.

@fmstrat each tensor is small. The problem is when you have millions of them and you have to compare each image to each. You can't index this. It has to be one by one. And you still need to covert the new image to tensors as well,which still needs gpu. I just don't see anything useful here. The current system would be faster.

Good point. I wonder how the commercial hash-based systems are doing it..

https://blog.cloudflare.com/the-csam-scanning-tool/

Does this not work?

Not for all csam and not at all for novel generative ai csam. It's also not for all countries nor is it easy for everyone to join it and not everyone wants to be on cloudflare. Same is true for other tools like photodna

False positive rate ~1 % False negative rate?

thank you for making the fediverse a safer place to be

I would love some form of this for Mastodon

I don't know the architecture of masto unfortunately. I guess it doesn't use pict-rs. What does it use for images?

AFAIK it is a built in system. But scanning the file folder or object storage probably works the same.

It would be definitely nice to have the option to scan multiple locations if you run more than one service.

Couldn't this be more efficiently solved by having only approved users post images? Like people with some posts/comments and positive karma (or whatever it's called on lemmy).

You still cache federated images. And not everyone can vet all their users like that, not to mention it's easy to fake it

1 more...

Wouldnt that just incentivice people to farm points? Better avoid dealing with reddit problems here as well.

1 more...

Fantastic work!

My first thought tho is of revenge. Is there any way to have it automatically report the incident to the FBI? Address of the uploader, etc

Not unless lemmy and pict-rs devs provide a lot more functionality. However be aware that most hits will be false positives. You would be sending a lot of garbage over to the FBI unless you review every hit manually.

What if the troll is uploading from outside the USA ? I don't think spaming authorities with reports will help...

You’re probably right

Also, if it reports a false positive, you’re sending an innocent person’s IP and other info to the authorities. I imagine the user won’t appreciate that.

1 more...

AI based with high false positive rate. Fantastic. These tools are great but impacting regular use of the platform is going to drive people away. Lemmy is at a critical state of needing to onboard users to plateau or grow, I feel this will not be good despite the good intentions.

It's not like you can just re-post what gets taken down. The magic box's response is final

I suppose on boarding users trumps dealing with CSAM. Thanks for clearing that up.

Literally unhinged take lmfao

Reasses your fucking priorities dude. Fucking child abuse or users on the website. Imo you deserve a global ban if you’re a serious on this take and not just being a 4head

Someone disagrees with me on the internet! Global ban!

Nah just people who are cool with sacrificing the removal of fucking child porn for user numbers.

Pedo.

Sounds like progress, but please consider using a term other than "whitelist" when describing a list of allowed values. While the use of blacklist predates references to black as a race, allowlist is a reasonable alternative that doesn't reinforce viewing black as less than or unwanted and white as allowed.

Making things that were never about race into things about race, just to have one more reason to be potentially offended by, is not productive and doesn't help anyone.

By exercising enough mental gymnastics almost any term could be twisted into something supposedly offensive. The real solution to that problem: don't do mental gymnastics.

Those are technical terms that have nothing to do with race or even humans.

Oh come on... The origin of blacklist was centuries before "black" became the term for a person of colour. And on a thread about CSAM...

Allowlist and Blocklist are also more intuitive to people who haven't heard the terms before.

I've honestly always found "allowlist" and "blocklist" to feel like forced compound words, and I don't see why "list" is necessary at all. For example, just saying "allowed" and "blocked" both implies it's a list and is more intuitive than any of the *list terms.

Personally I have no stake in the battle, but I do wish people would use the most intuitive terms for the situation at least (whatever they are, for example "enabled"/"disabled" or "included"/"excluded") instead of blanket ctrl+f on everything.

That's a good point, and I hadn't thought about that angle, that there just isn't a reason for the terms to exist in the first place.

"In the red" and "in the black" is another pair that isn't intuitive to me at all and I have to look up every time.

Do you ever get tired of twisting yourself into a pretzel every time you want to be offended?

Sounds like progress, but please consider using a term other than "whitelist" when describing a list of allowed values. While the use of blacklist predates references to black as a race, allowlist is a reasonable alternative that doesn't reinforce viewing black as less than or unwanted and white as allowed.