PSA: We're back, here's a post-mortem!
Hi Beeple!
Here's a vague version of events :
-
11PM EST: Lemmy.world got hacked
-
12:20AM EST: Blahaj.zone got hacked
-
12:25AM EST: I shut down the server
-
12:30AM EST: I make announcements to tell people about this
-
12:45AM EST: I have an idea of what the problem is but there is no fix
-
2:20AM EST: I go to sleep
-
8:50AM EST: The server is booted back up, steps are applied to mitigate issues (Rotating JWTs, Clearing DB of the source of vulnerability, deleting custom emoji), UI is updated with the fix, CSP and other security options are applied
-
11:40AM EST: We start testing things to make sure are working And well, now here we are.
If you have issues logging in or using an app:
-
Log out if you somehow are still logged in
-
Clear all cache, site data, etc.
-
Hard refresh Beehaw using CTRL+F5
-
Log back in.
If you still have issues, write to us at support@beehaw.org
To be clear : We have not been hacked as far as we know, we were completely unaffected. This was done preemptively.
Oh yeah, in case, you haven't, this is a good opportunity and reminder to follow us on Mastodon as the communication line was still up despite Beehaw being down : https://hachyderm.io/@beehaw
From what I can tell the whole point to the css class/formatting was controlling the size of the emojis. Depending on where they came from, I could see some being of random size and shape. Admins might not have the time or know-how to shrink them down, so css seems like a reasonable compromise as long as the files aren't huge.
I'm kind of bothered that the only fix seems to be on the frontend. Unfortunately, I haven't been able to stick with Rust long enough to take a reasonable crack at figuring out how to help on the backend. Input and output sanitization should ideally be handled in both places.
Lemmy's backend is kind of curious, in that it does the bare minimum to move content around and add some metadata fields.
For example, did you know that "deleting" a comment, only marks a "deleted: true" field, while the comment is still pushed in full to the frontend? Same thing happens with banned/mod removed comments, they just get marked as "removed: true" but otherwise still get pushed to the client, and the user can still edit them.
All the display processing is done in the frontend, or whichever app you happen to use.
I can maybe see marking it as deleted in case someone wants to creat undelete functionality later. I don't agree with it, but I can see why someone would do it.
It's just weird to still push it to the frontend.
Same with the removed stuff. All of that should be handled on the backend and never even sent to the frontend. Sometimes the reason for deletion is something you don't want getting grabbed by someone who is bored and poking around in developer tools, like doxxing information.
Since I don't have the time to do anything about it, though, I guess I don't have a place to complain. I have strong feelings about this stuff, but there's a limit to the number of things a single person can work on. If I were to hop on an open source project this minute, it would be helping migrate Cursorless to an LSP.
There is an undelete feature, at least the Liftoff app has it.
Right now, the only effective way to delete a comment, is to edit and blank it, then you can delete it (well, the content, you still get shown as the author). The bright side is that even when the comment gets mod-deleted, you can still edit it... and everyone can still read it in the JSON... wait 🤔
No need, just view source. In the Liftoff app there is a "nerd stuff" option where you can also read the content.
Anyway, once a comment gets federated, it's out of the original user's control, can't really take it back.
Hey, I wanted to thank you for this exchange. I learned a lot about lemmy.
Glad to be of assistance. There's plenty of it I still haven't looked into, though.