Lemmy.world (and some others) were hacked
While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.
As I am told, this was the issue:
- There is an vulnerability which was exploited
- Several people had their JWT cookies leaked, including at least one admin
- Attackers started changing site settings and posting fake announcements etc
Our mitigations:
- We removed the vulnerability
- Deleted all comments and private messages that contained the exploit
- Rotated JWT secret which invalidated all existing cookies
The vulnerability will be fixed by the Lemmy devs.
Details of the vulnerability are here
Many thanks for all that helped, and sorry for any inconvenience caused!
Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been 'stolen' and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).
For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.
I wish hackers would invest their time in clearing credit card debt, deleting hospital fees, or something else that actually serves the public good, instead of hacking ordinary people just trying to get by.
Deleting hospital fees/debt is very dangerous... In many HUGE regions in the US there's only one hospital and if that hospital suddenly can't pay its bills it could shut down, leaving a whole lot of completely innocent people in a very sad, people-are-dying sort of state.
In fact, something like this already happened:
https://www.cbsnews.com/chicago/news/st-maragrets-health-central-illinois-hospital-closing/
Hospitals are special in that they're often evil organizations (not all though) that are some of the easiest to hack but also provide critical services to the most vulnerable. One should tread lightly. Political solutions are better (hack some politicians that are against healthcare reform instead).
Clearing credit card debt via hacking is nearly impossible but I agree it would be a much more ethical choice for hackers to target. I used to work for the credit card industry. My unique insider perspective, deep industry knowledge, and personal experience is here to let you know they suck. They are just as evil and unethical and unnecessary as everyone thinks they are! Seriously: If Visa, MasterCard, American Express, and all the lesser players suddenly disappeared the world would be a better place.
Before that can happen though people need a backup payment method that doesn't go through their systems and no: Cash won't work (there's not enough in circulation and it's dangerous to carry large amounts of it). The credit card companies know this threat exists which is why they lobbied Florida (and probably other states) to outlaw alternative, government-run forms of payment (e.g. central bank currency).
As soon as people have a widely accepted payment option that doesn't go through Visa and MasterCard's middlemen (e.g. First Data) then hackers can take their gloves off! Until then though... Let's keep the payment infrastructure working, OK? Thanks!
There's no limit to the amount of good deeds hackers can do though. So let's encourage that! For example, there's plenty of cartels and evil religious organizations (e.g. Taliban, ISIS, Mormon Church, Prosperity Gospel scam artists) that have plenty of money to spare and enormous attack surfaces š
Just want to state the obvious and say, this is pretty much only the case in the US.
Considering some of the targeted instances and the stuff they left behind, it was likely some nazi.
Very impressed by how quickly action has been taken by this and other instances to patch the issue.
Very, seems like great work.
Hijacking the top comment to say I had problems with logging in to Lemmy.world today and liftoff was failing in odd ways.
I had to go into my web browser and clear my site cookies for lemmy.world to let me log in there.
In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)
Iām on iOS with the Memmy app. Itās a work in progress thatās officially unfinished so Iām not surprised but it has also been a bit buggy. Doesnāt seem that I can log out without deleting and reinstalling the app so hopefully this doesnāt happen too often XD
So I was actually just struggling with that myself, also in the Memmy app in case that isnāt clear
What I did was add my account (again)
There was no warning or anything, and it populated the list with two of me.
At that point, a ādelete accountā option appeared under both of them. So I guess in normal circumstances, it wants you to keep one account around at all times?
I deleted one of them, and the app basically reinitialized. Both were gone and it showed me the welcome screen.
I logged back in, and now everything is back to normal
I just did edit account and then saved, it seemed to trick it into logging in again (secrets on my instance were also reset).
I found I didnāt actually have to log out, just go into account settings and reconfirm everything without changing it
thanks for posting this, I wouldn't have figured that out lol
Thanks Ruud for fixing it! Just a reminder guys that If you are using a third party app you need to login again.
Further 3rd party heads up -- for us nontech refugees:
If it looks like you are logged in, you may not be. I use Connect, and at your reminder, I clicked my acct and it says I was logged in. I tried to comment that Connect login was working, and my comment didn't show up.
I tried again, only to see an āerror: not logged inā message pop up.
Signed out, signed in again manually, and all is well.
So do a double check, Lemurs. Trust in your actions, not your eyes.
Thank you.
In case anyone else is having trouble logging in, my password wasn't working so I had to reset it from the website.
So now I can log in via a 3rd party app but not the website (with the new password that I reset via the website.)
I'm currently posting from the 3rd party app. Digging around to try and find 2FA settings for Lemmy.
Someone else commented that cookies need to be cleared on the website (again, for anyone else reading this and having issues.)
For capable people this is a minor annoyance but whenever thereās an āeveryone needs to login againā issue, we will lose mere mortal users. In this case it wasnāt even clear that was needed - I appeared to be logged in but nothing worked. Ordinary users give up over things like this. Iāve seen it happen many times on sites where I had access to the analytics. I hope we regard this as a really bad thing to be avoided at all costs and not a āno big deal, just log in again.ā Easy for you, easy for me, many others will just bail.
Thanks for the heads-up.
IMPORTANT ANNOUNCEMENT: My account was not among those hacked. Any random bullshit appearing in my post/comment history was written by me.
Thatās something a hacker would say.
Thanks for letting us know - this is the kind of transparency that I wish the world had more of!
So what happened:
Am I right?
I'm old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:
Am I right? Correct me if I'm wrong.
Again, web is peace of sheet. This would never happen in desktop/server application. Any of the bullet points above would prevent this from happening. Even if the previous bullet point failed to do its job. Am I too naĆÆve? Maybe.
Marek.
I'm a modern web developer who used to be an old-school one.
Yeah - pretty much, though there are some mitigating factors.
Strictly speaking, it was the alt text for the emoji. Alt text is HTML, and rather than allow arbitrary HTML they allowed another language called Markdown. Markdown is "a plain text" language with human readable syntax specifically designed to be converted into HTML.
Markdown is the right format to use for emoji alt texts, but you do need to be careful of one thing - the original purpose of Markdown was to allow HTML content to be easier to write/read and it is a superset of the HTML language. So arbitrary HTML is valid markdown.
Virtually all modern Markdown parsers disable arbitrary HTML by default, but it's a behaviour which can be changed and that leaves potential for mistakes like this one here. Specifically the way Lemmy injected emojis with alt text into the Markdown content allowed arbitrary HTML.
This wasn't an obvious mistake - the issue over on Lemmy's issue tracker is titled "Possible XSS Attack" because they knew there was an XSS Attack somewhere and they weren't immediately sure if they had found it in the emoji system. Even now reading the diff to fix the vulnerability, it still isn't obvious to me what they did wrong.
It's fairly complex code and complexity is the enemy of security... but sometimes you have to do complex things. Back in the "old-school" days, nobody would have even attempted to write something as complicated as a federated social network...
Yeah - the Lemmy developers made a mistake there. There are a few things they aren't doing right around cookies and JWT tokens.
Hopefully they fix it. I expect they will... It was already actively being discussed before this incident, and those discussions have been seen by a lot more people now.
There are several levels of isolation that could have blocked this:
Yep - the modern best practice is for admins to manage the site via a completely different system. That adds considerable complexity and cost though, so it's rarely done unfortunately. But you know, Lemmy is open source... so if someone wants to take on that work they can do it.
I'll add one more - it should have taken less time to close the exploit... but given this is the first serious exploit I'll forgive that.
Ultimately several of failures contributed to this attack. I expect many of those failures will be corrected in the coming weeks, and that will make Lemmy far more secure than it is right now - so that next time there's a bug like the one in the Markdown parser it isn't able to cause so much disruption.
The good news is no harm was done, and a lot of people are going to learn some valuable lessons as a result of this incident. Ultimately the outcome is a positive one in my opinion.
Oh I forgot another line of defense / basic security mitigation. If a server produces an access token (such as JWT or any other old school cookie / session ID), pair it with an IP address. So in case of cookie theft, the attacker cannot use this cookie from his computer (IP address). If the IP changes (mobile / WiFi / ADSL / whatever), the legitimate user should log-in again, now storing two auth cookies. In case of another IP change, no problemo, one of the stored cookies will work. Of course limit validity of the cookie in time (lets, say, keep it valid only for a day or for a week or so).
JavaScript needs access to the cookies, they are the data storage for a given site.
To protect them, the browser silos them to the individual site that created them, that's why developers haven't been able to easily load cross domain content for years, to mitigate XSS attacks.
The security relies on the premise that the only valid source of script is the originating domain.
The flaw here was allowing clients to add arbitrary script that was displayed to others.
You're dead right that only the way to fix this is to do away with JavaScript access to certain things, but it will require a complete refactor of how cookies work.
I haven't done any web dev in a few years, this might even be a solved problem by now and we are just seeing an old school implementation. š¤·
Yes, it is called
HttpOnlyand is decided by the server who is sending the cookie to you in HTTP response header. I believe there are also HTTPS-only cookies that when received via HTTPS, cannot be used from HTTP, but I cannot find it right now.You're totally right. I just looked at my old jwt cookie and was susceptible to CSRF (cross site request forgery) by virtue of not having the SameSite flag being set. This has since been fixed, but it looks like there might still be changes pending as Javascript is currently able to read the cookie value (the HttpOnly flag is currently set to false, meaning that it is able to be accessed by the browser). While this isn't a major risk, it does increase the attack surface a bit.
"never happen in a desktop/server application" Just don't run mystery floppys and CDs right :)?
This is really good to see such transparency from admins
Ikr! This is really great to see!
Love the transparency!
We need to learn more about the exploit. Lemmy essentially runs on donations, we all need to support this site. I have a lot of faith in lemmy. I am a reddit refugee and i think it is reminiscent of the early days of reddit and not the pos that it became.
Once it's fixed in Lemmy I'm pretty sure we'll see in-depth explanations, at the moment though it's being discussed briefly on GitHub if you're interested
Good thing we all use randomly generated passwords for every account and always remember to change them every few months.
Passwords are safe, it's temporary session tokens (already invalidated) that were exposed.
Good call.
However, that wouldn't have prevented this from happening. Not even MFA can protect you in case of an exploit like this.
My password is 4, it was randomly generated with a dice. Every few months I roll the dice again.
I'm not perfect as far as password opsec goes, but I did indeed use a randomly generated unique password here.
Hell, even if your password safe is only what's built into Firefox or Chrome, that's still better than using easy-to-guess passwords or sharing passwords.
Damn, I go to bed early and I miss everything! Thanks for the quick resolution and transparent disclosure, this place is great!
Don't worry, all you missed was a lemonparty redirect and some weird picture of some guy's face and a caption. It was just some basic disruption, glad the staff resolved it quickly though.
FYI: I had to clear my lemmy.world cookies in order to be able to successfully log back in.
(This was with Firefox)
(Edit: I also shift-clicked reload, which somebody pointed out does clean the cache for that page, so I also cleaned the cache).
I had to do the same with chrome. The past week was not enough, so I had to choose to clear all time.
First - really good summary and sounds like everyone is working hard.
Cross posting the below comment.
Under GDPR if you have had a data breach you have a legal obligation to assess whether you need to report it and you must make the report within 72 hours of discovering the breach.
There are other types of reportable breaches too, I only mention data as it sounds most likely. You may or may not be subject to PECR which may also have been breached although less likely. I donāt really have enough familiarity with the regulation to discuss that one.
If you are not sure if there has been a breach you may also need to discuss it with the relevant body or make a report.
Please can you update what action you have taken regarding this and if the incident was reportable or not and the reasons why. Edit - from that new information, it sounds like this is a reportable breach.
For a full understanding, it would be good to know if you had 2FA enabled on the compromised account particularly as it had admin privileges and if so how 2FA was circumvented with this exploit.
It would also be good to know what measures you have in place to prevent the same or other malicious attempts on your Open Collective and Patreon accounts as issues with those are potentially more serious. They may not be vulnerable to this, but it is going to be reassuring to know there is good security practice, 2FA protection etc enabled and you have robust procedures in place.
Thanks for the info. We're looking into this.
If a valid browser token gets stolen like in this case, then MFA won't do much good because the stolen token will already have been authenticated. Linus Tech Tips experienced the same thing recently, you can check out their channel.
That makes sense, thanks so much - there's a few good explanations here which really help! Would it be right in saying that all affected servers should be logging off all users - some have but not sure if all.
The fix is to force the use of a new JWT encryption key which--when set--would immediately invalidate all existing user cookies, forcing all users to relogin.
Lemmy has a few weaknesses related to their use of JWT in cookies that need to be addressed... The biggest one being that they use the same secret key for all user cookies (every user should have their own unique session key). I'm pretty sure that if they implemented that the scope of this vulnerability would be drastically reduced (but I haven't looked at the precise mechanism of the vulnerability yet).
They also need to provide tools in the GUI for admins and users to invalidate all issued sessions (cookies) and a mechanism for regularly rotating session secrets (the cookie currently lasts for a year and even if the session token gets regenerated it'll still use the same secret).
They also need to make the expiration times configurable so that security-focused servers can set short expiration times. Related, they need to force the use of unique secrets for every session (even if it's the same user using different devices/apps).
I guess that would depend on the specific case. If you physically went on my computer to steal my token or infected my computer with a virus to do it then we can assume that no other tokens have been compromised. But if the malicious actor has managed to steal tokens from the actual server (which seems to be the case here) and not the client then yes, as the admin I would certainly require that everyone log in again as a safety measure.
Out of curiosity, where would the regulators go for a case like this? There's no "company" running it per. se.
It seems the general consensus is GDPR applies even to OSS non company entities, but it would appear that there's very little being done to honor it.
https://www.zwilnik.com/better-social-media/activitypub-conference-2019/oss-compliance-with-privacy-by-default-and-design/#:~:text=Although%20GDPR%20directly%20applies%20only,sysadmins%2C%20including%20in%20the%20Fediverse.
This article outlines Fediverse and responsibilities, I think it mostly requires someone to file a lawsuit before there's any action.
In another case a man had cameras in his back yard that could also see a public area and was fined and forced to move them.
https://www.termsfeed.com/blog/gdpr-exemptions/
Mainly it just seems to be fodder to be used in lawsuits to make people comply with others security wishes. Not certain how all that works since cities are covered in public cameras.
I am not sure how a platform like this will work with GDPR - each server will be responsible themselves, but how it works with the flow of data between servers and who the regulators would have cases against - I think that is to be tested at some point.
They will go after a person instead.
Can 2FA be enabled for all users? I don't see the link to activate it after saving.
edit
Yeah, this doesnāt work at all. The apps donāt open links anymore. I tried some github site that reads the link and generates a QR, but the codes donāt work. This is a complete waste of time.
Just reload the settings page after saving and you'll see the activation link. Just now enabled 2FA for my account.
Don't log out! Open private tab and try logging in to test that it works. Lemmy uses SHA-256 TOTP digest which may not work correctly with some authenticators, only generating useless codes.
How do we know that this isn't a fake announcement as well, trying to give us a sense of security???
Just kidding, thanks for letting us know! Thank god I haven't been too active the last few days! Can't afford my credentials being leaked, maybe I should be proactive and change my password anyways.
The hacker appeared to be the reigning (sic) "Spelling Be" champion of South Sudan.
Thank the heavens the meme community stayed safe through this without my daily dose of cybersecurity memes idk how I would function ;)
Hopefully with more attention on the source code scary hacks like this doesnāt happen again.
Do we have any details on how Michelle's account was compromised? Right now in the GitHub issue about the vulnerability they're clueless about how the custom emoji exploit could be performed without first an already compromised admin account.
EDIT: yeah here's how: https://github.com/LemmyNet/lemmy-ui/issues/1895#issuecomment-1629326627
You do NOT need an admin account to do that. Any normal user could have done that.
what steps are being taken to ensure it doesn't happen again? was any personal data compromised for users?
Good point, I'll update the post.
Also I am curious, what's the easiest way to currently reach the admins in case this happens again somehow? Two of them on their account have been seemingly inactive for a month and as per your own statement you rarely check your notifications and dms. Is there a discord somewhere for it?
Mail: info@lemmy.world Mastodon: @mwadmin@mastodon.world Matrix: https://matrix.to/#/#lemmy-support-general:discuss.online
Would it be a good idea to have a secondary email not attached to lemmy.world in case of a domain hack?
info@mastodon.world
Why wasn't there an info on /lemmy-world.statuspage.io ?
I think the admins that were on it didn't think of updating the status page...
Nice work on the recovery, especially from a 0-day.
Thanks! Is there any understanding as to why? Or are we thinking some script kiddies because they can?
They defaced it with dicks and changed the federation list to be only threads.net. I don't think it was a state sponsored chinese hacking group. :)
I'm ok with the dicks but the threads are TOO FAR!!! shuffles off to the angry done**
Thank you all for staying on top of it.
So all our cookies are negated now with the JWT changed, and we just needed to login again? Can attackers have stolen our cookies in order to use our accounts to post as if it was us? I'm sure they were only interested in admin cookies, so most others were "useless" to them? I see nothing wrong with my posts so I should be safe, right?
Prior to the JWT secret being rotated, yes, they could have authenticated as you. The tokens are now all invalid and useless
If you think they could change your password:
YES, they could.
They could have changed the email => "Forgot PW" and with that you lost ur account.
So thats why MalwareBytes gave me this message yesterday.
Wow that's cool. How did malwarebytes know the website was compromised ?
probably looking for obvious patterns like
"onload="...in image and link tags, because an onload event handler would usually never be put in those tags otherwise so the only plausible explanation is that it's a XSS attackI think it sees that the browser is trying to execute code that is suspicious (the payload of the XSS was pretty obvious).
I wish I knew. I tried logging into Lemmy yesterday and I was kept giving this message. I thought it might be relevant and saved this snip. I am only about to post this pic now. I did whitelist Lemmy on Malwarebytes after as well.
Interestingly, here is a log when browsing Lemmy over the last week or so.
Believe the derp.foo and .today are both federated instances. Don't know what the other rows are.
One wonders how much spez paid for that hack.
lots of
You guys really have my highest respect for spending so much time to keep this running, despite all the recent trouble and now even an attack.
Thank you very much <3 You guys are awesome and I really appreciate how publicly you deal with this.
That was scary and exciting. Response seems competent and transparent. I ā¤ļø this place.
I think this is a strong reminder: We shouldn't put all our eggs in one basket. This will happen again. Unlike Reddit, we don't need to concentrate all communities on one instance. We should all make an effort to spread out. Some other general use instances are:
Again, for those new, you can post content to any of these instances and interact with content from other instances at the same time, just like you can send an email from your Gmail account to your ProtonMail account.
This was an exploit in lemmy software and affected multiple instances though
Register on 5 so you're vulnerable on 5, got it.
The malicious Javascript was in a post however, and that post spread to all Lemmies that saw the post.
I'm pretty sure all web browsers who saw that post passed the JWT cookie to the hacker. The hacker was looking for admin accounts and got the Lemmy.world admin first.
Would be great to have identity independent from any particular servers...
That is currently not possible in the fediverse. Might be some day.
Yeah, I know. It is possible to implement it though. Posts and comments are cached in every federated server, the only thing you need is attribution to the author. Your key/certificate could be your identity.
Despite the fact that Lemmy is a fairly new piece of software, which makes these issues more likely, I am really grateful for it being open source, and I really appreciate this level of transparency.
Any truth to what I've heard this may have been done by a group we defederated with?
It's not truth if it's not confirmed.
Took me a bit to realize I actually had to log out and log in on Jerboa since it looked it I was still logged it but some interactions didn't work
Thanks for fixing and being so open about it
I just disabled whole "/admin" section on my instance and added nice message š
I would put a link to šš porn disguised as a "go to admin panel" link for them there
Updated it little bit, probably later add some "nice" links too š
Had to clear my browser catch to log in, Jerboa still shows as not logged in even after logging out which you do by clicking the hamburger menu then click the top banner to change/log out of accounts. This post is a test to see if my account works again via browser lol.
Edit: clearing app data/cache for Jerboa fixed the login issue.
Thanks for this comment, I had to wipe cache and cookies to login on desktop. Wasn't sure what was preventing me from logging in.
Test
The quick fix is much appreciated, thank you and everyone that helped for your hard work!
On Liftoff, I had to clear cache and storage in order to log back in. Still having issues with the website on Chrome, which keeps telling me I'm not logged in after clearing cache and logging back in.
Just wanted to say thanks for telling me this! Solved my issue and I wouldn't have thought of it straight away at all.
At least now we can mark off the "disruptive website defacement attack" line on the checklist of (relatively) new website growing pains. Better to have them make lots of noise and get fixed quickly than quietly do sneaky things in the background.
If we get substantial downtime from threads federating, I get BINGO with 4 corners.
Thank you for the transparency and swift solution!
Vulnerability strikes. Open source's lightning response strikes back. Again.
@nonagonOrc @ruud šš»
That doesn't surprise me. Especially the "homemade" instances. The documentation is severely lacking and I had to fix lots of stuff in the instructions with try&despair to make my instance run.
There's not a great focus in security if your application starts with "step 1: install docker"
Thanks for the transparancy about this.
Good communication is key to effective incident response. This is a good example of that. Kudos.
Can we get another admin to sign off on this being authentic? In other words, short of a signed GPG signature how do we trust announcements after a breach where admin accounts are compromised?
Good point. I did post about this on Mastodon @mwadmin@mastodon.world
Thanks for the reply!
Don't fall for it. They're also an admin on mastodon.world! :)
Now I don't know who to believe! Is Lemmy even real?
Are we all just lemmings? Oh...
So, do we change passwords, esp those who logged on during the attack? (I created this acct right before the attack happened tho.)
I think it's good practice to change passwords after an attack no matter what
No, passwords weren't compromised
Thanks for the transparency.
Ugh, people should not go after systems trying to give a free service to the internet. It just ruins everything.
How do we know you're the real you? This all could be part of the plan!
I'm not even sure myself..
Once again, thank you guys for all that you do. As many other people are saying, appreciate the transparency about these things.
Yah, I noticed my Lemmies auto-corrupted to Lemurs.
I don't care. I'm keeping it.
Lemurs are cute.
I had to create a new account. I tried enabling 2FA on my main account a week ago, but was never able to generate a token. Now when I try logging in it is asking for my 2FA token. Is there any way to get my account back. I'm a moderator of a community.
Send an email to info@mastodon.world - a ticket will be created.
If you had 2FA enabled and aren't logged into a browser anymore I think your pretty much just fucked. I've tried to get 2FA working with bitwarden and found out that even if you don't set up 2FA it's still enabled on the account once you check the box.
Soo it looks like the entry for this instance was also changed on https://lemmyverse.net/ . At least I hope it is the hack
Yeah I have no idea how to contact the owner of that, a refresh would be nice...
The website says
so hopefully it'll be changed back soon. edit: formatting
Updated :) https://lemmyverse.net/instance/lemmy.world
FYI I think owner is here: @tgxn@lemmy.tgxn.net
yes. it was updated 10 mins ago :)
Thanks for the great work. The response time was awesome, considering you were asleep as well.
Could admins sign announcements with a PGP key to mitigate false admin posts and the consequences this might have? Or is this no longer necessary?
It's probably a good idea to have official announcements be signed, that way it's obvious when they're actually posting officially or if they are compromised.
What's the difference? JWT is already cryptographically signed, but tokens were stolen. That's the issue.
No one store their PGP private key in cookies, at least I hope so.
Ideally someone wants to send signed message doesn't store the signing key in their browser, sandbox their browser even.
I think the lemmy.world admin posted on his official Mastodon.
https://mastodon.world/@mwadmin/110688515627268847
Can I ask some possibly dumb questions?
Thank you!
When you connect to a site, there are a few steps to validate that you are who you say you are (identification and authentication). Something like inputing you login/password. Since it would be tedious to do that on every requests, the first time you give your login/password to the server (this is the simplified version, this exchange is a bit more complex usually) the server gives you that JWT. For every subsequent requests, your browser automatically send that JWT that is simple to handle but hard to counterfeit, and the server safely knows that you're whoever is written in that JWT.
I assume there will be a post here when more details are known, or that this post itself will be updated. As with any online service, it's up to the service to decide if they want to communicate. (it may also be a legal requirements in some places to tell user when such an event occurs). Since we're talking about obtaining other user's authentication token including an admin, it is safe to assume that whatever an admin can see has leaked. This can range from basic user informations to more private stuff, although I am not familiar with the software behind lemmy. Note that this is a worst-case scenario; an admin impersonator could have access to anything an admin could see, it does not mean they immediately dumped everything. It depends on their motivation.
Protection against this kind of stuff Compromission of the JWT can happen in many ways and I don't know which way was used. But if there's a flaw in the software used (the lemmy's client-side code, for example) there is not much you can do. JWT can leak through many things :
Basically, stay up to date and don't use shady stuff. Easy to say, I know.
This is great thank you!
I can only log in on incognito mode, which makes me think my cookie has been stolen or whatever. So my question is, what should I be doing about that?
Your session token is still being cached by your browser. Its trying to authenticate with the server with that. The server is rejecting that token. If you can access the logout button, click it. If not, clear your cookies and cache for lemmy.world and refresh the page
I could only log on after clearing the app data and cache on Android, try that
Clear your cookies.
This is so sad lmao rip. With any site growing as fast as these instances (because of the Reddit folk) Ig these attacks are to be expected. Hope everyone's accounts and personal info are okay
Pardon the ignorance, but how do I know if I was compromised? what do?
What could a hacker even do with your Lemmy info? Unless you are posting critical information about yourself I don't see this as a huge issue. More of a concern for the mod and admin teams imo
Ah okay, I just see ''hacked'' and I freak out every time.
Thank god I used Firefox Relay to mask my true email... and a random password, I'll change it just in case.
Just to clarify... I don't need to delete cookies and sign out of every account on my PC to be safe? me don't understand
The most I see is that they could see your IP address. Not good if you are being stalked. Again I don't think this is a very big deal. If you are worried turn on a VPN.
IP addresses aren't logged by Lemmy
You would see an overdraft and a fee charge on your account where you had 37 dollars.
If you opened an infected page they were able to use your session. That means acting as your account, including seeing and changing your settings. (Which could have allowed them to change your account email or password.)
Nothing beyond that.
If information is complete, attackers gained access to active user sessions by leaking cookies via script injection, access would have been terminated when the cookies were invalidated. They likely did not have access to your password (changing it never hurts though).
Well that's just great it really is a shame though how some people would actively want to ruin something free like this just because they can.
Thanks for the transparency. Was having issues with Lemmy, now seems everything back to normal. Got a question, Just to add an extra layer of security, Do i need to use ToR or VPN with Lemmy ?
That wouldn't have helped. Don't consider this a secure messaging platform, or use this to communicate banking details or something.
Just assume everything you post is public.
Must have been jealous spez
Thank you for your work š
This is why I've decided against running my own Lemmy instance. Too much work to have to keep up constantly with updating, too big of an attractive target for attackers.
For now. I feel like Lemmy world and their admins are doing a great job of stress testing this and working out all the links.
A year from now it may be way easier to host your own instance.
If you run the instance only for yourself then I'd say it makes you an unattractive target. Why do a lot of work to hack an instance with one user?
But yeah, since Lemmy's code is not super mature there'll be some pains in the short term.
It seems there is no way in Lemmy to invalidate all your session cookies? Without that, how can you secure an account which has a stolen session cookie?
The 'secret' in the database was changed, causing all existing cookies to become invalid.
Very good. I think a feature where a user can revoke all their cookie sessions is still worthwhile, and maybe I'll look at raising a feature request for that, but it is good to know that cookies stolen during the recent hack have already been addressed.
Excellent, thanks for the quick response ruud and admins.
Thanks for the work. As a heads up it appears most of the block instances are back however I believe explodingheads is still missing which you may want to confirm.
EDIT: it has been added back to the block list.
Congratulations everyone on the quick fix/mitigation!
Had to re-login in the Connect app
Interesting.
So at least that wasn't 100% malicious, otherwise they could've kept the vuln hidden and just collect data and whatnot.
On the other hand, who cared enough about Lemmy to hack it? Weird.
This one: https://lemmy.world/u/LMAO
Why those kind of people exist? GET A F"+$+ LIFE
Lack of friends compensated with an inflated sense of entitlement.
Wow ... what a loser.
Thanks for the info, Ruud. I just put in for a monthly donation to you all -- I appreciate you.
Would it be a good idea to force a login if the users IP or device suddenly changes?
Does an admin account have any permissions to view email addresses or data of registered users?
Did MichelleG not have 2FA enabled?
Now that this has happened, it's be worth pushing this issue through as high priority. If
HttpOnlywas enabled, then an admin takeover would not have been possible.https://github.com/LemmyNet/lemmy-ui/issues/1252
The JWT exploit bypasses 2FA requirements. It basically steals your active session and allows a third party to use it.
TY to everyone itt who commented on how to fix the 3rd party app issues.
I was panicking when liftoff went wonky
Great lemmies! Thanks for uniting us.
Testingā¦ I have to keep trying this because memmy is being a dick.
the details of the vulnerability are already known now anyway since there's a fix that was proposed on the Lemmy GitHub so I don't think it will hurt others to talk about it
Could you please link the issue? Thanks!
https://github.com/LemmyNet/lemmy-ui/pull/1897/files found it myself
yup that's the one
what I find weird is that the "fix" still focuses only on the front-end, the issue is still that unescaped HTML is being stored in the database and still trusting the front-end is nuts
I mean, I'm pretty sure that for an XSS attack that's fine. The entire problem is that somebody posts e.g. a comment that contains code that is automatically run in users' browsers. If you make the front end just not execute that code then it's fine. Who cares what's stored in the back end?
I mean, it would still be better to have multiple fail-safes, and they probably should still sanitize text entering the database.
But this is sufficient for a quick fix.
Let me introduce you to my friend, Little Bobby Tables... :)
ALWAYS SANITISE!
I think people are forgetting that it's somewhat obvious the hackers or whomever, I don't really care honestly are Lemmy users considering they did this at night and got into the site so quickly to begin with, they'd have to have been familiar with it to get into it as fast as they did.
If anything everything should be fixed.
Seem to have a hard time loging in
Iāve been unable to login on desktop since this happened. Only been able to login via Memmy on IOS.
I put in my info and it kicks me back to the front page and doesnāt log me in.
Iāve tried clearing cache too
EDIT: Switching browser to Edge seemed to let me. Weird. Even reinstalled Firefox and still won't let me.
How does this impact those using mobile apps like Jerboa or Liftoff, instead of the website directly?
Um, probably coincidence or a false posi, but malwarebytes is labeling lemmy.
worldtoday as being compromised / malicious when following external links, it's only popped up twice, but here's a slightly redacted log file:-Log Details- Protection Event Date: 7/10/23 Protection Event Time: 1:24 PM
-Software Information- Version: 4.5.33.272 Components Version: 1.0.2069 Update Package Version: 1.0.72209 License: Premium
-System Information- OS: Windows 11 (Build 22621.1928) CPU: x64 File System: NTFS User: System
-Blocked Website Details- Malicious Website: 1 , C:\Program Files\Google\Chrome Beta\Application\chrome.exe, Blocked, -1, -1, 0.0.0, ,
-Website Data- Category: Compromised Domain: lemmy.today
(end)
That mentions lemmy.today, not lemmy.world.
thank you for the heads up, still getting used to instances
That explains why I had to clear my browser cache, I was unable to login until I did.
Passwords were leaked?
No, just browser session tokens if you loaded the site during the hack. And the tokens were reset after the site came back up.
Databases don't store passwords to begin with so passwords can't really be leaked. They store a hash of your password. So if your password is of sufficient length it could take 100s or 1000s of years to find a match to the hash. If you have a really short password or if the hash is using an insufficient hash algorithm then there exist these things called rainbow tables which are a list of every hash and it could simply be looked up what your password is.
Probably not, it seems only temporary "session" cookies (JWT) were leaked for those affected, so they were revoked for everyone. If you wanna feel safer, changing your password is a good idea "just in case".
Based on what I read I'd say no. XSS usually just affects browsers or anything that runs JS so it mainly affects the client side. JWTs don't (if following the spec) contain passwords, it's just a short lived token that identifies someone. If you sent your password as a DM you're dumb, and they got your password if they compromised your account.
I noticed this morning for a small amount of my posts with pictures, maybe 5-10%, the pictures were deleted or missing. Not sure if this is related to the incident.
What are the risks for people who use Jerboa for Lemmy? I logged put and back in and there doesn't seem to be any issues, so are the app users excluded from this?
Not too promising, hopefully it's fixed quickly.
Poor beehaw.org is still down
They don't know what's happening cause they cut themselves off from the greater Lemmy community.
They are aware. Someone (I believe an admin even) posted there last night about the lemmy.world issue.
Occasional cookie deletions I understand, but will sign-ins persist in the future?
Thanks all working again. Had to clear my browser cache in order to login again and had to resign in to memmy too.
I guess its early days for lemmy for incidents like this, fingers crossed something like this doesn't happen again :)
I wasn't using webpage, I was only using mobile app (Connect). Could my coockie be also stolen that way or was that only possible on webpage?
No, the vulnerability was due to a client-side bug in the Lemmy web UI. Mobile apps render content in a different way, and are not vulnerable to this kind of attack (apart from in exceptional circumstances).
Should probably log out and back in still though.
My main is locked up and it sucks, i hope this all gets fixed.
Here's a relevant post that talked about this with @AlmightySnoo@lemmy.world I think is worth looking into for anyone curious what exactly happened.
https://sh.itjust.works/post/923025
please don't visit the legal section of the website or anything confirmed compromised if anything.
Should I change passwords or no?
No, unless you see posts that you didn't create in your history, you should be fine.
Afaik, there's no way to get your password from the JWT and access to account, right?
Correct, the password is not viewable. But using this exploit a hacker would have access to the settings page of your account and modify password/email address etc.
I think you need the old password to make a new one!
You could change the email and then request a password reset. But yes you're right, that wouldn't work from the settings page! :)
Ah, makes sense
If you have to ask, just change it. It's free and quick, and it's good practice.
I hope devs will examine all parts of the code that display content to make sure proper sanitization
Thanks for your efforts. I know that Lemmy was put in place rather quickly as a Reddit alternative. But I'm genuinely hopeful that this will be a good alternative.
I appreciate the transparency. Hopefully with more eyes on the source code hacks like this will not happen again.
Just for a bit of perspective -- Scary hacks like this happen all the time in the for-profit corporate world, too. They just don't tell us about it. It will continue to happen as technology constantly grows and evolves.
Praise to the Admins and Devs who play this constant game of whack-a-mole!
Good job. I don't understand very much of that, so that makes me all the more grateful. Thank you.
So that was why the logo and name was changed to israel. And for some reason getting redirected to a gif that was from lemmy
I heard that some instances were defaced. Any examples of this? I wasn't online this noon so I never got to see any action.
No need if it was JWT token. After you use your password to log in, the server send your browser/app a JWT token. It uses this token whenever it requests anything from the server, to show that your logged in for this session, and the server can look at the token and tell who it gave the token to, show it knows you're logged in.
All that is to say, logging out will mark any leaked token as no longer valid, and when you log back in you'll have a new, non-hacked login token.
I mean, I want to see screenshots of the "fake announcements" the hackers posted through compromised admin accounts
It was some antisemitic bullshit. I have a screenshot, but it contains homophobic and racist phrases, so I don't think it's worth posting.
Oh sorry, I totally replied to the wrong comment lol
Was gonna go refind the right parent and reply, but tbh I'm in bed right now and too lazy for that, so I'm just leaving the non-sequitur there for anyone interested
May I ask was is the JWS coockinand if it is automatically changed or if we have to change it in a way?
Damnit, spez.
Amazing how you quickly reacted to this!! Bravo!!
TIP: if you can't login after what happened, clear out your browser cache including ALL cookies, that fixes it (it did for me at least). I believe it's also advisable to change lemmy password.
Well done on acting on it so quickly. I think I did see some of the fake announcements you were referring too but were taken down very quickly. Keep up the good work team and thanks for everything you are doing!
I had an issue of being logged out of my account and could not log back in, after closing and reopening the site, closing browser, etc until I cleared my cookies, then it let me back in. If that helps anyone.
Is this why Jerboa seems to not work any more? It keeps insisting I'm not logged in, when I am, showing me as anonymous, but also showing my profile details, not letting me interact with things, etc... It's been a big problem these past few days making Lemmy unusable :-(
Had the same with liftoff. Probably the app thinking the JWT is valid but the server declining it out of precaution.
I just logged out and back in again.
Log out and back in again