Lemmy.world (and some others) were hacked

Ruud@lemmy.worldmod to Lemmy.World Announcements@lemmy.world – 2319 points –

While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

  • There is an vulnerability which was exploited
  • Several people had their JWT cookies leaked, including at least one admin
  • Attackers started changing site settings and posting fake announcements etc

Our mitigations:

  • We removed the vulnerability
  • Deleted all comments and private messages that contained the exploit
  • Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been 'stolen' and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

671

You are viewing a single comment
View all comments

Could admins sign announcements with a PGP key to mitigate false admin posts and the consequences this might have? Or is this no longer necessary?

It's probably a good idea to have official announcements be signed, that way it's obvious when they're actually posting officially or if they are compromised.

What's the difference? JWT is already cryptographically signed, but tokens were stolen. That's the issue.

No one store their PGP private key in cookies, at least I hope so.

Ideally someone wants to send signed message doesn't store the signing key in their browser, sandbox their browser even.

Imagine you have Permission to post as axzxc1236@lemmy.world as an object that is locked in a safe deposit box, the key is JWT token.

The key, by how it works, can be obtained by (1) knowing axzxc1236@lemmy.world's password (+2FA token if configured) (2) have a copy of the token (3) somehow cheat lemmy.world's authentication scheme (by somehow skipping authentication or acquiring magic tool to generate random token .... etc).

What happened today is scenario (2), by a single line of JavaScript planted by hacker, and if you have access to my unlocked desktop computer, you can see and copy the key by using F12 (developer tool).

Scenario (2) can be mitigated by tying the sending computer IP to JWT token, but as far as I can tell lemmy.world doesn't have the option, and most of people get random new IP address every now and then because how Internet currently operates, so this feature can be very annoying.

now introduce Permission to post PGP signed message as axzxc1236@lemmy.world as an object that is locked in a safe deposit box, this deposit box has two key slot (like a bank deposit box), JWT token and PGP private key.

This PGP private key doesn't have to sit in your browser, it can be on a burner phone which it's only purpose is a APP for signing message, it can be on a USB thumb drive that is stored in your parent's basement, or usb thumb drive that sits in bank deposit box which requires a bank employee to authenticate your ID to access.........

PGP private keys are harder to steal than JWTs, as they are not generally stored as a long-term cookie but briefly just to sign something. Through XSS (the vulnerability in this case), cookies are relatively easy to steal, but to steal a PGP key would require a more complex script able to steal the key at the time it is loaded in the browser. It's a bit more sophisticated, but not totally bulletproof.

First of all, it's still possible, as you said. Second, we must always take humans into consideration. Let me explain.

Most people don't know what PGP is, thus you can sign your posts with any crap and most won't care.

Out of those who know what PGP is and how to use it, most won't be checking the posts as they will auto-assume that if it's signed - it's ok.

There will be a small minority of those who care, but they can be silenced easily through social engineering.

It is possible to integrate server side signature checks, but then again you're vulnerable when someone gains unrestricted access to your account.

The correct solution requires multiple features to be implemented.

  1. Lemmy web UI should move to an industry standard framework, which disables XSS and other attack vectors by default. For example, Angular.
  2. Official admin announcement, as well as important admin tasks, should require 2FA on each action. Like it's done on GitHub, for example.
  3. Cookies should not contain JWT tokens, they should only cotain session id for read only purposes. JWT token should be moved to sessionStorage instead. It's a lot more secure by design than cookies. You can still exploit it, but it's noticeably harder.

Agree with the points on PGP and other features. I almost made a lengthier reply mentioning the signing issues, which seems appropriate now. It would not be easy, but a successful implementation would definitely need clients to automatically detect and verify signed content, due to the human issues you mention. A problem is obtaining public keys from a trusted source. Maybe it could be attached to profile information with a 2FA requirement to modify it. Just an idea. In this way, verification is not dependent on the user to perform.