Lemmy.world update: Downtime today / Cloudflare

Ruud@lemmy.worldmod to Lemmy.World Announcements@lemmy.world – 2311 points –

Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.

Most of these 'attacks' are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.

For the other attacks, we are using them to investigate and implement measures like rate limiting etc.

445

You are viewing a single comment

You should change the public IP of the server if you haven't already

What happens tomorrow? Change the IP again? And again? It's not a long term solution.

They mean after adding a ddos mitigation like cloudflare, you should rotate the origin server IP so the origin server's IP is no longer publicly known and thus not directly reachable by ddos attackers. The only way to now interact with the application is though Cloudflare's network. You should only have to do this once as long as the origin IP doesn't publicly leak.

Another step would be to add firewall rules to only allow inbound traffic from cloudflare IPs: https://www.cloudflare.com/ips/

I recall a certain amount of overhead in IPTables "allow only from" situations but I'm not sure whether it's enough to make a DDOS any kind of viable on a server in this configuration.

Do you happen to know how effective the strategy is?

If your origin servers IP is never revealed then all traffic goes through cloudflare regardless. Firewall restricting the IPs is just good practice since cloudflare is the only IP that is supposed to talk to that server anyway, but it's not a requirement.

I can see some overhead if you're maintaining a large blacklist, but I don't see it happening with a small whitelist and default inbound DROP

Oh absolutely, I agree with the best practice! I just didn't know the real world efficacy of dropping packets near the NIC to mitigate DDOS load. There is certainly a performance limit but where that limit exists has been nebulous for me.

Cloudflare masks the origin IP address and has DDoS protection. Unless it's a DoS against the software, yes, it is a long term solution.