Lemmy.world update: Downtime today / Cloudflare

Ruud@lemmy.worldmod to Lemmy.World Announcements@lemmy.world – 2311 points –

Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.

Most of these 'attacks' are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.

For the other attacks, we are using them to investigate and implement measures like rate limiting etc.

445

Don't forget. Donate to them. There are no ads here. So we have to maintain the staff and servers.

Lemmy World

https://www.patreon.com/mastodonworld?utm_campaign=creatorshare_fan

Lemmy Devs

https://www.patreon.com/dessalines?utm_campaign=creatorshare_fan

Didn't the admins for Lemmy[.]world post their expenses recently-ish? I can't remember how much it would be for a single user to donate. I'd want to donate, but I'd like to know how much of my contribution would affect operation of the server.

Wow if I'm reading their expenses correctly, the maintenance bill doubled from May to June...

Probably because they have to keep growing the instance size due to the influx of users.

1 more...
1 more...

Imagine hosting a service for anyone else to use it, free of charge, no ads, free & open API, yet some idiots think it's fair to (D)DOS it.

There are more "interesting" targets, worst case - Reddit, who thinks everyone is just a number/noise.

Just leave Lemmy alone. :(

we will all still be here when their hyperactivity wears off.

with the old Reddit simulator, personally I'm not going anywhere anytime soon. This place has a great user base and it feels so old-school.

The new layout with old.lemmy I came back, and new apps coming out for it. It's been a good replacement. Was on tildes, but got banned for just discussing difficult topics....the admin there is just ban happy and yea he owns the site but will just ban people for no reason. Not to mention that the users over there, assuming new people are using the malicious tag as a down vote button which probably goes right to the admin. So you step out of line and you get banned. I really liked the place too, but it's not wanting to be a serious place to discuss topics with an admin like that.

I wonder if the owners of deddit, fb, tweetster, et al, might think it financially worthwhile to cause disruption in the fediverse, and even its ultimate failure.

I wouldn't be surprised, we didn't take their whole user base of anything but it's in their interest to keep viable competitors out of the way.

Every account they lose hits them in the pocketbook. The bigger the fediverse gets, the more adherents, the greater the momentum it will have and the harder it will be to stop.

Nipping it in the bud is the best, easiest, and least expensive place to nip it.

The downvotes suggest their operatives are reading the comments.

Counterpoint- people are down voting because they think its unlikely and many people are inherently gaurded against conspiratorial thinking- especially if they think it's unrealistic.

Whether you think its happening or not, the idea that the only reason anyone would downvote is because they're "opperatives" of the big social platforms is kind of out of touch with the fact that there are lots of people who don't think like you do. I'm a real person, love open source, and love the fediverse (have 3 lemmy accounts, plus an account for mastodon and pixelfed each) and I was tempted to down vote certain comments just because they seemed silly and a bit like fearmongering that there's a big bad boogey man out to get us.

I hope I'm being clear, communicating on the internet devoid of tone or facial expressions is hard- my point isn't that your perspective is silly, my point is that there are lots of people who would sincerely see it that way and disagree with you. Assuming that being disagreed with is a sign of the sort of conspiratorial situation you're describing is a self fulfilling prophecy. I hope I'm not coming across as hostile, that isn't my intent

Personally I think the other platforms are unlikely to see the fediverse as a problem until it proves it can be, because CEOs are stupid, and after eons of not having meaningful competition in this space I think they're likely to be overly proud and look down on our nice little platform. I think its far more likely its just the internet being shitty because lots of people on the internet like breaking or ruining anything they can, regardless of whether its a good thing to have exist. I could very easily be wrong, and perhaps other platform's owners do want to kill what we have before it can manifest into something bigger, but either way there are lots of sincerely held perspectives that might drive someone to down vote some of the comments here just because they think the situation being described is unrealistic.

Points well made and taken, thanks. No hostility perceived at all.

Reasonable minds can differ and frequently do. And it could be that people may think my suggestion is unrealistic or even silly.

There's no shortage of miscreants out there who just like to mess with things, thrown wrenches into spokes, etc. And these types could well be behind the daily local issues.

But here's an important point, and no offense intended. Corporations are like The Terminator. But instead of getting Sarah Connor, they purse profits. And regardless of CEO intelligence or accumen, every Fortune 500 company has a department that deals in these areas. They all have their skunk works and use them. It's been this way for centuries. A primer: https://en.m.wikipedia.org/wiki/Industrial_espionage

So whether they're operating here atm or not, there is nothing paranoid about assuming they are. If they're not, they will be. It's what they do.

Thanks for the input. :)

But here's an important point, and no offense intended. Corporations are like The Terminator. But instead of getting Sarah Connor, they purse profits. And regardless of CEO intelligence or accumen, every Fortune 500 company has a department that deals in these areas. They all have their skunk works and use them. It's been this way for centuries. A primer: https://en.m.wikipedia.org/wiki/Industrial_espionage

Lol, all very fair, corporations suck and are prone to doing anything shitty they can think of to even marginally improve their bottom line. Its an understandable sentiment.

I'm glad I was able to convey what I meant without it coming across as my being a dick :)

Take care! ❤️

It's an important life skill, being able to plant a thought in the mind of another and in a way that is likely to be accepted.

It crossed my mind since my last writing that, in the 80s, I got a money back guarantee for any counter-surveillance equipment purchased that didn't reveal surveillance equipment in a Fortune 100 facility. It was that pervasive back then. And my perception is that morals and business ethics have not improved in the interim. Far from it.

Good luck and thanks for the valuable, respectful input.

I agree, many of them appear to be edgy script kiddies upset that people don’t wanna use their precious reddit anymore

The downvotes suggest their operatives are reading the comments.

Let's not do this. People are allowed to downvote without being a paid operative. This was a very common mentality on Reddit I would like to avoid here.

What makes Lemmy interesting is that you can see the combined upvotes and downvotes. It’s not a “net” votes system like some shithole site whose name I will not mention. So I think people can read into the voting system much more than they might have been able to do on some other awful and alienating place.

But, I too disagree with the conspiratorial comment that there are operatives downvoting people on Lemmy, as if that could do anything meaningful. I think the notion that Lemmy is being hacked because the major social media companies are afraid of it, is also very extreme and conspiratorial.

I agree we should support this community and people’s ability to react positively or poorly to a post or comment.

Wondering if reddit or Musk are behind the attacks?

Most likely their parasocial fans. The Reddit stans who want to be edgy and follow their meme leader. Who will never acknowledge them no matter how much they do.

It's sad that they could target the real people making the world worse, yet only prop up the people who are oppressors.

I don't understand why people want to take down websites. Especially sites like Lemmy, which isn't exactly sticking it to anyone because no one owns it!

Are they just Reddit groupies?

For most hackers or wanna-bes (often called Script Kiddies, that is, people (generally young, even children thus the "Kiddies") who are not technologically inclined enough to be real hackers and see a tutorial online on how to run pre-written scripts that repeatedly perform various functions), the answer to "Why do you do it?" is often:

  1. "Because I was bored."

  2. "Because I can."

Very rarely are other reasons given.

More like "I get zero action, so I take my anger out on other people"

Some people enjoy causing suffering to others. On the internet they are termed trolls. Irl people usually just call them assholes. Most people have encountered them before.

I think they are far more common and likely than anyone giving two shits about reddit.

They're just trolls. Lemmy is popular enough that it's fun target for them, but still small and infantile enough that you don't have to be hackerman to ddos it. Reddit, twitter, etc... would be constantly getting ddos'd just for the lulz by people if they didn't have the infrastructure to make it a challenge.

I was using voip.ms last year when they were DDoS'd for over a week, by a group demanding payment via anonymous crypto. The DDoS ended when they switched to CloudFlare (which was probably pretty difficult because they're a SIP provider.)

Almost any website with a small number of servers is vulnerable to this attack, which happens to be great business for CloudFlare. I wonder which companies are most effectively competing with CloudFlare?

There are others, but I think the craziest thing about Cloudflare is its basic level of protection is free. Free, unmetered, DDOS protection. It's so popular because so many hobbyists use it for free, and are familiar with it. Then they convince their workplaces to adopt it when the need arises because they are already familiar with it.

They make money by selling support to companies, and selling access to some more advanced features (that often have a free tier as well). It's honestly so impressive, it made me wonder how much they actually make because it seems unnecessary for most to pay at all. Turns out they cleared almost a billion dollars in revenue in 2022.

Nah, it's not the 00s anymore. Hacker gangs are a real thing today.

I'm not actually in the security field so take this with a grain of salt. But I believe that these attacks play a similar role to random attacks in low level gangs. It proves that your criminal group has power and the ability to deface a website.

So if you publish that Lemmy.world will go down next week because your hackers are on it.... It's advertising. Its just business. It proves that your hackers have an ability and that you are up for sale.

1 more...

You don’t think just being bored is enough reason for some?

If I'm bored I find something productive and/or fun to do.

Launching a DDoS attack is neither.

You, sure. It’s not difficult to imagine a teen who’s not you

11 more...

In case you haven't considered this, some helpful advice. To keep them from the lemmy.world door after the CDN installation

  • Change the public IP addresses
  • rotate your certificates
  • block all traffic appart from the CDN and only allow a limited known good IP addresses (like yours and your support team). These steps will make your server harder to find, hopefully they move on.

You might have Cloudflare add a request header to the origin request, like x-cloudflare-key: <somesecret>, and then configure nginx on the server to block everything not containing that header.

just block ingress to your server from non cloudflare IPs or use argo tunnel.

Growing pains. This server and the platform will be better for it. If not for these script kids, some other attacker would eventually be motivated to try it.

Thank you as always for the transparency. This instance is going to be the most targeted because of its size. Y’all dealing with this is hard but you’re going to figure things out that will help the other instances.

Anything we can do as "users" to help, other than donating?

Hmm, best would be if those kids find a real hobby so they stop bothering us. On the other hand, it helps us understand Lemmy better and secure it.

That's true. Free stress testing the system I guess? Still they need to touch grass lol

If it’s the same people, they’ll probably get tired of it and move on. But the more we talk about it, the more likely it is that new people want to get in on the “fun”. I’d say to not make memes about the downtime and pretty much act like it doesn’t exist (as users, obviously the admins should take action as necessary to mitigate it and post to be transparent).

It's not. People hate large companies that have a dominant position in their industry. Usually, that's fair. However, in the case of DDoS protection, you have to have a large overbearing presence to be able to have the capacity to withstand such attacks. People don't know how to see through what's typically true for what's true in this case. Do I like having a dominant player in an industry? Not particularly. Do I understand why it's necessary in this case? Yes.

Come on everyone, let's be better than this. Ruud literally said script kids, why do yall have to go and blame reddit? The Lemmy gets more attention, and chaotic dumbasses do their thing. You don't have to do any mental gymnastics to tie it back to spez.

3 more...

Excellent! CDN and DDoS protection are essential. Also would recommend looking into load balancing if you haven’t.

Load balancing applications is significantly more complex than most people anticipate. In the naive implementation it typically increases database loads and reduces site performance. Static content balancing is trivial, and cloudflare will do that by default, but implementing the hard part will require careful software development to prevent a naive implementation from bringing down the database. Sticky sessions are just the beginning.

12 more...
12 more...

That's for for always keeping everyone up date. Sucks that you have these people wanting to DDOS a free community of people, I don't get it.

Either way thank you. Now to just somehow find a decentralized version of CloudFlare so we don't have to deal with there trackers that they have.

Commercial interest wants to see free communities like this die out. I won't name names.

Thank you for your hard work, and for keeping us updated on the situation.

Thank you! I will donate tomorrow

Be aware that you use another server so you might consider donating to them instead.

I have an account on yours too, but I might split it between both indeed :)

I'm curious, why bother with multiple accounts? It seems counterintuitive when taking federation into account

Edit: All of these reasons are why I host my own instance

I do it for the following reasons:

  1. The big main lemmy servers can and do go down regularly. having accounts on other instances still gives me the ability to log in and participate on the communities I care about. I sure do wish the underlying federation logic allowed for associating selected logins on various instances together somehow.
  2. Testing: often, I will create a post or comment into a community hosted on a different lemmy server and not see it update. When this happens, I'll log in on the other instance to see what that post or comment looks like from there.
  3. Insurance against defederation: I participated/followed some beehaw.org communities before they defederated, and then I was forced to open an account there to continue participating in those communities.

Taking some load from the biggest servers such as LW. I still have a community on LW however, and mod with my local account

Well, I can’t answer for them but this situation in particular makes it nice to have accounts on different instances. If I can’t log on to/load my lemmy.world account then I can switch to my lemm.ee account and load content there.

3 more...
3 more...
3 more...
3 more...

Cloudflare isn’t bad per se, but having huge amounts of the public internet behind a centralized provider is bad for the flexibility and resiliency of the internet as a whole.

Maybe one day we'll work out a distributed version. The upside is, they're a filter, not the actual site. If we work out better long term strats they're disposable. If they're worse than not having them at any point, it's just a dns change to kick em to the curb

Wonder why this wasn't done earlier. Hopefully we'll see less of the 404-type pages that has plagued this instance.

It costs more money right?

Not necessarily. I have several servers behind Cloudflare for free. I'm just limited on analytics, some advanced firewall settings, advanced cache management and maybe a few other features that I don't use. But the basic service is free.

https://www.cloudflare.com/plans/free/

Nothings free, someone's paying cloudflare something to operate

I imagine this is more of a "If we give people the basic stuff for free when they are small, they are more likely to buy our better stuff when they grow and need to update"

6 more...

Man I would love to know how/why doing that is enjoyable to some people. Like how sad and pathetic is your life that that is what is fun to you?

9 more...

On the plus side watching you all tackle and solve these problems gives me confidence in the long term viability of Lemmy and the fediverse. The transparency and often detailed technical discussion definitely helps a lot too.

You’re doing a great job so far. Thanks for the update.

Thank you for your efforts, work and results. Those "attackers" only deserve disgust.

Maybe they don't deserve as much, pity would be enough.

Benefit of using Cloudflare CDN:

Commenting and editing took about 0.5 seconds!

Also, ping is now from 200-300 miliseconds to just between 50 and 60 (depending on your ISP):

64 bytes from 172.67.218.212: icmp_seq=1 ttl=64 time=56.2 ms
64 bytes from 172.67.218.212: icmp_seq=2 ttl=64 time=60.2 ms
64 bytes from 172.67.218.212: icmp_seq=3 ttl=64 time=55.8 ms
64 bytes from 172.67.218.212: icmp_seq=4 ttl=64 time=58.9 ms
64 bytes from 172.67.218.212: icmp_seq=5 ttl=64 time=60.6 ms
64 bytes from 172.67.218.212: icmp_seq=6 ttl=64 time=60.5 ms
64 bytes from 172.67.218.212: icmp_seq=7 ttl=64 time=60.1 ms
64 bytes from 172.67.218.212: icmp_seq=8 ttl=64 time=55.0 ms
64 bytes from 172.67.218.212: icmp_seq=9 ttl=64 time=60.0 ms
64 bytes from 172.67.218.212: icmp_seq=10 ttl=64 time=61.4 ms
64 bytes from 172.67.218.212: icmp_seq=11 ttl=64 time=59.3 ms
64 bytes from 172.67.218.212: icmp_seq=12 ttl=64 time=58.5 ms
64 bytes from 172.67.218.212: icmp_seq=13 ttl=64 time=56.0 ms
64 bytes from 172.67.218.212: icmp_seq=14 ttl=64 time=60.6 ms
64 bytes from 172.67.218.212: icmp_seq=15 ttl=64 time=58.7 ms

The bugs in Lemmy are such that you don't even need to touch a server for it to be vulnerable. Cloudflare does not defend against such mistakes. Other servers can trigger deep PostgreSQL logic problems within Lemmy. Growing pains, a lot of the federation code was never tested, and today's crash is due to a logic issue with lemmy_server mistakenly updating 1700 servers it knows of through federation for a delete instead of the 1 local server.

I'm learning a lot by following lemmy.worlds actions. Appreciate the transparency!

Where can we donate toward server costs?

I hope lemmy.world can avoid using Cloudflare which goes against the spirit of Fediverse as it's just an objectively evil company.

Agreed. This is an emergency fix. Will look for final solution later.

Can you give some insight to this?

There are thousands of reasons from centralizing internet, abusing their market power, implementing barriers on web automation that can only be bypassed by the priviledged to fingerprinting and tracking users across the whole internet. It's a major for-profit market capture corporation - it's evil by design.

What would the alternative be? DDOS protection inherently benefits from a centrally controlled network for defense, and also from a single entity handling as many of the defenses as possible so they can see them all being used.

I guess I could trivially see the need for a not-for-profit version of this, but that'd still be a central entity, just mandated by law and funded from taxpayer money or something.

But back to the question, what is the alternative? There's a good reason everyone goes with Cloudflare, it's about defending from DDOS attacks, and they do it better than others.

The real alternative is super simple. It requires just a little bit of knowledge. All we would need is to have someone who is an enterprise grade sysadmin with nothing but free time and a willingness to do something they will barely get paid for, if not lose money on. Then we also need to hire out a dedicated network and security engineer as well as a dedicated network traffic monitor. Then we would need to implement and setup our own hosting, as well as servers and configure our own databases. Of course all of this has to be done as cheaply as possible by people who are so good at multiple different sectors of IT and could easily be making more money doing work, but obviously out of the kindness of their hearts want to progress the fediverse and Lemmy rather than realizing they could be making 200k+ doing the same thing for a private company rather than a hobby.

In short: we need a network engineer, a security analyst, a sysadmin (or maybe 2?) all of whom work 24/7 for free and then purchase all of the physical hardware with the knowledge and capacity to set it up and maintain it to nearly break even just so we can shitpost rather than those people working and making 200k+ a year.

The problem is not the service is that Cloudflare is a mega corporation. Having anti-ddos service which does nothing else is perfectly fine. Having one that also fingerprints everyone and does who-knows-what with all that absurd amount of data and control is a different issue entirely.

Then you give them an effective DDoS protection measure instead of posting things without evidence.

Well, it's not without evidence, we have plenty of that through the years. Unfortunately, we also don't have any real alternatives either, so the choice is take the DDoS or get Cloudflare. Not much of a choice.

There are alternatives. Akamai has a similar product. It's not free, but it works. Also doesn't require all traffic to go through them all the time, you can repoint your traffic at them on the fly and have them mitigate by scrubbing the unwanted traffic until the attack ends and then switch back, and this can be automated. My ISP at work uses this as they have large swaths of public IP address space to protect for vulnerable members.

Well, it’s not without evidence I mean without providing evidence. Dr. Moose said that Cloudflare abuses their market power without showing anything. I think we should do better than that. If you think that someone is doing a bad thing you can either refer back to the article that you read it from or at least google something that supports your claim. I don't like being a part of a community that just believes things because they sound real.

Honestly, if anything, Cloudflare uses their power too little. Where they allow just about anyone to use their services unless they get a complaint from the government. You can read here about how they don't shut down websites from their services unless they are breaking the law. I would assume that abusing market power is something like disallowing services because they criticize Cloudflare or some other arbitrary reason. I think that a good example of abusing your market share is Amazon where they forced merchants to keep their prices on other stores higher. But don't take my word for it, you can read all about it here.

2 more...
2 more...
2 more...

Damn these script kiddies.. I don't like Cloudflare at all but it does its job well. It may just be my paranoia, but putting a single entity in control of so many websites seems dangerous. I think we have all learned about the intentions of big corporations. But hey, it's better than being taken down tbf.

What are your reasons for hating cloudflair? Best i can tell they run a good service and their free offerings have been great (1.1.1.1)

We said the same thing about chrome 10 years ago. It's not the quality of the product, which is excellent. It's the concentration of control.

1 more...

Maybe is that Reddit dude, jealous of Lemmy's increasing popularity.

It seems like you made this comment in jest, but I wouldn't say it's outside the realm of possibility. We can't fly off the handle and lob accusations absent any sort of proof, but it would hardly be the first example of a corporation targeting an up-and-coming disruptive service run by amateurs.

I wonder now with the semi-adversarial/semi-cooperative nature between lemmy instances, if wer'e not going to see more DDOS and other types of raids happening because a different instance has an ax to grind against yours. Say between you defederated them, or they consider your instance too big etc.

I saw a post "Lemmy seems to have way more leftists than xyz." So I wouldn't be surprised to find it's political nonsense, people attacking a site they see as "the left."

I can see it now, a lemmy cold war of sorts where the "leftists" and "socialists" and "communists" are arguing non-stop over the definition of those terms and who is worst amongst them

I put this site behind cloudflare in response to this post. Other than having to change SSL/TLS encryption mode to Full, it seemed easy. I turned on bot fight mode and I'm using the managed WAF ruleset that comes with the free tier. Any configuration recommendations anywhere in the panel?

If your site was already accessible by HTTPS before you put it behind Cloudflare, try Full (Strict).

Hmmm, we're getting a fuckload of web requests on our Lemmy too... I think I'll enable CloudFlare too! :)

Just make sure you do your research before you do - people have broken federation by enabling it without due care in the past.

people have broken federation by enabling it without due care in the past

Any links regarding this? It sounds concerning and my instance admin uses Cloudflare as well.

If your instance uses Cloudflare, then you have nothing to worry about, as your admin clearly has federation working. You wouldn't be posting here if they didn't.

That seems troublesome. Is it more of an issue with Cloudflare or just improper configuration by the instance admin?

I would assume improper configurations on the part of the admin. I've never tried setting up Lemmy, but if I had to take a wild guess people aren't configuring the SSL right and other instances aren't talking to them because the domain on the cert is wrong.

If you decide to use Akamai, hmu. I'm not an Alamai guru, but I do it professionally.

I joined July 1 for obvious reasons, I love it here.

I'm asking how many times has lemmy had to deal with these kinds of attacks prior to the "Date shall not be named"

Cause it kinda seems pretty coincidental for the amount of times I've been forced to be "offline" on this platform is gd laughable.

I thumbs up good content always.

Is it just me, or is old.lemmy.world still using its old IP address?

lemmy.world and m.lemmy.world

64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=1 ttl=56 time=46.9 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=2 ttl=56 time=50.3 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=3 ttl=56 time=48.0 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=4 ttl=56 time=50.1 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=5 ttl=56 time=50.1 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=6 ttl=56 time=50.2 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=7 ttl=56 time=47.0 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=8 ttl=56 time=54.0 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=9 ttl=56 time=50.1 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=10 ttl=56 time=49.8 ms
--- lemmy.world ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9010ms
rtt min/avg/max/mdev = 46.893/49.635/54.048/1.956 ms

old.lemmy.world should be cloudflared, too

Thanks a lot for all the work you folks are doing to keep this instance up.

Yeah, first make sure it doesn't show anywhere anymore

I assume you are rotating ip addresses after swapping to cloudflare?

CloudFlare IP ranges can be found here. The DNS entry can point to any one of those IP addresses.

I think Ryan is referring to the usual requirement that the server's IP address is changed if switching to a CDN to avoid DDoS, since otherwise the attackers can usually just bypass the CDN by sending requests to the original IP of the server.

Not an issue if you only accept request from the cloudflare IPs and reject everything else

Depends on how big the attack is I think - inbound connection handling is not free, even if you're just rejecting

I mean, on your origin you can control the firewall of your own webserver. If you only accept https from the cloudflare IPs everyone using your Url should be patched thought cloudflare without issue and the attack wouldn't be much of a problem as they would be rejected. I use this method on some of my website at work.

Whats the motivation to DDOS? How mutch is specific malice to lemmy or lemmy.world itself and how much is genaric.

The kinds of people who do these things can have different motivations.

Some DDOS operators are "hired goons" who will DDOS whomever they're paid to. However, in order to demonstrate their capabilities, they need to do some damage first. If they can cause a big outage, they can later point to that outage and say "we did that" as proof that they're capable of doing damage.

Some DDOS operators are ideological or identity/drama-driven. They decide that they have a Cause, and that this justifies doing some damage. The same groups might do DDOS and also harassment, doxxing, spamming, etc. — their goal is to cause misery to the Bad People and "drive them off the Internet" by whatever means they find handy.

Some DDOS operators are just plain extortionists. They crash a site once or twice, then threaten to keep doing it forever until the site owner pays them off.

Some DDOS operators are bored kids making trouble.

Some DDOS operators are nation-state agencies trying to censor foreign sites that say things they don't like. In one case, the China government attacked GitHub to get at the anti-censorship site GreatFire.

8 more...
8 more...

Yeah, this is just growing pains for any website. Get popular enough for it to be "fun" to target. Then get enough data that it's "profitable" to target. Etc. And the usual way to deal is to first use an external solution at least until it becomes too expensive due to traffic volume. Then make your own solutions for problems you can solve yourself and pay external companies for the ones you can't.

Not sure if it’s related, but today on Mastodon, I’m unable to upload photos. Also can’t see pics from other users. Profile pics are mostly greyed out too.

Script Kiddies are definetely some of the saddest people on the internet. If you're gonna be an unethical hacker at all, actually do it. Don't be a sissy.

It's about skills. Script kiddie can download and run a script written by someone else but that's pretty much it.

Downloads Uber hax. Run script. Computer dies. "Heh, I am a god."

thank you guys for your work! is it possible to disable the cloudflare analytics/telemetry aka cloudflareinsights?

Do you run a reverse proxy infront? Eg. nginx is pretty performant at dropping unwanted traffic.

That doesn't help with volumes of otherwise legit looking traffic right? The problem that Cloudflare and Akamai etc address is usually content that is otherwise static that can be cached. Say the front page of hot lemmy.world is updated every few minutes with the newest hot item. That page is otherwise distributed by the CDN so the CDN can just direct the traffic to access it, and no requests are made to lemmy.world.

nginx would be helpful for any attacks located from a single address trying to making large numbers of connections, but without reading more into the attack I can suspect that this isn't what the attacker did.

Any news? I'm still seeing empty pages sometimes (db errors I think), s6 wonder if the kiddies are somehow getting through despite cloudflare.

You should change the public IP of the server if you haven't already

What happens tomorrow? Change the IP again? And again? It's not a long term solution.

They mean after adding a ddos mitigation like cloudflare, you should rotate the origin server IP so the origin server's IP is no longer publicly known and thus not directly reachable by ddos attackers. The only way to now interact with the application is though Cloudflare's network. You should only have to do this once as long as the origin IP doesn't publicly leak.

Another step would be to add firewall rules to only allow inbound traffic from cloudflare IPs: https://www.cloudflare.com/ips/

I recall a certain amount of overhead in IPTables "allow only from" situations but I'm not sure whether it's enough to make a DDOS any kind of viable on a server in this configuration.

Do you happen to know how effective the strategy is?

If your origin servers IP is never revealed then all traffic goes through cloudflare regardless. Firewall restricting the IPs is just good practice since cloudflare is the only IP that is supposed to talk to that server anyway, but it's not a requirement.

I can see some overhead if you're maintaining a large blacklist, but I don't see it happening with a small whitelist and default inbound DROP

Oh absolutely, I agree with the best practice! I just didn't know the real world efficacy of dropping packets near the NIC to mitigate DDOS load. There is certainly a performance limit but where that limit exists has been nebulous for me.

Cloudflare masks the origin IP address and has DDoS protection. Unless it's a DoS against the software, yes, it is a long term solution.

How does cloudflare work? Do you install the private SSL certificate there and so cloudflare can see all traffic, including passwords, in plain text or is the path from browser through to your server still encrypted?

Cloudflare decrypts to do the ddos protection, then reencrypts to the server.

If you are worried about security, cloudflare is provably more secure than any lemmy server.

But it still is a really bad idea to route big parts of the internet through one proprietary system. There have to be other ways to solve this.

1 more...
1 more...

Cloudflare is a proxy, so by its very nature it has to decrypt traffic. (I believe their enterprise plans may offer a way around this, but don't quote me.)

I wouldn't worry, however. If someone wanted to attack this site (or any site, really) they're almost certainly going to have an easier time going after the origin rather than trying to take on a juggernaut like Cloudflare.

Other posters are correct that cloudflare decrypts traffic. BUT it is highly unlikely that they will see your password in plaintext, since it is best practice to hash the password first on the front-end.

1 more...

Also when will CloudFlare drop lemmy as a 'Nazi' site?

Cloudflare makes the website feel dirty, but it'll protect the site until a better option is found.

Cloudflare makes the website feel dirty, but it'll protect the site until a better option is found.

Can you elaborate what you mean by this? Lots of sites use cloudflair and most users of those sites would never even know. What makes it dirty?

Its not. I explained in the below comment why Cloudflare and others like them helps you decentralise. And other benefits.

It was an intentionally loaded question for the commentor I was replying to. I've used cloudflair services many times because they provide outstanding service...they are a company who consciously decided they wanted to excel in just a few things, they got very good at those few things, and they are still very good at those few things.

I speculate people are kneejerking against the notion of "putting all your eggs in one basket", whithout stopping to think that cloudflair's entire service offering is about spreading your eggs out across multiple baskets. And it's not like you're stuck with them - companies can and do partner with multiple cdn's all the time to diversify their peering arrangements.

Cloudflare definitely has a great service and the positives probably overweight the negatives in this situation. But the potential for an attack from within cloudflare itself via trackers or a probably very low chance of a letter man being in the middle can feel a little tense. It boils down to not trusting the company. I especially do not like those outages, captchas, cookies and a centralized web. Cloudflare will help Lemmy stay on top of everything and keep stability though.

Maybe being addicted to uBlock having only green and no detections makes me worried. It's like a little bit of dirt on the floor.

2 more...
2 more...

It's been feeling sluggish all day long as well. I've been trying to post from my phone and PC, and it seems it's really slow from time to time.

I had a hard time signing in the other day as I got confused in the instances but otherwise I'm enjoying the experience browsing here using the summit app.

now I'm convinced that that cunt spez paid those brats to sabotage us

Tagging @ernest in case instance owners don't have a larger community in which they share news like this with each other.

Well I signed today and I got an error saying rate limit earlier for using these types of symbols "î¦âö)ééäë((ºÜݨ¿ã¿ï" I'm assuming It has nothing to do with this but just In case I'm making a comment about it edit:also just realized It may have been from how long the password was (33 characters)

This might be related. Encrypting passwords is resource intensive, and longer passwords need more resources.

Specifying really long passwords, repeatedly, is one way to DDoS a server. Maybe they're blocking unnecessarily long passwords.

http://crimeflare.eu.org list reasons why not to use Cloudflare, though IDK if it's just ultra-privacy oriented warnings or something else...

Not sure if I should be upset, although the claim of CF potentially sniffing passwords/credit card details/other sensitive information across various websites sounds plausible to me (some websites even have a TLS cert verified by "Cloudflare, Inc."!) 🤷‍♂️

They would be completely ruined if they were doing any of these things and proven to be doing them. Nothing to worry about for now.

Yeah, you're right. Just because they can doesn't mean they have to. We might not know what they're up to behind doors and for me it's horrifying to know the potential damage but hopefully it's in good faith.

they have a bad habbit of getting Brain bleeds

1 more...

A lot of people don't like that they've basically inserted themselves as a man in the middle to a lot of internet traffic.