TikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25-times faster than OpenAI

Luu Tuyen@lemmy.world to Technology@lemmy.world – 561 points –
TikTok’s parent launched a web scraper that's gobbling up the world’s online data 25-times faster than OpenAI
fortune.com
131

You are viewing a single comment

They obfuscate their traffic by randomizing user agents, so it's either add a global rate limit, or let them ass fuck you

the article told all source IPs can be tracked back to bytedance. Wouldn't it be possible to block them? maybe even blocking all IPs of a specific ASN

They can be tracked back one by one but if you have any amount of traffic it's a constant game of cat and mouse.

You can block entire ASNs until they start using residential proxies provided by less ethical companies. Then you end up blocking all of France or destroying user experience by enforcing a captcha on everyone.

Why do they need to hit a website like that? Wouldn't it just need to scrape the data and frig off. What is the point of creating that much traffic