AI companies are violating a basic social contract of the web and and ignoring robots.txt

Andy Reid@lemmy.world to Technology@lemmy.world – 933 points –
The rise and fall of robots.txt
theverge.com
198

You are viewing a single comment

What social contract? When sites regularly have a robots.txt that says "only Google may crawl", and are effectively helping enforce a monolopy, that's not a social contract I'd ever agree to.

When sites regularly have a robots.txt that says “only Google may crawl”

Is that actually true?

If so, why would they do that?