AI companies are violating a basic social contract of the web and and ignoring robots.txt

Andy Reid@lemmy.world to Technology@lemmy.world – 933 points –
The rise and fall of robots.txt
theverge.com
198

You are viewing a single comment

The open and free web is long dead.

just thinking about robots.txt as a working solution to people that literally broker in people's entire digital lives for hundreds of billions of dollars is so ... quaint.

It's up there with Do-Not-Track.

Completely pointless because it's not enforced

Do-Not-Track, AKA, "I've made my browser fingerprint more unique for you, please sell my data"

I bet at least one site I've visited in my lifetime has enforced it