AI companies are violating a basic social contract of the web and and ignoring robots.txt

Andy Reid@lemmy.world to Technology@lemmy.world – 933 points –
The rise and fall of robots.txt
theverge.com
198

You are viewing a single comment

The funny (in an "wtf" not "haha" sense) thing is, individuals such as security researchers have been charged under digital trespassing laws for stuff like accessing publicly available ststems and changing a number in the URL in order to get access to data that normally wouldn't, even after doing responsible disclosure.

Meanwhile, companies completely ignore the standard mentions to say "you are not allowed to scape this data" and then use OUR content/data to build up THEIR datasets, including AI etc.

That's not a "violation of a social contract" in my book, that's violating the terms of service for the site and essentially infringement on copyright etc.

No consequences for them though. Shit is fucked.

Corporations are people except when it comes to liability. Compare the consequences of stealing several thousand dollars from someone by fraud vs. stealing several thousand dollars from someone by fraud as an LLC.

Just thought of a nasty hack the browser makers (or hackers) could use to scrape unlisted sites - by surreptitiously logging user browser history for a crawl list

1 more...