Reddit shares plunge almost 25% in two days, finish the week below first day closeHubi@feddit.de to Reddit@lemmy.world – 834 points – 6 months agocnbc.com172Post a CommentPreviewYou are viewing a single commentView all commentsShow the parent commentWasn't there robots.txt at least?Robots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.There was, and still is
Wasn't there robots.txt at least?Robots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.There was, and still is
Robots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.
Wasn't there robots.txt at least?
Robots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.
There was, and still is