you think either of those companies pays attention to robots.txt? its not legally binding or anythjng
At the executive level, no I don't think they care or pay attention, but considering both have said "here's how to block our crawler," I do hope that that some mistreated developer did actually program a check in to the crawler. I still think it's worth doing, even though I don't fully trust them.
you think either of those companies pays attention to robots.txt? its not legally binding or anythjng
At the executive level, no I don't think they care or pay attention, but considering both have said "here's how to block our crawler," I do hope that that some mistreated developer did actually program a check in to the crawler. I still think it's worth doing, even though I don't fully trust them.