I guess Reddit is permitted to only let Google index it
Are they though?
I don't know of any law that says that they can't.
I also don’t know if a law that says search engines have to honor a robots.txt file. I guess we will see what happens if Bing or some other service decides to ignore it.
You can just require a log in to view content, or just flat out auto ban indexing robots.
I guess Reddit is permitted to only let Google index it
Are they though?
I don't know of any law that says that they can't.
I also don’t know if a law that says search engines have to honor a robots.txt file. I guess we will see what happens if Bing or some other service decides to ignore it.
You can just require a log in to view content, or just flat out auto ban indexing robots.