Google does do the things you mention. I don’t understand how it would undermine the fabric of the internet. Google follows and abides by robots.txt.
You misunderstood. I’m not talking about what Google does or doesn’t do or even what they could do so much as what they could compel other sites to do.
There’s a difference if robots.txt becomes a legal document. In that case, there is an incentive for a player like Google to force sites to weaponize robots.txt.
Imagine if 3/4 of the sites on the internet were compelled set up their sites to disallow all, only whitelisting Google?
This may not be a particularly likely outcome, but a legally enforceable robots.txt file could lead to a world where people need to use 2,3, 15 different search engines to find relevant information.
Terms of Service may be hard to enforce against user, but not commercial services. Digital trespassing means a whole lot more when a paid-for commercial service is blatantly abusing it.
I have yet to see any evidence of that.
If you don’t see how all the big players are locking up, and putting a price on their public data, then I can understand how you can think that none of this happen.
How none of what happens?
Robots.txt plays a huge part in the digital space. It’s not simply just a terms of service. Data scraping is a growing industry that is potential lost profit.
Sure, but… a legally enforceable robots.txt isn’t the answer, it’s unlikely to be about to actually be enforced, and it’s not something I think anyone is actually actively considering.