robots.txt will never be enforceable by law. It would completely undermine the fabric of the internet as we know it.
Additionally, it would make it impossible for any new company to enter the search space.
Beyond that, imagine if it were.
Google (for instance) could then, in theory, promote sites with which they had exclusivity arrangements and even de-list those with which they didn’t.
Creating a bifurcated internet.
No, I think the unintended consequences of a legally enforceable robots.txt file would be far too grievous to ever even entertain that as a possibility.
Besides, website terms of service have already been found to be unenforceable, and what is robots.txt but a very terse terms of service?