Related articles.
The modern robots.txt file needs rules for a dozen AI-specific bots. We go through each one: who owns it, whether it honors robots.txt, the exact user-agent string, and the rule you need.
