Hacker News (AI, 50+ points)Article
Stay discoverable in search while disallowing AI training
The robots.txt mechanism is crude but it's the tool everyone has, and Cloudflare's framing of "accountable mixed-use" signals that the search and training tension is now a business decision, not a technical problem. For builders: if you're scraping for training data, you need a policy for respecting robots.txt or you will face attrition. For publishers: understand that blocking training crawlers has a real cost in SEO and visibility. This is a permanent tradeoff, not a temporary friction.