Most robots.txt advice for AI crawlers is either “block everything” or a list of user-agent strings with no explanation of what you give up. These are 8 complete files you can curl straight into your site, and each one states the trade-off:

curl -s https://www.pathwren.workers.dev/robots/block-ai-training.txt >> robots.txt

• block AI training but keep the assistant fetchers that follow a person’s link and cite you
• block dataset crawlers only, leave search alone
• block everything automated except the classic search engines
• allow everything explicitly, which is a real choice and is written out as one

Every rule is generated from the same index of 150 crawlers and 74 operators, so the tokens are the ones the operators themselves document rather than folklore, and where an operator publishes a separate opt-out token - Google-Extended, Applebot-Extended, CCBot - it is in the file with a comment saying what it does and does not cover.

Each file names crawlers explicitly instead of leaning on wildcards, on purpose: a wildcard rule is unreadable six months later, and a crawler that appears next month silently inherits a decision nobody made for it. Named tokens mean the next change is a one-line diff you can review.

Two honest caveats. robots.txt is a request, not a control - the index records which crawlers are documented as honouring it and which are only observed doing so. And a user-agent is free to type, so 1987 IPv4 and 1062 IPv6 published prefixes from 15 operator endpoints are mirrored here too, re-fetched every six hours, for when you want to check the claim.

Static files, no key, CC0.

https://www.pathwren.workers.dev/c/lemmy/policy/

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)