New: one shared opt-out for AI training
Type your website. Doorman reads its robots.txt and tells you who can train on it, who can find and cite it in AI search, and who can read it for people. Then pick what you want and copy a fixed file.
Doorman keeps your own rules, drops old AI-bot lines and adds one clear block. Google Search, Bing and Apple search are never touched.
Put it at the root of your site, so it opens at yoursite.com/robots.txt. robots.txt is a request, not a lock: the big AI companies say they honor it.
GPTBot trains models; OAI-SearchBot finds pages for ChatGPT search; ChatGPT-User opens a page because someone asked. You can keep one out and let the others in.
“User-agent: __usage_ai-training” is a new shared line Google, Microsoft, OpenAI and Anthropic say they will honor. Microsoft starts Feb 1, 2027 and Google in early 2027. A “User-agent: *” rule never counts as it.
Doorman fetches a site’s public robots.txt, nothing else, and caches it for 10 minutes. A pasted file is checked in your browser and never uploaded.
Doorman reads any website's robots.txt and shows which AI bots it lets in: who can train on it, which AI search engines can find and cite it, and which assistants can read it for people. Pick what you want and copy a fixed robots.txt, including the new shared AI-training opt-out line. No account.
Add a robots.txt group that disallows the training crawlers by name (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider and others) plus the shared line "User-agent: __usage_ai-training" with "Disallow: /". Doorman writes this for you and keeps the rest of your file.
A shared robots.txt "policy token" from an IETF draft submitted on Oct 30, 2026. A group like "User-agent: __usage_ai-training / Disallow: /" states one AI-training preference for every provider that reads it. Google, Microsoft and OpenAI said they will honor it (Microsoft from Feb 1, 2027, Google in early 2027) and Anthropic has started implementing it. A "User-agent: *" group never counts as this preference.
Yes. Training and search use different bots. Keep GPTBot, ClaudeBot and Google-Extended out, and let OAI-SearchBot, Claude-SearchBot and PerplexityBot in. Doorman calls this the "Guest list" setup.
No. Google-Extended only controls Gemini training and grounding. Google Search uses Googlebot, which Doorman never changes. Blocking Googlebot would remove you from Google Search, and Doorman warns you if your file does.
It is a request that the major AI companies say they honor. Some bots ignore it, and assistants fetching a page because a person asked may not follow it. For hard blocking you need a firewall or bot management rule.
Yes. There is a public JSON API (GET https://doorman.clawhard.com/api/v1/check?url=example.com, see /openapi.json) and the tools doorman_check_site, doorman_check_robots and doorman_build_robots on the Clawhard MCP server at https://mcp.clawhard.com/mcp.
For AI assistants and developers: llms.txt · API (OpenAPI) · MCP server · sitemap