GitHub Add to your assistant
Menu

aeo.robots-blocks-training-bot

InfoAEO audit

robots.txt blocks a crawler that collects text to train AI models (GPTBot, ClaudeBot, Applebot-Extended, CCBot, meta-externalagent, Amazonbot, MistralAI-Training).

What to change

Nothing to change if you mean to opt out of model training: these bots do not decide whether answer engines cite the site. Allow them if you want the content in future models.

The documentation on developers.google.com

A finding of this rule on a page carries the exact edits for that page: the markup it quotes, and what to write instead.

When it runs

Runs on every AEO audit: of a page, a crawl, a build directory or HTML. Info is a best practice: worth doing, rarely urgent, and --no-info leaves it out. Each finding takes 1 point off the page's score.

The severity here is the rule's usual one: a finding can be raised or lowered when the page gives a reason.

Run it, or skip it

Check a page for this rule alone, after a fix for instance:

npx @rivi-software/agentic-seo-check aeo https://example.com --rule aeo.robots-blocks-training-bot

Or from an assistant, with the MCP tool seo_aeo_audit:

{
  "url": "https://example.com",
  "rules": [
    "aeo.robots-blocks-training-bot"
  ]
}

To leave it out of an audit, pass --disable aeo.robots-blocks-training-bot on the command, or disableRules: ["aeo.robots-blocks-training-bot"] to a tool.

All 32 rules