GitHub Add to your assistant
Menu

aeo.robots-blocks-search-crawler

ErrorAEO audit

robots.txt blocks a search crawler whose index AI answers draw on: Googlebot (AI Overviews, AI Mode), Bingbot (Copilot) or Applebot (Siri, Spotlight).

What to change

Allow these crawlers in robots.txt: remove the Disallow lines that match the audited pages in their groups, or in `User-agent: *` when they have no group of their own. Applebot follows the Googlebot group when no group names Applebot.

The documentation on developers.google.com

A finding of this rule on a page carries the exact edits for that page: the markup it quotes, and what to write instead.

When it runs

Runs on every AEO audit: of a page, a crawl, a build directory or HTML. An error is a problem that keeps a page out of search or out of answers, or breaks how it shows there. Each finding takes 15 points off the page's score.

The severity here is the rule's usual one: a finding can be raised or lowered when the page gives a reason.

Run it, or skip it

Check a page for this rule alone, after a fix for instance:

npx @rivi-software/agentic-seo-check aeo https://example.com --rule aeo.robots-blocks-search-crawler

Or from an assistant, with the MCP tool seo_aeo_audit:

{
  "url": "https://example.com",
  "rules": [
    "aeo.robots-blocks-search-crawler"
  ]
}

To leave it out of an audit, pass --disable aeo.robots-blocks-search-crawler on the command, or disableRules: ["aeo.robots-blocks-search-crawler"] to a tool.

All 32 rules