GitHub Add to your assistant
Menu

site.robots-txt-blocks-all

ErrorSEO auditcrawl

robots.txt disallows crawling of the whole site for all user agents.

What to change

Remove "Disallow: /" for "User-agent: *" unless the site is intentionally private (staging).

The documentation on developers.google.com

A finding of this rule on a page carries the exact edits for that page: the markup it quotes, and what to write instead.

When it runs

Runs on a crawl or a build directory. An error is a problem that keeps a page out of search or out of answers, or breaks how it shows there. Each finding takes 15 points off the page's score.

The severity here is the rule's usual one: a finding can be raised or lowered when the page gives a reason.

Run it, or skip it

Check a page for this rule alone, after a fix for instance:

npx @rivi-software/agentic-seo-check crawl https://example.com --rule site.robots-txt-blocks-all

Or from an assistant, with the MCP tool seo_crawl_site:

{
  "url": "https://example.com",
  "rules": [
    "site.robots-txt-blocks-all"
  ]
}

To leave it out of an audit, pass --disable site.robots-txt-blocks-all on the command, or disableRules: ["site.robots-txt-blocks-all"] to a tool.

All 49 rules