GitHub Add to your assistant
Menu

site.robots-blocked-pages

WarningSEO auditcrawl

Pages linked internally (or listed in the sitemap) are disallowed by robots.txt.

What to change

Allow the paths in robots.txt, or stop linking to them and remove them from the sitemap if they should stay hidden.

The documentation on developers.google.com

A finding of this rule on a page carries the exact edits for that page: the markup it quotes, and what to write instead.

When it runs

Runs on a crawl or a build directory. A warning is a problem that costs a page ranking, clicks or citations. Each finding takes 5 points off the page's score.

The severity here is the rule's usual one: a finding can be raised or lowered when the page gives a reason.

Run it, or skip it

Check a page for this rule alone, after a fix for instance:

npx @rivi-software/agentic-seo-check crawl https://example.com --rule site.robots-blocked-pages

Or from an assistant, with the MCP tool seo_crawl_site:

{
  "url": "https://example.com",
  "rules": [
    "site.robots-blocked-pages"
  ]
}

To leave it out of an audit, pass --disable site.robots-blocked-pages on the command, or disableRules: ["site.robots-blocked-pages"] to a tool.

All 49 rules