site.robots-txt-missing

What to change

A finding of this rule on a page carries the exact edits for that page: the markup it quotes, and what to write instead.

When it runs

Runs on a crawl or a build directory. A warning is a problem that costs a page ranking, clicks or citations. Each finding takes 5 points off the page's score.

The severity here is the rule's usual one: a finding can be raised or lowered when the page gives a reason.

Run it, or skip it

Check a page for this rule alone, after a fix for instance:

npx @rivi-software/agentic-seo-check crawl https://example.com --rule site.robots-txt-missing

Or from an assistant, with the MCP tool seo_crawl_site:

{
  "url": "https://example.com",
  "rules": [
    "site.robots-txt-missing"
  ]
}

To leave it out of an audit, pass --disable site.robots-txt-missing on the command, or disableRules: ["site.robots-txt-missing"] to a tool.

• Warningsite.robots-txt-errorrobots.txt returned a server error or could not be fetched. Google treats 5xx robots.txt as "do not crawl anything".
• Errorsite.robots-txt-blocks-allrobots.txt disallows crawling of the whole site for all user agents.
• Warningsite.robots-blocked-pagesPages linked internally (or listed in the sitemap) are disallowed by robots.txt.
• Warningsite.robots-txt-too-largerobots.txt is larger than 500 KiB; Google ignores everything after the first 500 KiB.
• Warningsite.robots-txt-unsupported-rulerobots.txt uses rules Google does not support: noindex or nofollow lines (retired in 2019, so the pages stay indexable; warning) or crawl-delay (Google ignores it, Bing reads it; info).
• Warningsite.robots-txt-invalid-rulerobots.txt has lines crawlers cannot use: an Allow or Disallow path that starts with neither / nor * (a full URL included), which matches no URL; an Allow or Disallow above the first User-agent line, which belongs to no group; or a Sitemap that is not a full URL, which search engines ignore.
• Warningsite.robots-blocks-resourcesrobots.txt keeps Googlebot from scripts or stylesheets the pages load, so Google renders the pages without them.
• Warningsite.sitemap-missingNo XML sitemap was found (robots.txt Sitemap: line or /sitemap.xml).