GitHub Add to your assistant
Menu

site.robots-txt-too-large

WarningSEO auditcrawl

robots.txt is larger than 500 KiB; Google ignores everything after the first 500 KiB.

What to change

Shorten robots.txt: group paths under shared prefixes, use wildcards (Disallow: /*?sort=) instead of listing URLs one by one, and remove rules for paths that no longer exist.

The documentation on developers.google.com

A finding of this rule on a page carries the exact edits for that page: the markup it quotes, and what to write instead.

When it runs

Runs on a crawl or a build directory. A warning is a problem that costs a page ranking, clicks or citations. Each finding takes 5 points off the page's score.

The severity here is the rule's usual one: a finding can be raised or lowered when the page gives a reason.

Run it, or skip it

Check a page for this rule alone, after a fix for instance:

npx @rivi-software/agentic-seo-check crawl https://example.com --rule site.robots-txt-too-large

Or from an assistant, with the MCP tool seo_crawl_site:

{
  "url": "https://example.com",
  "rules": [
    "site.robots-txt-too-large"
  ]
}

To leave it out of an audit, pass --disable site.robots-txt-too-large on the command, or disableRules: ["site.robots-txt-too-large"] to a tool.

All 49 rules