site.crawl-stopped

InfoSEO auditcrawl

The hosted server stopped requesting the site before the crawl was done: the site refused its requests (a challenge, HTTP 429), robots.txt asked for a Crawl-delay longer than a hosted crawl waits, or the server reached its request limit for the site. Site-wide checks (broken links, duplicates, sitemap coverage) are partial.

What to change

The message says which. The hosted server's own limit for a site lifts within a minute: crawl again then. A refusal or a Crawl-delay is the site's choice, which the hosted server keeps.

A finding of this rule on a page carries the exact edits for that page: the markup it quotes, and what to write instead.

When it runs

Runs on a crawl or a build directory. Info is a best practice: worth doing, rarely urgent, and --no-info leaves it out. Each finding takes 1 point off the page's score.

The severity here is the rule's usual one: a finding can be raised or lowered when the page gives a reason.

Run it, or skip it

Check a page for this rule alone, after a fix for instance:

npx @rivi-software/agentic-seo-check crawl https://example.com --rule site.crawl-stopped

Or from an assistant, with the MCP tool seo_crawl_site:

{
  "url": "https://example.com",
  "rules": [
    "site.crawl-stopped"
  ]
}

To leave it out of an audit, pass --disable site.crawl-stopped on the command, or disableRules: ["site.crawl-stopped"] to a tool.

All 49 rules