AI Robots.txt Checker
Paste your robots.txt, catch the mistakes that quietly deindex a site.
No live URL fetch needed — paste the file directly.
Most specific path wins, not first-listed rule — a common source of unexpected crawling behavior.
A stray Disallow: / or a blocked CSS/JS path gets flagged before the smaller issues.
Most robots.txt checkers fetch a live URL and run it against a matrix of a dozen-plus named crawlers, returning a pass/fail grid. This tool works from pasted robots.txt content and focuses on explaining why the flagged lines matter, not just listing which bots are blocked. Given a staging-leftover file — `User-agent: *`, `Disallow: /wp-admin/`, `Disallow: /`, plus a `Sitemap:` line — the explanation doesn't just flag `Disallow: /` as an issue; it states plainly that this line alone blocks every path on the site for every crawler matched by `User-agent: *`, and — critically — that the narrower `/wp-admin/` rule directly above it does NOT limit or override the broader one. Both rules apply simultaneously; the broad rule is enough on its own to deindex the entire site, which is precisely the trap that makes a leftover staging robots.txt so dangerous when it ships to production unchanged. That leads into the real matching rule most people get wrong, per the official Robots Exclusion Protocol (RFC 9309): the most specific — meaning longest — matching path wins, not whichever rule happens to be listed first in the file. The tool also treats AI crawlers as their own category rather than lumping them under general search bots: GPTBot, Google-Extended, ClaudeBot, and CCBot are distinct named user-agents serving different purposes — model training versus real-time browsing and citation — and a blanket `User-agent: *` rule covers all of them unless a more specific rule targets one by name, which is a distinction most site owners don't realize they're making by default. It's built for anyone shipping a robots.txt change — a developer double-checking a deploy, an SEO reviewing a site migration, or anyone who's ever accidentally copied a staging config into production. One honest limitation: this checks the file's own logic and known crawler-handling conventions; it doesn't verify the file is actually being served at the correct URL or with the correct content-type, which needs a live fetch to confirm. Built by EngraveOcean as part of a small, growing set of free AI developer utilities, alongside AI JSON Schema Explainer and AI Schema Markup Generator — no login, no daily cap.