Skip to content
Docs/Utility Tools

Robots.txt Checker

The Robots.txt Checker fetches and analyzes a site's robots.txt file, showing exactly what search engine crawlers are allowed or blocked from accessing.

How to Use

  1. Go to /utilities/robots-checker
  2. Enter a domain or URL and click Check
  3. The tool fetches the robots.txt file and parses its directives

What the Results Show

  • User-agent rules - Which bots have specific rules and what those rules are
  • Allowed paths - URLs and directories that crawlers can access
  • Disallowed paths - URLs and directories that are blocked from crawling
  • Sitemap references - Any sitemap URLs declared in the file
  • Crawl-delay directives - Whether a crawl delay is specified for any bot
  • Syntax issues - Invalid lines, typos, or formatting problems

The tool highlights potential issues like accidentally blocking important pages or missing sitemap declarations.

More in Utility Tools

SEO AuditPageSpeed CheckerKeyword ResearchContent Brief GeneratorSchema GeneratorBroken Link FinderRedirect CheckerDNS LookupGlobal Latency TestHTTP Header InspectorMetadata CheckerSitemap CheckerKeyword CombinerSERP PreviewHeading AnalyzerContent AnalyzerInternal Link AnalyzerOG Tag DebuggerBulk Status CheckerCanonical CheckerAnchor Text AnalyzerStructured Data ValidatorUptime CheckerAccessibility Checker