Search access
Check AI Crawler Access in Your Robots.txt
Check crawler rules separately from website responses, indexing and AI citations. Learn what a missing robots.txt means and how to test a path.
By Starry Research Labs · Published and reviewed
What does a robots.txt check tell you?
A robots.txt check tells you which published rules apply to a crawler and a URL path. It helps you find a broad Disallow rule or a more specific exception. It does not show whether a crawler has visited your website.
Our checker displays the winning directive and its line number. You can paste draft rules before publishing or fetch the public file after a change. Keep the exact path you tested with the result.
Does a 404 robots.txt block crawling?
Google treats a 404 robots.txt response as an absent file and applies no crawl restrictions from it. A page can still have its own indexing restriction, require authentication or be blocked by hosting rules.
A server error needs a different diagnosis. If the file cannot be fetched normally, a checker should report the failed response rather than inventing an allowed or blocked verdict.
Why do AI crawler names matter?
Different crawler names can have different jobs and different rules. Test the actual agent token against the file. A rule for GPTBot, for example, should not be treated as a measurement of whether ChatGPT mentioned your business.
Use our crawler comparison to see each rule result. Then check your hosting settings and verified request logs if you need evidence of real access. Sending an HTTP request with a bot-shaped user agent does not authenticate that request as the real crawler.
What should you save after a change?
Save the file, the tested URL path, the agent token and the timestamp. Re-run the public check after deployment. If a page remains absent from search, inspect its response, canonical and indexing directives next.
For AI discovery, follow this technical check with real answer samples. Keep crawler access results and observed citations in separate columns.
Sources and method
External references were reviewed for this guide. Examples and tool workflows are our own. The launch audit records our public website observations, with no ranking or citation claim.