Skip to content
Starry Research Labs

Check search access · Free tool

Robots.txt checker

Test a public URL or pasted robots.txt against Googlebot, Bingbot and AI crawler rules. See the matching directive and its line number.

Choose a source

The URL is sent to our server for this check. Results are not saved as a public report.

Your evidence will appear here, with the URL and time of the check.

What this tool does

Fetches the public robots.txt using our checker user agent or evaluates pasted rules. It matches agent groups and URL paths. A rule allowing a crawler does not prove that crawler visited, that a firewall allows it or that the page is indexed.

Method reviewed 2026-10-17. Version 1.0 · First public release.

Before you use the result

Does a missing robots.txt block Google?

A 404 response means Google treats the file as absent, with no crawl restrictions from that file. Other access and indexing controls can still apply.

Does allowing an AI bot guarantee a citation?

No. Access, indexing and selection as a cited source are separate checks.