Worked example
Robots.txt Allows Crawling, but the Page Returns 403
A controlled example showing why crawler rules and HTTP access need separate checks, with a repeatable before-and-after test.
By Starry Research Labs · Published and reviewed
The test setup
This is a controlled example from our checker tests, run on October 17, 2026. It uses synthetic responses and reserved example domains. It is not a customer case or a measurement of a real Googlebot request.
The robots.txt response contains User-agent: * and Disallow: /private. The tested path is /. Its rule result is allowed. In a separate page-response test, the same public URL returns HTTP 403. The website checker returns a partial access diagnosis and does not invent a title, canonical or passing result from the denied page.
Why both results can be correct
Robots.txt publishes crawl instructions. A firewall, an authentication requirement or the page server can still refuse an HTTP request. An allowed rule is evidence about the file, not evidence that the response can be fetched.
Our checker identifies itself as StarryResearchLabs-Checker. Its request is not an authenticated request from Google, OpenAI or another crawler operator. A refusal can therefore describe access for our checker without establishing how a verified crawler is treated.
How to investigate your own site
Run the robots.txt checker on the exact URL path, then run the homepage website check. Keep the returned status, tested URL and check time. If the page is refused, inspect the relevant hosting and firewall logs before changing a rule.
Allow the public page to return the intended HTML, then run a fresh check. Confirm the response changed before interpreting page metadata. A successful check still does not prove that the page has been indexed or cited.
Sources and method
External references were reviewed for this guide. Examples and tool workflows are our own. The launch audit records our public website observations, with no ranking or citation claim.