When Lighthouse can’t fetch robots.txt but curl returns HTTP 200
A short diagnostic for separating a browser policy error from the server’s response.
Get Mohammed Teto’s stories in your inbox
Join Medium for free to get updates from this writer.
Suppose the Lighthouse robots.txt audit reports a Content Security Policy (CSP) violation, but a direct request returns HTTP 200 and a plain-text robots.txt body. Keep both results. The audit describes what Lighthouse could fetch in that browser context. The direct request shows what the server returned to that client.
1. Record the audit error
Save the tested page URL, date, Lighthouse and browser versions, and the exact audit message. If it mentions CSP, check the policy on the page. Look in the response headers and, if necessary, in the HTML for a CSP meta element.
2. Request robots.txt directly
Use the final origin of the page you’re authorized to inspect. Replace example.org with that origin.
curl -sS --max-time 10 -i https://example.org/robots.txtRead the response body along with the status. A 200 can still return the wrong content. If the server redirects, record the status and Location header, then inspect the destination separately. Check the directives for the crawler you’re evaluating.
3. Keep the findings separate
For this example, the report would say: “Lighthouse’s fetch failed with a CSP violation. A direct GET returned HTTP 200 and robots directives.” Leave the production CSP in place while you investigate. Use separate evidence for any conclusion about crawler access, indexing or citation in AI answers.
I checked the direct request with a local fixture using connect-src 'none' and connect-src 'self'. Both returned the same robots.txt body with HTTP 200. The Lighthouse behavior is documented in the upstream report.
Read the reproducible diagnostic on GitHub for the commands and the evidence checklist. Follow Mohammed Teto on GitHub for more search and AI audit examples.
