robots.txt
Free AI robots.txt checker
See whether GPTBot, ClaudeBot, PerplexityBot and the rest can read your site. Copy an allowlist patch. Nothing is written to your server.
Questions
What does Allowed, Blocked and Not mentioned mean?
Allowed means a matching group lets the crawler read the homepage. Blocked means a matching Disallow wins. Not mentioned means the file never names that crawler and there is no * group, which crawlers treat as allowed.
There is no robots.txt. Is that bad?
No. A missing file is treated as allow-all. We still show the table so you can see that, and a patch you can add if you want the named crawlers written down.
Will this open my site to every bot?
No. The patch adds named groups for AI crawlers. If User-agent: * currently disallows everything, we say so, and the named groups override that for those crawlers only.
Can you write the file onto my server?
No. Copy the patch and paste it yourself. We never write to your site.