This is the heaviest check in the scan, and it is the one everything else depends on. An engine that cannot fetch your pages cannot quote them, so it answers the question from somebody else's site instead. Fixing it is usually two minutes.
Two different sets of crawlers
Most advice online treats these as one thing. They are not, they fail for different reasons, and the scan scores them separately.
Answer crawlers read your site to answer a question somebody is asking right now.
OAI-SearchBot,ChatGPT-User,PerplexityBotandPerplexity-User. Blocking these is why you are absent from answers today.Training crawlers gather material that shapes what future models know about businesses like yours.
GPTBot,ClaudeBot,Google-ExtendedandCCBot. Blocking these is a legitimate choice some businesses make over content licensing.
That is why one is a failure and the other is only ever a warning. If you blocked the training crawlers on purpose, leave them blocked. The scan is not going to argue with you about it.
The block is in one of two places
Read the technical detail on the finding first, because it names which. They need different fixes and editing the wrong one changes nothing.
Your robots.txt file. It lives at
yourdomain.com/robots.txt. Open that address in a browser to see what is currently there.Your firewall or CDN. If the finding says a crawler was turned away by your firewall, editing robots.txt will not help. The request never reached your site.
Fixing robots.txt
Add the block the Fix Pack gives you rather than replacing the file. The rest of it is doing a job, usually keeping search engines out of your admin pages.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /If there is an existing rule naming one of those crawlers with Disallow: / under it, delete that rule. An allow and a disallow for the same crawler is ambiguous, and the engines do not all resolve it the same way.
Fixing the firewall
Bot protection that blocks unknown crawlers catches these too, and it is usually switched on by whoever set the site up rather than by you. Look for bot protection, known bots, or a user agent rule, and allow the four names above.
On Cloudflare it is under Security. If you did not configure it, forward the crawler names to whoever hosts your site. That is all they need.
Check it worked
Wait a few minutes, then run a free scan again. This is the check everything else depends on, so it is worth confirming rather than assuming.
If your screens do not look like this, the wording has moved rather than the setting. Search your platform for the words in bold above. If you still cannot find it, send us the scan link and we will tell you where it is on your setup.