Does your website block AI bots (GPTBot, ClaudeBot)?

Possibly, and you likely wouldn’t know without checking. Many WordPress security plugins and some hosting firewalls block crawlers like GPTBot and ClaudeBot by default, treating them as unwanted traffic. If they’re blocked, no AI assistant can read your site at all, no matter how good the content is, and it usually takes one line in robots.txt to fix.

Why this happens without anyone deciding it

Nobody sits down and decides to block ChatGPT from reading their website. It happens as a side effect of something else: a security plugin’s default rule against ‘unknown bots’, a hosting firewall tuned to stop scraping, or a robots.txt file copied from a template years before these crawlers existed. The block is usually inherited, not chosen.

That makes it easy to miss. The site loads fine for a person, ranks fine on Google, which uses a different crawler, and nothing looks broken. The only symptom is an AI assistant that never mentions the business, and that symptom looks exactly like a content problem, so most owners go looking in the wrong place first.

The crawlers that actually matter

GPTBot reads for OpenAI’s models, ClaudeBot reads for Claude, and there are a handful of others doing similar work for Google’s and Perplexity’s assistants. If any of these are disallowed in your robots.txt, or blocked at the server level by a firewall rule, that assistant has nothing of yours to read and nothing to recommend, regardless of how well written the page is.

This is different from blocking spam or scraper bots, which is often sensible. The AI crawlers above are the same systems answering your future customer’s question, so blocking them is closer to unlisting yourself from the phone book than to stopping a nuisance.

How to actually check

Open yoursite.com/robots.txt directly in a browser and read it. Look for any line naming GPTBot, ClaudeBot, Google-Extended, or PerplexityBot followed by Disallow. If the file is empty or doesn’t mention them at all, you’re likely fine on that front, but the block can also happen at the firewall level, invisible in robots.txt, which needs a check with your host or security plugin settings directly.

A quick way to catch a firewall level block: check your site’s access logs, if you can, for requests from those bot user agents and see whether they’re returning a normal 200 or a blocked 403. If you can’t reach the logs yourself, this is one of the checks I run as part of an AI Visibility Snapshot.

Fixing it takes minutes, not a rebuild

If robots.txt is the problem, removing the Disallow line or adding an explicit Allow takes a couple of minutes and no development work. If a security plugin or firewall is the cause, it’s usually a checkbox in the plugin’s bot management settings, sometimes labelled ‘AI crawlers’ or listed by name. Either way, this is one of the few AI visibility fixes that is genuinely quick once you know it’s needed.

It’s also one of the first things worth checking, before spending time rewriting content, because no amount of good writing helps if the crawler carrying it to the assistant never got through the door.

Once the crawlers can actually reach your pages, what content AI needs before it will recommend you covers what to do with that access.