Is AI Allowed to Read Your Site? A 20-Minute Check for Marketing Teams

Updated July 2026 · Ambika Sharma, Founder, Chief Strategist at Pulp Strategy Communications and Product Architect of NeuroRank^®

AI can only cite a page it is allowed to read, and many sites quietly block the very crawlers that feed AI answers without knowing it. The AI fetchers are separate from Googlebot, so a site that Google indexes perfectly can still be invisible to ChatGPT, Gemini, Claude, and Perplexity. This is a 20-minute check any marketing team can run to confirm the right agents have access. You do not need engineering for the check itself, only for some of the fixes.

Know the agents, and the three jobs they do

Each AI vendor runs a small fleet split by job. Training crawlers collect content for future models: GPTBot for OpenAI, ClaudeBot for Anthropic, and Google-Extended for Google. Search crawlers build the index the answer engines cite from: OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude, and PerplexityBot for Perplexity. User agents fetch a page live when someone asks: ChatGPT-User, Claude-User, and Perplexity-User. The search and user agents are the ones that decide whether you can be cited today.

Avoid the most common and costly mistake

Blocking GPTBot does not remove you from ChatGPT Search. Training and search are separate agents, so a site can block GPTBot to stay out of model training and still be cited in ChatGPT Search through OAI-SearchBot. The costly version of this mistake runs the other way: blocking the search or user agents, which quietly removes you from the answers. If citation is the goal, the search and user agents must be allowed.

Check your robots.txt

Open your-domain.com/robots.txt and look for any Disallow rule under GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, or Google-Extended. A Disallow under a search or user agent is the line costing you citations. Fixing this is a text edit, but change it carefully, because a stray Disallow under the wrong agent, or under Googlebot, can de-index a site.

Check for a CDN or firewall that overrides the file

A clean robots.txt is not enough if your CDN is blocking bots above it. Cloudflare and similar tools ship a one-click “block AI bots” control that is easy to enable by accident, and it silently overrides your file. Confirm no firewall or bot-management rule is treating the search agents as scrapers. This is where a marketing team usually needs a quick word with whoever owns the CDN.

Confirm the models can actually read the content

Access is not the same as legibility. If your key content loads through JavaScript that the fetchers do not run, the page can be reachable and still effectively empty to a model. Confirm your important facts are present in the served HTML, not painted in after load.

Verify from your logs, and know the limits

Filter your server logs for the agent names above to see which bots are actually visiting and how often. Verify a suspicious visitor with a reverse DNS lookup, because user-agent strings can be spoofed. And know the honest limit: robots.txt is a request, not a lock. Some crawlers have been documented ignoring it, so anything you truly need to keep out belongs behind server or firewall rules, not just the file. Blocking the wrong bots has a measurable cost, with one late-2025 study finding publishers that blocked AI crawlers saw a 23.1 percent traffic decline (Rutgers and Wharton, 2025). Once access is confirmed, NeuroRank^® tracks whether that access is turning into actual citations across the four models.