Allow AI search/retrieval bots in robots.txt

How to fix

Let AI search and agent bots in through robots.txt so they can read your page live. This removes a blocker; it doesn't promise a mention.

1. Open https://yoursite.com/robots.txt. It should load (200) or not exist (404). A server error (5xx) makes crawlers treat your whole site as off-limits, so fix that first with your host.

2. Look for rules that block OAI-SearchBot, PerplexityBot, Claude-SearchBot, Claude-User, Googlebot or Bingbot, and for User-agent: * with Disallow: /, which blocks every bot without its own group.

3. Give those bots their own group that allows the page. A bot follows only the group that names it, so repeat any private paths:

   User-agent: OAI-SearchBot
   User-agent: PerplexityBot
   User-agent: Claude-SearchBot
   User-agent: Claude-User
   User-agent: Googlebot
   User-agent: Bingbot
   Allow: /
   Disallow: /admin/
   
   User-agent: *
   Disallow: /admin/
   

4. Remove any Disallow for ChatGPT-User or Perplexity-User. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules". To block them, use your firewall.

5. If the check says your site turned it away (403), see "Firewall and CDN rules" below: robots.txt isn't the cause.

6. Save and deploy. OpenAI and Perplexity say changes can take up to about 24 hours to reach them.

How to check it worked: run the free check again. The "AI agents are allowed in" row should pass: "Your robots.txt lets every AI search and agent bot we track open this page."

Firewall and CDN rules

robots.txt only asks; your firewall, CDN (the network that serves your site) or bot filter decides who gets in. Let AI search and agent bots through, and allow them by the vendors' published IP addresses or your provider's verified-bot setting, not by name alone.

1. Find the rule. In your firewall or CDN log, filter for the time of the check and the bots named in the row (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Claude-User, Google-Agent). Note which rule fired: an AI bot setting, bot protection, a rate limit, or a country or user-agent block.
2. Cloudflare. In Security Settings, open the AI bot policies and set Search and Agent to Allow (Training is your call, see next section). Since September 15, 2026, Cloudflare offers new domains recommended AI settings. For sites with ads, that means search crawlers allowed, AI training disallowed, and AI agents blocked on pages that carry ads. If you added your site after that date, check these settings even if you never changed them. To let verified bots past your custom rules, add a WAF rule on cf.client.bot.
3. Vercel. Open Firewall, then Rules. If the AI Bots Managed Ruleset (Allow by default) says Deny, set it to Log or Allow. If Bot Protection is on Challenge, add a custom Bypass rule for the AI bots you want.
4. Akamai. In Bot Management, set the AI bot categories you want to Allow or Monitor, not Deny.
5. Other firewalls. Allow each bot only when its IP address is on the vendor's published list. OpenAI, Perplexity and Google each publish one.
6. Check for HTTP 402. It means pay per crawl is on: bots that don't pay get a payment request instead of the page. Keep it only if that's your plan.

How to check it worked: run the free check again. The "Your firewall passed our AI-crawler test" row should pass: "Our AI-crawler test got through: your page loaded when we visited as each of the {n} AI agents we test." Our test runs from our own servers, so if you verify AI crawlers by IP address, a Watch here is expected. If you don't, check your firewall rules. This row never fails: the worst it shows is Watch.

Training bots are a policy choice

Decide whether AI companies may train on your pages. Most use separate bots for training and search, so you can block training and stay findable. The exception is Google-Extended, which also decides whether Gemini can use your pages in its answers. The free check reports your choice but doesn't grade it.

To change your policy:

1. Choose: training allowed (the default with no rule) or blocked.

2. To block training, put the training bots in one group:

   User-agent: GPTBot
   User-agent: ClaudeBot
   User-agent: Google-Extended
   User-agent: Applebot-Extended
   User-agent: CCBot
   Disallow: /
   

3. Keep search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) out of that group, or the "AI agents are allowed in" row will fail.

4. On Cloudflare, check its Training setting too. It can block training bots at the edge and write these robots.txt lines for you.

How to check it worked: run the free check again. The "AI training bots" row should read "You block {bots} from training on your pages.", and "AI agents are allowed in" should still pass.