Short answer: if your site sits behind Cloudflare, AI crawlers can get a 403 from the edge even when your robots.txt says Allow: /. Test with curl -A "GPTBot" and compare against a normal browser user agent. On my zone the cause was Cloudflare's AI bot settings (ai_bots_protection, ai_training: disallow, managed robots.txt). Turning them off took one API call, and every AI crawler I tested went from 403 to 200 within a minute.

I write this site partly so that people, and the AI assistants they ask, can find what I've built. So it was a surprise to find out this morning that some of those assistants had never been allowed in.

How did I notice?

I was auditing jqzdev.com for search and AI-answer visibility. robots.txt was as open as it gets:

User-agent: *
Allow: /

Sitemap: https://jqzdev.com/sitemap.xml

Then I requested the home page with a few crawler user agents. This is the check I now run on every domain:

for ua in "GPTBot/1.1" "ClaudeBot/1.0" "CCBot/2.0" "Bytespider" \
          "PerplexityBot/1.0" "OAI-SearchBot/1.0" "Googlebot/2.1"; do
  code=$(curl -s -o /dev/null -w '%{http_code}' \
    -A "Mozilla/5.0 (compatible; $ua; +http://example.com)" https://jqzdev.com/)
  echo "$ua $code"
done

Results on 2026-10-19, before the fix:

User agent Status
GPTBot 403
ClaudeBot 403
CCBot 403
Bytespider 403
PerplexityBot 200
OAI-SearchBot 200
Googlebot 200

The 403 body was Cloudflare's plain Your request was blocked. with server: cloudflare and no via: Caddy header, so the request never reached my origin. My own products on the same Cloudflare account (vaneform.com, meetnewai.com, earlylore.com) returned 200 for the same test, which narrowed it to a per-zone setting.

What was blocking them?

Reading the zone's bot management settings through the Cloudflare API (GET /zones/<zone_id>/bot_management) showed the difference. On jqzdev.com:

{
  "ai_bots_protection": "only_on_ad_pages",
  "ai_training": "disallow",
  "is_robots_txt_managed": true,
  "bot_preference_sync_enabled": true
}

On vaneform.com, where AI crawlers were getting through, ai_bots_protection was disabled, ai_training was disabled and is_robots_txt_managed was false.

I don't know when those values were set on jqzdev.com; I never picked them deliberately. The setting name suggests it only applies to pages with ads. My site has no ads, and the training crawlers were blocked anyway. I'm reporting what I observed, not how Cloudflare decides internally.

How do you turn it off?

In the dashboard, look for the AI crawler controls in the zone's bot settings. With the API it's one call:

curl -X PUT "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/bot_management" \
  -H "Authorization: Bearer $CF_TOKEN" -H 'Content-Type: application/json' \
  --data '{"enable_js":false,"fight_mode":false,"ai_bots_protection":"disabled",
           "crawler_protection":"disabled","is_robots_txt_managed":false}'

Save the GET output before you change anything so you can put it back. Turning off ai_bots_protection also flipped ai_training to disabled in the response. About 20 seconds later the same curl loop returned 200 for GPTBot, ClaudeBot, CCBot, Bytespider, anthropic-ai, meta-externalagent and Applebot-Extended.

Two notes:

  • This is zone-wide. Every subdomain on the zone now lets these crawlers through. My internal tools on subdomains sit behind a login, so a crawler only sees a sign-in page.
  • is_robots_txt_managed: false means Cloudflare stops prepending its own rules to your robots.txt. Your file is the only policy.

Why not block AI crawlers?

Blocking is a reasonable choice for some sites. For a solo developer whose products need to be discovered, I think it's the wrong default:

  • AI assistants answer "what's a free tool for X" questions from what they've crawled and from live fetches. If your pages return 403, you aren't in that answer.
  • Search-style fetchers (PerplexityBot, OAI-SearchBot) were already allowed here, but training crawlers decide what a model "knows" about you without a live fetch.

The real cost of AI crawlers is load, not principle. I learned that on Vaneform, my free website traffic checker. In one three-day window in October 2026, ClaudeBot made roughly 1.56 million requests there, and about 67% of them hit a bulk-compare tool page with endless ?q= variations. Those pages rendered at the origin every time.

How do you survive the load without blocking?

What worked on Vaneform, with the evidence I have:

  1. Edge-cache public HTML. Long max-age plus stale-while-revalidate and stale-if-error, and never cache 5xx. On 2026-10-15 PerplexityBot made 13,765 requests in 80 minutes while ClaudeBot made 5,000+; over four hours that was about 34,000 requests, all 200, no 5xx, and no visible latency change.
  2. Give crawlers somewhere better to go. On the pages crawlers hammered, I added a collapsed footer linking the leaderboards, recent reports and guides, plus an llms.txt listing the same.
  3. Turn off Bot Fight Mode. On Cloudflare's free plan it can't exempt verified bots, so it can catch Bingbot and Googlebot too.
  4. Don't add Disallow rules for crawl pressure. Fix the endless URLs instead (cache them, or stop linking them).

Uncached long-tail pages are still a risk: one night in October, ClaudeBot crawling uncached report pages pushed average lookups to 3–4 seconds until it passed. Caching is a mitigation, not a cure.

What I check now on every domain

  • robots.txt allows / and lists the sitemap.
  • The curl loop above returns 200 for every crawler.
  • GET /zones/<id>/bot_management shows ai_bots_protection: disabled and Bot Fight Mode off.
  • Public HTML has edge cache headers; error pages don't.
  • /llms.txt exists and points at the pages worth reading. Mine is at jqzdev.com/llms.txt.

If you want to see whether AI assistants are actually sending you visitors after this, Vaneform has a guide on measuring AI referral traffic.

FAQ

Does robots.txt Allow mean AI crawlers can reach my site?

No. robots.txt is a request to well-behaved crawlers. A CDN or firewall in front of your site can still return 403 before the crawler ever sees your content. Test with the crawler's user agent.

Does Cloudflare block AI crawlers by default?

On my jqzdev.com zone, AI training crawlers were blocked without me ever choosing that setting, while older zones on the same account were open. Check your own zone's bot settings instead of assuming either way.

Will allowing GPTBot and ClaudeBot slow my site down?

It can, if crawlers hit pages that render at the origin. On Vaneform, ClaudeBot made about 1.56 million requests in three days. Edge caching public HTML with stale-if-error kept a later 34,000-request burst at 100% 200 responses.

Is a spoofed user agent test reliable?

It shows whether the edge blocks by user agent, which is what these settings do. Real crawlers also come from known IP ranges, so a site can behave differently for them. Check your CDN's bot analytics for the real traffic.