Cloudflare WAF & Bot Fight Mode Blueprint: Unblock OAI-SearchBot, ChatGPT-User & PerplexityBot
Updated 2026-11-01 · Grounded in 2459 audited B2B SaaS domains · Related: 27-Bot robots.txt Policy
Even when /robots.txt explicitly allows AI search crawlers, Cloudflare Managed Challenges or aggressive WAF rules frequently serve HTTP 403 Just a moment... HTML shells to headless retrieval bots, dropping citation share to zero. Add a high-priority WAF Custom Rule skipping Managed Challenges on deterministic discovery paths and verified search agents:
# Cloudflare WAF Custom Rule -> Action: Skip (All remaining custom rules + Super Bot Fight Mode)
(http.request.uri.path in {"/llms.txt" "/llms-full.txt" "/pricing.md" "/answers.md" "/.well-known/ai.txt" "/.well-known/mcp.json" "/.well-known/server.json" "/.well-known/agent-card.json" "/.well-known/openapi.json" "/ai/summary.json" "/ai/faq.json" "/ai/service.json"})
or (cf.client.bot and http.user_agent contains "OAI-SearchBot")
or (cf.client.bot and http.user_agent contains "ChatGPT-User")
or (cf.client.bot and http.user_agent contains "Claude-SearchBot")
or (cf.client.bot and http.user_agent contains "PerplexityBot")