AI can't read what it can't reach

GPTBot tried to fetch your site this morning. So did ClaudeBot, PerplexityBot, and a dozen others. Whether they succeeded is the entire ballgame for AEO — and most enterprise teams have no idea what the answer is.

This is The Access Layer — the technical plumbing that determines whether AI crawlers can reach your site, and whether the content they fetch is parseable once they do. It's the most invisible part of the methodology, and the easiest to break by accident. A Cloudflare rule a developer enabled in 2022 to block scrapers. A robots.txt directive that hasn't been touched since the last SEO audit. A “Block AI Bots” toggle someone clicked once and forgot about. Each of these silently returns 403s to the AI crawlers that matter, and most teams have no way of knowing it's happening.

The fix usually takes an afternoon. The cost of not fixing it — of doing every other piece of AEO work upstream of an Access Layer that's quietly broken — is invisibility you've paid good money to create.

Why this stage exists at all

The Access Layer is the precondition for every other piece of AEO work. The Brief tells you which prompts to target. The Verification Stack tells AI who you are. Neither matters if the AI crawlers that build training data and real-time answer indexes can't actually fetch your content. Stage 1 and Stage 2 are upstream of citation. Stage 4 is the gate.

There are now at least fifteen AI-specific crawlers reaching enterprise sites, and the list keeps growing. They split into two categories that behave differently:

You need both. Most teams know they need to allow Googlebot. Fewer realise they also need to think about Google-Extended, which is what controls AI training use, separate from the search crawler. Almost none have audited whether all fifteen are getting through.

There's also a two-layer technical reality worth naming now, because it's the source of most of the failure modes downstream. robots.txt is advisory — it's a polite request, and well-behaved bots respect it but it can't enforce anything. The CDN layer (Cloudflare, Fastly, Akamai, AWS WAF, plus standalone bot-protection services like DataDome and PerimeterX) is enforced — it can actually block at the network level. When the two contradict, the CDN wins. Which is why a perfectly-written robots.txt can sit next to a Cloudflare toggle that's quietly blocking every AI crawler in scope, and the marketing team has no idea.

The playbook

Five moves. Total time for a serious audit and fix is 15–25 hours over two to three weeks — most of that is verification rather than implementation, because the work itself is fast and the validation is what protects the engagement.

Before auditing anything, build the list of crawlers in scope. This is the reference document every other move runs against. The list as of mid-2026:

Sixteen bots. The list shifts every few months — new bots launch, old ones get renamed, user-agent strings change. Treat this as a living document. Quarterly review minimum.

This is the diagnostic step. Three sub-audits, all run before anything is changed.

If server log access isn't available, Cloudflare and most CDN dashboards provide bot analytics that can substitute. Worth confirming access during scoping rather than mid-engagement.

Screenshot everything. The before/after gap is what justifies the engagement, and a lot of the impact shows up weeks after deployment, when AI responses start improving and the screenshots are what tie cause to effect.

Build a clean, explicit robots.txt that addresses each AI crawler by name. Don't rely on the wildcard default — be specific. Specific directives override general ones, but ambiguity causes problems and crawler operators occasionally shift their default behaviour.

The template for “allow all AI crawlers” (the most common case):

# Standard crawlers User-agent: * Allow: / Disallow: /admin/ Disallow: /checkout/ Disallow: /*?session= # AI crawlers — explicitly allowed User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: CCBot Allow: / User-agent: Bytespider Allow: / User-agent: Meta-ExternalAgent Allow: / Sitemap: https://www.your-domain.com/sitemap.xml

For selective blocking — publishers, for example, sometimes have legitimate reasons to block training crawlers while allowing real-time fetch crawlers — swap Allow: / for Disallow: / on the specific bots being blocked. The strategic decision should be deliberate, in writing, signed off by the client.

“We are allowing GPTBot for training because we want to be cited in ChatGPT responses, and we accept the implication of being included in training data” is a written position. “We never thought about it” is not.

After deployment, re-run the curl tests from Move 2 to confirm each bot now returns 200.

This is where the real fix usually lives. robots.txt is advisory. The CDN is enforcement. If the CDN is blocking, nothing in robots.txt matters.

For Cloudflare specifically — which is where the majority of these blocks happen — three things to check:

For other CDNs — Fastly's Edge Security, Akamai's Bot Manager, AWS CloudFront with WAF managed rules — the equivalent settings exist under different names. For standalone bot-protection services (DataDome, PerimeterX, HUMAN), explicit allowlisting of AI bot user agents is usually required; coordinate with the security team.

After changes, retest. The same curl commands from Move 2 should now return 200s across the full reference list, and server logs over the following 48–72 hours should show new crawl activity from previously-absent bots.

llms.txt is the proposed standard (originated by Jeremy Howard at Answer.AI in late 2024) for telling AI systems what a site is about — separate from robots.txt, which only controls whether they can crawl. Adoption is still emerging — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot all read it; many other bots don't yet — but it's a low-effort future-proofing play.

Deploy a /llms.txt file at site root with the standard format: an H1 with the company name, a blockquote with a one-sentence description, a short paragraph of context, then organised sections listing products and services, key resources, about pages, and contact information. Each link is markdown-formatted with a short description.

For larger sites with substantial documentation, optionally deploy /llms-full.txt — a longer version that includes condensed markdown of the most important pages. Skip it if maintenance bandwidth is limited.

Validate via llms-txt-validator.com or equivalent. Confirm the file is accessible at the root URL.

Don't oversell this move to internal stakeholders. llms.txt adoption is genuinely in flux. Frame it as future-proofing — when adoption increases (which is likely), the brand is already positioned. Don't frame it as “this will move citation rates next month.”

What to measure

Five Access Layer signals. Report monthly — access either works or it doesn't, so there's less noise here than in the citation-rate metrics from The Brief.

Common failure modes

Five patterns, almost all of them invisible to marketing teams until someone runs the audit.