Google says don't chunk your content for AI. Its own retrieval systems chunk it anyway.

Executive summary

There is a live, checkable contradiction sitting inside Google's own published guidance. Google Search Central's AI optimization guide tells publishers, in plain language, that "there's no requirement to break your content into tiny pieces for AI to better understand it" (Confidence: High — primary source, fetched and verbatim-confirmed). The same page adds that publishers don't need "new machine readable files, AI text files, markup, or Markdown" for Google's generative AI features, because "Google Search itself doesn't use them" (Confidence: High — primary source, verbatim). Read at face value, this is Google closing the case on chunking: write normal pages, skip the llms.txt theatre, move on.

Except Google itself has been ranking sub-page passages since 2020. Passage Ranking — announced at Google's October 2020 "Search On" event and rolled out through early 2021 — indexes and ranks individual passages within a page, not just the page as a whole, and was reported at the time to affect roughly 7% of search queries globally (Confidence: Medium — corroborated across five-plus independent industry outlets citing the same Google announcement; the original Google blog post could not be directly re-fetched this session). And outside Google entirely, the retrieval systems that power AI answers — from the foundational Dense Passage Retrieval (DPR) architecture to Anthropic's production Contextual Retrieval pipeline — operate exclusively on chunks, typically well under a full page, because that is a structural requirement of embedding-based search, not a stylistic choice (Confidence: High — primary sources fetched directly).

The honest synthesis is not "Google is wrong" or "chunking research is wrong." It's that Google's guidance addresses whether you need to manually pre-chunk your writing — and correctly says no, because the systems handle segmentation themselves. It does not, and cannot, address what those systems do internally once your page reaches them. Ahrefs' analysis of 174,048 pages cited across 560,346 AI Overviews found essentially zero relationship between total word count and citation likelihood (Spearman correlation of 0.04), with the average cited page running 1,282 words and just over half under 1,000 (Confidence: High — primary vendor source, directly fetched). That is consistent with a system pulling a passage, not judging a document. This piece lays out what's verified on each side of that line, and a framework for writing one page that survives contact with all three readers: a human, Google's classical ranking systems, and whatever retrieval layer sits underneath an AI answer.

A timeline of the mechanics

Part 1 — What Google's guidance actually says, verbatim

It's worth being precise here because a lot of secondary SEO commentary paraphrases Google loosely, and at least one claim from an earlier research pass on this exact topic turned out to be misattributed to the wrong Google document. The corrected record, from Google Search Central's "Guide to Optimizing for Generative AI Features on Google Search," fetched and verbatim-confirmed directly:

"There's no requirement to break your content into tiny pieces for AI to better understand it." (Confidence: High.) "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." (Confidence: High.) "There's no ideal page length, and in the end, make pages for your audience, not just for generative AI search." (Confidence: High.) "The best practices for SEO continue to be relevant because our generative AI features on Google Search are rooted in our core Search ranking and quality systems." (Confidence: High.)

Notably, the guidance about writing for humans — "make sure the content is well written and easy to follow. People generally appreciate it when web pages are organized by paragraphs and sections, along with headings that provide a clear structure to navigate content" — also lives on this same AI optimization guide page, not on Google's separate "Guidance on Generative AI Content on Your Website" document, as an earlier draft of this research incorrectly claimed (Confidence: High — both URLs directly re-fetched and cross-checked; the words "paragraphs," "sections," and "headings" do not appear together anywhere on the other document). That second document is narrower than its name suggests: it's about disclosure and quality standards for AI-generated content (labeling, metadata accuracy, avoiding scaled abuse), not about how to structure a page for AI retrieval (Confidence: High — directly fetched). Conflating the two documents produces exactly the kind of confident-sounding-but-wrong citation this piece is trying to avoid.

Taken together, Google's actual position is narrower than "don't worry about structure at all." It's: don't manually pre-fragment your writing into artificial chunks, don't build a parallel AI-only file, and don't chase a magic word count — because the ranking and retrieval systems that feed AI features are the same systems that already crawl, parse, and rank the page. That's a statement about your workflow. It is silent on what those systems do to the page after ingestion.

Part 2 — The precedent Google didn't mention: Passage Ranking

Six years before this guidance existed, Google told the industry something structurally different. At "Search On" in October 2020, Google described a ranking breakthrough that let it "not just index web pages, but individual passages from the pages" (Confidence: Medium — quote corroborated across Search Engine Land, Search Engine Journal, SEOSLY, SearchAtlas, Justia Onward, and Sixth City Marketing, all citing the same Google announcement; the original blog.google post could not be located or fetched directly despite repeated attempts, so this is capped below High). It rolled out in US English by December 2020–February 2021 and was reported to affect roughly 7% of queries across all languages once fully live globally (Confidence: Medium — same sourcing caveat).

That system predates the current AI Overview era, but it establishes something important: Google's classical ranking stack is already capable of scoring a passage independently of the page it sits in, for a non-trivial share of queries. Passage Ranking didn't require publishers to chunk anything either — it operated on ordinarily structured pages and extracted the relevant unit itself. That's the same architecture Google's current AI guidance describes: publishers write normally, the system does the segmentation. The guidance isn't new in kind; Passage Ranking is the four-year-old proof that it's technically credible.

Part 3 — How retrieval actually works underneath an AI answer

Passage-level retrieval isn't a Google-specific quirk — it's close to a hard requirement of how embedding-based search works everywhere. Dense Passage Retrieval, the 2020 paper that underpins most modern RAG and semantic-search systems, works by splitting a corpus into fixed-size blocks before anything else happens: the DPR paper split a December 2018 Wikipedia dump into 21,015,324 disjoint 100-word passages and beat the BM25 keyword baseline by 9–19 percentage points (absolute) on top-20 retrieval accuracy (Confidence: High — the accuracy gap confirmed directly from the paper's abstract; the exact passage count is corroborated across independent secondary sources describing the paper, not yet re-verified against the full PDF body in this session). You cannot embed "a page." You embed a chunk, because the vector has to represent something bounded enough that similarity search means anything.

Anthropic's Contextual Retrieval, published in September 2024, is the clearest documented example of a production lab wrestling with exactly this problem — and its existence is itself evidence that chunking creates real information loss. The technique prepends each chunk with an LLM-generated summary of where it sits in the source document, using a prompt that asks the model to "give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else." (Confidence: High — primary source, fetched, prompt confirmed verbatim.) The results: Contextual Embeddings alone cut the top-20 retrieval failure rate 35%, from 5.7% to 3.7%; adding contextual BM25 pushed that to 49% (2.9%); adding a reranking step on top reached 67% (1.9%) (Confidence: High — same primary source).

Anthropic's own counsel, worth taking seriously as a genuine counter-argument to "always chunk": "If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods." (Confidence: High — verbatim, primary source.) Chunking exists to solve a context-window and cost problem, not because fragments are inherently a better representation of meaning. For a small enough corpus, the correct answer is: don't chunk at all. Most public websites are not a 500-page knowledge base competing for one model's context window — they're one page among billions competing for a retrieval slot — so the comparison isn't a perfect match, but the underlying point holds: chunking is an engineering workaround, not a philosophy of good writing.

On sizing, the literature converges loosely rather than precisely: production systems commonly use chunks in the range of roughly 64/128 up to 512–1,024 tokens, with smaller chunks favoured for narrow factual lookups and larger ones for broader-context tasks, and the optimal size is domain- and embedding-model-dependent rather than universal (Confidence: Medium — corroborated across multiple arXiv papers including Bhat et al.'s "Rethinking Chunk Size For Long-Document Retrieval," synthesized from fetched summaries rather than full manual read-throughs; treat any specific token range as indicative, not a hard rule). One of those papers found that context-window length itself — say, 2,048 versus 8,192 tokens — has a larger and more monotonic effect on retrieval quality than chunk size does (Confidence: Medium, same sourcing basis). There is no single "correct" chunk size to write toward, which is itself an argument against trying to author paragraphs to hit one.

Part 4 — What gets cited in the wild

Ahrefs' December 2025 analysis is the largest independently-verifiable dataset connecting page characteristics to actual AI Overview citations: 174,048 cited pages drawn from 1,677,876 cited URLs across 560,346 AI Overviews. Word count barely mattered — a Spearman correlation of 0.04 against citation — with 53.4% of cited pages under 1,000 words, 30.6% between 1,000 and 2,000, and only 16.0% over 2,000; the average cited page ran 1,282 words (Confidence: High — primary vendor source, all figures directly fetched and confirmed). Their stated recommendation: "Prioritize structure and clarity. Use headings, lead with your main point, and write in declarative sentences that are easy to parse." (Confidence: High — verbatim.)

That recommendation is functionally identical to Google's own human-audience guidance quoted in Part 1 — organize by paragraphs and sections, use clear headings — which is a genuinely reassuring convergence. It suggests the three readers this piece is named for aren't actually pulling in three directions on the writing level. Where the tension is real is architectural: something in the retrieval pipeline is selecting a citable unit from inside a 1,282-word average page, and it is very unlikely that unit is the whole page. The GEO paper's finding — that specific optimization techniques (the paper doesn't disclose which techniques translate to which exact percentage in the fetched material) can shift visibility in generative engine responses by up to 40% — is separate evidence that the retrieval and ranking layer inside these systems is sensitive to how content is written and organized at a granularity below "the page," even though the paper's technique-by-technique breakdown could not be verified this session and is not claimed here (Confidence: High for the top-line "up to 40%" figure and KDD 2024 acceptance; the underlying mechanism is not itemized in what was verified).

What everyone is missing

Google's guidance and Google's own systems are answering different questions, and treating them as one instruction is the error. The AI optimization guide tells you what to do as a writer: don't manually fragment, don't build parallel AI files, write for humans. Passage Ranking and whatever underlies AI Overview citation tell you what the system does after you hand it a page: it isolates a unit smaller than the whole. Both can be true at once. The mistake is assuming "don't chunk" means "structure doesn't matter below the page level" — Google never said that, and its own 2020 passage system contradicts it directly.

The llms.txt debate has absorbed attention that should have gone to heading structure. llms.txt is a real, named, dated proposal — Jeremy Howard, September 2024, via llmstxt.org — with real adopters (Anthropic, Hugging Face, Perplexity, Zapier as of March 2025) and a real gap: no Google, no OpenAI, no IETF or W3C ratification (Confidence: High — all directly fetched). Google's guidance explicitly says it doesn't use "new machine readable files, AI text files, markup, or Markdown" for its own generative features. Building an llms.txt file may still be a reasonable bet on other vendors' systems, but it is not a substitute for the thing every verified source agrees actually matters: whether a human reader (and, per Ahrefs, the citation layer) can find your main point inside a clearly headed section without reading the whole page.

"Passage" and "chunk" are the same idea wearing two industries' vocabulary, and that's the whole insight. Google's search engineers call it passage ranking. RAG engineers call it a chunk. Both describe a sub-page unit of text that a system scores or embeds independently of its neighbours. Once you see them as the same mechanism, the practical brief becomes obvious: write so that any contiguous 100–300 word span of your page — a section under one heading — could stand alone and still make its point, attribute its claim, and be understood without the paragraph before it. That is not "chunking your content." It's writing sections that don't depend on invisible context, which is also, not coincidentally, good writing.

Nobody has published what an AI system's actual chunk boundaries look like for a real page, and that gap is being filled with confident-sounding numbers that don't survive scrutiny. Multiple specific figures circulated in earlier research for this piece — a 5,000–6,000 character reading window, exact percentage citation rates by section position, a named "context cliff" at 2,500 tokens — traced back to unsourced marketing blogs, tiny unreplicated samples, or claims that turned out to be third-hand once checked. None of them are used here. The absence of a verified, primary-sourced answer to "where exactly does an AI system draw the chunk boundary on a real page" is itself the most important unresolved fact in this space, not a footnote.

Future predictions

• Google will keep publishing guidance framed around "don't do X for AI specifically" (no AI files, no forced word counts) because its retrieval already runs on the same infrastructure as classical Search — the guidance is a byproduct of shared architecture, not a policy choice, and that framing will persist as long as AI Overviews stay bolted to core Search ranking (Confidence: Medium — inference from the stated architecture, not itself a sourced prediction).
• llms.txt adoption will likely stay concentrated among AI-native vendors (Anthropic, Hugging Face, Perplexity, Zapier) rather than spreading to Google or OpenAI, absent formal IETF/W3C ratification (Confidence: Medium — extrapolated from the confirmed March 2025 adopter list and lack of ratification; no forward-looking statement from either company was in the verified set).
• Expect more vendor-published retrieval-improvement techniques in the Contextual Retrieval mold (context-aware chunking, hybrid reranking) as labs continue treating the 2020-era DPR baseline as a floor to beat rather than a ceiling (Confidence: Medium — reasonable extrapolation from Anthropic's published trajectory, not a sourced forecast).
• Large-scale citation studies like Ahrefs' will likely become a standard GEO research format, given how directly the word-count finding contradicted prevailing "longer is better" assumptions (Confidence: Low — speculative about industry behaviour, no source predicts this).
• The gap between Google's "we don't require chunking" guidance and its unexplained internal retrieval mechanics is unlikely to close through official disclosure; expect continued reliance on secondary observational studies (Ahrefs-style) rather than primary confirmation from Google of how AI Overview citation actually selects text (Confidence: Medium — grounded in the pattern of Google declining passage-level mechanics disclosure to date).

Practical takeaways

1. Write each page in clearly headed sections that could each stand alone if pulled out of context — this satisfies Google's human-readability guidance and hedges against however passage-level retrieval actually segments your page. This is core Technical Foundations work, not an AI-specific add-on.
2. Don't build an llms.txt file as your primary AI strategy. It has real but narrow vendor adoption (Anthropic, Hugging Face, Perplexity, Zapier) and zero confirmed use by Google or OpenAI — treat it as a minor supplementary bet, not a foundation.
3. Stop optimizing for a target word count. Ahrefs' 0.04 correlation between length and AI Overview citation, and Google's own "there's no ideal page length" line, both point the same direction: length is not the lever.
4. Lead each section with its main declarative claim before the supporting detail — this mirrors Ahrefs' own stated recommendation and gives any passage-level or chunk-level retrieval system a clean sentence to lift. This is the practical core of Generative Engine Optimization.
5. Treat entity clarity — naming the subject, source, and claim explicitly inside each section rather than relying on pronouns tied to an earlier paragraph — as a retrieval hedge, since a system may only ever see one section at a time. See Entity & Knowledge Architecture for how this compounds across a site.
6. Track whether your pages are actually being cited in AI answers rather than assuming good structure equals citation — the mechanics here are inferred from published research, not confirmed against your specific content, so measurement through LLM Visibility Monitoring is the only way to know if the framework is working for you.