Google says don't chunk your content for AI. Its own retrieval systems chunk it anyway.

Executive summary

Google's own published guidance contains a live, checkable contradiction. Google Search Central's AI optimization guide tells publishers, in plain language, that "there's no requirement to break your content into tiny pieces for AI to better understand it". The same page adds that publishers don't need "new machine readable files, AI text files, markup, or Markdown" for Google's generative AI features, because "Google Search itself doesn't use them". Read at face value, this is Google closing the case on chunking: write normal pages and skip the llms.txt theatre.

Except Google itself has been ranking sub-page passages since 2020. Passage Ranking, announced at Google's October 2020 "Search On" event and rolled out through early 2021, indexes and ranks individual passages within a page, not just the page as a whole, and was reported at the time to affect roughly 7% of search queries globally. And outside Google entirely, the retrieval systems that power AI answers, from the foundational Dense Passage Retrieval (DPR) architecture to Anthropic's production Contextual Retrieval pipeline, operate exclusively on chunks, typically well under a full page, because embedding-based search structurally requires it. Style has nothing to do with it.

The honest synthesis sits between "Google is wrong" and "chunking research is wrong." Google's guidance addresses whether you need to manually pre-chunk your writing, and correctly says no, because the systems handle segmentation themselves. It does not, and cannot, address what those systems do internally once your page reaches them. Ahrefs' analysis of 174,048 pages cited across 560,346 AI Overviews found essentially zero relationship between total word count and citation likelihood (Spearman correlation of 0.04), with the average cited page running 1,282 words and just over half under 1,000. That pattern fits passage-level selection far better than whole-document judgment. This piece lays out what's verified on each side of that line, and a framework for writing one page that survives contact with all three readers: a human, Google's classical ranking systems, and whatever retrieval layer sits underneath an AI answer.

A timeline of the mechanics

Part 1 — What Google's guidance actually says, verbatim

Precision matters here. Secondary SEO commentary often paraphrases Google loosely, and at least one claim from an earlier research pass on this exact topic turned out to be misattributed to the wrong Google document. The corrected record, from Google Search Central's "Guide to Optimizing for Generative AI Features on Google Search," fetched and verbatim-confirmed directly:

"There's no requirement to break your content into tiny pieces for AI to better understand it." "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." "There's no ideal page length, and in the end, make pages for your audience, not just for generative AI search." "The best practices for SEO continue to be relevant because our generative AI features on Google Search are rooted in our core Search ranking and quality systems."

The guidance about writing for humans ("make sure the content is well written and easy to follow. People generally appreciate it when web pages are organized by paragraphs and sections, along with headings that provide a clear structure to navigate content") also lives on this same AI optimization guide page, not on Google's separate "Guidance on Generative AI Content on Your Website" document, as an earlier draft of this research incorrectly claimed. That second document is narrower than its name suggests: it covers disclosure and quality standards for AI-generated content (labeling, metadata accuracy, avoiding scaled abuse), and says nothing about how to structure a page for AI retrieval. Conflating the two documents produces exactly the kind of confident-sounding-but-wrong citation this piece is trying to avoid.

Taken together, Google's actual position is narrower than "don't worry about structure at all." Don't manually pre-fragment your writing into artificial chunks. Don't build a parallel AI-only file, and don't chase a magic word count. The stated reason: the ranking and retrieval systems that feed AI features are the same systems that already crawl, parse, and rank the page. That is a statement about your workflow. It says nothing about what those systems do to the page after ingestion.

Part 2 — The precedent Google didn't mention: Passage Ranking

Six years before this guidance existed, Google told the industry something structurally different. At "Search On" in October 2020, Google described a ranking breakthrough that let it "not just index web pages, but individual passages from the pages". It rolled out in US English by December 2020–February 2021 and was reported to affect roughly 7% of queries across all languages once fully live globally.

That system predates the current AI Overview era, but it establishes one thing clearly: Google's classical ranking stack can already score a passage independently of the page it sits in, for a non-trivial share of queries. Passage Ranking didn't require publishers to chunk anything either. It operated on ordinarily structured pages and extracted the relevant unit itself. Google's current AI guidance describes the same architecture: publishers write normally, the system does the segmentation. The guidance isn't new in kind. Passage Ranking is the four-year-old proof that it's technically credible.

Part 3 — How retrieval actually works underneath an AI answer

Passage-level retrieval comes close to a hard requirement of embedding-based search everywhere, well beyond Google. Dense Passage Retrieval, the 2020 paper that underpins most modern RAG and semantic-search systems, splits a corpus into fixed-size blocks before anything else happens: the DPR paper split a December 2018 Wikipedia dump into 21,015,324 disjoint 100-word passages and beat the BM25 keyword baseline by 9–19 percentage points (absolute) on top-20 retrieval accuracy. You cannot embed "a page." You embed a chunk, because the vector has to represent something bounded enough that similarity search means anything.

Anthropic's Contextual Retrieval, published in September 2024, is the clearest documented example of a production lab wrestling with exactly this problem, and its existence is itself evidence that chunking creates real information loss. The technique prepends each chunk with an LLM-generated summary of where it sits in the source document, using a prompt that asks the model to "give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else." The results: Contextual Embeddings alone cut the top-20 retrieval failure rate 35%, from 5.7% to 3.7%; adding contextual BM25 pushed that to 49% (2.9%); adding a reranking step on top reached 67% (1.9%).

Anthropic's own counsel, worth taking seriously as a genuine counter-argument to "always chunk": "If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods." Chunking exists to solve a context-window and cost problem; nothing about fragments makes them inherently better representations of meaning. For a small enough corpus, the correct answer is: don't chunk at all. Most public websites are one page among billions competing for a retrieval slot. The 500-page case is a single knowledge base competing for one model's context window, so the comparison is imperfect, but the underlying point holds: chunking is an engineering workaround, never a philosophy of good writing.

On sizing, the literature only converges loosely. Production systems commonly use chunks in the range of roughly 64/128 up to 512–1,024 tokens, with smaller chunks favoured for narrow factual lookups and larger ones for broader-context tasks. The optimal size depends on the domain and the embedding model; no universal one exists. One of those papers found that context-window length itself (say, 2,048 versus 8,192 tokens) has a larger and more monotonic effect on retrieval quality than chunk size does. There is no single "correct" chunk size to write toward, which is itself an argument against trying to author paragraphs to hit one.

Part 4 — What gets cited in the wild

Ahrefs' December 2025 analysis is the largest independently-verifiable dataset connecting page characteristics to actual AI Overview citations: 174,048 cited pages drawn from 1,677,876 cited URLs across 560,346 AI Overviews. Word count barely mattered. The Spearman correlation against citation came in at 0.04, with 53.4% of cited pages under 1,000 words, 30.6% between 1,000 and 2,000, and only 16.0% over 2,000; the average cited page ran 1,282 words. Their stated recommendation: "Prioritize structure and clarity. Use headings, lead with your main point, and write in declarative sentences that are easy to parse."

That recommendation is functionally identical to Google's own human-audience guidance quoted in Part 1 (organize by paragraphs and sections, use clear headings), and the convergence is reassuring. It suggests the three readers this piece is named for aren't actually pulling in three directions on the writing level. Where the tension is real is architectural: something in the retrieval pipeline is selecting a citable unit from inside a 1,282-word average page, and it is very unlikely that unit is the whole page. The GEO paper found that specific optimization techniques can shift visibility in generative engine responses by up to 40%, though the paper doesn't disclose which techniques translate to which exact percentage in the fetched material. That is separate evidence that the retrieval and ranking layer inside these systems is sensitive to how content is written and organized at a granularity below "the page," even though the paper's technique-by-technique breakdown could not be verified this session and is not claimed here.

What everyone is missing

Google's guidance and Google's own systems are answering different questions, and treating them as one instruction is the error. The AI optimization guide tells you what to do as a writer: don't manually fragment, don't build parallel AI files, write for humans. Passage Ranking and whatever underlies AI Overview citation tell you what the system does after you hand it a page: it isolates a unit smaller than the whole. Both can be true at once. The mistake is assuming "don't chunk" means "structure doesn't matter below the page level." Google never said that, and its own 2020 passage system contradicts it directly.

The llms.txt debate has absorbed attention that should have gone to heading structure. llms.txt is a concrete, dated proposal from a named author: Jeremy Howard, September 2024, via llmstxt.org. It has genuine adopters (Anthropic, Hugging Face, Perplexity, Zapier as of March 2025) and a genuine gap: no Google, no OpenAI, no IETF or W3C ratification. Google's guidance explicitly says it doesn't use "new machine readable files, AI text files, markup, or Markdown" for its own generative features. Building an llms.txt file may still be a reasonable bet on other vendors' systems, but it is no substitute for the thing every verified source agrees actually matters: whether a human reader (and, per Ahrefs, the citation layer) can find your main point inside a clearly headed section without reading the whole page.

"Passage" and "chunk" are the same idea wearing two industries' vocabulary. Google's search engineers call it passage ranking; RAG engineers call it a chunk. Both describe a sub-page unit of text that a system scores or embeds independently of its neighbours. Once you see them as the same mechanism, the practical brief becomes obvious: write so that any contiguous 100–300 word span of your page, a section under one heading, could stand alone, make its point, and attribute its claim without leaning on the paragraph before it. Nobody should call that "chunking your content." Sections that don't depend on invisible context are, not coincidentally, good writing.

Nobody has published what an AI system's actual chunk boundaries look like for a real page, and that gap is being filled with confident-sounding numbers that don't survive scrutiny. Several specific figures circulated in earlier research for this piece: a 5,000–6,000 character reading window, exact percentage citation rates by section position, a named "context cliff" at 2,500 tokens. They traced back to unsourced marketing blogs, tiny unreplicated samples, or claims that turned out to be third-hand once checked. None of them are used here. The absence of a verified, primary-sourced answer to "where exactly does an AI system draw the chunk boundary on a real page" is itself the most important unresolved fact in this space.

Future predictions

• Google will keep publishing guidance framed around "don't do X for AI specifically" (no AI files, no forced word counts), because its retrieval already runs on the same infrastructure as classical Search. The guidance follows from shared architecture more than from any policy decision, and that framing will persist as long as AI Overviews stay bolted to core Search ranking.
• llms.txt adoption will likely stay concentrated among AI-native vendors (Anthropic, Hugging Face, Perplexity, Zapier) rather than spreading to Google or OpenAI, absent formal IETF/W3C ratification.
• Expect more vendor-published retrieval-improvement techniques in the Contextual Retrieval mold (context-aware chunking, hybrid reranking) as labs keep treating the 2020-era DPR baseline as a floor to beat rather than a ceiling.
• Large-scale citation studies like Ahrefs' will likely become a standard GEO research format, given how directly the word-count finding contradicted prevailing "longer is better" assumptions.
• The gap between Google's "we don't require chunking" guidance and its unexplained internal retrieval mechanics is unlikely to close through official disclosure. Expect continued reliance on secondary observational studies (Ahrefs-style) over primary confirmation from Google of how AI Overview citation actually selects text.

Practical takeaways

1. Write each page in clearly headed sections that could each stand alone if pulled out of context. This satisfies Google's human-readability guidance and hedges against however passage-level retrieval actually segments your page. It is core Technical SEO work, not an AI-specific add-on.
2. Don't build an llms.txt file as your primary AI strategy. It has real but narrow vendor adoption (Anthropic, Hugging Face, Perplexity, Zapier) and zero confirmed use by Google or OpenAI. Treat it as a minor supplementary bet, nothing more.
3. Stop optimizing for a target word count. Ahrefs' 0.04 correlation between length and AI Overview citation, and Google's own "there's no ideal page length" line, both point the same direction: length is not the lever.
4. Lead each section with its main declarative claim before the supporting detail. This mirrors Ahrefs' own stated recommendation and gives any passage-level or chunk-level retrieval system a clean sentence to lift. It is the practical core of Generative Engine Optimization.
5. Treat entity clarity as a retrieval hedge: name the subject, source, and claim explicitly inside each section instead of relying on pronouns tied to an earlier paragraph, since a system may only ever see one section at a time. See Entity & Knowledge Architecture for how this compounds across a site.
6. Track whether your pages are actually being cited in AI answers; don't assume good structure equals citation. The mechanics here are inferred from published research, not confirmed against your specific content, so measurement through LLM Visibility Monitoring is the only way to know if the framework is working for you.