Technical SEO

Otterly's crawl found roughly 73% of sites blocked from AI citation entirely, not because their content was weak but because the crawlers could not get to it. Two causes dominate: robots.txt rules excluding AI user agents, and content that only exists after JavaScript executes.

This is the cheapest problem in search to fix and the most commonly missed, because it does not show up in any ranking report. Your classical positions can be healthy while you are absent from every AI surface, and nothing in a standard SEO dashboard will tell you why.

The most expensive version is blocking Google-Extended. It controls Gemini and AI Overviews grounding and has no effect whatsoever on classical Search ranking, so blocking it removes you from the AI surfaces while protecting nothing. It usually gets switched on by someone reacting to a headline about AI scraping, and it usually goes unnoticed for months.

We check crawler access on day one of every engagement, before anyone writes a word. It takes an afternoon and it determines whether the rest of the programme can work.

A modern JavaScript framework will happily show you a complete page while serving a crawler an empty container. Google renders JavaScript, eventually and inconsistently. Most AI retrievers do considerably less, and some do none at all.

The test is not whether the page looks right in your browser. It is whether your actual copy is present in the initial HTML response. We check the raw response, not the rendered DOM, and on a surprising number of sites the answer is that the headline is there and the body is not.

The fix depends on the stack: server rendering, static generation at build time, or in stubborn cases moving the content that matters into the initial payload while the interactive parts hydrate afterwards. What we do not do is accept a rendering gap on the grounds that Google mostly handles it, because Google is no longer the only reader that counts.

Performance is a ranking factor with a modest weight and a large indirect effect. A slow page loses people before it can convert them, which shows up in the metrics search engines watch even when the speed itself was never the deciding signal.

We work to the three thresholds and stop there. Largest Contentful Paint under 2.5 seconds, Interaction to Next Paint under 200 milliseconds, Cumulative Layout Shift under 0.1. Field data from real users rather than lab scores, because a perfect lab score on a page nobody can use on mobile data is theatre.

The usual culprits are dull and consistent: images served at the wrong size or in the wrong format, layout shift from elements without reserved dimensions, and third-party scripts blocking the main thread. Most sites get from failing to passing on those three alone, without a rewrite.

Crawl budget is a real constraint on a large site and mostly irrelevant on a small one, which does not stop it being sold to both. The problems worth attention are different.

Duplicate and near-duplicate URLs split signals between pages that should be one page. Orphan pages sit in the sitemap with no internal link pointing at them and get treated accordingly. Redirect chains lose a little equity at every hop and slow the crawler down. Parameter variants create dozens of URLs for one piece of content. Each is unglamorous and each is quietly costing something.

Then there is thin content sitting in the index. A cluster of near-identical short pages does not add coverage, it competes with itself and drags on the pages around it. The fix is consolidation into one substantial page with redirects from the rest, which is exactly what we did to this site's own process section: six near-duplicate stubs under a parent that returned a 404, now one page with the old URLs redirecting into it.

We treat schema as infrastructure rather than as a citation tactic, and the distinction matters because the evidence on the second claim is poor. Ahrefs' controlled test found AI Overview citations declined after schema was added, and a controlled enterprise-retrieval study found JSON-LD barely moved accuracy while legible entity pages did.

What markup does reliably is tell a parser what a thing is. Breadcrumbs that match the visible trail, an FAQ block built only from questions actually rendered on the page, an article with real publication and modification dates, an organization declared once with an identifier that every other node references.

Validation is not optional. Invalid JSON-LD is functionally the same as no JSON-LD, and it fails silently, so it stays broken until someone checks. Every schema change goes through the Rich Results test before it ships.

The audit itself is fast, because the collection is automated: a real crawl rather than a checklist, field performance data, raw-HTML rendering checks across page types, log analysis where logs are available, and the index picture from Search Console. Days, not weeks. What takes weeks is everything after it.

What comes back is a backlog ordered by impact against effort, not a 200-item PDF. Crawler access and rendering sit at the top because they gate everything else. Then indexation problems, then performance, then the long tail of hygiene.

Most of it is a few days of engineering rather than a rebuild. Where a fix genuinely needs a platform change, we say so plainly and price the alternative, because the honest answer is sometimes that the CMS is the problem and no amount of tuning around it will help.

The fastest way to undo a year of ranking work is a site migration nobody mapped. New URLs, no redirects, and the accumulated authority of every old page evaporates over the following weeks while everyone wonders why traffic fell off a cliff.

The mapping is the whole job. Every old URL gets a destination that serves the same intent, and a redirect to a generic parent is not that. Sending forty retired pages to the homepage tells a crawler those pages had nothing worth keeping, and it treats them accordingly.

We use permanent redirects, keep chains to a single hop, and keep the retired URLs out of the sitemap while leaving them redirecting for as long as anything still links to them. Then we watch Search Console for the errors that only appear once real crawlers hit the new structure, because the problems that matter tend to surface in week two rather than on launch day.