How to evaluate an AI visibility tracking tool

Three reasons, all practical. The category is moving fast enough that feature and price comparisons published a quarter ago are now partly wrong, and a buyer acting on a stale table makes a worse decision than one acting on criteria. We sell a managed service that overlaps with what these tools do, so our ordering of them would not be disinterested. And the dominant business model for tool round-ups is affiliate referral, which reliably produces rankings shaped like commission rates.

The criteria below do not decay the same way. What a measurement of AI visibility has to do to be sound is a property of the problem, not of this quarter's products, and it will still be true when half the current names have merged or closed.

If you want a shortlist of products, the fastest honest route is to search the category, take the first five names that appear repeatedly across sources with no referral disclosure, and run every one of them through the table above. That takes an afternoon and produces a better answer than any ranking, including one we could write.

The composite score with no published formula. Almost every product in this category offers a single number summarising your AI visibility. Some are reasonable weighted aggregates. Others are unfalsifiable. The test is simple: ask how it is calculated. A vendor who will not say has given you a number you cannot audit, cannot reproduce and cannot act on, and its main function is to move reassuringly.

The single run presented as a position. Ask an engine the same question three times and you can get three different answers. A tool that samples once per period and draws a line between the points is charting noise with confidence. What is meaningful is the trend across many runs of a stable panel, with the variance visible rather than smoothed away.

The quietly mutable panel. If prompts can be added or removed without a version record, the baseline is not a baseline. This rarely happens through bad faith — someone notices the panel missing an obvious question and adds it — but the effect on the trend line is the same, and only a visible edit log prevents it.

Share of voice with no denominator. A share figure requires a defined universe: which prompts, which engines, which competitor set. Presented without those, it is a percentage of something unspecified, and it can be moved by changing the denominator rather than the performance.

Dedicated AI-visibility platforms. Built for this problem specifically, generally the strongest on panel management, per-engine breakdown and competitor capture. They are also the youngest companies in your stack, which is a real procurement consideration if you need the historical data to still exist in three years. Export capability matters most here.

Established SEO suites with AI modules added. The advantage is consolidation: one contract, one login, and AI metrics sitting beside the Search Console data they have to be read against. The trade-off is depth, since these modules are typically newer and shallower than the dedicated products, and panel control is often the first thing simplified away.

Do it yourself. A scheduled job that runs a fixed prompt list against each engine's API, stores the raw responses, and extracts mentions and competitor names. Entirely feasible for a team with an engineer, and it gives you complete control of the panel and permanent ownership of the history.

Most businesses over a certain size end up with two of the three, usually a suite for the search data and something dedicated or homemade for the panel, because the suites are rarely deep enough on their own and the dedicated products do not replace Search Console.

Building is cheaper than it looks and more work than it sounds. The engineering is genuinely modest: a scheduler, API calls to each engine, a store for raw responses, and extraction logic for mentions and competitor names. A capable developer has a first version running in days.

The costs people forget are the ongoing ones. API calls for a real panel across several engines, run often enough to see through the variance, are an actual monthly line item. Extraction logic breaks when response formats change, which they do without notice. And somebody has to read the output every week and care about it, which is the part that quietly fails first — a homemade tracker nobody opens is worse than no tracker, because it produces the feeling of measurement without any.

Buy if you want the reading habit enforced by an interface someone else maintains, or if nobody on the team will own it. Build if the panel needs to be unusual, if the data must be yours permanently, or if you are already running scheduled jobs and this is one more.

Either way the panel is the asset and the tool is the container. Write the prompts from recordings of your own sales calls, freeze them, and keep them somewhere that survives changing vendors. That list is worth more than any dashboard rendering it.