Before you buy an AEO tool, ask what its score measures

A page-readiness score summarises configured checks on your URL; an observed AI citation records a link in one answer. Neither tells you whether a buyer visited, trusted you or bought. Before you pay for an AEO tool, make the vendor show which measurement each number represents and open the records behind it. Then name the person who will act on what the tool finds.

Google warns that third-party tools have no access to its internal ranking or AI systems. A vendor can inspect a page, run prompts and record answers. It cannot turn those external observations into an internal Google metric. That boundary is the starting point for a useful demo.

Here AEO means work to make a page findable, clear and useful as a source for an answer engine. The first screen an answer engine can quote covers the page craft. This article is a buying test for the software offered to measure it.

What is the tool actually measuring?

A proposal may bundle three jobs. Separate them before looking at the price.

1. Page checks inspect specified URLs against specified rules. A readiness score is a summary of those checks. Ask which URLs were read, which checks passed, how each was weighted and when the rules last changed. The score says what the checker found; it does not say an engine used the page.
2. Answer monitoring sends selected questions to selected AI services and records the responses. A visibility figure needs its prompt list, full answers, modes, dates, cited URLs and a denominator: how many runs were attempted and how many completed. A brand mention and a source link are different observations. This distinction matters when a dashboard counts both as “visibility.”
3. Work management assigns edits, technical fixes and retests. A task list can help a team finish work. It does not show that the work changed what an answer engine displayed.

One product may do all three. A high mark on the first cannot stand in for evidence from the second. Neither stands in for traffic or revenue. If the seller calls every number an “AI visibility score,” ask them to give each one its own name and definition.

Why is one AI answer a weak baseline?

In SparkToro’s 2026 brand-recommendation study, 600 volunteers ran 12 prompts a combined 2,961 times across ChatGPT, Claude and Google’s AI. The authors put the chance that a repeated prompt in ChatGPT or Google’s AI returns the same brand list twice at less than one in 100. The order was even less stable. Their test counted recommended brands, not source links, and it does not tell us how every other kind of answer behaves.

That is enough reason to challenge a precise “AI ranking position” derived from a handful of runs. For a fixed prompt set, count how many completed runs named you and how many linked to you. Show both numerators and the denominator. Keep failed runs visible. The resulting share describes that sample under those conditions; it is not a population-wide rank.

Which six questions belong in the demo?

Ask the seller to use your own page and one question a buyer would actually ask. Have them open the record on screen.

1. What produced this number? For a page score, request the checks, weights, inspected URL and calculation version. For an answer score, request the prompts, runs and full responses. If the vendor combines them, ask for the components separately.
2. Who chose the prompts? Get the exact text and a versioned list. Can you add a question from a real sales call, remove one that does not fit and keep the old list for comparison? A topic label is too broad to reproduce a run.
3. How many runs sit behind the chart? Request scheduled, successful and failed counts per prompt and period. Ask what happens when a response has no link, or a run times out. A rate without its denominator conceals how much was observed.
4. Under which conditions were the answers collected? Record service, model or mode when available, date, language and location. OpenAI notes that Search can be selected manually and may use approximate location from an IP address. A location-sensitive prompt deserves that context.
5. Can I inspect and retain the raw answers? Ask for the complete text, exact cited URLs, timestamps and run conditions in an export you can keep if the contract ends. If it reports a citation, open the cited page and check whether the passage supports the answer. A mention without a link should have its own count.
6. What changed, and who will make the next change? Ask the seller to explain one score movement using its records: page edit, prompt change, run conditions or calculation. Then name who will edit the page, approve factual claims and retest. If managed editing is included, ask to see a documented example.

The seller need not promise a higher score. They do need to make the evidence and the work legible. A slide about a proprietary method does not answer these questions.

What can each record prove?

Record	What you can check	What it does not establish
Page-check result	Which configured rules passed on one URL at one time	That an answer engine cited it
Full answer with prompt and conditions	What that recorded run displayed	What every user saw
Cited URL and passage	Whether a linked page supports the answer	Why the engine chose it
Before-and-after score	A difference under the stated formula	That your edit caused the difference
Analytics visit or event	A visit or action recorded under your tracking rules	The value of an answer that sent no click

Read this table as a limit on interpretation, not a dismissal of the data. A tool can make each record easier to collect. The buying question is whether that convenience is worth the fee and the team’s time.

What do your existing reports already show?

Check these first-party reports before paying another service to repeat their view:

Report	What its documentation describes	The gap for this buying test
Google Search Console: generative AI performance	Impressions from links to your site in AI Overviews and AI Mode, with page, country, date and device views	It reports impressions, not the full answers or your own fixed prompt runs
Bing Webmaster Tools: AI Performance	Citation counts, cited pages and sampled grounding queries across Copilot, Bing AI summaries and selected partner integrations	The launch post does not describe full answer transcripts or click outcomes
Google Analytics 4: AI Assistant channel	Visits from the AI assistants on Google’s list	It cannot count answers that sent no visit; Google AI Overviews and AI Mode visits fall under Organic Search

Google says its generative AI report was rolled out worldwide by 31 August 2026, but warns that a site may not see it when impressions are too low. An absent report is not a zero-citation finding. Google announced the AI Assistant channel in its 13 May 2026 GA4 release note; keep that date in mind when you compare periods.

These reports answer different questions. A monitor may add a saved record of what a selected prompt returned, including a response that never sent a visit. Judge that addition against the coverage you need, rather than counting every chart as new evidence.

Who will edit the page?

A monitoring subscription can tell a founder the same opening is weak every week. The subscription cannot assign authority inside the founder’s team. Put names next to five jobs before signing: choose the questions, inspect cited passages, edit the copy, resolve access or rendering problems, and approve factual claims. One person may hold several jobs. Record when a change is made so the next observation has context.

Start with your most important URL. The citation-readiness audit is a fifteen-minute pass/fail check for one URL; run it before the sales call. If the page lacks a direct answer or a supporting source, fix that known problem and date the change. Do not wait for a dashboard to tell you it exists.

What would a four-week pilot decide? (hypothetical)

An installation firm in Eindhoven considers a monitor for two service pages: one on heat pumps, one on noise. It gives the vendor ten fixed buyer questions to run three times a week for four weeks. That schedules 120 runs. The agreed export must include every attempted run, including failures, plus the full answers and source links. The firm’s editor owns page changes.

At the midpoint, the vendor’s page-check score for the noise page is 64 out of 100. The page-check report shows which checks failed. The editor rewrites an opening that asked for a consultation before answering the question, adds a checked source and logs the date. The page score then rises to 82. That demonstrates a change under the vendor’s page-check formula.

Suppose two runs fail in each half. Of the 58 completed runs before the edit, two link to the noise page. Of the 58 completed runs after, two link to it. The monitored sample shows no increase in observed links. It also cannot prove the edit had no effect elsewhere. The pilot delivered a documented repair and an auditable record of 120 attempted runs. It did not deliver evidence for a citation-growth claim.

All numbers in this example are invented for the decision test. Before the pilot starts, agree what would justify renewal: reliable exports, useful page repairs, a measured change under stable prompts, or saved staff time. Do not replace that criterion with a new promise after the results arrive.

What belongs in the order form?

Five lines are enough to expose an unclear offer:

1. Coverage: named URLs, exact prompt list and version, services, modes, languages, locations and frequency.
2. Records: full answers, cited URLs, timestamps, failed runs, the score formula and export format.
3. Owners: named people for editorial changes, source checks, technical repairs and approval.
4. Comparison: a saved baseline, dated page edits, stable comparison prompts and a log of collection or scoring changes.
5. Exit: when exports can be downloaded, what remains available after cancellation and what the fee does and does not cover.

Read the owners line aloud. If the answer is “your team,” budget your team’s hours. If the answer is “we do it,” ask who checks the proposed edits before they go live. A subscription with no owner for the page is a better record of unfinished work, not a page improvement.

Try this today (20 minutes)

1. Write down three questions you have heard from actual buyers. Pick one important page they ought to find. Save the questions exactly as written.
2. For each question, open a fresh Temporary chat, choose Unpersonalized before sending the first message and select Search manually. Note the date, language and approximate location or IP country for the run.
3. Save each full answer. In a small table, mark whether your firm was named, whether your page was linked and whether the linked passage supports the answer. Keep a missing answer or failed run in the table too.
4. Open the first-party reports you have access to. If Search Console’s generative AI report is absent, record that rather than entering zero. Then send the same three questions to the vendor you are considering and ask for its export of those runs, with their conditions. Compare its records with your table, column by column.

Three answers are a sample, not a trend. The table tests the seller, not your visibility. In 20 minutes you recorded, by hand, the question, its conditions, the full answer, and whether a link was there and supported the answer. A tool that cannot show you the same for its own runs has not yet earned its fee. AEO, GEO and SEO are three jobs, not three synonyms; whichever job you buy software for, the decision rests on the records behind the offer.

Cite this:Before you buy an AEO tool, ask what its score measures.Len P. van der Hof. https://lenvanderhof.com/en/blog/aeo-tool-buying-test/ · Published 27 October 2026.