At a glance
- Access comes first: blocks in robots.txt or at the CDN keep AI crawlers away, often without anyone noticing.
- Without an entry in the Google and Bing indexes, the basis for answers that cite sources is missing.
- A fixed set of questions, asked regularly in several assistants, produces a comparable series of measurements.
- Visits from AI answers can be recognised in web analytics by sources such as chatgpt.com or perplexity.ai.
Test 1: Can AI crawlers reach the website?
Every provider sends its own crawlers. Some collect texts to train the models, others build the index for web search, and others fetch a page when someone asks about it in a chat. For answers that cite sources, the last two groups matter most.
- OAI-SearchBot
- OpenAI, ChatGPT search index
- ChatGPT-User
- OpenAI, retrieval on request in the chat
- GPTBot
- OpenAI, model training
- PerplexityBot
- Perplexity, search index
- Perplexity-User
- Perplexity, retrieval on request
- Claude-SearchBot
- Anthropic, search index
- Claude-User
- Anthropic, retrieval on request
- ClaudeBot
- Anthropic, model training
- Bingbot
- Microsoft, Bing index, also the basis for Copilot
- Googlebot
- Google, search index, also the basis for AI Overviews
- Google-Extended
- Not a crawler of its own: a token in robots.txt that controls use by Gemini
Three places need checking: the website’s robots.txt (at /robots.txt), the settings of the CDN or firewall (in Cloudflare under AI Crawl Control) and, if accessible, the server logs. They show whether the crawlers actually come by and which response they get. The status code 403 means: turned away.
Test 2: Is the website in the search indexes?
A search for site:example.co.uk on Google and on Bing gives a rough idea of how many pages each one knows. Google Search Console and Bing Webmaster Tools are more precise: both show which pages are indexed, which are not and why. If important service pages are missing, the problem comes before any AI optimisation.
Test 3: Do the assistants name the company?
The test only becomes meaningful with fixed questions, asked regularly in the same form. Three kinds of questions cover the essentials:
- Brand: ‘What does Example Ltd do?’ shows what an assistant knows about the company.
- Service and region: ‘Who maintains commercial refrigeration in Manchester?’ shows whether the company is named when there is a concrete need.
- Problem: ‘How can a cold room use less electricity?’ shows whether expert content serves as a source.
The questions are asked in ChatGPT with web search, in Perplexity, in Microsoft Copilot and in Google Search with AI Overviews and AI Mode, ideally without being signed in or in a new chat, so that earlier conversations do not colour the result. For each question, record: named or not, in which position, with or without a link and which sources the assistant cites.
Answers vary. The same question can get a different answer the next day. What is meaningful is therefore the trend over months, not a single test.
Test 4: Are the facts right?
Assistants sometimes repeat outdated services, wrong addresses or details of companies with the same name. This can only be corrected at the sources: on your own website, in the structured data, in the Google Business Profile, in business directories and in profiles. The more consistently name, address and services appear everywhere, the less often assistants mix a company up.
Test 5: Do visits from AI answers arrive?
In web analytics, visits from AI answers appear with their own sources, such as chatgpt.com, perplexity.ai, copilot.microsoft.com or gemini.google.com. ChatGPT usually adds utm_source=chatgpt.com to links. In Google Analytics 4, a dedicated channel group brings these sources together and makes them comparable over months.
Clicks from Google AI Overviews cannot be reported separately: Search Console counts them together with the rest of web search.
Cadence and effort
About an hour is enough to get started: check access and the index, define ten to twenty questions, record the first measurement. After that, a monthly round with the same questions is enough, plus one after every major change to the website. Specialised tools ask such questions automatically and at scale; for the trend, a spreadsheet will do at first.
The fundamentals behind this are explained in the article SEO and GEO.
Frequently asked questions
Why does ChatGPT not name a company that Google finds?
ChatGPT search uses its own crawlers and different indexes than Google. Common causes are a block on OAI-SearchBot, gaps in indexing at Bing or details on the website that cannot be clearly attributed.
Should the crawlers for training be allowed as well?
For answers that cite sources, the search crawlers are what counts. Training works in the long run: what goes into the training data, a model can repeat even without web search. Whether that is wanted is a trade-off between reach and control over your own content.