The AI Search Landscape
AI search is not one system. Each assistant finds, reads and cites sources in its own way, which is why Cited checks several engines and more than one model from each.
How each engine works
ChatGPT (OpenAI)
ChatGPT searches the web when a question needs current information, reads a handful of pages, and writes an answer with linked sources. OpenAI runs its own search index, fed by OAI-SearchBot, and fetches pages on demand with ChatGPT-User when a chat needs them.
ChatGPT tends to cite pages that answer the question directly: clear service and product pages, comparison articles with concrete details, and FAQ pages. Review sites and directories show up often for local and "best X" questions.
Gemini and Google's AI answers
Google's AI shows up in three places: the Gemini app, AI Overviews at the top of Google results, and AI Mode, a chat-style search. All three draw on Google Search, so the pages Googlebot can crawl and Google already trusts have a head start. Google's Knowledge Graph, Business Profiles and reviews feed into the answers too, which matters a lot for local businesses.
Structured data (Schema.org) helps Google understand who you are and what you offer. The Google-Extended token in robots.txt controls whether Google may use your pages for Gemini.
Claude (Anthropic)
Claude searches the web for questions that need it, reads the results and cites the pages it uses. Anthropic's Claude-SearchBot builds its search index, and Claude-User fetches pages on demand for a chat. Claude favours clear, specific, well-sourced pages and is quick to skip vague marketing copy.
Same question, different answers
The same question often gets different businesses from different engines, and even from different models of the same engine. A fast model (what many people get in a free app) can cite different sources from the flagship model. That's why a Cited audit asks a flagship and a fast model from each engine, three times per question.
How AI decides what to cite
Across engines, a few things decide which sources make it into the answer:
Relevance
The page must answer the question. Pages that answer it plainly, early on the page, are easier to pick.
Authority and trust
Engines prefer sources other sites already trust: established publications, well-known directories and review sites, and businesses with consistent details across the web.
Freshness
For anything that changes (prices, opening hours, "best of this year"), engines prefer recently updated pages with clear dates.
Structure
Clear headings, short paragraphs, lists, tables and schema markup make it easy for an AI to pull out a fact and cite it.
AI crawlers
| Bot | Operator | Purpose |
|---|---|---|
| OAI-SearchBot | OpenAI | Search index for ChatGPT answers |
| ChatGPT-User | OpenAI | Fetches pages on demand during a chat |
| GPTBot | OpenAI | Training data |
| Claude-SearchBot | Anthropic | Search index for Claude answers |
| Claude-User | Anthropic | Fetches pages on demand during a chat |
| ClaudeBot | Anthropic | Training data |
| Googlebot | Google Search, which Gemini, AI Overviews and AI Mode draw on | |
| Google-Extended | Robots token for using your pages in Gemini |
The search and user bots decide whether you can appear in answers today. The training bots decide whether your pages help train future models. Many businesses allow all of them; if you want to opt out of training, block only the training bots.
robots.txt examples
Allow AI search and chat fetchers, and opt out of training:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Allow everything (the simplest choice for most small businesses):
User-agent: *
Allow: /
Firewalls and CDNs
robots.txt isn't the only gate. Cloudflare's "Block AI bots" setting, other firewalls and some hosts turn AI bots away before they reach your site, often without the owner knowing. The free AI bot check tests both robots.txt and firewall blocks for all eight bots, and the site check in every paid audit includes it.