# AI search landscape

> How ChatGPT, Gemini, Claude and Google's AI answers pick what to cite, and the bots behind them.

Source: https://youcited.com/docs/geo-concepts/ai-search-landscape
Section: GEO Concepts
Last updated: 2026-10-29
All docs: https://youcited.com/docs/index.md

## The AI Search Landscape

AI search is not one system. Each assistant finds, reads and cites sources in its own way, which is why Cited checks several engines and more than one model from each.

## How each engine works

### ChatGPT (OpenAI)

ChatGPT searches the web when a question needs current information, reads a handful of pages, and writes an answer with linked sources. OpenAI runs its own search index, fed by **OAI-SearchBot**, and fetches pages on demand with **ChatGPT-User** when a chat needs them.

ChatGPT tends to cite pages that answer the question directly: clear service and product pages, comparison articles with concrete details, and FAQ pages. Review sites and directories show up often for local and "best X" questions.

### Gemini and Google's AI answers

Google's AI shows up in three places: the Gemini app, **AI Overviews** at the top of Google results, and **AI Mode**, a chat-style search. All three draw on Google Search, so the pages Googlebot can crawl and Google already trusts have a head start. Google's Knowledge Graph, Business Profiles and reviews feed into the answers too, which matters a lot for local businesses.

Structured data (Schema.org) helps Google understand who you are and what you offer. The **Google-Extended** token in robots.txt controls whether Google may use your pages for Gemini.

### Claude (Anthropic)

Claude searches the web for questions that need it, reads the results and cites the pages it uses. Anthropic's **Claude-SearchBot** builds its search index, and **Claude-User** fetches pages on demand for a chat. Claude favours clear, specific, well-sourced pages and is quick to skip vague marketing copy.

### Same question, different answers

The same question often gets different businesses from different engines, and even from different models of the same engine. A fast model (what many people get in a free app) can cite different sources from the flagship model. That's why a Cited audit asks a flagship and a fast model from each engine, three times per question.

## How AI decides what to cite

Across engines, a few things decide which sources make it into the answer:

### Relevance

The page must answer the question. Pages that answer it plainly, early on the page, are easier to pick.

### Authority and trust

Engines prefer sources other sites already trust: established publications, well-known directories and review sites, and businesses with consistent details across the web.

### Freshness

For anything that changes (prices, opening hours, "best of this year"), engines prefer recently updated pages with clear dates.

### Structure

Clear headings, short paragraphs, lists, tables and schema markup make it easy for an AI to pull out a fact and cite it.

## AI crawlers

| Bot | Operator | Purpose |
|---|---|---|
| OAI-SearchBot | OpenAI | Search index for ChatGPT answers |
| ChatGPT-User | OpenAI | Fetches pages on demand during a chat |
| GPTBot | OpenAI | Training data |
| Claude-SearchBot | Anthropic | Search index for Claude answers |
| Claude-User | Anthropic | Fetches pages on demand during a chat |
| ClaudeBot | Anthropic | Training data |
| Googlebot | Google | Google Search, which Gemini, AI Overviews and AI Mode draw on |
| Google-Extended | Google | Robots token for using your pages in Gemini |

The search and user bots decide whether you can appear in answers today. The training bots decide whether your pages help train future models. Many businesses allow all of them; if you want to opt out of training, block only the training bots.

## robots.txt examples

Allow AI search and chat fetchers, and opt out of training:

```
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /
```

Allow everything (the simplest choice for most small businesses):

```
User-agent: *
Allow: /
```

## Firewalls and CDNs

robots.txt isn't the only gate. Cloudflare's "Block AI bots" setting, other firewalls and some hosts turn AI bots away before they reach your site, often without the owner knowing. The free [AI bot check](/ai-bot-check) tests both robots.txt and firewall blocks for all eight bots, and the [site check](/docs/feature-guides/site-check) in every paid audit includes it.