Where does ChatGPT get its information? How AI search chooses sources

Perplexity, Copilot and Google's AI features work the same way at a high level, but each draws on a different index. The distinction matters because only live retrieval can cite your latest page, and retrieval only reaches pages a crawler has fetched. This guide explains where each AI engine gets its sources, which AI crawlers do what, and what makes a page more likely to be chosen.

Two sources of knowledge: training data and live retrieval

Every AI assistant answers from two sources: knowledge baked into the model during training, and pages it retrieves at answer time. Training data is a snapshot that ends at the model's cutoff and has no links. Live retrieval pulls current pages from a search index and can cite them.

So when people ask how ChatGPT gets its information about their brand, the useful answer is retrieval. You can opt out of training, but you can't opt a page in. A page reaches a live answer only if a search crawler has fetched it and an index has stored it. That's why AI answers cite what's been crawled, and they lean toward pages that are kept current.

ChatGPT: GPTBot, OAI-SearchBot and search partners

ChatGPT uses separate agents for training, search and live fetching, plus outside search providers and partner content. OpenAI documents each agent in its crawler docs:

• GPTBot crawls content "that may be used in training" OpenAI's models. Disallowing it signals your content shouldn't be used for training.
• OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features." Sites that opt out of it "will not be shown in ChatGPT search answers, though can still appear as navigational links."
• ChatGPT-User visits a page when a user's request calls for it. OpenAI says it "is not used for crawling the web in an automatic fashion," that "robots.txt rules may not apply," and that it "is not used to determine whether content may appear in Search."

OpenAI says changes to robots.txt take about 24 hours to reach its search systems.

Does ChatGPT use Bing? At the October 2024 launch of ChatGPT search, OpenAI said it "leverages third-party search providers, as well as content provided directly by our partners" (OpenAI). Microsoft announced in May 2023 that "Bing is coming to ChatGPT as the default search experience" (Microsoft). OpenAI hasn't confirmed which providers it uses today, and third-party estimates of how closely ChatGPT's citations follow Bing's rankings conflict sharply. The safe reading: allow OAI-SearchBot, and don't treat any one search index as a confirmed route into ChatGPT. Our page on Bing, IndexNow and AI answers covers the Bing side, and ChatGPT SEO covers the full playbook.

Perplexity: PerplexityBot and its own index

Perplexity answers from its own web index, built by its own crawler, PerplexityBot. Perplexity says the index tracks "over 200 billion unique URLs" and that machine learning predicts which URLs need indexing and when (Perplexity). Its systems process "tens of thousands of index update requests" every second (Perplexity).

Perplexity documents two agents (Perplexity docs):

• PerplexityBot surfaces and links websites in Perplexity's search results and is "not used to crawl content for AI foundation models." Perplexity recommends allowing it.
• Perplexity-User fetches pages to answer a user's question. Because a person requested the fetch, it "generally ignores robots.txt rules."

Perplexity isn't an IndexNow participant and has no URL submission tool, so discovery depends on its crawler finding you: through links, sitemaps and mentions on pages it already visits.

Copilot and Bing

Microsoft Copilot and Bing's AI summaries are grounded in Bing's index. The AI looks up pages with search queries, then cites what it finds. Bing Webmaster Tools' AI Performance report, in public preview since February 10, 2026, shows citations across "Microsoft Copilot, AI-generated summaries in Bing, and select partner integrations," plus a sample of "grounding queries": the phrases the AI used to retrieve the pages it cited (Bing Webmaster Blog).

Microsoft is the most explicit engine about freshness. Its July 2025 guidance says "freshness signals directly influence how quickly updates are reflected in search results and AI generated answers." It recommends IndexNow for real-time URL submission and sitemap lastmod values that reflect true content changes. It also says "no tool can guarantee when or how your content will appear in AI-generated results" (Bing Webmaster Blog). Bingbot is the crawler behind all of this, so a page Bingbot hasn't fetched can't be cited in Copilot.

Google AI Overviews and AI Mode: built on Google's index

Google AI Overviews and AI Mode link only to pages that are in Google's Search index, and Google is still the biggest gateway to the web: it sends about 9 in 10 of the visits that search engines send to websites worldwide (StatCounter, September 2026). Google's rule is simple: to appear as a link in AI Overviews or AI Mode, a page must be "indexed and eligible to be shown in Google Search with a snippet." It adds that there are "no additional technical requirements," and that "indexing and serving isn't guaranteed" (Google Search Central). Being indexed is the first step. Whether an AI answer cites your page still depends on quality, relevance and authority.

Both features may use "query fan-out," running several related searches across subtopics, so a page can be cited for a sub-question even if it doesn't rank for the user's exact wording. Google's 2026 guide says its AI features retrieve "relevant, up-to-date web pages from our Search index." It also says optimizing for them "is optimizing for the search experience, and thus still SEO." No special files, markup or chunking are required, and Google Search ignores AI text files such as llms.txt (Google AI optimization guide).

Gemini works similarly when it grounds an answer. Google's API docs say the model decides whether a Google Search would help, runs its own queries and cites the URLs it gets back (Gemini API docs). Google's crawler documentation says Gemini Apps can be grounded in "content from the Google Search index" (Google's common crawlers). So one Google index feeds Search, AI Overviews, AI Mode and, when it decides to search, Gemini.

AI crawlers compared: who they are and what they do

AI crawlers fall into three groups: search crawlers that decide whether you can appear in AI answers, training crawlers that collect data for models, and user-triggered fetchers that read a page while answering. Only the first group controls whether you can be cited in search answers.

For user agents, IP ranges and robots.txt rules for each bot, see our AI crawlers guide and the pages on OAI-SearchBot, GPTBot, ChatGPT-User and ClaudeBot.

Sources: Google, Google crawlers, OpenAI, Perplexity, Bing.

If I block GPTBot, will ChatGPT stop citing me? Not on its own. Search citations depend on OAI-SearchBot and the indexes ChatGPT draws on. OpenAI says each setting is independent.

What makes a page more likely to be chosen

A page is more likely to be chosen when it ranks well in the engine's underlying search, comes from a source the web already trusts, matches the question closely and states the answer clearly. Freshness helps at the margin, mostly for time-sensitive questions. The evidence, roughly in order of strength:

1. Search ranking. A NeurIPS 2025 benchmark found a document's position in the model's context, which traditional search ranking drives, mattered far more for citation than on-page "conversational SEO" (C-SEO Bench). In July 2025, Ahrefs found 76.1% of AI Overview-cited pages ranked in Google's top 10 (Ahrefs). In a March 2026 update, after AI Overviews moved to Gemini 3 in January, that share was 37.9%, partly because Ahrefs also improved its parsing method (Ahrefs).
2. Brand and authority. Across 75,000 brands, web mentions correlated with AI Overview visibility far more strongly (ρ = 0.664) than backlinks did (0.218) (Ahrefs). A 2025 University of Toronto preprint found AI search heavily favors earned media, meaning third-party coverage, over brand-owned pages (Chen et al.).
3. Fit to the question. An AirOps study of 548,534 pages ChatGPT retrieved found pages whose titles overlapped the query by 50% or more were cited at 20.1%, against 9.3% for pages under 10% overlap (AirOps). Engines also disagree: in Kevin Indig's May 2026 analysis of ChatGPT, Perplexity and Google AI Overviews, 91% of citations appeared in only one engine (Growth Memo). Each engine needs checking on its own.
4. Quotable content. The original GEO study found adding quotations, statistics and source citations raised visibility in generative answers on its benchmark (Aggarwal et al.). Our LLM SEO guide covers how to write passages a model can quote.
5. Freshness. Seer Interactive's July 2026 vendor study of 7,683 cited pages found 75% had been updated within a year (Seer Interactive). Freshness matters most for time-sensitive queries, and it only counts once the update has been crawled.

None of this is a guarantee. Each engine picks its own sources, and citations shift from day to day.

How to let the right AI crawlers in

To let the right AI crawlers in, allow the search crawlers in robots.txt and at your firewall, decide separately on training crawlers, and confirm the pages you care about are actually indexed. A practical sequence:

1. Audit robots.txt. Make sure Googlebot, Bingbot, OAI-SearchBot and PerplexityBot aren't disallowed, including by a broad User-agent: * rule.
2. Decide on training crawlers separately. If you want to opt out of training, disallow GPTBot or add Google-Extended rules. Don't touch the search crawlers.
3. Check your CDN and firewall. Bot-protection rules often block crawlers that robots.txt allows. OpenAI and Perplexity publish IP ranges you can allowlist.
4. Wait, then verify. OpenAI and Perplexity say robots.txt changes can take up to about 24 hours. Check your server logs for the user agents afterwards.
5. Keep key content in HTML text. Don't hide your main answer in images, PDFs or scripts that only load after user interaction.
6. Confirm indexing. Access only makes crawling possible. Check that your important pages are actually in Google's and Bing's indexes, in their current version.

Step 6 is where many teams find surprises, and it's what FreshlyCited handles on the Google side. Paste your pages or backlinks into a visibility check to see which pages Google has indexed, push only the invisible ones and get a re-check. FreshlyCited doesn't track AI citations. It helps the pages you want cited get found and indexed by Google, which makes them eligible for AI Overviews and AI Mode, and for verified sites it also sends them to Bing through IndexNow, free. ChatGPT, Perplexity and other assistants use their own crawlers and search partners. We don't submit to them, and we don't claim to. For the full strategy, start with our AI search optimization guide. Check your own links free.