ChatGPT vs Claude vs Gemini vs Perplexity for Web Research

A market research team asks four assistants the same question: "How much does this product cost today at this shop in Türkiye?" One returns a price with numbered sources, one reads the page but does not say when, one gives a range based on Google results, and one says it cannot reach the page. All four are sold as "connected to the web"; what separates them is not how clever the model is but how each product relates to the web.

This post compares ChatGPT, Claude, Gemini and Perplexity from that single angle: how live search works, how sources are shown, whether a URL you give them can be read and where they stop, what the "deep research" and agent modes do, and what the web tools look like on the API side. Then we turn to the site owner: bot identities, robots.txt behaviour and the August 2025 dispute between Cloudflare and Perplexity. The last section is for data teams: why these assistants are not scraping tools and where a proxy fits. Vendor details were checked on their own pages on 7 October 2026; we leave out model version names.

How do the four assistants reach the web?#

All four have live search; the difference is when it runs and on what infrastructure. The model itself is not connected to the web: in every product a search layer sits in front of it, turns the question into queries, fetches results and hands them to the model. Whatever is current in the answer comes from that layer.

ChatGPT, according to OpenAI's help page, searches the web when needed, sometimes works with other search providers to do so, and rewrites your question into targeted queries. Search is available on every plan; sources arrive as clickable citations in the text, and a "Sources" view lists related links.

Claude gained web search in March 2025 and has offered it globally on all plans since May 2025. You enable "Web search" from the "+" menu in a chat, and Claude searches when the question depends on current information: recent events, prices, facts about organizations and products that may have changed, or an explicit request to look something up. It does not search for settled knowledge, mathematics or creative writing.

Gemini grounds its answers in Google Search. According to Google's help page, when sources are available a "Sources" button appears below or inside the answer and opens the links in a side panel; if there is no button, no links were provided for that answer. Not every answer rests on a search.

Perplexity starts from a different place. The product is an answer engine: every question goes to search first, and the model then writes an answer with numbered sources from the pages it found. There is no "turn off the web" option; the web is the precondition for the answer.

How do their citation formats differ?#

All four cite sources, but the relationship between source and answer differs. Perplexity puts a number next to each sentence and treats the source list as part of the answer. Claude cites at sentence level: which passage came from which URL, with a 150-character excerpt, travels inside the response. ChatGPT places citations as links inside the text and adds a source list. Gemini collects sources in a separate panel and treats an unsourced answer as normal.

The location column matters. OpenAI's help page says ChatGPT passes an approximate location derived from your IP address to its search partners, without the IP itself. A price question asked from Istanbul may not return the same results as one asked from Berlin. We found no page documenting this for the other three products, so we make no claim about them.

What happens when you paste a URL?#

Reading a page (fetch) is a separate capability from search: search returns summaries from an index, fetch goes to one specific address and retrieves the page itself. All four products use a separate bot identity for this when acting on a user's behalf; the mechanism is the same everywhere:

1. The address has to appear in the conversation. Claude's API tool makes this a rule: only URLs that appeared in a user message, in search results or in tool results can be fetched; an address the model wrote itself cannot. The reason given is the risk of data exfiltration.
2. Domain and robots.txt are checked. Claude's tool returns url_not_allowed for a path closed by robots.txt, a private network address or a URL that carries a credential. Gemini's URL context tool excludes local network addresses and tunnelling services.
3. The page is retrieved. Text, HTML and PDF are read. Claude's documentation states that pages rendered with JavaScript are not supported; Gemini's says content behind a login or a paywall is not read.
4. The content is trimmed. Claude has a max_content_tokens limit; Gemini allows 34 MB per URL and 20 URLs per request. The model reads the part that fits, not the whole page.
5. The answer is written and cited. Citations are optional for Claude's fetch; Gemini returns which URLs were read successfully alongside the answer.

Then there is caching. Claude's documentation says fetched content is cached and may not reflect the latest version at the URL; use_cache can be turned off for fresh content. Gemini first looks in its own index cache and fetches live only if the page is not there. For data that changes by the hour, such as a price, this decides how old the answer is.

What do deep research and agent modes do?#

All four products have a long-form research mode, and all run the same loop: plan, search, read, search again for what is missing, write the report. Gemini opened this mode as "Deep Research" in December 2024; OpenAI launched "deep research" in February 2025, Perplexity its "Deep Research" the same month, and Anthropic its "Research" feature in April 2025. The difference lies in how many pages are read, but also in how sources travel: Perplexity and Claude tie the citation to the sentence, ChatGPT and Gemini gather them under the report. We described the shape of this loop in how AI agents work.

Agent modes go one step further: the assistant browses, fills in forms and clicks. Perplexity shipped its own browser, Comet, for this; Google lists a separate bot identity (Google-Agent) for agents that browse on a user's behalf; OpenAI's agent mode lives inside ChatGPT. To a site owner such a visit resembles a human one, but the request still comes from a program and carries its identity in the user agent header.

What do the web tools look like on the API side?#

On all four APIs, web search is a server tool: the vendor's infrastructure runs the search and feeds the results to the model. The developer controls three things: the number of searches, a domain filter and the location.

The same request with and without country: "TR" can return different pages. Claude's documentation asks for at least one field and takes the country as a two-letter ISO code:

{
  "type": "web_search_20250305",
  "name": "web_search",
  "max_uses": 5,
  "user_location": {
    "type": "approximate",
    "city": "Istanbul",
    "country": "TR",
    "timezone": "Europe/Istanbul"
  }
}

In Google's grounding tool, the groundingMetadata returned with the answer carries the search queries, the source chunks and a searchEntryPoint, and Google asks that these "Search Suggestions" be shown to the user. Perplexity's search cannot be turned off, but search_recency_filter limits how old the sources may be.

Which bot identities matter to site owners?#

Each vendor uses three identities for three jobs: crawling for training, crawling for a search index, and opening a page on a user's behalf. Each is managed separately in robots.txt; blocking the training bot does not block the search bot. Our robots.txt guide covers how the file is written; here we list only the names and the stances.

OpenAI defines its three identities in its bots documentation. GPTBot crawls for training; OAI-SearchBot finds and links sites in ChatGPT search and is not used for training; ChatGPT-User visits a page when a user asks a question and does not crawl automatically. Because a user initiates those visits, the document says robots.txt rules "may not apply" to ChatGPT-User. A site that blocks OAI-SearchBot is not shown in ChatGPT search answers; a change takes about 24 hours to be picked up. IP ranges are published in separate JSON files.

Anthropic defines all three bots on its help page and says all three honour robots.txt; it is the only vendor that says "obeys" for its user-triggered bot. The page also says the bots support the Crawl-delay directive and publishes the IP ranges in a single JSON list.

Google's Gemini does not crawl with a separate crawler. Googlebot builds the index; Google-Extended is a robots.txt token with no user agent string of its own, and it governs whether crawled content may be used to train Gemini models and for Google Search grounding in Gemini Apps and on Vertex AI. The documentation states two things plainly: Google-Extended does not affect a site's place in Search, and user-triggered fetchers generally ignore robots.txt rules because a user requested the fetch. Blocking Google-Extended does not touch Googlebot or visits made on a user's behalf.

Perplexity writes in its crawler documentation that PerplexityBot crawls to surface sites in search results, is not used for training and respects robots.txt. The sentence for Perplexity-User is different: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." Both bots' IP ranges are published in separate JSON files.

The practical conclusion: robots.txt governs automated crawling, and three of the four vendors treat a single visit made on a user's behalf as outside that file. The way to limit those visits is a server-side rule based on the user agent and the published IP lists. A common setup blocks the training bots and leaves the search bots open:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

What happened between Cloudflare and Perplexity?#

On 4 August 2025, Cloudflare published a post alleging that Perplexity used undeclared crawlers. In its account, sites that had blocked PerplexityBot reported that their content kept appearing in Perplexity's answers, and new test domains Cloudflare had closed to all bots were still retrieved by Perplexity. The post said that once the declared user agent was blocked, requests arrived with a generic Chrome-like user agent and from changing networks (ASNs), and that robots.txt was sometimes not fetched at all; the same day Cloudflare removed Perplexity from its verified bots list.

Perplexity answered the next day with a post titled "Agents or Bots? Making Sense of AI on the Open Web": the traffic in question was not automated crawling but "user-driven agent" requests opening a page for a user's immediate question, nothing was indexed, and Cloudflare had confused part of the traffic with a third-party cloud browser service. The two accounts do not agree, and this post is not the referee. The lasting result is that the identity question moved towards a technical answer: the same month, Cloudflare opened a "signed agents" programme in which agents sign their requests with Web Bot Auth, an IETF draft. A bot copying a user agent string cannot produce that signature.

Are these assistants data collection tools?#

No. The limits above are design decisions: the number of searches per request, the cache, reading only URLs that appear in the chat, not running JavaScript and refusing paths closed by robots.txt all scale these tools to one user's immediate question. A hundred pages take a hundred questions, and you cannot control which version of a page was read. Our post on safe web access for LLMs describes what a model should watch for when it reads live pages.

For regular, large-scale collection there are three routes: a source that explicitly permits it, the source's official API, or your own crawler. On the third route, reading robots.txt, respecting the site's rate limits and identifying yourself with an honest user agent are your job. Running the crawler through a Datacenter Proxy separates your exit addresses from your application servers and makes it easier to identify yourself with a fixed IP list; it does not raise the number of requests a site allows per minute.

A proxy's real place is geography. A shop can show different prices, currencies and campaigns depending on the visitor's country. The assistants fetch a page from wherever their own infrastructure sits; they publish their IP ranges but make no promise about the country. In your own crawler that control is yours: a Residential Proxy with an exit in Türkiye returns the page as a buyer in that country sees it and lets you record "seen from which country" with the data. That an assistant's answer, in ChatGPT's case, may vary with where the question was asked is a separate matter.

Common mistakes#

• Recording an assistant's price without a date. The answer may come from a cache or from a page indexed days ago; store the source URL and the time of the answer together.
• Taking an unsourced answer for a sourced one. In Gemini, no "Sources" button means the answer did not rest on a search; in Claude, with web search off, the answer comes from training data.
• Trying to feed an assistant hundreds of URLs. The tools cap URLs and tokens per request; once the job grows, move to the official API or your own crawler.
• Reporting a JavaScript-rendered page as "the assistant could not read it". The page returns as an empty shell; the problem is how the page is built.
• Blocking only the training bot and considering the job done. Search bots and user-triggered bots are separate identities, and most of the latter do not read robots.txt at all.
• Trusting the user agent string. Anyone can write it; verification needs an IP list or a signed request.

Decision guide#

Frequently asked questions#

Is ChatGPT connected to the internet?#

Yes. ChatGPT searches the web when needed, on every plan, and attaches clickable sources to the answer. It sometimes works with other search providers and rewrites your question into targeted queries.

Can Claude search the web?#

Yes. Web search has been in Claude since March 2025 and available on all plans in all countries since May 2025. You enable it from the "+" menu in a chat; Claude searches when a question needs current information and ties every citation to its source.

What is the difference between Perplexity and ChatGPT?#

Perplexity starts every answer with a search and makes the numbered sources part of the answer; ChatGPT searches only when needed and gives sources as inline links. Perplexity suits a fast, sourced answer; ChatGPT suits a long conversation and a wider set of tools.

Can AI assistants read pages behind a paywall?#

No. Gemini's URL context documentation states that content behind a login or a paywall is not read, and the other tools request the page without a session as well. The assistant sees the version of the page the site shows to everyone.

Does blocking Google-Extended affect Google Search rankings?#

No. Google's documentation says Google-Extended has no separate user agent string and does not affect a site's place in Google Search. The token only governs whether crawled content is used for Gemini training and grounding.

Can I block AI bots completely with robots.txt?#

Partly. Training and search bots honour robots.txt; most bots that open a page on a user's behalf do not, with Anthropic's Claude-User as the exception. To limit user-triggered visits you need server-side rules based on the user agent and the published IP lists.

Summary#

All four assistants reach the web: Perplexity makes search the precondition for every answer, ChatGPT and Claude search when needed, and Gemini leans on Google Search and treats an unsourced answer as normal. For reading URLs they share documented limits: no JavaScript, no logins, no paths closed by robots.txt, and a cache. For site owners the real split is in the user-triggered bots; every vendor except Anthropic treats those visits as outside robots.txt, so control shifts to user agents and IP lists. For data teams these tools are research assistants, not collectors; regular collection needs a permitting source, an official API or your own crawler, and to see a page from the right country you can look at our data scraping infrastructure.