What Is Common Crawl? The Open Web Archive Behind AI

Published:

14 minute read

Acar Diveroli
Written by: Acar Diveroli
Web page cards ride a belt into a CC-MAIN-2026-43 machine; out come WARC and WAT blocks and a blue WET block of plain text

A line in your server log reads CCBot/2.0 (https://commoncrawl.org/faq/). Or the data section of an AI model paper says that most of its training text came from "filtered Common Crawl". Both lead to the same place: a non-profit that has been copying the public web since 2008 and gives the copies to anyone who wants them.

This post covers what a crawl contains, how the crawler behaves, how big and fresh the data is (with the official crawl list as we read it on 17 November 2026), who uses it, where it falls short, how site owners opt out and how it compares with the Wayback Machine.

What is Common Crawl?

Common Crawl is a 501(c)(3) non-profit that crawls the web and publishes the pages it collects as an open dataset. Its website describes a corpus of more than 300 billion pages, free and open since 2007 and cited in over 10,000 research papers (read on 17 November 2026).

The reasoning is practical: crawling billions of pages takes servers and engineers that only large companies can afford, so one group does it and shares the result. The data sits on Amazon S3 in the US East (N. Virginia) region and can also be downloaded over HTTPS from data.commoncrawl.org. It is not a search engine: there are no ranked results, only an index that looks up addresses.

What is inside a Common Crawl crawl?

Each crawl is published as files in three formats, plus indexes that tell you where a given page is stored.

  • WARC (Web ARChive) is the raw record: for every page, the HTTP request CCBot sent and the full response, headers and HTML included.
  • WAT files hold metadata computed from each WARC record, stored as JSON: response headers, the links found on the page and other details.
  • WET (WARC Encapsulated Text) files hold only the plain text extracted from each page.
  • The URL index answers "was this address captured, and where?". A lookup returns the capture time, the HTTP status and the file, offset and length of the record. A second form, the URL Index in Parquet format, suits filtering millions of rows with database tools.

Robots.txt files and non-200 responses, such as redirects and errors, are published as separate sets. The main parts of the October 2026 crawl, with compressed sizes from Common Crawl's announcement:

FileWhat it holdsSize (compressed)Typical use
WARCRequests, responses, raw HTML86.19 TiBRe-reading pages your own way
WATHeaders and links as JSON14.06 TiBLink graphs, site research
WETExtracted plain text5.85 TiBLanguage data, text analysis
URL indexCaptured addresses and locations0.43 TiBFinding pages before download

A tebibyte (TiB) is about 1.1 terabytes. The text is about a fifteenth of the raw archive, so a team that only needs words starts from WET files, and one that needs page structure or a better text extractor starts from WARC.

How does CCBot collect pages?

Common Crawl's crawler, CCBot, reads robots.txt first, fetches plain HTML and slows down when a server struggles. The details below come from Common Crawl's FAQ, read on 17 November 2026.

  1. It checks robots.txt. If the file disallows CCBot, the page is not fetched; Common Crawl rechecks the file from time to time.
  2. It identifies itself. Requests carry the user agent CCBot/2.0 (https://commoncrawl.org/faq/) and come from a few dedicated IP ranges listed in the FAQ. A reverse DNS lookup on a real CCBot IPv4 address returns a name ending in crawl.commoncrawl.org, which separates it from bots that only borrow the name.
  3. It finds pages through links and sitemaps. CCBot follows links from other sites, so it can reach pages you never linked yourself, and reads any sitemap listed in robots.txt. It respects nofollow links.
  4. It fetches HTML without running scripts. No JavaScript, no cookies: a page whose content is built in the browser can be almost empty in the archive.
  5. It paces itself. It waits a few seconds between requests to the same site, backs off on 429 (too many requests) and 5xx errors, and obeys Crawl-delay.
  6. It samples. Common Crawl does not archive whole websites; it keeps a randomly selected subset of each site's pages.
  7. It publishes. Captures are written to WARC files; WAT, WET and the indexes are derived from them, and the crawl is released under an ID such as CC-MAIN-2026-43.

How big is Common Crawl, and how often is it updated?

Common Crawl publishes a new crawl every month; in 2025 and 2026 one appeared for each month. The official list of crawls is a JSON file at https://index.commoncrawl.org/collinfo.json. We read it once on 17 November 2026 at 02:02 UTC; these are its first two entries, unchanged:

json
[
  {
    "id": "CC-MAIN-2026-43",
    "name": "October 2026 Index",
    "timegate": "https://index.commoncrawl.org/CC-MAIN-2026-43/",
    "cdx-api": "https://index.commoncrawl.org/CC-MAIN-2026-43-index",
    "from": "2026-10-05T10:56:23",
    "to": "2026-10-18T05:34:47"
  },
  {
    "id": "CC-MAIN-2026-39",
    "name": "September 2026 Index",
    "timegate": "https://index.commoncrawl.org/CC-MAIN-2026-39/",
    "cdx-api": "https://index.commoncrawl.org/CC-MAIN-2026-39-index",
    "from": "2026-09-04T13:16:03",
    "to": "2026-09-17T05:59:21"
  },

The list held 129 entries. The newest, CC-MAIN-2026-43, was crawled between 5 and 18 October 2026; the oldest covers 2008 to 2009 in ARC, the format used before WARC. Each entry has a cdx-api address, the index for that crawl, and the dates show the rhythm: about two weeks of crawling, then a pause until the next month.

Common Crawl's announcement of the October 2026 crawl, published on 21 October, gives its size: 2.16 billion pages, or 371.19 TiB uncompressed, from 41.4 million hosts and 34.3 million registered domains. Of those pages, 830.1 million were at addresses no earlier crawl had visited; the rest let researchers compare the same URL across months.

A separate news dataset, CC-NEWS, was launched in 2016 to publish articles from news sites soon after capture instead of waiting for the monthly release.

Who uses Common Crawl?

Common Crawl is used mostly by people who need web pages in bulk and cannot crawl billions of them on their own:

  • Language model training. A large language model (LLM) learns by predicting text and needs hundreds of billions of words; Common Crawl is the largest free supply.
  • Academic research. Linguists and web researchers study how the web is built, which languages it uses and how pages change.
  • SEO and link analysis. WAT files list the links on each page, and Common Crawl builds host- and domain-level web graphs from them with rank scores such as harmonic centrality and PageRank; the May to July 2026 host graph had 240.4 million nodes and 3.7 billion edges.

The best-documented use is in model papers. OpenAI's GPT-3 paper describes 41 monthly Common Crawl shards from 2016 to 2019: 45 TB of compressed plain text before filtering and 570 GB after, making up 60% of the training mix. The authors kept pages similar to higher-quality text and removed near-duplicate documents.

Google's T5 paper built C4, the Colossal Clean Crawled Corpus, from a single month of Common Crawl, April 2019, keeping about 750 GB of English text after cleaning. Cleaned sets like C4 let teams use Common Crawl text without processing a raw crawl themselves.

Training is a different job from answering questions about today. A model trained on a 2019 snapshot knows nothing newer, so many AI tools fetch current pages when a question is asked, a pattern called retrieval-augmented generation (RAG). Our guide to web scraping for RAG shows how those pages become clean text.

What are the limits of Common Crawl?

Common Crawl is a large sample of the public web, taken once a month from one place, and each part of that sentence sets a limit.

  • A sample, not the web. Each site is represented by a random subset of its pages. Sites that block CCBot are missing, pages behind a login are never fetched, and content built by JavaScript is thin.
  • Snapshots, not live data. A page that changed yesterday shows its old version, or none, until the next crawl. What happens between two crawls is not in the main dataset.
  • Uneven languages. Common Crawl detects each page's language with CLD2, an open-source detector. Its language statistics for the October 2026 crawl show 42.56% English, then Russian (6.30%) and German (5.77%); Turkish was 1.14%.
  • Noise and duplicates. The T5 authors wrote that most text extracted from Common Crawl is not natural language: menus, error messages and repeated text. Serious uses filter and deduplicate first.
  • One viewpoint. CCBot works from a handful of IP ranges. A site that changes prices, language or content by visitor location answers CCBot with the version meant for those addresses.

Is Common Crawl data free to use?

Common Crawl data is free to download, but the pages in it still belong to the people who published them. Its terms of use (last updated on 7 March 2024) say Common Crawl does not own the crawled content and that users must respect third parties' copyrights and carry the risk of their own use. The terms do not forbid AI training, but users must indemnify Common Crawl against claims arising from it.

Using crawled pages to train AI is disputed. In April 2026 the News/Media Alliance, a US publishers' group, wrote to Common Crawl asking it to remove publishers' content on request and to forbid AI use in its terms. Whether training on copied pages is lawful is still argued in courts and legislatures and differs by country; for a commercial project, ask a lawyer in your market.

Pages can also contain personal data such as names and email addresses. Processing it falls under privacy laws such as the GDPR in the EU or KVKK in Türkiye, whoever collected the page first.

How can site owners opt out of Common Crawl?

To keep CCBot out of your site, add a group for it to robots.txt. Common Crawl's FAQ gives these two lines:

text
User-agent: CCBot
Disallow: /

To slow it down instead, give it a delay in seconds between requests:

text
User-agent: CCBot
Crawl-delay: 2

Our robots.txt guide explains how groups and rules are read. Blocking CCBot does not stop GPTBot, ClaudeBot or any other bot; each needs its own group.

A robots.txt rule governs future visits; pages captured before you added it are already in published crawls. For legal removal requests, Common Crawl keeps a public list called the Opt-Out Ledger.

Common Crawl vs Wayback Machine vs your own crawl

Common Crawl, the Wayback Machine and a crawl of your own all store copies of web pages, but they answer different questions.

Common CrawlWayback MachineYour own crawl
Made forBulk analysis by programsViewing old pagesYour specific question
How you use itDownload files, query the indexEnter an address, pick a dateCollect the pages you choose
FreshnessMonthly snapshotsWhen a page was savedAs often as you schedule it
CoverageA sample of many sitesPages that were savedOnly your target sites
CostFree data, your computingFree to browseYour setup and upkeep

The Wayback Machine, run by the Internet Archive, is built for people: you type an address, pick a capture from a calendar and see the page as it looked, and anyone can ask it to save a single page with Save Page Now. Our Wayback Machine guide covers that side. Common Crawl has no such viewer; its files are meant for code that reads millions of pages at once.

Why Common Crawl cannot give you fresh, local data

Common Crawl cannot show you today's price of a product in Madrid or how a page looks to visitors in Seoul, because it is crawled once a month from a few IP ranges. Data like that has to be collected for your question, at the time and from the place that matter.

Teams that track prices or check localized pages collect those public pages themselves: on a schedule, at a polite rate, within robots.txt and the site's terms. To see a page as a visitor in a given country or city sees it, the request has to come from there; a Residential Proxy with country and city targeting sends it through a home connection in that place.

Our web crawler solution page shows where proxies fit in such a crawler. They do not belong in front of Common Crawl itself: its FAQ asks users not to query the index through proxy networks, to send one request at a time with a clear user agent and to slow down on 503 errors.

Common mistakes

  • Treating it as the whole web. It is a sample; a page missing from a crawl proves nothing.
  • Training on raw WET text. Most of it is menus, notices and duplicates; filter and deduplicate first.
  • Hammering the index server. It is heavily rate limited, and an IP that sends too many requests can be blocked for 24 hours. For broad filtering, use the Parquet URL Index with a tool such as Amazon Athena or Apache Spark.
  • Expecting robots.txt to clean old crawls. A new rule affects future visits only.

Decision guide

Your needBest option
Broad web text for research or model trainingWET files or a filtered set such as C4
How one page looked on a past dateThe Wayback Machine
Which sites link to whichWAT files or Common Crawl's web graphs
Whether your site is in Common CrawlA slow lookup in the URL index
Keep your site out of future crawlsA CCBot group in robots.txt
Today's prices, or pages as seen from one countryYour own polite crawl or a managed service

Frequently asked questions

Is Common Crawl free?

Yes. Anyone can download the data over HTTPS or from Amazon S3 without paying Common Crawl; you pay only for your own storage and computing, and Common Crawl recommends processing it in us-east-1, the AWS region where it is stored. The pages keep their copyright.

Does Common Crawl respect robots.txt?

Yes. CCBot reads robots.txt before fetching, skips what is disallowed for CCBot, obeys Crawl-delay and respects nofollow links. The FAQ lists its IP ranges, so you can check whether a visitor calling itself CCBot is real.

Does Common Crawl include images?

Practically no. In the October 2026 crawl, 98.31% of captures were HTML pages and 1.00% PDF files, while images made up about a thousandth of a percent. The HTML still contains image addresses, but not the image files.

What is the difference between Common Crawl and C4?

Common Crawl is the raw monthly crawl. C4 is a cleaned English dataset that Google researchers built from one month of it, April 2019, by removing boilerplate lines, very short pages, code, offensive words and repeated text, which left about 750 GB.

Can you search Common Crawl like a search engine?

No. There is no full-text search and no ranking; the index only finds captures by address or address pattern. Searching the text means processing WET files with your own tools.

How can I check if my website is in Common Crawl?

Open index.commoncrawl.org, choose a crawl and search for your domain with a wildcard, such as example.com/*. The results list each captured URL with its capture time and HTTP status. Send one query at a time, because the server is rate limited.

Summary

Common Crawl is a free, open copy of a large sample of the public web, captured about once a month by CCBot and published as raw pages (WARC), metadata (WAT), plain text (WET) and URL indexes. It sits behind much of AI training and web research, but it is a periodic sample from one viewpoint, it needs heavy filtering, and its pages keep their copyright. Site owners control CCBot through robots.txt. When your data has to be current or local, collect it at a polite rate, or ask our sales team about managed page extraction.

Ask ChatGPTAsk Claude