--- title: "HTTP 402 Payment Required: Pay Per Crawl and x402 Explained" description: "HTTP 402 Payment Required means the server wants payment before it serves a resource. Why the code sat unused for decades and how pay per crawl and x402 use it." url: https://proxynet.io/blog/http-402-payment-required date: 2026-10-06 author: "Acar Diveroli" category: "Web Scraping, AI" lang: en --- # HTTP 402 Payment Required: Pay Per Crawl and x402 Explained A crawler that has fetched the same news site for months starts logging a new answer: `402 Payment Required`, with a `crawler-price` header attached. Elsewhere, a developer sees the same three digits on a Vercel project, or a Stripe call fails with a 402 that has nothing to do with crawlers. The code is almost as old as the web, yet until recently hardly any server sent it. This post covers what 402 means in the HTTP standard, why it sat unused, and where it appears now: Cloudflare's pay per crawl, the open x402 protocol, AWS WAF and a few billing APIs. Then come what a 402 should change in a crawler or an AI agent and what site owners can decide. Vendor details were checked on their own pages on 6 October 2026. > **Note: Short answer** > > HTTP 402 Payment Required is a status code that RFC 9110 still lists as "reserved for future use", so no standard format stands behind it. Cloudflare's pay per crawl now answers verified AI crawlers with a 402 and a `crawler-price` header, and the open x402 protocol, used by Cloudflare and AWS WAF, puts payment terms in a `PAYMENT-REQUIRED` header. For a crawler, a 402 is a price: pay through the site's scheme or stop. Stripe, Vercel and npm also send 402 for their own billing problems. ## What does HTTP 402 Payment Required mean? HTTP status codes are the three-digit answers a server sends with every response: 200 means "here is the page", 404 means it does not exist, 403 means you may not have it. Codes from 400 to 499 put the problem on the client's side. 402 sits among them with a one-sentence definition: [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#section-15.5.3), the current HTTP standard from June 2022, says only that the code "is reserved for future use". That sentence goes back to January 1997, when RFC 2068, the first HTTP/1.1 specification, reserved 402 in the same words. A W3C page of status codes from 1994 had imagined more: a 402 listing acceptable "charging schemes" and a `ChargeTo` header the client could send back to pay. That header never reached a standard. Because the standard defines no body, no header and no way to pay, every system that sends a 402 invents its own format. Browsers do nothing special with it: a person who meets one sees whatever page the server sent, as with any other 4xx error. ## Why did 402 sit unused for so long? A status code is useful only if the client can act on it. After a 401 it can log in; after a 402 it would need a way to pay inside a single HTTP exchange, and for most of the web's history there was none. Payments ran through checkout pages built for people, and paywalls answer with a normal 200 page and a sign-up form. Two things changed. A large share of requests now comes from software: AI crawlers collecting pages for training or search, and AI agents fetching a page for a user. They read pages at volumes no person would and never see the ads that pay for them. Meanwhile, signed bot identities began to tell sites who is asking, and payment protocols let one program pay another without a checkout page. ## Where you will see a 402 today Not every 402 is about crawlers. Billing platforms used the code long before pay per crawl, each with its own meaning: | Where | What the 402 means | What to do | |---|---|---| | Cloudflare pay per crawl | The site charges this AI crawler per page | Pay if enrolled, otherwise stop | | x402 (Cloudflare, AWS WAF) | Payment terms in a `PAYMENT-REQUIRED` header | Pay with an x402 client or stop | | Stripe API | "Request Failed": valid request, failed payment | Read the error `code` | | Vercel | `DEPLOYMENT_DISABLED`: payment required | Owner checks usage and plan | | npm publish | Scoped packages are private by default | Use `npm publish --access public` | | Other paid APIs | A billing or quota problem on the account | Check the API's error docs | On Stripe, a declined card is the typical case, and `decline_code` says why the issuer refused. Vercel disables a deployment when the team exceeds its usage limits. On npm, private packages need a paid account, so a new `@scope/name` package published without `--access public` is refused. If a website shows you a 402 page, only its owner can fix the payment problem behind it, often with the hosting plan. ## How does pay per crawl work? Pay per crawl is Cloudflare's system for charging AI crawlers per request. Announced in July 2025, it is still in closed beta according to Cloudflare's documentation. As a reverse proxy in front of the site, Cloudflare answers the crawler before the request reaches the site's server, and as merchant of record it bills the crawler's operator and pays the site. 1. **The site owner sets a price.** Under **AI Crawl Control** › **Payments** in the Cloudflare dashboard, the owner enables pay per crawl, sets a default price for the domain and picks Allow, Charge or Block for each AI crawler. 2. **The crawler proves who it is.** Its operator creates a Cloudflare account, connects billing through Stripe and signs every request with Web Bot Auth: a private key signs parts of the request, the `Signature-Input` and `Signature` headers carry the result, and the public key is published on the operator's domain. Without signatures, anyone could pose as a paying crawler. 3. **A request without payment intent gets a 402** with the price in a `crawler-price` header. 4. **The crawler accepts the price.** It repeats the request with `crawler-exact-price` (this exact price) or `crawler-max-price` (any price up to this ceiling) and includes that header in the signed parts, so nobody can change it on the way. 5. **Cloudflare serves the page** with a `200` and a `crawler-charged` header showing the amount billed. 6. **Anything else is another 402,** this time with a `crawler-error` header that explains the problem. Cloudflare's [guide for crawler operators](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages/) shows the exchange with these values (signature headers left out): ```text HTTP/2 402 crawler-price: USD 0.01 # the crawler repeats the signed request with: crawler-exact-price: USD 0.01 HTTP/2 200 crawler-charged: USD 0.01 ``` Charge also applies to crawlers with no billing relationship with Cloudflare. They cannot pay, so for them it works like a block, but the 402 tells them that paid access exists. Block refuses the request with no price on offer. ## What is x402? x402 is an open payment protocol that turns the 402 response into a machine-readable invoice. Coinbase created it and contributed it to the x402 Foundation, which the Linux Foundation launched on 2 April 2026 with members such as AWS, Cloudflare, Google, Stripe and Visa. The [version 2 specification](https://github.com/x402-foundation/x402/blob/main/specs/x402-specification-v2.md) is published under the Apache-2.0 licence. Over HTTP, a payment takes four steps: 1. The client requests a resource. 2. The server answers `402` with a `PAYMENT-REQUIRED` header, Base64-encoded JSON listing the ways it accepts payment. 3. The client signs a payment authorization with its wallet and sends the request again with a `PAYMENT-SIGNATURE` header. 4. The server checks and settles the payment, usually through a facilitator, a payment service with `/verify` and `/settle` endpoints, and returns the resource with a `PAYMENT-RESPONSE` header. If either step fails, the answer is another 402. Decoded, the payment terms look like this (trimmed from the specification's own example, addresses shortened): ```json { "x402Version": 2, "resource": { "url": "https://api.example.com/premium-data" }, "accepts": [ { "scheme": "exact", "network": "eip155:84532", "amount": "10000", "asset": "0x036C...CF7e", "payTo": "0x2096...287C", "maxTimeoutSeconds": 60 } ] } ``` `scheme` is the pricing model: `exact` for a fixed price, `upto` for a variable price with a ceiling. `network` names the payment network, here Base's test network. `amount` is counted in the token's smallest unit, not in dollars, and `payTo` is the seller's wallet. Version 1, which older clients still send, used `X-PAYMENT` and `X-PAYMENT-RESPONSE` headers instead. The protocol calls itself independent of network, token and currency, and it also runs over MCP and A2A, two protocols AI agents use to call tools and each other. The deployments below settle in USDC, a stablecoin pegged to the US dollar. This describes how the protocol works; it is not advice to buy, hold or use any digital currency. ## Cloudflare and AWS: who sends a 402 today? **AWS WAF.** Since 15 June 2026, AWS WAF can monetize AI traffic on sites behind Amazon CloudFront. A rule with the Monetize action returns a 402 with x402 terms: a price per request in USDC, the accepted networks (Base and Solana), the publisher's wallet, a time limit and a scheme. According to the [AWS documentation](https://docs.aws.amazon.com/waf/latest/developerguide/waf-ai-traffic-monetization-how-it-works.html), payment settles only after the site's server answers with a 2xx, so a failed page costs nothing. Owners can price by identity, for example one price for verified AI search crawlers and another for unverified agents. **Cloudflare.** On 30 September 2026, Cloudflare opened two more betas. The [Monetization Gateway](https://blog.cloudflare.com/monetization-gateway-beta/) charges agents per request, per search query or per token, using x402; payments settle in USDC on Base through Coinbase's x402 Facilitator, and the closed beta admits only eligible US-based sellers and buyers. Pay Per Use sends no 402 at all: a buyer offers a price for a specific use of the content, reports each use, and Cloudflare bills the buyer and pays the publisher. | | Pay per crawl | x402 (Gateway, AWS WAF) | Pay Per Use | |---|---|---|---| | Price signal | `crawler-price` header | `PAYMENT-REQUIRED` header | Offer agreed in advance | | Client needs | Web Bot Auth keys, Stripe | A wallet that signs payments | A buyer agreement | | Money moves through | Cloudflare | USDC via a facilitator | Cloudflare | | Charged for | Each successful crawl | Request, query or token | Each reported use | | Status, 6 Oct 2026 | Closed beta | Closed beta; live on AWS | Beta | ## What a 402 means for AI crawlers and scrapers A 402 is neither an outage nor a block. It is a price, and there are two sound reactions: pay through the site's scheme, if your operator has agreed to, or stop requesting that resource. - **Do not retry it like a 429 or a 503.** Those say "come back later"; a 402 will repeat itself. Log the URL, the price header and the time, then move on. - **Do not try to get around it.** The price is attached to the crawler's identity and the site's rules, not to an IP address. Rotating IPs or dropping the signature turns a known crawler into an unidentified bot, and AWS lets owners charge unverified agents too. Taking priced content without paying also breaks the site's terms. - **Cap the spend.** With pay per crawl, `crawler-max-price` sets a ceiling per request. With x402, give the agent's wallet a budget and a list of approved sites before it signs anything. - **Tell a price from Cloudflare's other answers.** Challenges, blocks and rate limits look different in status and headers; our [Cloudflare scraper guide](/blog/cloudflare-scraper) shows how to tell them apart in code. Before any of that, read robots.txt. Pay per crawl does not replace it: a path robots.txt rules out should not be requested at any price. Our [robots.txt guide](/blog/robots-txt) explains how crawlers apply the rules. Proxies play a narrower role here than many expect. They change the route and the country a request comes from, which matters for public pages that differ by location, such as local prices. A [Residential Proxy](https://proxynet.io/residential-proxy) shows a crawler the page a visitor in that country sees. It does not change who the crawler is to a payment scheme, and it does not make a priced page free. ## What site owners can decide For a site owner, 402 adds Charge to Allow and Block. Allow suits crawlers whose visits bring readers back, such as search engines. Charge earns money only from crawlers that sign their requests and have a billing relationship; for every other crawler it works like Block. Charging needs something in front of the site that can price requests: Cloudflare (pay per crawl or the Monetization Gateway) or AWS WAF with CloudFront. A site on plain hosting can send a 402 from its own code, but it would also need to identify crawlers and collect money. robots.txt and llms.txt keep their roles. robots.txt tells crawlers which paths they may fetch. llms.txt, a Markdown file that gives AI tools a short summary of the site and its key pages, says nothing about access or price; [our llms.txt guide](/blog/what-is-llms-txt) covers what it does. Neither file can charge anyone. ## Limits and open questions - **No standard format.** Cloudflare's `crawler-*` headers and x402's `PAYMENT-*` headers are separate schemes; a crawler must support each one it wants to pay through. - **Narrow availability.** Most of these programs are closed betas, and the Monetization Gateway is limited to the US. - **Payment rails.** x402 deployments settle in a stablecoin today. Wallet, tax and accounting rules differ by country and are a question for finance and legal teams. - **Identity first.** Billing assumes signed requests. The IETF draft for Web Bot Auth only became a working group document on 1 September 2026. ## Common mistakes - **Retrying a 402 in a loop,** as if it were a temporary overload. - **Treating a 402 as a block** and changing IPs or user agents. - **Letting an agent sign x402 payments** without a spending cap or a list of approved sites. - **Charging crawlers that cannot pay** and expecting income: for them, Charge is a block. - **Reading Stripe's 402 as "you must pay":** Stripe calls it "Request Failed", typically a declined card. ## Decision guide | Your situation | What to do | |---|---| | Your crawler gets a 402 with `crawler-price` | Stop, or pay through Cloudflare if enrolled | | Your agent gets a 402 with `PAYMENT-REQUIRED` | Pay only within a set budget and site list; otherwise stop | | Stripe, Vercel or npm returns 402 | Fix the payment, plan or package access (see the FAQ) | | You want AI crawlers to pay for your pages | Compare pay per crawl, the Monetization Gateway and AWS WAF | | You only want AI crawlers kept out | Use robots.txt and blocking; no payment system needed | ## Frequently asked questions ### What does 402 Payment Required mean? The server wants payment before it delivers the resource. The HTTP standard reserves the code without saying how to pay, so each system adds its own headers: `crawler-price` for pay per crawl, `PAYMENT-REQUIRED` for x402. ### Is a 402 an error? It belongs to the 4xx class of client-side problems, and browsers show it like any error page. For a crawler built for paid access, though, it is an ordinary step: price, payment, page. ### How do I fix a 402 Payment Required error? It depends on the sender. On Stripe, read the error code; on Vercel, check your team's usage and plan; on npm, publish scoped packages with `--access public`. If a website shows you a 402, only its owner can fix it. ### What is the difference between 401, 402 and 403? 401 means the server wants you to authenticate, with a login or an API key. 402 means it wants payment. 403 means it understood the request and refuses it, with no price on offer. ### Do I need cryptocurrency to use x402? The specification leaves room for other networks and currencies, but the live deployments, AWS WAF and Cloudflare's Monetization Gateway, settle in USDC, so a paying client needs a wallet that holds it. This describes the systems; it is not a recommendation. ### Can a proxy get around a 402? No. The price is tied to the crawler's identity and the site's rules, not to an IP address. Hiding your identity turns a paying client into an unidentified bot and breaks the site's terms. If you do not want to pay, stop requesting the page. ## Summary HTTP 402 Payment Required was reserved in 1997 and left undefined; pay per crawl, x402, AWS WAF and Cloudflare's Monetization Gateway have now given it a job. For a crawler or an agent, a 402 is a price: pay within a budget, or stop. For a site owner, it adds Charge to Allow and Block, next to robots.txt rather than in place of it. Stripe, Vercel and npm use the same code for their own billing problems, so read the headers first. For collecting public data that carries no price, see our [data scraping](/data-scraping) page.