---
title: "What Is Web Bot Auth? Signed Identity for Bots"
description: "Web Bot Auth lets bots and AI agents sign their HTTP requests so a site can check who sent them. How it works, its IETF status and who supports it."
url: https://proxynet.io/blog/what-is-web-bot-auth
date: 2026-11-19
author: "Acar Diveroli"
category: "AI, Security"
lang: en
---

# What Is Web Bot Auth? Signed Identity for Bots

Your access log shows 40,000 requests from yesterday that all say `ExampleBot/1.0 (+https://bot.example/about)`. Some come from the ranges the bot's operator publishes, many from a cloud provider's shared range, a few from home connections. Which ones are the real crawler, and which are someone borrowing its name? The `User-Agent` line cannot tell you: any client can type it.

Web Bot Auth is the answer the IETF, the body behind many internet protocols, is now standardising: the bot signs each request with a private key, and your site checks the signature against a public key the operator publishes. This post covers what it is and where the standard stands, why older checks fall short, how a signed request looks, who supports it, what it means for site owners and bot builders, and what a signature does not prove.

> **Note: Short answer**
>
> Web Bot Auth is a way for bots and AI agents to prove who sent an HTTP request. The operator publishes a public key in a key directory at an HTTPS address; the bot signs parts of every request with the matching private key, using HTTP Message Signatures (RFC 9421), and adds three headers: `Signature-Input`, `Signature` and `Signature-Agent`. The site, or the CDN or firewall in front of it, fetches the key and checks the signature, so identity comes from a key instead of an IP address or a `User-Agent` string. As of 19 November 2026 it is an IETF working-group draft, not a finished RFC; Cloudflare and AWS WAF verify it, and Google signs part of its agent traffic experimentally.

## What is Web Bot Auth?

Web Bot Auth is a protocol in which an automated client signs its HTTP requests so that the receiving site can verify which operator sent them. It is built on HTTP Message Signatures, an IETF standard published as RFC 9421 in February 2024 for signing selected parts of an HTTP message. Web Bot Auth adds the bot-specific rules: which parts must be signed, a fixed `tag` value, a header that points to the operator's keys and a format for publishing them.

The name is also that of an IETF working group, `webbotauth`. Its charter covers search crawlers, web archivers, link checkers, AI training crawlers and AI agents acting for a user, on sites that mainly serve people; API authentication between services and human logins are out of scope.

As of 19 November 2026, the group's only adopted document is the Internet-Draft [HTTP Message Signatures for automated traffic](https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/) (`draft-ietf-webbotauth-httpsig-protocol-01`, 29 October 2026), written by authors from Cloudflare and Google and aimed at the Standards Track. It replaces two earlier drafts, one on the architecture and one on the key directory. An Internet-Draft is work in progress, so field names and rules can still change.

## Why are User-Agent strings and IP lists not enough?

The `User-Agent` header and IP allowlists identify a bot by things other clients can copy or share, so neither proves who sent a request. The `User-Agent` is a line of text the client writes itself; we explain how it is built in [What Is a User Agent?](/blog/what-is-user-agent) A well-behaved crawler puts its name and an info URL there, but nothing stops another client from writing the same line.

On our test machine, we started a tiny server on 127.0.0.1 that echoes request headers back, then sent `curl -A "ExampleBot/1.0 (+https://example.com/bot)"` to it. The server received that exact string as the `User-Agent`, with nothing in the request linking it to example.com.

IP lists are stronger, but they break in three common cases. Bots on shared cloud infrastructure leave from addresses other customers also use. AI agents that act for a user may run in that user's browser, so they leave from a home or office address. And operators change their ranges, so last month's list can be wrong today.

The classic check for large crawlers is forward-confirmed reverse DNS: the site looks up the hostname behind the IP, checks that it belongs to the operator's domain, then resolves that hostname and confirms it returns the same IP. It works for operators that own their address space, but not for an agent on a shared cloud address or a user's laptop.

## How does Web Bot Auth work?

Web Bot Auth works as a signing step on the bot's side and a lookup-and-verify step on the site's side:

1. **The operator creates a key pair.** The private key stays with the bot. Ed25519, a modern signature algorithm with short keys, is used in the draft's examples and is the one Cloudflare accepts.
1. **The operator publishes the public key.** It goes into a key directory, a JSON Web Key Set (JWKS) served over HTTPS; the draft also defines a well-known path for it, `/.well-known/http-message-signatures-directory`.
1. **The bot signs each request.** It covers at least the host (`@authority`) or the full target URI, plus the `Signature-Agent` header, adds a creation time, an expiry time, a key ID and the tag `web-bot-auth`, then signs.
1. **The bot sends three extra headers.** `Signature-Input` lists what was signed, `Signature` holds the signature, and `Signature-Agent` gives the address of the key directory.
1. **The site fetches the key.** The verifier, often a CDN or web application firewall, downloads the directory, caches it and picks the key that matches the `keyid`.
1. **The site verifies and decides.** It rebuilds the signed data from the request it received, checks the signature and the time window, then applies its own policy: allow, rate-limit or block.

Because the signature covers the host, a signature made for one site does not verify on another, and the draft recommends that it expire within 24 hours.

## What does a signed request look like?

A signed request is an ordinary HTTP request with three more headers. The example uses made-up, shortened values for a fictional bot at `bot.example` fetching a page from `shop.example`.

```text
GET /products HTTP/1.1
Host: shop.example
User-Agent: ExampleBot/1.0 (+https://bot.example/about)
Signature-Agent: sig1="https://bot.example/.well-known/http-message-signatures-directory"
Signature-Input: sig1=("@authority" "signature-agent";key="sig1");created=1795046400;expires=1795050000;keyid="x7Qp…3kLa";tag="web-bot-auth"
Signature: sig1=:jR3x…Q2Bw==:
```

The label `sig1` ties the three headers together, so one request can carry more than one signature. `created` and `expires` are Unix timestamps, here one hour apart.

| Header | What it carries | Who reads it |
|---|---|---|
| `Signature-Agent` | Address of the key directory | Verifier, to fetch keys |
| `Signature-Input` | Signed parts, times, key ID, tag | Verifier, to rebuild the data |
| `Signature` | The signature itself | Verifier, to check it |
| `User-Agent` | Bot name and info URL | Logs and people |

The `keyid` is the key's JWK thumbprint, a SHA-256 hash of the public key. On our test machine, a throwaway Ed25519 key made in Node.js gave a 43-character key ID and a 64-byte signature, 88 characters in base64. When we changed the signed host from `shop.example` to `shop.example.net`, verification failed.

## How do the old and new checks compare?

Web Bot Auth moves the proof of identity from the network address to a key the operator controls.

| Check | What the site looks at | Where it breaks |
|---|---|---|
| User-Agent | A text header | Anyone can copy it |
| IP allowlist | The source address | Shared cloud, user devices |
| Reverse DNS | Hostname of the IP | Needs own IP space |
| Web Bot Auth | A signature and a key | Sites must verify it |

In practice these checks are layered, not swapped: Google advises keeping IP, reverse DNS and user-agent checks alongside signatures, because not every request is signed yet.

## What does a valid signature prove, and what not?

A valid signature proves that the request was signed with a key listed in the directory at the `Signature-Agent` address, and nothing more. The draft says so plainly: the signature does not tell you who operates the agent, whether it is benign or whether the request is authorised.

- **Identity is not permission.** A verified bot can still be rate-limited, kept out of paths or blocked; robots.txt and a site's terms still apply.
- **The directory owner is the identity.** If a key is stolen, requests signed with it verify until the operator removes it from the directory.
- **Signed requests are linkable.** Sites can correlate requests that carry the same identifier, so the protocol does not suit agents that need to stay anonymous.

Nor does it cover what an agent does with the content it fetches: a verified agent can still be misled by instructions hidden in a page, a separate risk called prompt injection.

## Who supports Web Bot Auth today?

As of 19 November 2026, Cloudflare and AWS WAF verify Web Bot Auth signatures, Google signs part of one agent's traffic, and Shopify asks bots to sign. Details change with the draft, so check each vendor's current page.

- **Cloudflare.** According to its [Web Bot Auth documentation](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/), Cloudflare uses the method to verify bots in its Verified bots programme: operators host a key directory, register it and sign with Ed25519. Since 1 July 2026, signed agents are filed under Verified bots too.
- **AWS WAF.** The [AWS WAF Bot Control rule group](https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-bot.html) checks signatures from version 4.0 and labels each request `verified`, `invalid`, `expired` or `unknown_bot`, so site owners can write rules on the result.
- **Google.** Google's [Web Bot Auth page](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth) calls its implementation experimental: a subset of `Google-Agent` requests is signed as `https://agent.bot.goog`.
- **Shopify.** A May 2026 developer changelog asks bots and agents on Shopify storefronts to sign with Web Bot Auth; unsigned traffic gets the strictest rate limits.

## What does Web Bot Auth mean for site owners?

For site owners, Web Bot Auth turns "is this really that bot?" from a guess into a check, and most of the work happens in the CDN or firewall, not in your application.

A sensible starting policy: let the verified crawlers you want through at a normal rate, and give unsigned automation a lower rate limit rather than a blanket block, because most bots do not sign yet. Watch `invalid` and `expired` results, which point to broken setups or copied headers. If you verify signatures yourself, cache key directories, do not follow redirects when fetching them, and answer unsigned requests to protected paths with a 403 and an `Accept-Signature` header, as the draft recommends.

Signed identity also makes your published preferences easier to enforce. robots.txt states which paths each bot may crawl, as covered in [What Is a robots.txt File?](/blog/robots-txt), and Web Bot Auth lets you check that a bot claiming to follow those rules is the bot it says it is.

## What does it mean for bot and agent builders?

For crawler and agent operators, Web Bot Auth is a way to be recognised without every site keeping an IP list for you, at the cost of managing keys.

- **Use separate identities.** The draft says a crawler and an agent acting for a user should use distinct `Signature-Agent` URLs, keys and directories.
- **Keep signatures short-lived.** Set `expires` close to `created`.
- **Rotate keys.** Publish the new key before you use it, then remove the old one.
- **Protect the private key.** Store it like any credential; shared secret keys (HMAC) are not allowed.
- **Keep your manners.** A clear `User-Agent`, robots.txt and polite request rates still matter.

For agents that shop or pay for a user, signing is only half the story; the payment side is in [Why Are AI Shopping Agents Blocked on Websites?](/blog/ai-shopping-agents-blocked)

## Does a proxy change a bot's identity under Web Bot Auth?

No. With Web Bot Auth, a bot's identity lives in its key, not in the IP address its traffic leaves from, so a proxy decides where a request exits but not who the bot is. The draft states that the signature does not depend on the IP and that a proxy should not strip the signature headers. With HTTPS, a proxy only tunnels the encrypted connection, so the signed headers arrive as the bot sent them.

That split is useful for crawler teams: the signature answers "who is this?", while the exit location answers "what does this page show visitors in Germany or Brazil?", for example when checking localised prices. Our [web crawler proxy](/web-crawler) page describes geo-targeting and adjustable wait times between requests for this kind of polite crawling.

When a crawler needs a fixed, known exit address, for example to match a firewall rule on your own sites, a [Datacenter Proxy](https://proxynet.io/datacenter-proxy) gives dedicated IPv4 addresses. Either way, a signed request through a proxy is still subject to the site's own policy.

## Where will you meet Web Bot Auth?

- **CDN and firewall dashboards:** verified bot categories, signature results and rules built on them.
- **Search and AI crawlers:** operators that publish key directories next to their crawler documentation.
- **AI agents acting for users:** browsing and shopping agents that sign some or all of their requests.
- **Store platforms:** storefronts that give signed bots higher rate limits.
- **Monitoring and link checking:** uptime monitors that want to be told apart from attacks.

## Common mistakes

- **Calling it an RFC.** Web Bot Auth is a working-group draft built on RFC 9421; it can still change.
- **Treating a valid signature as permission.** It proves the key, not good intent or a right to crawl.
- **Blocking all unsigned bots.** Most automated traffic is not signed yet; rate limits are a gentler first step.
- **Dropping the old checks.** Google itself advises keeping IP, reverse DNS and user-agent checks during the rollout.
- **Long-lived signatures.** A signature valid for days gives copied headers a longer life.
- **One key for everything.** A crawler and a user-directed agent should not share an identity.

## Decision guide

| Your situation | What to do |
|---|---|
| You run a site behind a CDN or WAF | Turn on its signature check, write rules on the result |
| Fake crawler traffic fills your logs | Allow verified bots, rate-limit the unverified |
| You operate a crawler | Publish a key directory, sign every request |
| You run an AI agent for users | Use a separate key and directory from your crawler |
| You verify signatures yourself | Cache directories, no redirects, short time windows |
| You need a fixed or local exit IP | Use a proxy; the signature stays the same |

## Frequently asked questions

### Is Web Bot Auth an official standard?

Not yet. It is an Internet-Draft of the IETF Web Bot Auth working group, aimed at the Standards Track. The signing method underneath it, HTTP Message Signatures, is already a standard: RFC 9421.

### Is Web Bot Auth the same as Cloudflare verified bots?

No. Web Bot Auth is an open protocol that any site, CDN or firewall can verify. Cloudflare's Verified bots programme is one place that uses it, next to other checks such as published IP lists and reverse DNS.

### Does AWS WAF support Web Bot Auth?

Yes. AWS WAF Bot Control verifies signatures from rule group version 4.0 onward and adds a label with the result, such as `verified` or `unknown_bot`, that your own rules can act on.

### Do I need Cloudflare to sign requests with Web Bot Auth?

No. You need a key pair, a key directory on an HTTPS address you control and signing code in your bot. Shopify, for example, says bots do not have to enrol with Cloudflare to sign requests for its storefronts.

### Does Web Bot Auth replace robots.txt?

No. robots.txt tells bots which paths they may crawl; Web Bot Auth tells the site which bot is asking. The two work together.

### Can someone copy a signed request and reuse it?

Only within narrow limits. The signature covers the host, so it fails on another site, and it carries an expiry time. Sites that need stricter protection can ask for a `nonce`, a one-time value, and refuse signatures they have already seen.

## Summary

Web Bot Auth gives bots and AI agents a verifiable identity: the operator publishes a public key, the bot signs each request with HTTP Message Signatures, and the site checks the signature instead of trusting a `User-Agent` string or an IP list. It is still an IETF draft, already verified by Cloudflare and AWS WAF and tested by Google. A signature proves who is asking, not what they may do, so robots.txt, rate limits and site policies still decide. Where your crawler's requests exit is a separate question, and our [proxy services](/proxy) cover that side.
