When a user tells an AI assistant "find me waterproof hiking boots in size 42 that fit my budget and add them to the cart", the assistant tries to do it the way a person would: it searches, opens store sites, compares product pages, picks the size and moves towards checkout. And very often it stops somewhere. Instead of the product page a challenge screen appears, the add-to-cart request is refused, or the payment page flags a suspicious transaction. In the user's eyes the assistant has failed; in the store's eyes a bot has been stopped.
In this article we explain what an AI shopping agent is, why sites block these agents, and the heart of the problem: why it is hard for a site to tell a human, a malicious bot and an agent acting with the user's authorisation apart. Then we cover the solutions emerging for this problem (signed agents, Web Bot Auth and new protocols on the payment side), and the right path for merchants and agent developers. This article does not explain how to get agents past bot protection; it is built around agent traffic identifying itself properly.
What is an AI shopping agent?
An AI shopping agent is an AI system that takes a user's shopping goal and carries it out by acting on websites itself. The general mechanism is the perceive-plan-act-evaluate loop we describe in How Do AI Agents Work?; in a shopping agent, the loop's tools are store websites and payment systems.
What a shopping agent does falls into three levels:
- Research: searching for products, comparing prices and features, reading reviews.
- Preparation: choosing size, colour and quantity, adding to cart, evaluating shipping options.
- Transaction: completing the order using payment details.
Each level carries a different risk for the site. Research looks like what a scraper does. Adding to cart affects the site's stock and session systems. Payment means money and fraud risk. Sites react differently to agents at each level.
Why do sites block agents?
When an e-commerce site stops agent traffic, it is usually not because of a special stance against agents but because defences that have existed for years catch agents too.
Bot protection. E-commerce sites use bot management systems against price scraping, inventory hoarding (adding limited products to carts so others can't buy them), account takeover and card testing attacks. These systems score requests with signals such as speed, IP reputation, browser fingerprint and behaviour. A shopping agent often runs from a data center, with a headless browser, opening pages far faster than a human; in other words, it carries almost every signal bot protection systems look for. For an example of how these signals are evaluated, see Cloudflare Precursor.
Payment fraud risk. Payment systems verify with various signals that a card is being used by its holder: device, location, habits and extra verification steps such as 3-D Secure. When an agent tries to pay with card details, most of these signals don't match the cardholder's normal profile. From the payment system's point of view, the picture resembles automated shopping with a stolen card.
Terms of service. Many e-commerce sites' terms of use restrict automated access, placing orders automatically or collecting site content with automated tools. Under the site's own rules, an agent is an unauthorised automation tool even if it acts on the user's behalf.
Customer relationship and data. For the merchant, the agent is an intermediary that comes between the customer and itself. A large part of the sales experience, such as the product page, recommendations, campaigns and loyalty programmes, is bypassed when the agent shows the user only a summary. Some merchants are cautious about agent traffic for this reason.
Unclear liability. When the agent adds the wrong size to the cart, buys a product the user didn't approve or misreads the price, who handles the return and the dispute? While these questions have no clear answers, merchants and payment companies may prefer blocking to reduce risk.
Why is it hard to tell humans, bots and authorised agents apart?
Historically, a site's bot protection works with two categories: human and bot. A shopping agent is a third category that doesn't fit that split.
| Feature | Human visitor | Malicious bot | Agent acting for a user |
|---|---|---|---|
| On whose behalf does it act? | Itself | An attacker | A real user |
| Traffic pattern | Browser, human speed | Automation, high speed | Automation, high speed |
| IP source | Home or mobile connection | Usually data center or proxy | Usually the agent provider's servers |
| Intent | Shopping | Scraping, hoarding, fraud | Shopping |
| Payment | The cardholder | A stolen or test card | With the cardholder's authority |
| What the site wants | Allow | Block | Allow, but verify |
The problem is that in the table's first four rows the agent looks like a malicious bot, and in the last two it looks like a human. Because bot protection systems can only look at behaviour and network signals, the information they need to tell the agent from a bot, namely "who is behind this automation and what authority does it have", isn't in the HTTP request.
This gap can't be filled with the User-Agent header, because that header is plain text and anyone can write any value in it. For a site to believe that a request saying "KnownShoppingAgent/1.0" really comes from that company's agent, it needs verifiable proof. IP address lists help to a degree, but addresses are shared and change in cloud infrastructure.
That is why the agent hiding itself or trying to look human doesn't solve the problem; it makes it worse. An agent that fakes its browser fingerprint and switches IPs with proxy pools turns into exactly the kind of traffic bot protection systems are designed to block. The solution goes the opposite way: the agent identifying itself in a verifiable way.
Signed agents and Web Bot Auth
The main approach emerging in this direction is for the agent to cryptographically sign every HTTP request. Underneath it is the IETF's HTTP Message Signatures (RFC 9421) standard. The standard defines signing selected components of an HTTP message (method, address, certain headers) with a private key and the recipient verifying the signature with the public key.
Web Bot Auth is the name of the approach that uses this standard to authenticate bots and agents. According to Cloudflare's Web Bot Auth documentation, it works like this:
- The agent operator creates a key pair and publishes the public key as a key directory on its own domain, at
/.well-known/http-message-signatures-directory. - The agent adds three headers to every request:
Signature-Input(the components the signature covers, the key ID, creation and expiry times, and a nonce),Signature(the signature itself) andSignature-Agent(the address of the key directory). - The site, or the CDN in front of it, verifies the signature: it reads the key directory, checks the signature with the public key and makes sure the signature hasn't expired.
- The verified request is attributed to a known agent. Based on this, the site can allow the request, limit its speed or restrict its access to certain paths.
An important feature of this approach is that verification doesn't depend on the IP address: whichever server the agent runs from, the signature points to the same operator. The key directory itself is signed too, which makes it harder for someone else to impersonate the operator with a fake directory.
Since July 1, 2026, Cloudflare has handled agents that identify themselves cryptographically in this way under its verified bots classification. In practice, this means site owners can open a separate path for verified agents in their bot protection rules.
New protocols on the payment side
An agent being able to reach a site is half the problem. The other half is verifying the agent's authority to pay. Several protocols have appeared in this area since 2025. They are less competitors than protocols focused on different steps of the process:
- Visa Trusted Agent Protocol: according to Visa's announcement, the protocol Visa announced in October 2025 is built on the HTTP Message Signatures standard, is aligned with Web Bot Auth and was developed together with Cloudflare. Its goal is to let merchants tell agents recognised by Visa and acting with shopping intent apart from malicious automation.
- Agentic Commerce Protocol (ACP): an open specification maintained by OpenAI and Stripe. According to the project page, it standardises the purchase flow between buyer, agent, merchant and payment provider; the agent shows the checkout interface to the user, while the merchant keeps its own infrastructure and payment processing.
- Agent Payments Protocol (AP2): the protocol announced by Google with its partners aims to carry an agent's authority to pay on the user's behalf with cryptographically signed authorisation documents. That way, the merchant and payment company can verify that the transaction is within the limits the user approved.
Some of these protocols are still in beta, and their scope is changing fast. If you are planning an integration, check the relevant protocol's current documentation.
Who is solving what?
| Party | The problem it has | The solution emerging |
|---|---|---|
| Merchant (e-commerce site) | Can't tell an agent from a malicious bot | Verifying signed agent traffic and managing it with separate rules |
| CDN and bot management service | Automation whose identity can't be verified | Web Bot Auth signature verification, verified bot classification |
| Payment network | The agent's authority on behalf of the cardholder is unknown | Recognised agent protocols, verifiable authorisation documents |
| Payment provider | No standard payment flow between agent and merchant | Open purchase protocols |
| Agent developer | Requests get blocked, transactions get cut off | Signing its requests, using official integrations |
| User | No control over what the agent will buy | Spending limits, approval steps, verifiable authorisation |
What can merchants do?
Blocking agent traffic completely or leaving it completely open may not be right for a merchant. A step-by-step approach is healthier:
- Measure the traffic. See how much of the automated traffic reaching your site is search engines, known AI crawlers and automation of unknown identity.
- Write your policy. Decide which agents can reach which pages (catalogue, cart, checkout). Keeping catalogue pages open and tying checkout to verified protocols is a common starting point.
- Update robots.txt and your terms. State your preferences for agents both in machine-readable form and clearly in your terms of use. We explain how the file is written in What Is a robots.txt File and How Do You Read It?.
- Recognise signed agents. Use the settings your bot management service offers for verified agents; evaluate agent traffic with verified identity separately from traffic of unknown identity.
- Offer structured data. Schema.org data on product pages and official product feeds, if you have them, let agents reach correct information without having to scrape the page.
- Follow your payment partners' agent protocols. If your payment provider supports these protocols, it becomes possible to handle agent-originated transactions in a flow separate from fraud rules.
For jobs such as monitoring agent and bot traffic on your own site and verifying ad and price views from different countries, see the scenarios on our e-commerce proxy solution and ad verification solution pages.
The right path for agent developers
If you are developing a shopping agent, the wrong answer to getting blocked is hiding the agent better. Faking the browser fingerprint, having challenge screens solved or spreading traffic across proxy pools to get around bot protection goes against sites' rules and puts your agent in exactly the class of traffic that should be blocked. We explain how browser fingerprinting works in Browser Fingerprinting, and why challenge screens appear in Puppeteer and CAPTCHA.
The right path goes through these steps:
- Identify your agent. Use a
User-Agentwith a defined product token and a contact address; if possible, sign your requests with Web Bot Auth. - Prioritise official integrations. If the merchant has an API, a product feed or a supported purchase protocol, use those instead of scraping pages.
- Follow robots.txt and the terms. Don't force paths a site has closed to agents, even on a user's behalf.
- Make user approval part of the process. Get explicit approval from the user before irreversible steps such as payment; let the user, not the agent, set spending limits.
- Limit speed. You don't need to open dozens of pages in seconds for one user's purchase.
- Accept failure. If a site blocks your agent, tell the user clearly and offer an alternative; don't try to get around the block.
We explain how to make agents' web access safe with rate limits, allowlists and logging in Safe Web Access for LLMs: Rate Limits and Permissions. In that setup, a proxy is used not to hide the agent but to control and log which address and location agent traffic leaves from.
Common mistakes
- Trying to make the agent look human. A fake fingerprint and IP rotation put the agent in the same class as malicious bots.
- As a merchant, treating all automated traffic as one category. Not separating verified agents from bots of unknown identity can shut legitimate sales channels.
- Treating the User-Agent header as authentication. Anyone can write the header; verification needs a signature.
- Paying without user approval. It triggers fraud rules and creates a serious trust problem with the user.
- Planning an integration without checking protocol status. Most protocols in this area are changing fast.
- Treating a block as a technical error. It is usually a deliberate policy of the site.
Decision guide
| Your situation | Recommendation |
|---|---|
| Your agent only researches products | A User-Agent that identifies you, robots.txt, rate limits; a product API if available |
| Your agent will add to cart and pay | The purchase protocol the merchant supports, user approval |
| Your agent gets blocked often | Sign requests with Web Bot Auth; don't try to get around the block |
| You're a merchant and agent traffic is growing | Measure the traffic, write a policy, manage signed agents with separate rules |
| You're a merchant and agent transactions get declined at checkout | Evaluate your payment provider's agent protocols |
| You want to control your agent traffic's egress | A fixed, logged exit address and an allowlist |
Frequently asked questions
Why do AI shopping agents hit challenge screens?
Bot protection systems evaluate agent traffic with signals such as speed, browser characteristics and IP source. Because agents look like malicious automation on these signals, they meet challenge screens or blocks. As long as the request doesn't carry verifiable information about who is behind it, the site can't tell the two apart.
What is Web Bot Auth?
It is an approach that lets bots and agents prove their identity by cryptographically signing their HTTP requests. It uses the RFC 9421 HTTP Message Signatures standard. The agent publishes its public key on its own domain, and the site or CDN verifies every request's signature with that key.
Does a signed agent work on every site?
No. A signature only proves who the agent is; it doesn't grant access to the site. Each site decides how much to allow signed agents. On sites that don't verify signatures, the signature has no effect.
Can my agent get past a block by using a proxy?
Trying to get around bot protection by changing IPs through a proxy means ignoring a site's explicit preference and puts your agent in the same category as malicious automation. The legitimate use of a proxy here is to control and log agent traffic's egress and to appear from a specific location when needed.
Should merchants block agent traffic completely?
It is a business decision. Blocking completely reduces fraud risk but can lose customers who use agents. For many merchants the balanced route is keeping catalogue pages open, managing verified agents with separate rules and tying checkout to supported protocols.
Are these protocols used in Türkiye?
Most of these protocols are new, and their scope and regional support are changing fast. As a merchant or developer in Türkiye, the most reliable way is to ask your own payment provider and CDN directly which agent verification and payment protocols they support.
Summary
AI shopping agents get blocked on sites because bot protection systems can't tell an agent working for a user apart from malicious automation, and payment systems can't verify that the agent is acting with the cardholder's authority. The solution isn't hiding the agent but having it identify itself verifiably: signed requests with RFC 9421-based Web Bot Auth, recognised agent frameworks such as Visa Trusted Agent Protocol, and purchase and payment authorisation protocols such as ACP and AP2 are moving in that direction. Merchants should measure agent traffic and set a policy, and agent developers should rely on official integrations and user approval. To verify price and content views from different locations within the rules, take a look at our price monitoring solution page.
