---
title: "How Bot Detection Works: The Logic of Anti-Bot Systems"
description: "Bot detection is not a single check: IP reputation, TLS fingerprint, headers, browser signals and behaviour are measured separately, then merged into one score."
url: https://proxynet.io/blog/how-bot-detection-works
date: 2026-09-19
author: "Acar Diveroli"
category: "Web Scraping"
lang: en
---

# How Bot Detection Works: The Logic of Anti-Bot Systems

The manager of an online shop looks at the morning report: at three in the morning visitor numbers quadrupled, the number of add-to-cart events did not move, and the login page shows thousands of failed attempts. How much of that traffic is a search engine, how much a price comparison service, how much a script trying stolen passwords? Bot detection is the site owner's attempt to answer that question for every single request, within milliseconds.

This article describes bot detection from the site owner's side: which layers a request passes through, what each layer measures, how the signals turn into a single score, and who pays when that score is wrong. We summarise six layers in order and leave the detail to each layer's own article. How headless browsers are recognised, how good bots prove themselves, and what the legitimate route looks like for someone running automation are all here. Ways to get around protections are not the subject of this article.

> **Note: Short answer**
>
> Bot detection works in layers. Before a connection is established, the IP address's reputation and the network (ASN) it belongs to are checked. During the connection, the TLS handshake, the HTTP/2 settings and the consistency of the headers are read. Once the page opens, JavaScript probes the browser environment, and throughout the session the speed and navigation pattern are watched. No signal decides on its own: they all merge into one score, and the site owner uses that score to allow, rate limit, challenge or block.

## What is bot detection?

Bot detection is the work of telling whether a request arriving at a site came from a browser a person is using or from an automated program. The software that does this is called an anti-bot system, bot protection or bot management; most of it runs not in the site's own code but in the CDN or web application firewall (WAF) in front of it. Cloudflare, Akamai and DataDome are frequently named providers in this field. The system looks for answers to two questions: is the request automated, and if so, is it the kind of automation the site wants?

## What is bot traffic: how are good bots told apart from bad ones?

Bot traffic is every request produced by a program's decision rather than a person's click. Three kinds sit side by side in the logs:

- **Bots the site wants.** Search engine crawlers, uptime monitoring services, messaging apps generating link previews, the site's own test automation.
- **Bots the site weighs up.** Price comparison services, archiving tools, AI crawlers, research scripts. Some sites open the door, some accept them with a rate limit.
- **Automation the site does not want.** [OWASP's automated threats project](https://owasp.org/www-project-automated-threats-to-web-applications/) catalogues this group by naming it: bulk testing of stolen passwords (credential stuffing), validating stolen card numbers (carding), grabbing limited stock with automation (scalping) and copying content in bulk (scraping) are all on that list.

The difficulty is that all three look alike on the wire. That is why the system does not look at a single sign but at the sum of the layers.

## In what order is a request evaluated?

1. **The connection arrives.** The source IP is known before any content is read; its reputation, country and network are looked up.
2. **The TLS handshake happens.** The first message (ClientHello) carries a clue about which software is speaking.
3. **The HTTP connection is established.** In HTTP/2 the client announces its connection settings; then come the method, the address and the headers.
4. **The server-side score is calculated.** Signals from the first three steps combine with the address's recent history; the request can already be rejected here.
5. **The page reaches the browser.** A small script on the page probes the browser environment and reports the result back.
6. **The session is watched.** Request frequency, navigation order and interaction update the score throughout the session.
7. **If the threshold is crossed, an action follows.** A rate limit, a challenge screen or a block.

The first four steps happen server-side without the visitor noticing anything; the fifth and sixth produce data only on clients that run JavaScript.

## The layers in one table: which layer measures what?

| Layer | What it measures | Its strength | False positive risk |
|---|---|---|---|
| Network | IP reputation, ASN, country, past complaints | Works cheaply, before any content is read | Shared IPs: CGNAT, corporate networks, VPNs |
| Protocol | TLS ClientHello, HTTP/2 settings | Shows the client software independently of headers | Older devices, networks doing corporate TLS inspection |
| Header | `User-Agent`, Client Hints, header set and consistency | Compares the declaration against reality | Privacy extensions, little-known browsers |
| Browser | JavaScript environment, automation flags, fingerprint | Separates a real browser from an imitation | Script blockers, accessibility tools |
| Behaviour | Request rate, navigation order, interaction | Looks at the whole session, not a single request | Very fast real users, keyboard-only navigation |
| Challenge | Visible or invisible test | Gives suspicious traffic a second chance | Every test creates friction for real customers |

The last column carries the point of this article: every layer has a group of humans it gets wrong, and thresholds are tuned in full knowledge of that cost.

## The network layer: what do IP reputation and ASN say?

The IP address is the earliest piece of information about a request. The system looks at the address's abuse history, its country and, through the ASN (autonomous system number), which network it is registered to. A request from a hosting company's network and a request from a block an ISP has set aside for home subscribers are not treated the same: home users do not reach the internet from a data centre. How sites classify an address as "ISP" or "hosting" is covered in [ISP Proxy vs Residential Proxy](/blog/isp-vs-residential-proxy).

Reputation comes from two sources: blacklists that record whether the address has been reported as a source of spam or attacks ([IP Blacklists](/blog/ip-blacklist)), and risk score services that reduce the address type, and whether it is a proxy or VPN exit, to a single number ([IP Fraud Score](/blog/ip-fraud-score)). The layer's weakness is that it measures the address, not the person. Mobile operators and many ISPs put large numbers of subscribers behind the same address ([CGNAT](/blog/what-is-cgnat)); a script running from that address drags down the reputation of everyone sharing it.

## The protocol layer: TLS fingerprint and HTTP/2 settings

In the first message of an HTTPS connection, the client openly sends the cipher suites it supports, the extensions and their order. The list differs from browser to browser and from library to library; when the server condenses it into a short summary (JA3 and JA4 are common formats), it has a marker that is independent of the headers. A request whose `User-Agent` header says Chrome but whose handshake resembles a Python library stands out through that inconsistency. How the calculation works and where it falls short is covered in [TLS Fingerprinting and JA3](/blog/tls-fingerprinting).

[RFC 9113](https://www.rfc-editor.org/rfc/rfc9113#name-settings) requires both parties to send a `SETTINGS` frame at the start of an HTTP/2 connection. The values in that frame, and the order in which the pseudo-headers (`:method`, `:authority`, `:scheme`, `:path`) are sent, differ in every HTTP stack. The layer's value is that it is independent of any declaration; its risk is that employees behind corporate security appliances, which decrypt and re-establish TLS traffic, arrive with a handshake that does not match their browser.

## The header layer: User-Agent and consistency

Headers are the client's declaration about itself: `User-Agent` states the browser and the operating system, the Client Hints headers (the `Sec-CH-UA` family) repeat the same information in a structured form, and `Accept-Language` announces the language preference. Nobody verifies this declaration; any program can write whatever value it likes. How the string is read is covered in [What Is a User-Agent?](/blog/what-is-user-agent).

That is why an anti-bot system looks not at the declaration itself but at how it fits the other layers. Does the declared browser version really send that header set in that order? Does the language preference match the country of the IP, and the time zone match what the browser reports? No single mismatch is proof on its own (someone living abroad connects from another country with a Turkish-language browser), but each one pulls the score down a little.

## The browser layer: JavaScript signals

When the page opens, the protection gains access to something it could not see server-side: the browser itself. A script added to the page reads the screen dimensions, the installed fonts, the drawing output of the graphics hardware, the time zone and the supported APIs; the combination of these values is the browser fingerprint. The fingerprint serves two purposes: recognising the same client even when the IP changes, and testing whether the declared environment is real. An environment that claims to be Chrome on Windows but reports a graphics driver found only on Linux servers fails that test. The full list of signals is in [Browser Fingerprinting](/blog/browser-fingerprinting).

Clients that never run the script (HTTP libraries such as cURL or Python Requests) produce no data in this layer. On an API endpoint that is expected; on an HTML page, the complete absence of a script result is enough for the client not to be counted as a browser.

## What is a headless browser and how do sites recognise it?

A headless browser is a real browser that runs without opening a window on screen: it loads the page, runs the JavaScript and hands the result to your code. It is used in end-to-end testing, PDF generation, site monitoring and permitted data collection; the common tools are Playwright, Puppeteer and Selenium. When you need one is covered in [Static and Dynamic Pages](/blog/static-vs-dynamic-pages), and the difference between the two tools in [Playwright vs Selenium](/blog/playwright-vs-selenium).

Sites recognise a browser driven by automation from three marks:

| Signal | Where it comes from | How the site owner reads it |
|---|---|---|
| `navigator.webdriver` being `true` | [The W3C WebDriver standard](https://www.w3.org/TR/webdriver2/): the flag is set when the browser is remotely controlled | The browser's own declaration; test automation carries the same flag |
| Environment differences | Side effects of running without a window: window dimensions, plugin list, some APIs | Weak on its own, meaningful alongside other signals |
| Interaction style | Events produced by code rather than by a human hand | Handed over to the behaviour layer |

The flag in the first row is not a detection trick but part of the standard: the W3C text defines it as the standard way for the browser to tell the document that it is being controlled by WebDriver.

The weight of the second row has fallen over time. According to [Chrome's headless documentation](https://developer.chrome.com/docs/chromium/headless), the old headless mode was a separate application that did not share the browser's code; since Chrome 132 it is offered only as a separate binary named `chrome-headless-shell`, while `--headless` runs the real Chrome code. As the environment differences shrank, the weight shifted to behaviour.

The lesson for a site owner is this: detecting a headless browser is not the same as detecting bad intent. Your QA team and your monitoring service carry the same marks; separating that traffic with a known IP or a signed-request rule does less damage than looking at the flag and blocking wholesale.

## The behaviour layer: rate and navigation pattern

The earlier layers look at who the request came from; the behaviour layer looks at what it does. The simplest measure is rate: the number of requests arriving in a given period from the same address, session or account. When the limit is crossed, the server returns `429 Too Many Requests`. What the counters are kept against, and the fixed window, sliding window and token bucket algorithms, are covered in [429 Too Many Requests](/blog/http-429-too-many-requests).

Beyond rate there is the navigation pattern. A person moves from the home page to a category and from there to a product, and their browser also downloads the images and the style sheets; a session that visits only product addresses at steady intervals, requesting no side files at all, looks different. Requests going to hidden links a visitor could never see are also a signal in this layer ([Honeypot Traps](/blog/honeypot-traps)). Newer systems watch in-page interaction (cursor movement, whether the tab is visible) throughout the session; we examined a current example in [Cloudflare Precursor](/blog/cloudflare-precursor).

This layer says little about a single request and a great deal about a hundred. Its cost is delay: data has to accumulate before a decision can be made.

## How signals become a decision: scoring and challenges

None of the layers says "this is a bot" on its own. Their outputs merge into one score. [Cloudflare's bot score documentation](https://developers.cloudflare.com/bots/concepts/bot-score/) is a clear example of this approach: every request is given a score between 1 and 99, where 1 means there is high confidence that the request was automated and 99 that it came from a human; anything below 30 counts as "likely automated". According to the documentation, the score is fed by heuristics, machine learning, anomaly detection and JavaScript detections.

A score is not an action. The action is set by the rule the site owner writes, and the options form a ladder:

1. **Allow.** The score is high, or the request comes from a verified bot.
2. **Rate limit.** The request is accepted, its frequency is capped.
3. **Challenge.** An invisible test is given to the browser or a checkbox to the visitor; what that screen means on the visitor's side is covered in [Cloudflare "Verify You Are Human"](/blog/cloudflare-verify-you-are-human).
4. **Block.** The request is refused with a `403` or a block page.

The ladder is tuned per endpoint: on a blog page the cost of allowing a low-scoring request is a few kilobytes of traffic, on a login page it is a compromised account.

## The cost of a false positive: what happens when a real visitor is taken for a bot?

Bot detection has two kinds of error. Taking a bot for a human (a false negative) comes back as load, fake accounts or stolen content. Taking a human for a bot (a false positive) is lost revenue: the customer who gives up at the challenge screen, the payment that is never completed. The second error does not show up in reports, because a blocked visitor never reaches the analytics tool at all.

Its main sources are these:

- **Shared IP addresses.** Mobile operator subscribers and users on dorm, café and corporate networks pay the price of other people's behaviour.
- **VPNs and privacy tools.** Because the address looks like it belongs to a data centre, the network layer lowers the score ([VPN or Proxy Detected](/blog/vpn-or-proxy-detected)).
- **Script blockers, accessibility tools, keyboard navigation.** The browser layer gets no data and the behaviour layer does not find the pattern it expects.

On the visitor's side this appears as messages like "detected as a bot" or "unusual traffic detected": we covered two examples in [Google's "Unusual Traffic" Error](/blog/google-unusual-traffic-error) and ["Sorry, You Have Been Blocked"](/blog/sorry-you-have-been-blocked).

The practical conclusion: challenge before you block, and watch the share of requests that pass. Every request that passes is a record of your rule stopping the wrong person.

## Verified bots: how does a site recognise a good bot?

Writing "Googlebot" into the `User-Agent` header is a one-line job; that is why a good bot is recognised by proof, not by declaration. Three methods are in use today.

**Reverse DNS verification.** [Google's Googlebot verification documentation](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot) gives the method: a reverse DNS lookup is done for the IP in the log, the returned name is checked to be under `googlebot.com`, `google.com` or `googleusercontent.com`, and then that name is resolved back to an IP and confirmed to match the first address.

**Verified bot lists.** Bot management services do this check on the site owner's behalf; [Cloudflare's verified bots documentation](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) lists two conditions: the bot identifies itself honestly (through a cryptographic signature, a published IP list or reverse DNS) and does not abuse that trust, meaning it obeys robots.txt and keeps a reasonable request rate. Ignoring the `crawl-delay` directive, or producing traffic that does not match the declared purpose, is grounds for removal from the list.

**Signed requests (Web Bot Auth).** In the newer approach the bot signs every request with its private key, publishes its public key on its own domain, and the site verifies the signature. The underlying standard is [RFC 9421 HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421), and an IETF [Web Bot Auth working group](https://datatracker.ietf.org/wg/webbotauth/about/) has been set up to adapt it to bots. The group considers today's solutions, such as IP lists, `User-Agent` strings and shared API keys, inadequate; the identity of an agent making requests on a user's behalf is in scope, while the identity of the end user behind the agent is out of scope. We described the signature headers in [Why AI Shopping Agents Get Blocked on Websites](/blog/ai-shopping-agents-blocked).

In all three, identity is established by proof (a DNS record, IP ownership, a private key) rather than by the client's declaration.

## If you run automation, what is the legitimate route?

The map points to the same conclusion for someone running automation: the system is built to make way for traffic that identifies itself and behaves in moderation.

- **Use the official API if there is one.** Access rests on an agreement and bot detection does not come into play.
- **Obey robots.txt.** Its format and how to check it in Python are covered in [What Is the robots.txt File?](/blog/robots-txt).
- **Cap your rate per site.** Respect `429` and `Retry-After` responses.
- **Give your bot a name.** A bot name and a contact address in the `User-Agent` give the administrator the option of writing to you instead of blocking you.
- **Ask permission for high-volume access.** If you run a crawler that serves the public, apply to verified bot programmes.
- **Stop when you see a challenge screen.** That screen is the site's clear preference; we do not recommend CAPTCHA-solving services.

The implementation of these steps (concurrency, conditional requests, the difference between rotation and sticky sessions) sits with tested code in [How to Collect Data Without Getting Blocked](/blog/web-scraping-without-getting-blocked).

## Use cases: who needs this map, and what for?

- **Marketing and analytics teams:** to separate the source of traffic spikes and protect the ad budget from fake clicks. The advertising side is covered in [Google Ads Click Fraud](/blog/google-ads-click-fraud) and on our [ad verification solution](/ad-verification) page.
- **Data teams:** to understand which behaviour looks suspicious and why when collecting data from publicly available pages; the general setup is on our [data scraping solution](/data-scraping) page, and for content that varies by location a [Residential Proxy](https://proxynet.io/residential-proxy) is used.
- **QA and test teams:** your own site's bot protection can stop your own test automation. Route test traffic out of a fixed address and write an allow rule for that address; a [ISP Proxy](https://proxynet.io/static-isp-residential-proxy) or your own static IP is used for this ([app testing solution](/app-testing)).
- **Brand protection teams:** scans for fake shops and counterfeit listings are automation too and pass through the same layers; the scope is on our [brand protection solution](/brand-protection) page.

## Common mistakes

- **Blocking on a single signal.** A rule that looks only at a data centre ASN or the `navigator.webdriver` flag also stops monitoring services and your own tests.
- **Trusting the `User-Agent` header and opening the door to "Googlebot".** A declaration is not proof; use reverse DNS or a verified bot list.
- **Applying the same threshold to every endpoint.** A login page and a blog post do not carry the same risk.
- **Not measuring false positives.** If the share of requests that pass the challenge is not tracked, you do not know who your rule is stopping.
- **On the automation side: treating a block as a technical puzzle.** A challenge screen is not an error but the site owner's answer; review your rate, your scope and your permission status.

## Decision guide

| Your situation | Recommendation |
|---|---|
| You have unexplained traffic on your site | Break the logs down by ASN and endpoint; rate limit first, challenge second |
| You see bulk password attempts on the login page | A low threshold and a challenge on that endpoint; a per-account attempt limit |
| You worry about blocking search engine bots by mistake | A verified bot list or reverse DNS verification |
| Customers complain about the challenge screen | Look at the share that passes the challenge and loosen the threshold |
| Your own test automation is being blocked | Route the tests out of a fixed address and write an allow rule for it |
| You collect public data regularly | The API first; failing that, robots.txt, a low rate, a named bot identity |
| You run a crawler or agent that serves the public | A verified bot programme, signed requests |
| You see a "bot detected" message as a visitor | Try with the VPN and extensions switched off; if it persists, your network exits through a shared address |

## Frequently asked questions

### What is anti-bot, and what does it do?

Anti-bot is the software layer that classifies automated traffic arriving at a site and, according to the site owner's rules, allows it, slows it down, challenges it or blocks it. Its purpose is not to stop every bot but to separate harmful automation from wanted bots such as search engines.

### What does "detected as a bot" mean?

It means the site's protection layer gave your request a low score; this does not mean you were using a program. The most common causes are a VPN or a shared IP address, script-blocking extensions and opening many pages in a short time. The first step is to switch off the VPN and the extensions and reload the page.

### Why do Selenium and Playwright get caught by bot detection?

These tools drive the browser through WebDriver or a similar protocol, and the standard requires the browser to announce that with the `navigator.webdriver` flag. Environment differences and a pace unlike human interaction are added on top. If you are testing your own site, the solution is to write an allow rule for the test traffic on the protection side.

### Does bot detection only look at the IP address?

No. The IP is the earliest signal, not the only one: the TLS handshake, the consistency of the headers, the browser environment and behaviour throughout the session are each evaluated separately. That is why changing only the IP does not affect the other layers.

### How do I spot bot traffic in Google Analytics?

A sudden spike from a single city or network, an engagement time close to zero, and sessions with no relation to conversions are typical marks. An analytics tool is not enough for a definite separation; examine the same time window in your server or CDN logs, broken down by IP, ASN and `User-Agent`.

### Where do you start with protecting a small site from bots?

Start with three steps: put rate limits on expensive endpoints such as login, registration and search, turn on your CDN's basic bot protection in challenge mode rather than block mode, and check that search engine bots are passing through the verified list.

## Summary

Bot detection is not a wall but a series of measurements set one after another: the network layer reads the address's reputation and ASN, the protocol layer the TLS and HTTP/2 trace, the header layer the consistency of the declaration, the browser layer the reality of the environment, the behaviour layer the navigation pattern. They all merge into one score, and the site owner chooses allow, rate limit, challenge or block according to the endpoint's risk. Because every layer has real users it gets wrong, a well-built system challenges first and blocks second. Good bots prove their identity with reverse DNS, verified bot lists and RFC 9421 based signatures; for anyone running automation, the lasting route is the official API, robots.txt, a moderate rate and an open identity. For data collection work that follows the rules, you can find the proxy types in our [proxy services](/proxy).