What Is Agentic Web Scraping and How Does It Work?

Suppose you have two hundred different small sites and need to collect the same fields from all of them — product name, price, stock status. Writing a script per site means hundreds of selectors; feeding them all into a single scraping API means empty records for most of them, because their structure doesn't fit. Give the same job to an AI agent and the picture changes: the agent reads every page itself, decides where the data is by looking at that page, and if needed follows a link and fixes its plan halfway through.

That, in one sentence, is agentic web scraping (agent-driven scraping). In this article we explain what agent-driven scraping is, which steps the agent loop goes through in a scraping job, the building blocks of the system, a working skeleton in code, how cost and reliability compare to rule-based scripts, and the agent's typical failure modes. The other side of the "who chooses the route" question — the fixed pipeline where the model works only at the parsing step — is covered in our AI web scraper article.

What is agentic web scraping?#

There are three levels of collecting data from a page, and to use the term correctly you need to keep the three apart. At the first level everything is in the developer's hands: the script knows in code which address to visit and which element to take data from. At the second level the parsing work is delegated to the model, but the route is still fixed: the page arrives, the model extracts the fields, the flow ends — that was the subject of our AI web scraper article. At the third level the model enters the loop: which page to visit, which button to press, when to stop — these are decided at run time. This third level is agentic web scraping.

To put it in terms of the distinction in Anthropic's Building effective agents article: flows whose steps are drawn by code are workflows, and flows where the model manages its own process are agents. Agent-driven scraping is that definition applied to data collection. To compress the difference into one sentence: in an AI scraper the answer to "who parses the page?" is the model; in agentic scraping the answer to "who chooses the route?" also passes to the model.

How does the agent loop work in scraping?#

The agent's general way of working — perceive, plan, act, evaluate — and the theoretical basis of this loop were the subject of our How AI Agents Work article; instead of repeating it, let's look at its scraping-specific form. Each round runs through five steps:

1. Observe. What the agent "sees" is not the pixels of the browser window but the page's structural summary: the accessibility tree, headings, link lists, form fields. This summary enters the model's context window; the raw HTML does not enter in full.
2. Plan. Looking at the goal — "extract the price of every product in this list" — the model picks the next action: click the filter button, move to the second page of the pagination, or decide that the data is now visible.
3. Act. The action the model picks is a tool call: git(url), tıkla(betimleme), doldur(alan, değer). It is the browser that actually executes the call; the model only writes the instruction.
4. Extract. When the target data is visible, the model fills in the fields according to the schema you defined. This step is the parsing step of the fixed pipeline, embedded inside the agent.
5. Verify and repeat or finish. The output's types and required fields are checked in code; if something is missing, the model sees which information is missing and returns to the relevant page. A stop condition such as a step budget, a cost limit or a page count keeps the loop from running forever.

The fourth and fifth steps show that agent-driven scraping works with two separate disciplines at the same time: not trusting the model's decisions and output adds two layers of work at once — a validation layer and a stopping layer. Without both, the system looks like it "works" while quietly producing wrong data.

Building blocks#

• Browser tool. The agent's hands. An automation library such as Playwright provides the accessibility tree the model can see and the actions it can use; there is also a ready-made component that does this job as an MCP server. We covered its setup and proxy flags in our Playwright MCP article; we leave the protocol itself to our What Is MCP? article.
• Schema and validation. The JSON schema of the requested data binds the model's output to a contract. Type, required and range checks stay in code; suspicious records go to a separate queue instead of being written to the main store.
• Checkpoint and memory. Long jobs can be interrupted; the addresses the agent visited, the pages it completed and the records it collected are kept outside. The context window is short-term memory; the memory of a two-hundred-site job doesn't fit in the model's window, but it fits in a file.
• Security boundaries. The domains the agent may open, the number of steps it may take and the tools it may call are restricted in advance; the content of the page it scrapes enters the model as data, not as instructions. We built this whole layer, including the allowlist, the rate limit and injection sanitisation, in our Safe Web Access for LLMs article.

A small agent skeleton#

The Python sketch below shows the provider-independent skeleton of the loop above. The browser side is Playwright; the model_cagir function is filled in with the client of the provider you use, and the model is expected to return either an action or output that matches the schema.

import json
from playwright.sync_api import sync_playwright

PROXY = {"server": "http://pr.proxynet.io:8000",
         "username": "kullanici", "password": "parola"}
HEDEF = "https://example.com/urun-listesi"
SEMA = {"urun_adi": str, "fiyat": float, "stokta": bool}

def gozlemle(sayfa):
    # The agent's eye: not raw HTML, the page's structural summary.
    return sayfa.locator("body").aria_snapshot()

def model_cagir(gozlem, talimat):
    # Fill in with your provider's official client. gozlem + talimat is sent
    # to the model; the reply is either {"arac": ..., ...} or schema-valid data.
    raise NotImplementedError("the model call goes here")

def dogrula(kayit):
    for alan, tur in SEMA.items():
        if not isinstance(kayit.get(alan), tur):
            raise ValueError(f"{alan} is not of the expected type")
    return kayit

with sync_playwright() as p:
    tarayici = p.chromium.launch(proxy=PROXY)
    sayfa = tarayici.new_page()
    sayfa.goto(HEDEF)

    for adim in range(8):                       # stop condition: step budget
        karar = json.loads(model_cagir(gozlemle(sayfa),
                                       "Extract the products in the list."))
        if karar.get("arac") == "cikar":
            print(dogrula(karar["kayit"]))      # never trust the model directly
            break
        if karar["arac"] == "git":
            sayfa.goto(karar["url"])
        elif karar["arac"] == "tikla":
            sayfa.get_by_role(karar["rol"], name=karar["ad"]).click()

Three properties of the skeleton summarise the rest of the article: what the model sees each round is not raw HTML but a structural summary; the loop has a stop condition; the model's output is written nowhere before it passes through dogrula. We gave all of Playwright's proxy options in table form in our Playwright proxy article.

Side by side with rule-based scraping#

Putting the three levels in the same table clarifies which job belongs to which level:

The cost formula is simple: the agent's total fee is the number of turns multiplied by the tokens sent and received per turn. In a rule-based script this number is close to zero; in an AI scraper the number of turns is one (a single parsing call); in agent jobs the number of turns varies with how hard the page is — most pages finish in two turns, while a job that goes through filter menus can take six to eight turns. That's why the agent's budget is tied to a step budget rather than to the job itself: a system built without a ceiling like for adim in range(8) in the loop can spin for hours on a broken page. We covered how the model side's unit costs are calculated in our Web Scraping with GPT-6 Astra article.

The agent's failure modes#

The failures of agent-driven systems differ from a script's; a script breaks and stops, an agent can drift without breaking. The main modes and where they show up:

• Getting stuck in a loop. The agent moves back and forth between the same two pages or clicks the same button repeatedly. Its appearance is a swelling turn count and the same action sequence repeating in the logs; the cure is to keep the visited addresses and actions and show the model this list every turn.
• Goal drift. The model turns the "extract prices" goal into a "browse the site" goal; no records arrive but the turn is spent. The cure is to repeat the goal in every turn's context and to cap the number of empty turns.
• Made-up fills. When the data isn't visible, the model fills the fields with probability. The only cure is the validation layer; the suspicious-record queue is the only place that catches this failure mode.
• Cost blow-up. The combined result of looping and drift: a job written without a step ceiling and a budget ceiling can spend the day's budget in a single run. Both ceilings live in code, not in the model's conscience.
• Silent degradation. The sneakiest mode: the agent returns "successful" with few records. If thirty of two hundred sites come back empty and nobody looked, the error hides not in the log but in the report itself. The cure is a below-expectation alert: a job that falls under the expected record count is flagged separately.

What changed on the blocking side?#

To bot protection, the agent looks the same as every scraper that uses a headless browser: IP reputation, request rate and browser signals are measured the same way; the model's intelligence is invisible to those measurements. What's more, the agent produces more requests than the fixed pipeline — intermediate pages are opened and returned to for every decision. That's why session continuity matters in agent jobs: the same browsing session leaving from the same exit address means the requests in between don't get disconnected from each other. A Sticky Proxy that provides an address that doesn't change during the session is therefore suited to agent jobs; on wide target lists, a Residential Proxy is used to grow the pool.

Nothing gets easier on the legitimate-framework side either: the agent still has to follow robots.txt and site terms, behind-login and personal data remain a separate responsibility. One line has been added to these: sites have started to separate agents that announce themselves when they come (verified bot identity, signed requests) from those that don't. We covered why agents get blocked and this new arrangement in our Why Are AI Shopping Agents Blocked on Websites? article; the summary is this: the legitimate route is not evading detection but openly carrying your identity.

Use cases#

• Lists that go through filter menus. In catalogues where category, filter and pagination steps work differently on every site, the agent reduces hours of work per script to a single instruction; the script side of pagination logic is in our Pagination article.
• Long-tail catalogues. Collecting hundreds of small vendor sites into one schema is born from prototyping with an agent instead of writing selectors per site; the decision table for scaling with the fixed pipeline is in our AI web scraper article.
• First setup of price and stock monitoring. On a new target set the agent does the first discovery; if the route it found is stable, the same job is handed to a rule-based script; the end-to-end setup of monitoring is in our Competitor Price Tracking article.
• Live data for model applications. The agent can collect current page content for your language model application and serve it cleaned; in this use, the access layer must be limited by the rules in our Safe Web Access for LLMs article.

When agent, when script?#

The last-row rule can be summarised like this: the agent belongs to the discovery phase of a job, the script to the repetition phase. Turning the route the agent found once (the addresses visited, the elements clicked, the form values sent) into a script means repeating the same job from then on without tokens.

Legal and ethical framework#

The agent doesn't change the law of scraping: robots.txt, site terms, personal data and behind-login content rules apply exactly the same; we covered the framework in our Is Web Scraping Legal? article. Agent-driven work carries two extra responsibilities. First, page content is sent to the model API: on pages containing personal data, this transfer itself falls under legislation and cleaning is required before sending. Second, the agent doesn't just read — it clicks and can fill forms; the permissions given to it must therefore be limited to the minimum requirement and its actions must be logged.

Frequently asked questions#

Does agentic web scraping replace classic scraping?#

No. An agent pays off in jobs where the route isn't known in advance; on stable pages and at high volume, a rule-based script is still faster, cheaper and more predictable. The common arrangement is the two side by side: the agent discovers, the script repeats.

How is the cost of agent-driven scraping calculated?#

It is the number of turns multiplied by the token cost per turn. In a rule-based script the number of turns is zero, in an AI scraper one; in agent jobs it varies with how hard the page is. That's why the budget is tied to the loop, not to the job: the step ceiling and the spending ceiling are written into code.

In which cases is it sensible to use an agent?#

When the structure is different every time, the steps can't be written in advance and the job's volume covers the token cost. Filter menus, multi-step lists and hundreds of different small sites fit this definition; a single product page doesn't.

How is the data the agent collects validated?#

It's no different from validation on the fixed pipeline: a JSON schema, type and required-field checks, range checks and manual comparison on a sample. Additionally, in agent jobs the expected record count is monitored; a job that falls below expectation doesn't pass silently, it is flagged separately.

Sites are blocking agents — what is the legitimate route?#

Not hiding the agent's identity but carrying it openly; following the site's terms and robots.txt; using verified bot identity if it can be granted. We covered the reasons for blocking and the signed-agent arrangement in our AI Shopping Agents article.

Which tool should I start agentic scraping with?#

For the browser side, Playwright and an interface that connects it to the model: either a ready-made MCP server or your own loop. On the model side, any API with structured output support is enough; to start, you can use the skeleton above together with a step ceiling.

Summary#

Agentic web scraping puts the model into both the route and the parsing of scraping: it observes the page, decides the next step and extracts data against the schema. What it buys is variable, multi-step jobs fitting into a single instruction; the price is a cost that grows with the turn count and an unpredictability that must be reined in with code. With a step ceiling, a validation layer and logging in place, the agent is a strong tool for discovery; repeating stable jobs stays with the script. When setting up your data-collection pipeline, take a look at our data-scraping solutions.