Building a Web Scraping AI Agent

Architecture

Four parts, each with one job.

The loop is simple. The model receives the task and the tool descriptions, asks for a tool call, your code runs it and returns the result, and the model either asks for another call or answers. Every major model API supports this pattern under the name tool use or function calling, and the Model Context Protocol standardizes how hosts discover and call tools served by separate processes.

Two rules shape everything below. First, the model only proposes; your code disposes. A model can be confused by a page or manipulated by text planted in one (prompt injection), so it must never be the component that enforces scope. Second, keep the network decision outside the model. The agent should not choose proxies or rotate IPs by reasoning about blocks. It uses the line it was given, and routine line operations go through a controlled tool.

Browser tool and proxy

The fetch tool below is the core of the agent. It is plain Python with HTTPX and BeautifulSoup, and every limit lives in it.

"""Guarded fetch tool and a minimal tool-calling loop for a scraping agent.

The model is any object with a `next_step(messages)` method that returns either
{"tool": "fetch_page", "url": "..."} or {"final": "..."}. Swap in your model
provider's tool-calling API; the guardrails stay in this code, not in the prompt.
"""
import os
import time
from urllib.parse import urljoin, urlsplit
from urllib.robotparser import RobotFileParser

import httpx
from bs4 import BeautifulSoup

AGENT_UA = "example-research-agent/0.3 (contact: you@example.com)"

FETCH_TOOL_SCHEMA = {
    "name": "fetch_page",
    "description": "Fetch one page from an allowed site and return its visible text and links.",
    "input_schema": {
        "type": "object",
        "properties": {"url": {"type": "string", "description": "Absolute https or http URL"}},
        "required": ["url"],
    },
}


class FetchTool:
    def __init__(self, allowed_hosts, max_pages=10, min_interval=2.0, max_chars=4000):
        self.allowed_hosts = set(allowed_hosts)
        self.max_pages, self.min_interval, self.max_chars = max_pages, min_interval, max_chars
        self.pages_used, self.last_request, self.robots = 0, 0.0, {}
        proxy = None
        if os.environ.get("PROXY_URL"):
            proxy = httpx.Proxy(os.environ["PROXY_URL"], auth=(os.environ["PROXY_USER"], os.environ["PROXY_PASSWORD"]))
        self.client = httpx.Client(proxy=proxy, trust_env=False, follow_redirects=False,
                                   headers={"User-Agent": AGENT_UA}, timeout=httpx.Timeout(20.0, connect=5.0))

    def _robots_for(self, url):
        origin = "{0.scheme}://{0.netloc}".format(urlsplit(url))
        if origin not in self.robots:
            parser = RobotFileParser()
            response = self.client.get(urljoin(origin, "/robots.txt"))
            if response.status_code >= 500:
                parser.disallow_all = True
            elif response.status_code >= 400:
                parser.allow_all = True
            else:
                parser.parse(response.text.splitlines())
            self.robots[origin] = parser
        return self.robots[origin]

    def __call__(self, url):
        parts = urlsplit(url)
        if parts.scheme not in ("http", "https") or parts.hostname not in self.allowed_hosts:
            return {"error": "host not allowed"}
        if self.pages_used >= self.max_pages:
            return {"error": "page budget used up"}
        if not self._robots_for(url).can_fetch(AGENT_UA, url):
            return {"error": "disallowed by robots.txt"}
        wait = self.last_request + self.min_interval - time.monotonic()
        if wait > 0:
            time.sleep(wait)
        self.last_request = time.monotonic()
        self.pages_used += 1
        response = self.client.get(url)
        if response.status_code != 200:
            return {"error": f"HTTP {response.status_code}"}
        soup = BeautifulSoup(response.text, "html.parser")
        links = sorted({urljoin(url, a["href"]) for a in soup.select("a[href]")})
        return {"url": url, "text": soup.get_text(" ", strip=True)[: self.max_chars], "links": links[:50]}


def run_agent(model, task, tool, max_steps=8):
    messages = [{"role": "user", "content": task}]
    for _ in range(max_steps):
        step = model.next_step(messages)
        if "final" in step:
            return step["final"], messages
        result = tool(step["url"])
        messages.append({"role": "tool", "name": "fetch_page", "input": step["url"], "content": result})
    return "stopped: step limit reached", messages

A few choices are deliberate. Redirects are not followed automatically, so a redirect to another host cannot slip past the allowlist; the model sees the 3xx and has to ask for the new URL, which is checked again. Errors come back as data rather than exceptions, so the model can report them instead of retrying blindly. Page text is truncated, which caps token cost per step. And trust_env=False stops the client from silently picking up a proxy from the environment of whatever machine runs it.

We tested this file on October 30, 2026 with Python 3.12 and HTTPX 0.28.1, through a local authenticated proxy, against a fixture site whose robots.txt disallows a private path. A scripted model asked for four URLs: the site's index, the disallowed admin page, a host outside the allowlist and the pricing page. The tool returned the index, refused the admin page as disallowed by robots.txt, refused the outside host, and returned the pricing text. The fixture's log confirmed the admin page was never requested.

When the page needs JavaScript

For pages that render in the browser, give the agent a browser tool instead of more HTTP tricks. Microsoft's Playwright MCP server exposes a real browser to any MCP host. Its --proxy-server flag sets a proxy, and credentials go in its configuration file under the browser launch options, which follow Playwright's own proxy settings:

{
  "browser": {
    "browserName": "chromium",
    "launchOptions": {
      "headless": true,
      "proxy": {
        "server": "http://PROXY_HOST:PROXY_PORT",
        "username": "YOUR_PROXY_USERNAME",
        "password": "YOUR_PROXY_PASSWORD"
      }
    }
  }
}

Note what the Playwright MCP documentation says about its origin filters: it warns that --allowed-origins and --blocked-origins are not a security boundary and do not apply to redirects. Keep your real allowlist in a tool you control, or run the browser in an isolated environment that can only reach the hosts you intend.

Coronium MCP for line control

The agent's network exit should be stable within a task and easy to refresh between tasks. The Coronium MCP server gives an MCP host tools for exactly that: list your lines, test one, rotate its IP and poll the rotation status, request a replacement, and read plans and stock. A host configuration with both servers looks like this:

{
  "mcpServers": {
    "browser": {
      "command": "npx",
      "args": ["-y", "@playwright/mcp@latest", "--isolated", "--config", "/path/to/playwright-mcp.json"]
    },
    "coronium": {
      "command": "npx",
      "args": ["-y", "coronium-proxy-mcp@2.3.0"],
      "env": {
        "CORONIUM_API_TOKEN": "YOUR_CORONIUM_API_TOKEN",
        "CORONIUM_TOOL_PROFILE": "directory"
      }
    }
  }
}

The directory profile has no money tools at all. If you use claude.ai or the Claude apps, you can skip the local install and add the hosted server at https://api.coronium.io/mcp as a custom connector, signing in with your dashboard account; it runs the same directory profile and never spends.

Practical rules for an agent with line control:

• Rotate between tasks, not inside one. A login or a multi-page flow should keep one IP from start to finish.
• Respect the rotation rules: at least 60 seconds between rotations, and some devices rotate on request only. A rotate call returns done, rotating (poll the status tool, do not resubmit) or unchanged (the carrier kept the address).
• Set the host's tool timeout above 180 seconds so rotation and replacement calls can finish.
• Never hand a rotation link to the model as text it might paste somewhere; a link preview rotates the line.

Coronium lines are physical devices with their own SIM, one customer per line, over HTTP and SOCKS5, with data not metered on dedicated lines. The MCP server page documents every profile and tool, and our MCP proxy server guide compares MCP setups more broadly. For the product itself, see Coronium mobile proxies.

Guardrails: robots.txt, rate and scope

The tool above enforces four limits. Here is why each one exists and how to tune it.

• Scope. The host allowlist is the most important line in the file. An agent that follows links will eventually reach a page you did not intend, and a page can contain instructions aimed at the model. With an allowlist, the worst case is a wasted step.
• robots.txt. The Robots Exclusion Protocol is standardized in RFC 9309. It is a statement of the site owner's wishes, not an access control, and honoring it is the minimum courtesy for an automated client. The tool follows the RFC's handling of errors: an unreachable robots.txt (5xx) means do not crawl; a missing one (4xx) means no restrictions. Python's urllib.robotparser does the parsing.
• Rate. A minimum interval per host keeps the agent's footprint close to one person's. If a site sends Retry-After or a crawl delay, use the larger value.
• Budget. A page cap per task bounds cost and stops loops where the model keeps asking for the same few pages.

Two more limits belong outside the tool. The MCP tools specification says there should always be a human in the loop with the ability to deny tool invocations, and hosts implement that with confirmation prompts. Keep confirmation on for anything that spends money or changes state. And keep secrets out of the model's context: proxy credentials and API tokens live in environment variables the tools read, never in the prompt.

Finally, legality. An agent fetching pages is still a scraper in the eyes of a site's terms and the law. Use official APIs where they exist, avoid personal data you have no basis to collect, and read our overview of web scraping law before scaling. If a site blocks agents by policy, our guide to why AI agents get blocked explains the verified-agent route.

Cost per run

A run costs model tokens plus a share of the network. Both are measurable.

Model cost comes from usage. Every major model API returns input and output token counts with each response; log them per step. The fetch tool's text cap is the main lever: a page truncated to 4,000 characters is roughly a thousand tokens of input, and every later step re-sends the conversation so far, so input grows with each page the agent reads.

Network cost is the line price divided by the runs it carries. A Coronium line is billed for 24 hours, 7 days or 30 days rather than by data volume on dedicated lines, so the share per run falls as you run more tasks in the period. The cheapest 24-hour plan in the catalog right now: From $7. Prices vary by country and carrier, so use the price of the line you actually run.

"""Estimate the cost of one agent run from logged token usage and your line's price.

Fill the prices from your model provider's current price list and your proxy plan.
"""


def run_cost(usage, input_per_million, output_per_million, line_price, runs_per_line_period):
    tokens_in = sum(step["input_tokens"] for step in usage)
    tokens_out = sum(step["output_tokens"] for step in usage)
    model = tokens_in / 1e6 * input_per_million + tokens_out / 1e6 * output_per_million
    network = line_price / runs_per_line_period
    return {"tokens_in": tokens_in, "tokens_out": tokens_out, "model_usd": round(model, 4),
            "network_usd": round(network, 4), "total_usd": round(model + network, 4)}

Pass the usage entries you logged, your model's current per-million-token prices, the price of your line and how many runs it handles in its billing period. The function is deliberately boring: if a run costs more than you expect, the token columns show whether the agent read too many pages or re-sent too much context.

When a plain script is enough, use one. If you already know the URLs and the fields, a crawler like the one in our Python web scraping tutorial is cheaper and more predictable than an agent. Agents earn their cost when the path through a site is not known in advance. For managed extraction services that sit between the two, see the Firecrawl proxy setup and our look at scraping in the agentic era.

Frequently asked questions