Why AI Agents Get Blocked by Websites

Signals that give agents away

An agent built on a hosted model typically runs in a cloud container, drives a browser or an HTTP client, and moves through pages faster than a person. Each of those choices produces a signal.

The network

Cloud providers publish their address ranges, and IP databases label them as hosting. A request from a data center is not a crime, but almost no human browsing comes from one, so many sites score it as automated before anything else happens.

The browser

Automation frameworks announce themselves unless configured otherwise. The W3C WebDriver standard defines navigator.webdriver as true under remote control, headless modes render and time things differently from headed browsers, and an HTTP client without JavaScript cannot run the checks a protected page expects. Bot-defense vendors look for exactly these signs. DataDome's documentation, for example, says its page script runs consistency checks for headless Chrome, Puppeteer and Selenium.

The pattern

Agents parallelize: they open ten tabs at once, fetch the same endpoint at a fixed interval, skip images and scripts, and never scroll. Behavioral models are trained on what humans do, so this pattern stands out even when every individual request looks fine.

The declaration

Some agents identify themselves, which is the right thing to do and can still lead to a block if the site refuses that category. OpenAI documents a separate ChatGPT-User agent for user-initiated actions and notes that robots.txt rules may not apply to it, since a person triggered the request. Anthropic's crawler documentation lists Claude-User for user-initiated fetches alongside ClaudeBot and Claude-SearchBot, and says its bots do not try to get around CAPTCHAs. A site that blocks those names by policy is making a choice, not a mistake.

Site policies on AI agents

Policy changed fast in 2025 and 2026, and most agent blocks you see today are policy, not detection.

On July 1, 2025, Cloudflare announced it would block AI crawlers by default for new customers. On July 1, 2026, its changelog split AI traffic into three behaviors site owners control separately: Search, for crawlers that index pages to answer questions later; Agent, for software doing something live for a specific person, including browser-use agents; and Training, for crawls that feed model training. From September 15, 2026, new domains onboarding to Cloudflare get updated defaults: any page carrying advertising refuses Training and Agent traffic, and Search is still let through. The Agent setting has three choices: allowed, refused on every page, or refused only where ads appear.

Other vendors take the same line. DataDome's bot authentication page states that automated requests from an unauthenticated bot or AI agent are classified as threats and blocked by default. Kasada launched an AI Agent Trust product in January 2026 and argues on its blog that one AI provider can appear as a browser acting for a user, a scraper and a commerce client at different moments, each needing its own decision.

Courts are working through the same question. Amazon sued Perplexity over its Comet shopping agent in November 2025, and a federal district court granted a preliminary injunction in March 2026. The Ninth Circuit later vacated that injunction on the record before it, reasoning that the assistant acted as a tool following the user's instructions, and sent the case back for further proceedings, as reported by Retail Insight Network on August 5, 2026. The case is not over, and the appellate court said its ruling was limited to that record. For anyone building agents, the safe reading is that site terms and access policies matter and the law is still being written.

Reading the block

Before changing anything, capture what the agent actually received: status code, the first few hundred bytes of the body, the response headers and the URL. Then sort it.

The fourth and fifth rows are confused more often than any other pair. An agent framework that wraps both the model call and the page fetch may surface "429 Too Many Requests" without saying which side sent it. Log the URL with every error. For Cloudflare responses specifically, our guide to Cloudflare IP bans and challenge loops explains how to read 1020, 1015 and challenge pages, and the Turnstile guide covers widget errors.

If the block is policy (the site disallows agents, or Cloudflare's Agent category is blocked on that page), stop there. A different IP or a disguised browser does not turn a refusal into permission. Ask for access, use the site's API, or verify the agent so the owner can allow it by name.

Fixes: route, browser, pacing, identification

When the site allows agents and the block is detection, fix the four layers in this order.

Identification first

Identification is the fix that keeps working. Give the agent a stable, descriptive user agent with a contact URL. Then sign its requests. Web Bot Auth, being standardized by the IETF's webbotauth working group, uses RFC 9421 HTTP Message Signatures: the operator publishes an Ed25519 public key in a key directory at /.well-known/http-message-signatures-directory, and every request carries Signature-Input, Signature and Signature-Agent headers. Cloudflare's Web Bot Auth documentation, updated October 8, 2026, walks through registering a key directory through its bot submission form. DataDome accepts the same signatures, and Kasada lists Web Bot Auth among the standards its agent policies support. Our explainer on Web Bot Auth and verifiable agents has the full setup.

Browser

Use a real browser engine in headed mode with a persistent profile, not a raw HTTP client, for sites that run client-side checks. Microsoft's Playwright MCP server lets an MCP host drive such a browser, and its configuration accepts a proxy with credentials. Keep the profile's time zone and language consistent with the exit.

Route

Run the browser from a network that fits what the agent does. For an agent that acts for a person on mobile-first services, a carrier line gives it the same network label as that person's phone. Coronium's mobile proxies are dedicated devices with their own SIM, rented to one customer per line, over HTTP and SOCKS5. Keep one task or account on one line, and rotate between tasks rather than inside them.

Pacing

Serialize work per site. One tab, natural waits for pages to load, a per-host delay, and a cap on pages per task. Agents that act for one person should look like one person's activity, because that is what they are.

MCP setup

If the agent runs in an MCP host such as Claude, Claude Code or Cursor, it can manage its own Coronium line through the Coronium MCP server, so a human does not have to rotate IPs or check health by hand.

The hosted server needs no install. In claude.ai and the Claude apps, open Settings, Connectors, add a custom connector with the URL https://api.coronium.io/mcp, then sign in with your Coronium dashboard account. In Claude Code:

claude mcp add --transport http coronium-remote https://api.coronium.io/mcp

Then run /mcp to sign in. The hosted URL runs the directory profile, which never spends money: the agent can list owned proxies, test them, rotate and check rotation status, request a replacement, read plans and stock, get renewal quotes and open tickets. A separate buyer URL exists for agents that may purchase, with purchases off until you enable them for that connection.

To run the server locally instead, add it to the host's MCP configuration with your API token from dashboard.coronium.io, Account, API Access:

{
  "mcpServers": {
    "coronium": {
      "command": "npx",
      "args": ["-y", "coronium-proxy-mcp@2.3.0"],
      "env": {
        "CORONIUM_API_TOKEN": "YOUR_CORONIUM_API_TOKEN",
        "CORONIUM_TOOL_PROFILE": "directory"
      }
    }
  }
}

Set the host's tool timeout above 180 seconds, because a rotation waits for the carrier. A rotate call returns done with the new IP, rotating (poll coronium_get_rotation_status, do not resubmit) or unchanged (the carrier kept the address; wait about 60 seconds). Spending tools exist only when you opt in. The MCP server page lists every profile and tool, and Coronium for AI agents covers the agent-facing API and live stock.

Frequently asked questions