Web Scraping with Python

Setup

You need Python 3.10 or newer; we used 3.12. Create a virtual environment so the project's packages stay separate from the system ones, as the venv documentation describes, then install the libraries:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install requests httpx beautifulsoup4 lxml playwright
python -m playwright install chromium

The target is quotes.toscrape.com, part of the toscrape.com sandbox, which describes itself as a safe place for beginners learning web scraping. It has ten pages of quotes with pagination, plus variants that render with JavaScript, scroll infinitely or require a login. Nothing you do there affects a real business, which is the right place to make mistakes.

What each package does, with the version we tested:

First request and parse

Start with one page. Requests fetches it, and Beautiful Soup turns the HTML into a tree you can query with CSS selectors.

"""Fetch one page of the quotes.toscrape.com practice site and print each quote."""
import requests
from bs4 import BeautifulSoup

URL = "https://quotes.toscrape.com/page/1/"
HEADERS = {"User-Agent": "quotes-tutorial/1.0 (contact: you@example.com)"}

response = requests.get(URL, headers=HEADERS, timeout=(5, 20))
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
for card in soup.select("div.quote"):
    text = card.select_one("span.text").get_text(strip=True)
    author = card.select_one("small.author").get_text(strip=True)
    tags = [tag.get_text(strip=True) for tag in card.select("a.tag")]
    print(f"{author}: {text[:60]} {tags}")

Run it with python quotes_first.py. Our run printed ten lines, each with an author, the first 60 characters of the quote and its tags, starting with Albert Einstein.

What the code is doing:

• timeout=(5, 20) gives five seconds to connect and twenty to read. Requests has no default timeout, so a request without one can hang forever on a stalled server. Always set it.
• raise_for_status() turns a 4xx or 5xx response into an exception, so a block page or an error is never parsed as if it were data.
• The User-Agent names the script and a contact. Sites read it, and an honest one is the first courtesy a scraper owes.
• select("div.quote") finds every quote card; select_one inside each card finds its parts. Open the page in a browser, right-click a quote and choose Inspect to see why these selectors work.

Selectors are the fragile part of every scraper. Prefer classes and attributes that describe content (small.author) over position-based paths (div > span:nth-child(2)), because content classes survive redesigns more often.

Sessions and retries

A real crawl makes many requests to one site. A requests.Session reuses the TCP and TLS connection, keeps cookies, and applies headers and settings to every request. Mounting an HTTPAdapter with a urllib3 Retry policy makes it resilient to the temporary errors every site produces.

This is the complete crawler. It reads robots.txt, follows the Next link through every page, pauses between requests, writes a CSV and supports an optional authenticated proxy through environment variables.

"""Crawl every page of a quotes site politely and save the results as CSV.

Settings come from the environment:
  BASE_URL        site to crawl (default https://quotes.toscrape.com)
  DELAY_SECONDS   pause between requests (default 1)
  PROXY_URL       optional, scheme://host:port of an authenticated proxy
  PROXY_USER, PROXY_PASSWORD   credentials for that proxy
"""
import csv
import os
import sys
import time
from urllib.parse import quote, urljoin, urlsplit
from urllib.robotparser import RobotFileParser

import requests
from bs4 import BeautifulSoup
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

BASE = os.environ.get("BASE_URL", "https://quotes.toscrape.com")
AGENT = "quotes-tutorial/1.0 (contact: you@example.com)"
DELAY = float(os.environ.get("DELAY_SECONDS", "1"))


def proxies_from_env():
    url = os.environ.get("PROXY_URL")
    if not url:
        return None
    parts = urlsplit(url)
    user = quote(os.environ["PROXY_USER"], safe="")
    password = quote(os.environ["PROXY_PASSWORD"], safe="")
    full = f"{parts.scheme}://{user}:{password}@{parts.hostname}:{parts.port}"
    return {"http": full, "https": full}


def make_session():
    retry = Retry(
        total=4,
        backoff_factor=1.0,
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=("GET", "HEAD"),
        respect_retry_after_header=True,
    )
    session = requests.Session()
    session.headers["User-Agent"] = AGENT
    adapter = HTTPAdapter(max_retries=retry)
    session.mount("https://", adapter)
    session.mount("http://", adapter)
    session.proxies = proxies_from_env() or {}
    session.trust_env = False  # ignore HTTP_PROXY and similar variables
    return session


def load_robots(session):
    robots = RobotFileParser()
    response = session.get(urljoin(BASE, "/robots.txt"), timeout=(5, 20))
    if response.status_code == 407:
        response.raise_for_status()  # the proxy rejected our credentials: stop, do not guess
    if response.status_code >= 500:
        robots.disallow_all = True  # RFC 9309: treat an unreachable file as full disallow
    elif response.status_code >= 400:
        robots.allow_all = True  # no robots.txt: nothing is disallowed
    else:
        robots.parse(response.text.splitlines())
    return robots


def crawl(session, robots):
    url = urljoin(BASE, "/page/1/")
    while url:
        if not robots.can_fetch(AGENT, url):
            print(f"skipped by robots.txt: {url}", file=sys.stderr)
            return
        response = session.get(url, timeout=(5, 20))
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "html.parser")
        for card in soup.select("div.quote"):
            yield {
                "text": card.select_one("span.text").get_text(strip=True),
                "author": card.select_one("small.author").get_text(strip=True),
                "tags": ";".join(t.get_text(strip=True) for t in card.select("a.tag")),
                "page": url,
            }
        link = soup.select_one("li.next a")
        url = urljoin(url, link["href"]) if link else None
        time.sleep(robots.crawl_delay(AGENT) or DELAY)


def main():
    session = make_session()
    rows = list(crawl(session, load_robots(session)))
    with open("quotes.csv", "w", newline="", encoding="utf-8") as handle:
        writer = csv.DictWriter(handle, fieldnames=["text", "author", "tags", "page"])
        writer.writeheader()
        writer.writerows(rows)
    pages = len({row["page"] for row in rows})
    print(f"saved {len(rows)} quotes from {pages} pages")


if __name__ == "__main__":
    try:
        main()
    except requests.RequestException as error:
        # Do not print the exception text: it can contain the proxy URL with credentials.
        print(f"request failed: {type(error).__name__}", file=sys.stderr)
        sys.exit(1)

The Retry configuration is the part most tutorials get wrong:

• status_forcelist lists the responses worth retrying: 429 (too many requests) and the 5xx errors that usually mean a temporary problem. A 403 or 404 is not in the list, because retrying a refusal does not change it.
• backoff_factor=1.0 spaces attempts out exponentially, so a struggling server gets more breathing room on each retry rather than less.
• respect_retry_after_header=True makes the client wait as long as the server asks when it sends Retry-After, a header defined in the HTTP standard.
• allowed_methods limits retries to GET and HEAD, which are safe to repeat. Never blindly retry a POST that might create something twice.

We checked the retry policy against a local endpoint that answers 503 with Retry-After: 1 twice and then 200: the session waited, retried and returned 200 ok after retries in about two seconds.

The crawl itself, against the live sandbox through a local authenticated proxy, printed saved 100 quotes from 10 pages in about 13 seconds with a one-second delay between pages. Against the local fixture site, which has three pages and a robots.txt with a crawl delay, it saved 30 quotes from 3 pages.

Two error-handling details matter in production. The final except prints only the exception's type, because the text of a proxy error can include the full proxy URL with its password. And a 407 from the proxy while fetching robots.txt stops the crawl instead of being read as "no robots.txt", which would otherwise make the crawler treat the whole site as allowed. With a wrong password, our run stopped at once with request failed: HTTPError.

Async with HTTPX

When pages are independent, fetching several at once saves time. HTTPX offers an async client with an API close to Requests. The important part is the limit: concurrency without a cap is how a well-meaning scraper turns into a load test.

"""Fetch several pages concurrently with HTTPX, with a hard cap on parallel requests."""
import asyncio
import os
import sys

import httpx
from bs4 import BeautifulSoup

BASE = os.environ.get("BASE_URL", "https://quotes.toscrape.com")
PAGES = int(os.environ.get("PAGES", "5"))
LIMIT = asyncio.Semaphore(int(os.environ.get("CONCURRENCY", "3")))


def client_proxy():
    if not os.environ.get("PROXY_URL"):
        return None
    return httpx.Proxy(os.environ["PROXY_URL"], auth=(os.environ["PROXY_USER"], os.environ["PROXY_PASSWORD"]))


async def fetch(client, number):
    async with LIMIT:
        response = await client.get(f"{BASE}/page/{number}/")
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "html.parser")
        return number, len(soup.select("div.quote"))


async def main():
    async with httpx.AsyncClient(
        proxy=client_proxy(),
        trust_env=False,
        headers={"User-Agent": "quotes-tutorial/1.0 (contact: you@example.com)"},
        timeout=httpx.Timeout(20.0, connect=5.0),
        limits=httpx.Limits(max_connections=3, max_keepalive_connections=3),
    ) as client:
        results = await asyncio.gather(*(fetch(client, n) for n in range(1, PAGES + 1)), return_exceptions=True)
    for item in results:
        if isinstance(item, Exception):
            print(f"failed: {type(item).__name__}", file=sys.stderr)
        else:
            print(f"page {item[0]}: {item[1]} quotes")


asyncio.run(main())

Two limits work together here. The asyncio.Semaphore caps how many fetch tasks run at once, and httpx.Limits caps the connection pool. Set both to the same small number. return_exceptions=True keeps one failed page from cancelling the others, and the result loop reports failures by type only.

Through the local proxy against the live sandbox, five pages with three in flight completed in about a second, each reporting ten quotes. For a site you do not own, keep the cap at two or three and add a delay; for your own staging site, raise it until the server's latency starts to climb, then back off.

If you are writing worker services rather than scripts, our guides to HTTPX proxy configuration and aiohttp cover connection pools, timeouts and TLS in more depth.

JavaScript pages with Playwright

Some pages arrive as an empty shell and build their content in the browser. The sandbox's /js/ variant is one: fetch it with Requests and you get a script, not quotes. Playwright drives a real browser engine, waits for the content, and lets you read the rendered page.

"""Render the JavaScript version of the quotes site with Playwright and read the result."""
from os import environ

from playwright.sync_api import sync_playwright

URL = environ.get("JS_URL", "https://quotes.toscrape.com/js/")


def proxy_settings():
    if not environ.get("PROXY_URL"):
        return None
    return {
        "server": environ["PROXY_URL"],
        "username": environ["PROXY_USER"],
        "password": environ["PROXY_PASSWORD"],
    }


with sync_playwright() as playwright:
    browser = playwright.chromium.launch(proxy=proxy_settings())
    page = browser.new_page()
    page.goto(URL, wait_until="domcontentloaded", timeout=30_000)
    page.wait_for_selector("div.quote", timeout=15_000)
    quotes = page.eval_on_selector_all(
        "div.quote",
        "cards => cards.map(c => ({text: c.querySelector('.text').innerText, author: c.querySelector('.author').innerText}))",
    )
    browser.close()

for quote in quotes:
    print(f"{quote['author']}: {quote['text'][:60]}")
print(f"{len(quotes)} quotes rendered by the browser")

wait_for_selector is the line that makes this reliable. It returns as soon as the first quote exists in the rendered page, so the script never relies on a fixed pause that is too short on a slow day and wasteful on a fast one. eval_on_selector_all runs a small function in the page and returns plain data, which is faster than pulling elements back one at a time.

Our run through the local proxy against the live /js/ page rendered all ten quotes in under three seconds, including browser start-up. Against a local fixture page that builds its list from a JSON endpoint, it also returned ten.

Use a browser only when you need one. It costs far more memory and time per page than an HTTP request. Often the data a JavaScript page displays comes from a JSON endpoint you can see in the browser's network panel; if the site's terms allow it, requesting that endpoint directly is simpler and lighter. For sessions, contexts and rotation in Playwright, see our Playwright proxy guide.

Adding an authenticated proxy

A proxy sends your requests out through another network. Reasons to use one in a permitted project include testing how a site looks from another country, keeping a job's traffic off your office connection, or giving a service a stable, dedicated exit.

If you need a dedicated exit, Coronium's mobile proxies are physical devices with their own SIM, one customer per line, reachable over HTTP and SOCKS5, with the IP changed by dashboard button, API, rotation link or timer. A rotation link changes the IP on every GET request, so call it from code rather than pasting it anywhere a link preview could open it.

Four rules prevent most proxy bugs:

1. Keep credentials out of code. The scripts read PROXY_URL (scheme, host and port only), PROXY_USER and PROXY_PASSWORD from the environment. Put real values in your shell, a .env file excluded from version control, or your deployment's secret store.
2. Encode special characters. A password containing @, : or / breaks a proxy URL unless it is percent-encoded, which is why the crawler runs both values through urllib.parse.quote(value, safe=""). HTTPX and Playwright take credentials as separate fields and avoid the problem.
3. Turn off environment pickup. Requests and HTTPX read HTTP_PROXY, HTTPS_PROXY and NO_PROXY from the environment by default. trust_env=False makes the script use only the proxy you configured, so a variable left over on a server cannot reroute traffic silently.
4. Keep the gateway scheme as given. A proxy URL starting with http:// is correct for HTTPS sites too: the client sends a CONNECT request and then negotiates TLS with the site through the tunnel. For SOCKS5 proxies, Requests needs the requests[socks] extra and a socks5h:// URL so that DNS is resolved through the proxy.

To confirm the route, request an IP echo service through the proxy and compare the address with a direct request from the same machine. Our Requests proxy guide covers SOCKS5, environment precedence and error messages in detail.

Each library takes proxy settings differently, and the examples above already show all three:

Polite crawling

A scraper that works is half done. The other half is making sure it behaves well toward the site it visits, which is also what keeps it working.

• Read robots.txt. The Robots Exclusion Protocol is standardized in RFC 9309, and Python ships a parser in urllib.robotparser. The crawler fetches robots.txt with its own session (so the proxy and timeouts apply) and follows the RFC on errors: a missing file allows everything, an unreachable one disallows everything.
• Honor delays. If robots.txt sets a crawl delay, use it; otherwise pick one second or more between pages on a site you do not own. If the site sends Retry-After, wait at least that long.
• Cap concurrency. Two or three requests in flight are plenty for most sites. Your scraper competes with the site's real visitors for the same servers.
• Identify yourself. A User-Agent with your project name and a contact lets a site owner reach you instead of blocking you.
• Fetch less. Cache pages you have already processed, request only what changed, and stop when you have what you need.
• Read the terms. Some sites forbid automated collection outright; many offer an API that is better than any scraper. Personal data carries legal obligations regardless of how public it looks.

Our guide to whether web scraping is legal covers the law by region. The web scraping proxy guide covers network design for larger permitted projects, and mobile proxies for web scraping covers when a carrier line is the right exit.

Run the whole series yourself

The download at the top of this page contains a local origin server with fixture pages, an authenticated HTTP proxy with synthetic credentials, every example from this tutorial and the related guides in this series, and the runner that executes them. It needs Python 3.12; a few checks also use Node, Go, Java, Ruby and curl when they are installed and skip otherwise. Running python run_lab.py --offline keeps every request on your machine.

Frequently asked questions