How to Scrape a Website with Python: A Practical Guide to Requests, BeautifulSoup, and Playwright

Overview

Before you write a Python web scraper, define the exact data you need, the pages that contain it, and how often the process should run. A small, well-scoped extraction job is easier to test than a script that attempts to copy an entire site.

Most projects fit one of two patterns:

• Static or server-rendered HTML: the useful content is present in the response returned by the server. Use requests to download the page and BeautifulSoup to parse it.
• Client-rendered content: the initial response is mostly a shell, and JavaScript loads the data later or displays it after an interaction. Use Playwright when a normal HTTP request cannot access the required content reliably.

Start with the simpler option. Browser automation adds setup, execution time, and more possible failure points. It is useful, but it should solve a specific problem rather than become the default for every page.

Before collecting data, review the site’s terms, available APIs, robots.txt guidance, authentication requirements, and any applicable laws or contractual restrictions. A robots.txt file is an important signal about a site’s stated crawling preferences, but it is not a complete substitute for your own compliance review. The robots.txt guide for web scraping is a useful companion checklist.

Checklist by scenario

Scenario 1: Scraping ordinary HTML with Requests and BeautifulSoup

Use this path when viewing the page source shows the information you want. Install the basic packages in a virtual environment:

python -m venv .venv
source .venv/bin/activate
pip install requests beautifulsoup4

A minimal extractor should set a timeout, check the response, and isolate parsing from downloading:

import requests
from bs4 import BeautifulSoup

url = "https://example.com/products"
response = requests.get(
    url,
    headers={"User-Agent": "research-scraper/1.0"},
    timeout=20,
)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
items = []

for card in soup.select("article.product-card"):
    name_node = card.select_one(".product-name")
    price_node = card.select_one(".product-price")
    items.append({
        "name": name_node.get_text(" ", strip=True) if name_node else None,
        "price": price_node.get_text(" ", strip=True) if price_node else None,
    })

print(items)

Use specific selectors, but avoid selectors tied to incidental styling where possible. A semantic class, data attribute, or stable element relationship is usually easier to maintain than a long chain of positional selectors. Store raw URLs and normalized values separately when you may need to audit the result later.

Scenario 2: Handling pagination

Pagination should have a clear stopping condition. Do not assume that every site uses numbered links. A page may expose a next link, a cursor, a load-more endpoint, or a finite set of URLs.

from urllib.parse import urljoin

url = "https://example.com/products"
all_items = []

while url:
    response = requests.get(url, timeout=20)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")

    for card in soup.select("article.product-card"):
        name = card.select_one(".product-name")
        all_items.append({
            "name": name.get_text(" ", strip=True) if name else None
        })

    next_link = soup.select_one("a[rel='next']")
    url = urljoin(url, next_link["href"]) if next_link and next_link.get("href") else None

Add a maximum page limit, track visited URLs, and stop if the next URL repeats. These safeguards prevent malformed pagination from creating an unintended loop.

Scenario 3: Scraping JavaScript-rendered pages with Playwright

Choose Playwright when the required content appears only after JavaScript runs, a button must be pressed, or a page requires a browser context to expose the relevant elements. Install the package and its browser separately:

pip install playwright
playwright install

A basic Playwright workflow waits for a meaningful element rather than relying only on a fixed sleep:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/catalog", wait_until="domcontentloaded")
    page.locator("article.product-card").first.wait_for()

    rows = []
    for card in page.locator("article.product-card").all():
        rows.append({
            "name": card.locator(".product-name").inner_text()
        })

    browser.close()

print(rows)

For infinite scrolling, define a limit and verify that the number of extracted items increases. For a detailed Playwright workflow covering selectors, pagination, and export, see How to Scrape JavaScript Websites with Playwright.

Scenario 4: Exporting and validating the result

Export to JSON or CSV only after checking the shape of the data. Confirm that required fields are present, URLs are absolute when needed, duplicate records are handled, and text is normalized consistently. Keep a small sample of the source HTML or page URL with each run when reproducibility matters.

What to double-check

• Request behavior: Use timeouts, handle connection errors, and avoid sending requests faster than the site can reasonably handle. Add controlled retries only for temporary failures.
• HTTP responses: A successful connection does not guarantee useful content. Check the status code, content type, redirect destination, and whether the response is an error page.
• Selectors: Test selectors against more than one page. A selector that works for one product or article may fail when an optional field is absent.
• Encoding: Preserve Unicode text and inspect unusual characters before cleaning it. Over-aggressive whitespace or punctuation removal can damage names and descriptions.
• Pagination: Record the number of pages visited and items collected. Compare those totals with an expected range rather than assuming completion.
• Dynamic data: If Playwright displays data that Requests cannot find, inspect network requests and page state. An official endpoint may be more stable than scraping rendered markup, subject to permission and access constraints.
• Secrets and personal data: Do not place credentials in source code or logs. Minimize collection of personal information and define retention and access rules before the scraper runs.

When debugging, save the response body or a screenshot for the failing case. A captured artifact makes it easier to determine whether the issue is a changed selector, a redirect, a blocked request, or a page that has not finished rendering.

Common mistakes

Starting with browser automation. If the data is already in HTML, Requests and BeautifulSoup are usually easier to run and debug. Test the plain response first.

Using brittle selectors. Long selectors based on layout or element position often break after minor front-end changes. Prefer stable attributes and validate that the selector returns the expected count.

Ignoring missing fields. Calling a method on a nonexistent element can stop the entire run. Treat optional fields as optional and record missing values explicitly.

Hard-coding one page. Separate the URL, extraction function, pagination logic, and export step. This makes it simpler to change the target or reuse the workflow.

Relying on fixed delays. In Playwright, wait for a specific element, response, or state when possible. A fixed delay may be too short on one run and unnecessarily long on another.

Skipping validation. A scraper can finish without errors while returning empty fields or duplicate rows. Add assertions, sample checks, and basic logging before scheduling it.

Confusing encoding with security. If scraped data must be encoded or hashed for a downstream workflow, use the appropriate operation. The guides to Base64 encoding and decoding and SHA-256 hashing explain why these operations serve different purposes.

When to revisit

Revisit this scraping checklist before a seasonal planning cycle, before increasing the collection frequency, and whenever the target site changes its layout, navigation, authentication, or rendering approach. Also review the workflow when you move from a local script to a scheduled job or shared system.

At each review, run a small test and compare it with a known-good sample. Check the response status, item count, required fields, duplicate rate, pagination stop condition, and export format. Confirm that your use of the data still fits the site’s current terms, access controls, and internal retention rules.

A practical maintenance routine is to keep selectors and configuration in one place, log each run, preserve representative failure cases, and document why Requests or Playwright was selected. If the scraper no longer needs a browser, simplify it. If JavaScript rendering has become essential, update the browser workflow deliberately rather than adding more delays and fragile workarounds. For broader troubleshooting, consult Common Web Scraping Errors and How to Fix Them.

Use this article as a pre-run checklist: define the scope, choose the least complex method, verify access and permissions, extract defensively, validate the output, and schedule a review whenever the inputs or workflow change.