How to Scrape a Website with Python: Requests, BeautifulSoup, Pagination, and Exporting Data

Overview

A reliable scraper is more than a loop that downloads URLs. It is a small data pipeline with clear stages: identify an appropriate source, fetch a page, parse the fields you need, follow additional pages, normalize the values, validate the output, and save a useful result.

For mostly server-rendered HTML, Python's requests library and BeautifulSoup are a straightforward combination. Requests handles HTTP communication, while BeautifulSoup provides methods for locating elements with CSS selectors and inspecting their text or attributes. This approach is usually easier to test and maintain than introducing a browser when the required content is already present in the response.

Before collecting data, check the website's terms, published instructions, access controls, and technical expectations. Request only the pages and fields you need, use a descriptive user agent where appropriate, limit request frequency, and avoid collecting personal or restricted information without a valid reason and authorization. A scraper should be designed to reduce unnecessary load rather than maximize request volume.

Step-by-step workflow

1. Define the record before writing selectors

Start with the output you want. For a product listing, a record might contain name, price, category, and url. For an article index, it might contain title, author, published_at, and url. This prevents the scraper from becoming a collection of unstructured snippets.

Use stable fields and decide how missing values should be represented. Empty strings, null values, and omitted keys each communicate something different to downstream code.

2. Install the small set of dependencies

python -m pip install requests beautifulsoup4

Keep the script in a virtual environment if it will be reused or deployed. Pinning versions can also make scheduled runs more predictable, although selectors and website behavior still need periodic review.

3. Fetch a page with timeouts and a clear response check

import requests

url = "https://example.com/articles"
headers = {
    "User-Agent": "ExampleResearchBot/1.0 (contact: you@example.com)"
}

response = requests.get(url, headers=headers, timeout=20)
response.raise_for_status()
html = response.text

A timeout prevents a stalled connection from holding up the entire run. raise_for_status() turns common HTTP error responses into visible failures instead of allowing the parser to process an error page as if it were valid data. In a production workflow, catch request exceptions, log the URL and error, and decide whether a retry is appropriate.

4. Parse the HTML with explicit selectors

from bs4 import BeautifulSoup
from urllib.parse import urljoin

soup = BeautifulSoup(html, "html.parser")
records = []

for card in soup.select("article.card"):
    title_node = card.select_one("h2.card-title")
    link_node = card.select_one("a.card-link")

    if not title_node or not link_node:
        continue

    records.append({
        "title": title_node.get_text(" ", strip=True),
        "url": urljoin(url, link_node.get("href", ""))
    })

Prefer selectors tied to meaningful structure rather than fragile positional expressions. Test each selector against a saved HTML response, and keep extraction separate from downloading so you can improve parsing without repeatedly requesting the site.

5. Add pagination deliberately

Pagination can use numbered links, a next link, a page parameter, or a cursor. The safest first implementation follows the next link found on the current page and stops when that link is absent.

from time import sleep

base_url = "https://example.com/articles"
current_url = base_url
all_records = []

while current_url:
    response = requests.get(current_url, headers=headers, timeout=20)
    response.raise_for_status()
    page = BeautifulSoup(response.text, "html.parser")

    for card in page.select("article.card"):
        title_node = card.select_one("h2.card-title")
        link_node = card.select_one("a.card-link")
        if title_node and link_node:
            all_records.append({
                "title": title_node.get_text(" ", strip=True),
                "url": urljoin(current_url, link_node.get("href", ""))
            })

    next_node = page.select_one("a[rel='next']")
    current_url = urljoin(current_url, next_node["href"]) if next_node else None
    sleep(1)

Use a set of visited URLs to guard against a malformed site repeating the same page. Also define a maximum page count during development. If pagination is controlled by JavaScript or the HTML response contains no records even though a browser displays them, inspect the network requests or move to a browser automation workflow.

6. Clean values before exporting

Cleaning should be predictable and reversible where possible. Strip surrounding whitespace, collapse repeated spaces, normalize missing values, and preserve the original URL. If prices or dates will be analyzed, store a normalized representation in a separate field rather than silently replacing the displayed value.

def clean_text(value):
    return " ".join(value.split()) if value else None

for record in all_records:
    record["title"] = clean_text(record.get("title"))

all_records = [r for r in all_records if r.get("title") and r.get("url")]

7. Export to CSV and JSON

import csv
import json

with open("articles.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=["title", "url"])
    writer.writeheader()
    writer.writerows(all_records)

with open("articles.json", "w", encoding="utf-8") as file:
    json.dump(all_records, file, ensure_ascii=False, indent=2)

CSV is convenient for spreadsheets and simple imports. JSON is better when records contain nested values or will be passed to another program. Choose one as the primary output and generate the other only when it serves a real handoff.

Tools and handoffs

Requests and BeautifulSoup work well when the target content is available in the initial HTML. If a page depends on client-side rendering, login state, interaction, or infinite scrolling, a browser automation tool may be more suitable. See the Playwright scraping guide for a workflow based on selectors, pagination, and data export. Puppeteer is another browser-based option, particularly for JavaScript projects.

Use the browser's developer tools to compare the raw response with the rendered page. The Network panel can reveal whether the visible data comes from an HTML document or a separate JSON request. If a documented or accessible data endpoint provides the same information, it may offer a cleaner handoff than parsing rendered markup. Inspect URLs, headers, and response bodies with the web developer diagnostics toolkit.

Keep the workflow modular: one function for fetching, one for parsing, one for pagination, one for normalization, and one for export. This makes it easier to replace BeautifulSoup with another parser or change the output schema without rewriting the entire scraper.

Quality checks

• Count records: Compare the number of extracted records with the number visible on a small sample of pages.
• Check required fields: Flag missing titles, URLs, identifiers, or other fields needed by the next system.
• Detect duplicates: Deduplicate on a stable URL or source identifier, not only on title text.
• Validate pagination: Confirm that the scraper stops and does not revisit the same URL indefinitely.
• Inspect encoding: Check accented characters, symbols, and non-Latin text in both CSV and JSON output.
• Save failures: Record pages that could not be fetched or parsed so they can be reviewed separately.
• Test changes safely: Run against a small page limit and saved fixtures before processing the full collection.

Selectors are often the most fragile part of a scraper. Avoid treating a successful HTTP response as proof that extraction succeeded: a redesign can return status 200 while changing every relevant class or nesting relationship. A simple validation rule, such as requiring a minimum proportion of records to contain a title and URL, can catch silent breakage early.

When to revisit

Revisit the scraper whenever the source changes its layout, navigation, rendering approach, URL structure, or access requirements. Also review it before increasing the crawl scope, adding new fields, changing the output schema, or moving the job into a scheduled environment.

A useful maintenance routine is to keep a small set of representative HTML fixtures and run parser tests against them. Record the expected fields, pagination behavior, and duplicate rules alongside the code. When a run produces an unusual count or a spike in missing values, pause the export, inspect a raw response, and compare it with the last known-good fixture.

For a practical next step, choose one permitted page type, define three to five output fields, save a sample response, and implement the fetch-parse-export loop with a low page limit. Add validation before expanding pagination. If the content is rendered only after JavaScript runs, switch to the broader Python scraping guide and then follow the Playwright workflow. This staged approach keeps the scraper understandable, testable, and easier to update when its inputs change.