How to Scrape JavaScript Websites with Playwright: A Practical Guide to Selectors, Pagination, and Data Export

Overview

When you scrape a website built with client-side JavaScript, the HTML returned by the first request may contain little more than an application shell. Product cards, search results, tables, or article listings can be added later by browser scripts. Playwright solves this by controlling a real browser engine and letting the page complete enough of its normal loading process before you read the DOM.

A reliable workflow has six stages:

1. Confirm that automated access is appropriate and review the site’s published instructions.
2. Open the page with a controlled browser context.
3. Inspect the page and choose selectors based on stable structure or accessible roles.
4. Wait for a meaningful state, such as a visible result list, rather than adding arbitrary delays.
5. Collect normalized records and handle pagination deliberately.
6. Validate and export the data in a format suited to the next step.

Before coding, review the target site’s terms, access guidance, and robots.txt. Keep request volume reasonable, avoid collecting sensitive personal information without a legitimate reason, and identify yourself appropriately where the site’s rules require it. The goal is dependable data extraction, not aggressive traffic generation.

Set up a Python Playwright project

Install Playwright in an isolated Python environment, then install the browser binaries required by your project. The exact installation command can vary with your environment, so use the current Playwright documentation for the supported setup steps. A minimal project should also include a requirements file, a data directory, and a place for logs or screenshots.

Start with a small sample. Confirm that one page can be opened and one record can be extracted before adding pagination, retries, or scheduled execution. This makes selector and timing problems easier to diagnose.

A minimal browser pattern

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    context = browser.new_context()
    page = context.new_page()
    page.goto("https://example.com/list", wait_until="domcontentloaded")
    page.locator("article.card").first.wait_for(state="visible")
    print(page.locator("article.card").count())
    browser.close()

Use a locator that represents a real page concept, such as a card, row, heading, or navigation button. The example selector is only illustrative: inspect the target site and replace it with a selector that matches its structure.

Checklist by scenario

Scenario 1: Extracting a static-looking list rendered by JavaScript

• Open the page in a visible browser during development so you can observe loading behavior.
• Inspect whether the data appears after scrolling, filtering, or an interaction.
• Prefer role, label, test ID, or stable data attributes when available.
• Wait for the result container or a known record, not simply for a fixed number of seconds.
• Read text from individual locators and normalize whitespace before storing it.

Selectors based on presentation classes can break when a site’s design changes. A selector tied to a semantic role or a documented attribute is usually easier to maintain. If no stable selector exists, combine several modest signals rather than relying on a deeply nested CSS path.

Scenario 2: Collecting multiple fields from cards or rows

Scope each field lookup to its parent record. This prevents the title from one card being paired with the price or date from another. Build a small extraction function that returns the same schema for every record, including empty values when a field is absent.

def read_card(card):
    title = card.locator(".title").inner_text().strip()
    link = card.locator("a").first.get_attribute("href")
    return {
        "title": title,
        "url": link,
    }

cards = page.locator("article.card")
records = [read_card(cards.nth(i)) for i in range(cards.count())]

For production workflows, add normalization for URLs, dates, currency-like text, and whitespace. Preserve the original text when interpretation could be ambiguous, and store a crawl timestamp so later runs can be compared.

Scenario 3: Handling pagination

Pagination commonly appears as numbered links, a next button, an infinite scroll list, or a “load more” control. Choose the strategy that matches the page instead of assuming that every page uses a query parameter.

1. Extract and validate the current page’s records.
2. Identify whether a next control exists and is enabled.
3. Capture a marker from the current page, such as the first record URL.
4. Click or navigate to the next state.
5. Wait for the old content to change or for the new result state to appear.
6. Stop when the control is absent, disabled, or the marker repeats.

For a next button, wait for a meaningful change after the click. If the interface updates in place, compare a result count, a loading indicator, or the first record’s text. If pages have stable URLs, direct navigation can be easier to resume and audit than repeated clicking.

Scenario 4: Exporting data to JSON or CSV

JSON is convenient when records contain nested fields or when another program will consume the output. CSV is useful for flat tables and spreadsheet review. Before exporting, define a column order, convert missing values consistently, and decide how to represent line breaks and lists.

import csv
import json

with open("records.json", "w", encoding="utf-8") as file:
    json.dump(records, file, ensure_ascii=False, indent=2)

with open("records.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=["title", "url"])
    writer.writeheader()
    writer.writerows(records)

For larger jobs or repeated collection, consider a database rather than one large file. The guide to storing scraped data in CSV, JSON, SQLite, or Postgres can help you choose a destination based on volume, querying needs, and update frequency.

What to double-check

• Access and scope: Confirm that the pages and fields you plan to collect are within the permitted scope. Do not bypass authentication, paywalls, technical restrictions, or access controls.
• Selector quality: Test selectors against several records, not only the first visible item. Check the behavior when an optional image, label, or value is missing.
• Timing: Replace arbitrary sleeps with locator-based waits where possible. A fixed delay may be too short on a slow run and wasteful on a fast one.
• URL handling: Convert relative links to absolute URLs using the page’s base URL. Remove tracking parameters only when doing so will not change the resource’s identity.
• Pagination safety: Set a maximum page limit, detect repeated pages, and record the last successful URL. These safeguards prevent accidental loops.
• Data validation: Check required fields, record counts, duplicate identifiers, and unexpected empty results. A successful browser run does not guarantee a correct dataset.
• Observability: Save structured logs, failure URLs, and occasional screenshots during development. Redact sensitive values before sharing logs.
• Change tracking: Keep a sample output and a few representative page snapshots. Comparing them after site changes makes silent extraction failures easier to spot.

If a page exposes a documented data endpoint that you are allowed to use, it may be simpler and less resource-intensive than browser automation. Do not assume that an endpoint is available or permitted; inspect the application only within the limits that apply to your project.

Common mistakes

Using a plain HTTP client for a browser-rendered page

A request library may return the initial shell before JavaScript inserts the records. First verify whether the required data exists in the returned HTML. If it does not, use Playwright or another appropriate browser automation tool, or use an authorized data source.

Waiting for a fixed number of seconds

Long sleeps make a scraper slow, while short sleeps create intermittent failures. Wait for a specific locator, response, URL change, or state transition. Use a timeout as a failure boundary, not as proof that the page is ready.

Relying on fragile selectors

Selectors built from several generated class names or exact DOM depth often fail after a redesign. Prefer semantic selectors and keep all selectors in one clearly named section so they can be updated without searching through the entire script.

Ignoring partial failures

If one record fails, decide whether to skip it, retry it, or stop the run. Record the failure either way. A script that exits successfully while silently dropping half its fields is more dangerous than one that reports an explicit error.

Confusing encoding with security

Scraped text may contain encoded values, escaped HTML, or URL-encoded parameters. Decode only when the field’s meaning requires it, and do not treat Base64 as encryption. For related debugging workflows, see the Base64 encode and decode guide and the URL encoding guide.

Running at high speed by default

Concurrency, retries, and browser contexts should be introduced only after a single-page workflow is correct. Start conservatively, honor published limits, and make the rate and scope configurable. More parallelism is not automatically better data extraction.

When to revisit

Revisit this Playwright scraping checklist whenever the target site changes its layout, navigation, loading behavior, or authentication flow. It is also worth reviewing before a seasonal planning cycle, a scheduled collection run, or a move from a local experiment to a shared or automated environment.

Before each important run, perform a short maintenance check:

1. Run a smoke test against one known page and confirm that expected fields are populated.
2. Compare the record count and schema with a recent successful sample.
3. Verify that pagination stops correctly and does not repeat records.
4. Confirm that output files can be opened by the next system or person in the workflow.
5. Review logs for timeouts, redirects, empty selectors, and unexpected status changes.
6. Update browser dependencies and test them in a controlled environment before changing a production job.

Keep the scraper small and modular: browser setup, page navigation, extraction, pagination, validation, and export should be separable responsibilities. When a site changes, this structure lets you update a selector or wait condition without rewriting the entire workflow. For broader troubleshooting, consult common web scraping errors and fixes. A maintained checklist, a representative test page, and validated output are the foundation of a scraper that remains useful as both websites and Playwright APIs evolve.