Web data rarely arrives in the shape you need. This practical workflow shows how to identify HTML tables, embedded JSON, and CSV files, extract the relevant fields, normalize values, repair common encoding problems, remove duplicates, validate records, and convert the result into reliable JSON, CSV, or spreadsheet-ready data.
Overview
Data extraction is only the first part of a useful web scraping workflow. A table may contain headings in one format and values in another. A page may display product information in HTML while storing the same records inside a JSON script block. A CSV export may open incorrectly because its delimiter or character encoding does not match the spreadsheet application.
A repeatable process separates the work into five stages:
- Identify the source format and the fields you actually need.
- Extract records without losing their structure.
- Normalize names, values, dates, numbers, and missing fields.
- Validate the cleaned records and remove accidental duplicates.
- Export the result in a format suited to the next tool or person.
This approach applies whether you use a script, browser automation, a visual web scraping tool, or a set of online developer utilities. Before collecting data, check that your use of the site and its content is appropriate, and avoid collecting fields that are not necessary for the task.
Step-by-step workflow
1. Inspect the page before extracting anything
Start by determining where the data lives. An ordinary HTML table usually has a <table> element with header cells and rows. Some pages use lists or cards that look like tables but have no table markup. Other pages render an empty container and load records through JavaScript.
Use browser developer tools to inspect the page source and the live DOM. Look for stable attributes, repeated row containers, embedded script elements, and network responses containing structured data. Record the fields you need, such as name, url, price, category, and updated_at. Defining the target schema early prevents unnecessary extraction and makes later validation easier.
2. Extract data from an HTML table
For a conventional table, map each header to the corresponding cell in every row. Do not rely only on cell position if the page may add optional columns. A simple extraction should account for empty cells, row-spanning cells, links inside cells, and visible text that differs from the link destination.
Trim leading and trailing whitespace during extraction, but do not immediately remove internal spaces or punctuation. Those characters may be meaningful in names, addresses, or descriptions. Preserve the original URL when a cell contains a link, and consider storing both the displayed label and the absolute URL if the data will be reused.
If you are using Python, a requests-and-HTML-parser workflow may be sufficient for server-rendered tables. For a JavaScript-dependent page, browser automation may be more suitable. The Python web scraping guide covers common requests, parsing, pagination, and export patterns, while the Playwright scraping guide explains the additional considerations for pages whose content appears after rendering.
3. Parse embedded JSON carefully
Structured data may be present in a script element, a page state object, or a response returned by an API call. Embedded JSON is useful because it often contains cleaner fields than the visible page. However, not every script block is valid standalone JSON. It may be JavaScript with a variable assignment, escaped characters, or an object nested inside a larger structure.
First isolate the object or array, then decode it with a real JSON parser rather than relying on string splitting. A JSON formatter can help reveal missing commas, incorrect quotes, and unexpected nesting during investigation. Keep the original payload separately when possible so that you can compare the extracted records with the source if a parser fails later.
4. Normalize fields into a consistent schema
Cleaning scraped data means making equivalent values look equivalent without erasing useful information. Choose a field name convention and apply it consistently. For example, use one spelling for a field such as published_at instead of mixing publishedDate, date_published, and published.
- Whitespace: Trim surrounding spaces and collapse accidental line breaks where appropriate.
- Text: Decode HTML entities and preserve meaningful punctuation.
- Numbers: Remove display-only currency symbols or separators only after deciding how the value should be stored.
- Dates: Convert recognizable dates to one documented format, while retaining the original value if interpretation is uncertain.
- URLs: Resolve relative links against the source page and remove tracking parameters only if that is safe for your use case.
- Missing values: Choose one representation, such as an empty value or
null, and use it consistently.
Be cautious with automatic type conversion. A leading zero in a product code, postal code, or account identifier can be lost if the value is treated as a number. Preserve identifiers as text unless arithmetic is required.
5. Fix encoding and delimiter problems
Encoding issues often appear when characters such as accented letters, curly quotation marks, or non-Latin text are exported and reopened in another application. Confirm the source encoding and choose an output encoding supported by the receiving tool. When exporting CSV, also document the delimiter, quoting rules, line endings, and whether the first row contains headers.
A CSV file is not a universal data model. Fields containing commas, quotation marks, or line breaks must be quoted correctly. If the destination expects tab-separated values or a particular regional delimiter, configure the export for that destination rather than assuming a comma will work everywhere.
6. Remove duplicates using a deliberate key
Duplicate records can be created by pagination, repeated API responses, alternate URLs, or a scraper that retries without tracking what it has already saved. Select a stable deduplication key, such as a canonical URL or source identifier. If no single field is reliable, combine several fields and document the rule.
Do not automatically discard records that merely look similar. Two entries can share a title while representing different versions or locations. When duplicates conflict, keep the source record, compare timestamps or other fields, and define which value takes precedence.
Tools and handoffs
The best tool depends on the source and the next destination. A browser inspector is useful for locating selectors and examining live content. A JSON formatter helps inspect nested objects. A CSV viewer or spreadsheet can provide a quick visual review, while a script gives you repeatability, logging, and stronger validation for recurring jobs.
For small, one-off transformations, online developer tools can shorten the path between extraction and review. For example, an online JSON-to-CSV converter may be convenient when the object structure is already understood. A URL encoder and decoder can help inspect query parameters, and a text editor with regular-expression support can assist with controlled cleanup. Avoid pasting confidential, personal, or access-controlled data into an online tool unless its handling is appropriate for that data.
When the workflow becomes recurring, move the transformation into code or a versioned configuration. Store a sample input, the expected schema, and an example output. This creates a clear handoff between the scraper and the cleaning stage. For broader reliability concerns such as retries, caching, rate limits, and validation, see how to build a reliable web scraper.
Use JSON when nested objects, arrays, or machine-to-machine exchange matter. Use CSV when the data is tabular and likely to be reviewed in spreadsheet software. For a human handoff, include a short data dictionary describing each column, its type, permitted missing values, and the source field used to populate it.
Quality checks
A clean-looking export can still be incomplete or structurally wrong. Add checks before declaring the workflow finished:
- Compare the number of extracted records with the number of source rows or expected pages.
- Confirm required fields are present and that their types are consistent.
- Check that URLs are valid and that relative links were resolved as intended.
- Look for unexpected empty values, repeated identifiers, and unusually long text.
- Verify that numeric fields do not contain display labels or currency symbols.
- Open the exported CSV in the intended destination application and inspect special characters.
- Run a JSON parser against the final file rather than relying only on visual formatting.
Keep a small sample of known records for regression testing. If a selector, page layout, or embedded object changes, the sample can reveal the problem before an entire export is accepted. For related diagnostics involving headers, URLs, JSON, JWTs, and encodings, the web developer diagnostics toolkit provides a useful companion workflow.
When to revisit
Revisit this workflow whenever the source page changes its markup, moves data from HTML into JavaScript, changes an embedded JSON structure, adds pagination, or modifies its export behavior. You should also review it when a downstream spreadsheet, database, or API begins rejecting values that previously worked.
Make the process easier to maintain by recording the source URL, extraction date, selector or endpoint assumptions, schema version, and transformation rules. Schedule a small test extraction before a larger run if the data supports a recurring report. When a failure occurs, compare the raw response, parsed records, and final export to identify the stage that changed.
To put the workflow into practice, choose one representative page and create a schema with five to ten required fields. Extract a small sample, inspect whether the data is HTML or JSON, normalize the values, export both JSON and CSV, and run the quality checks above. Once the sample is correct, scale the process to pagination or additional pages. Treat the raw input as evidence, the cleaned records as a defined data product, and every conversion as a handoff that should be tested.
