Web Scraping Data Validation Guide: Schemas, Missing Fields, Duplicates, and Change Detection

Overview

A scraper can return a successful HTTP response and still produce unusable data. A selector may match the wrong element, a price may become an empty string, pagination may stop early, or a site redesign may shift every field into a different column. These failures are often more difficult to detect than an obvious request error because the output still looks structurally valid.

Web scraping data validation turns assumptions about the source into explicit checks. Instead of asking only whether a page loaded, you ask whether each record has the expected fields, types, formats, relationships, and level of completeness. The result is a repeatable quality gate that can run after every scrape.

A useful validation workflow has five layers:

1. Schema validation: Confirm that each record contains the expected fields and data types.
2. Completeness checks: Measure missing, blank, and placeholder values.
3. Content validation: Check formats, ranges, relationships, and business rules.
4. Consistency checks: Detect duplicates, unexpected row-count changes, and distribution shifts.
5. Source-change detection: Preserve representative HTML, API responses, or screenshots and compare them when results change.

Validation should not be treated as a final cleanup step. It belongs between extraction and export, with failed checks recorded alongside the run metadata. For broader reliability practices, see this guide to building a reliable web scraper and the related guide to testing scrapers with fixtures and regression checks.

What to track

1. Define a schema before collecting data

Start with a field inventory. For each field, document its name, type, whether it is required, its expected format, and any acceptable boundaries. For example, a product record might include source_id as a required string, name as a required non-empty string, price as a non-negative decimal, currency as a short code, available as a boolean, and collected_at as a timestamp.

Keep the schema close to the code or configuration that performs validation. A simplified schema might look like this:

{
  "source_id": {"type": "string", "required": true},
  "name": {"type": "string", "required": true, "min_length": 1},
  "price": {"type": "number", "required": false, "minimum": 0},
  "available": {"type": "boolean", "required": true}
}

The exact library is less important than making the rules explicit. In Python, teams commonly implement these rules with a validation library, a data-frame validation layer, or carefully scoped custom functions. Whatever approach you use, distinguish between a missing field, a null value, an empty string, and a placeholder such as “N/A.” They may require different responses.

2. Measure completeness and missing fields

For every run, record the number and percentage of null, blank, malformed, and default values per field. A field that is usually complete but suddenly falls to 60% coverage deserves investigation, even if the scraper still returns the expected number of rows.

Set thresholds according to the field’s purpose. A required identifier may have a zero-tolerance rule, while an optional description can tolerate missing values. Avoid one universal threshold for every column. Store the baseline from previous successful runs so a new result can be compared with normal behavior.

3. Detect duplicates and identity problems

Choose a stable identity key, such as a source-provided ID, canonical URL, or a carefully constructed combination of fields. Track both exact duplicates and probable duplicates. Exact duplicates may indicate pagination overlap; probable duplicates may occur when titles or addresses are normalized inconsistently.

Also check whether the identity key is unique within a run and whether previously known records suddenly receive new identifiers. If the source does not provide a stable ID, save the canonical URL and a normalized fingerprint of important fields. Do not use a volatile field, such as a timestamp or changing price, as the only identity key.

4. Validate formats, ranges, and relationships

Format checks catch values that technically exist but cannot be trusted. Examples include validating URLs, dates, email-like strings where appropriate, currency amounts, postal codes, and enumerated categories. Range checks can flag negative prices, impossible percentages, unusually long titles, or dates far outside the expected collection period.

Cross-field rules are equally useful. A record marked available=true should normally have an availability signal or inventory value. An end date should not precede a start date. A product with a discount should have a current price that is not greater than its previous price, unless the source’s meaning is documented differently.

5. Track volume and source structure

Record total rows, pages requested, pages successfully parsed, response status categories, and records per page. A sudden drop in rows may indicate pagination failure, a blocked request, a changed result set, or a legitimate source change. These causes should not be treated as equivalent.

Save a small set of representative fixtures from the source. Comparing stored HTML, JSON keys, JSON-LD fields, or selected text snippets can reveal a layout or API change before it affects every record. The guide to extracting structured data from web pages is useful when your validation includes JSON-LD, Open Graph, or microdata.

Cadence and checkpoints

Run basic validation on every scrape. The minimum checkpoint should occur after parsing and before data is written to a production table or exported file. A failed required-field or schema check should stop the export or route the run to quarantine, depending on the importance of the dataset.

Use a layered cadence:

• Every run: Check schema, required fields, types, duplicate keys, row count, and critical formats.
• Daily or weekly: Review completeness trends, category distributions, response outcomes, and validation failures.
• Monthly or quarterly: Revisit thresholds, inspect representative source pages, refresh fixtures, and review whether fields still serve the project.
• After a source change: Re-run a small sample, compare raw responses, and confirm selectors or API mappings before restoring the full schedule.

Keep a run manifest with the collection timestamp, scraper version, source or endpoint, requested page count, output row count, validation results, and a link to retained raw samples. This makes it possible to distinguish a scraper regression from a genuine change in the source. If requests begin failing, use the troubleshooting guide for 403, 429, 500, and timeout responses before interpreting the resulting data as a content change.

How to interpret changes

Not every difference is an error. A row-count increase may reflect a legitimate new listing, while a completeness drop across several unrelated fields is more likely to indicate a parser or source problem. Interpret changes by comparing multiple signals rather than reacting to one number.

A practical diagnosis sequence is:

1. Compare the current row count with recent successful runs.
2. Check request statuses, pagination coverage, and response sizes.
3. Compare missing-field rates by field and by page or category.
4. Inspect duplicate counts and identity-key behavior.
5. Review saved raw samples for changed HTML, labels, JSON keys, or content states.
6. Run the parser against the last known-good fixture and the newest sample.

Patterns help narrow the cause. A failure isolated to one field often points to a selector or normalization issue. A failure affecting every field on later pages may suggest pagination or rate limiting. A sudden change in all records from one category may be a source-side content change. A large increase in duplicate records may indicate overlapping pages or a changed sort order.

Use severity levels so alerts lead to consistent action. For example, a critical failure can block publishing when identifiers are missing or the parser returns no records. A warning can allow export while notifying an operator when an optional description becomes less complete. Document the decision for each rule instead of relying on an unexplained percentage.

When to revisit

Revisit your validation rules on a monthly or quarterly schedule, even when runs appear healthy. Data quality rules can become stale as the source adds fields, changes categories, removes pagination, or changes the meaning of an existing value. Review whether required fields are still required, whether acceptable ranges remain realistic, and whether old exceptions can be removed.

Also revisit the workflow after any scraper code change, source redesign, endpoint migration, unusual row-count change, or sustained shift in missing-field rates. Update fixtures whenever the source structure changes, and review validation-library syntax when dependencies are upgraded. Keep schema versions with the output so historical results can be interpreted correctly.

For a practical recurring checklist, take these actions:

1. Write or update the field schema and identify stable keys.
2. Run a small sample through type, format, range, and relationship checks.
3. Measure nulls, blanks, placeholders, duplicates, and row counts.
4. Compare the results with a recent successful baseline.
5. Inspect raw samples whenever a metric crosses its threshold.
6. Quarantine suspicious output rather than silently overwriting trusted data.
7. Record the decision, scraper version, source behavior, and any rule changes.

A scraper is ready for routine use when its expected data contract is documented, its failures are visible, and its output can be compared with a known baseline. Treat validation as part of the extraction workflow—not as an optional cleanup—and recurring runs become easier to trust, diagnose, and maintain.