How to Extract Structured Data from Web Pages: JSON-LD, Open Graph, and Microdata

Overview

Web pages commonly expose machine-readable information in several formats. JSON-LD usually appears inside <script type="application/ld+json"> elements. Open Graph and Twitter Card data use <meta> tags, while Microdata places attributes such as itemscope, itemtype, and itemprop directly on HTML elements.

These formats can describe similar concepts, including page titles, descriptions, images, authors, products, articles, and dates. They are not interchangeable, however. A page may contain more than one JSON-LD object, incomplete social metadata, nested Microdata, or values that disagree with the visible content. A reliable structured data scraper therefore needs a collection strategy, a precedence rule, validation checks, and a way to preserve the original source.

Start with the simplest HTTP request that returns the relevant HTML. If the metadata is inserted only after JavaScript runs, use a browser automation tool such as Playwright or Puppeteer, then parse the rendered HTML. For request handling, retries, caching, and rate limits, see the guide to building a reliable web scraper.

Checklist by scenario

Extracting JSON-LD

1. Collect every script[type="application/ld+json"] element rather than stopping at the first match.
2. Trim the script text and parse it as JSON. Record parse failures with the page URL and a short error message.
3. Handle an object, an array of objects, and an object containing an @graph array.
4. Keep @type, @id, and the original object so downstream code can distinguish entities.
5. Flatten only the fields your project needs, such as name, url, image, author, and datePublished.

for script in soup.select('script[type="application/ld+json"]'):
    raw = script.get_text(strip=True)
    try:
        data = json.loads(raw)
    except json.JSONDecodeError:
        continue

    objects = data if isinstance(data, list) else [data]
    for obj in objects:
        if isinstance(obj, dict) and isinstance(obj.get('@graph'), list):
            objects.extend(obj['@graph'])

The example is a starting pattern, not a complete schema parser. Some pages use escaped characters, multiple entities, or values represented as either strings or nested objects. Normalize those variations after collection rather than silently discarding them.

Scraping Open Graph and Twitter Card metadata

1. Select Open Graph properties using meta[property] and Twitter values using meta[name].
2. Store repeated values as arrays when the field can legitimately have more than one value, especially images.
3. Preserve the property name, content value, and source format for troubleshooting.
4. Use a clearly documented fallback order, such as JSON-LD first for a chosen entity field, Open Graph next, then the standard HTML title or description.

def meta_values(soup, attr, prefix):
    result = []
    for tag in soup.find_all('meta'):
        key = tag.get(attr, '')
        if key.startswith(prefix) and tag.get('content'):
            result.append(tag['content'].strip())
    return result

og_title = meta_values(soup, 'property', 'og:title')
twitter_title = meta_values(soup, 'name', 'twitter:title')

Parsing Microdata HTML

Find top-level elements with itemscope and an optional itemtype. For each item, collect descendants with itemprop, converting values according to their element: content for a meta element, href for a link, src for an image, and visible text for ordinary elements. Stop or recurse carefully when a nested itemscope represents a separate property value.

Microdata parsing is more involved than selecting a few attributes. If the project depends heavily on it, use a tested parser or create fixtures covering nested items, repeated properties, missing types, and URL-valued fields. The article on CSS selectors and XPath can help when you need to inspect the surrounding HTML.

Exporting normalized records

Define a stable output contract before scraping at scale. A practical record might include source_url, canonical_url, title, description, image_urls, entity_type, author, published_at, formats_found, and extracted_at. Keep arrays as arrays in JSON. For CSV, choose a convention such as a delimiter-separated image field or a separate child table.

What to double-check

• Page scope: Confirm that you are extracting the intended document, not an error page, login response, or consent shell.
• Encoding: Decode the response using its declared or detected character set before parsing text. This prevents damaged names and descriptions.
• URLs: Resolve relative image and canonical URLs against the page URL, then validate the resulting scheme and host according to your project rules.
• Dates: Preserve the original date string and store a normalized value only when its timezone and format are clear.
• Conflicts: Compare structured fields with visible page content and record disagreements instead of choosing silently.
• Duplicates: Deduplicate entities using a stable identifier such as @id or a carefully chosen canonical URL, not only the name.
• Validation: Check required fields, expected data types, URL syntax, array shape, and maximum lengths before export.

Save a small sample of raw HTML and parsed output for each important page type. These fixtures make it easier to detect a site redesign or parser regression. For a broader testing workflow, see how to test a web scraper with fixtures and regression checks.

Common mistakes

Assuming JSON-LD is always valid JSON: A malformed script can break a naïve parser. Catch errors per block so one bad element does not discard every other format.

Taking the first matching value: Pages can contain multiple images, authors, or entities. Preserve all candidates until your record-level rules identify the correct one.

Confusing metadata with visible truth: Structured data can be stale, incomplete, or intended for a different page state. Treat it as an input to validate, not an unquestionable authority.

Using regular expressions as an HTML parser: Regex is useful for cleanup and validation, but HTML nesting and quoted attributes require an HTML parser. Use the regex guide for web scraping for the narrower tasks where patterns are appropriate.

Ignoring rendered content: A request-based scraper cannot see data added by client-side JavaScript unless the server response already contains it. Compare raw and rendered HTML before choosing browser automation.

Dropping provenance: Store the source URL, format, selector or property name, extraction timestamp, and parser version. Provenance makes corrections and audits practical.

When to revisit

Revisit this checklist before a seasonal planning cycle, a new extraction project, or any change to your browser, HTML parser, schema-handling code, or export format. Also review it after a target site redesign, a sudden increase in empty fields, or a shift from server-rendered to client-rendered pages.

Keep a small monitoring set of representative URLs: one simple page, one page with multiple entities, one page with nested data, and one page known to be incomplete. Run the scraper against these fixtures whenever workflows or tools change. Compare field presence, data types, entity counts, and normalized URLs rather than checking only whether the request succeeded.

Before shipping a new structured data scraper, use this final sequence:

1. Inspect raw HTML and identify every format present.
2. Parse JSON-LD, social metadata, and Microdata independently.
3. Normalize fields with explicit precedence and conflict rules.
4. Validate records and retain warnings alongside successful output.
5. Export a small sample to JSON and CSV, then inspect both manually.
6. Add fixtures for the page patterns that caused problems.

This approach keeps extraction maintainable: formats may vary from page to page, but your collection, validation, and review process remains consistent.