How to Extract Metadata from Web Pages for SEO Audits

Overview

When people say they want to extract page metadata, they often mean more than just the <title> tag. For a useful SEO metadata scraper, you usually want a compact page record that can be compared across many URLs and revisited over time.

At minimum, a metadata record should capture:

• Final URL after redirects
• HTTP status code
• Page title
• Meta description
• Robots directives from meta tags
• Canonical URL
• Primary heading such as H1
• Open Graph and Twitter card metadata
• Structured data blocks such as JSON-LD
• Language and hreflang signals where relevant

For technical SEO and web diagnostics, extraction is not only about collecting values. It is also about collecting them in a normalized form so you can audit patterns. For example, a canonical tag is much more useful when you also track whether it is self-referential, relative, missing, duplicated, or pointing to another page.

A practical workflow usually follows this sequence:

1. Fetch the URL and store the final response details.
2. Parse the rendered or raw HTML depending on the site.
3. Extract predefined metadata fields with stable selectors.
4. Normalize values for comparison.
5. Flag issues with simple rules.
6. Store the output in CSV, JSON, SQLite, or a database for diffing and reporting.

If your target pages rely heavily on JavaScript, a simple HTTP request may miss important values. In those cases, use a browser-based approach such as Playwright or Puppeteer so you can inspect the final DOM. If you need a broader framework decision, Scrapy vs Playwright: Which Web Scraping Framework Should You Use? and Puppeteer Web Scraping Tutorial: Extract Data from JavaScript-Rendered Pages are useful next reads.

Before scraping at scale, it is also sensible to review crawl permissions and target behavior. For a quick preflight, see robots.txt for Web Scraping: What Developers Should Check First.

Checklist by scenario

Use the scenario that matches your audit. The goal is not to pull everything every time. The goal is to pull the right fields consistently for the decision you need to make.

1. Single-page QA before publishing

This is the simplest scenario and a good starting point for a reusable checklist.

• Confirm the page returns the expected status code.
• Extract the <title> and check that it is present, non-empty, and unique within the batch if you are comparing drafts.
• Extract the meta description and check that it exists and is not duplicated from another page template.
• Extract the canonical tag and verify that it points to the intended public URL.
• Extract the H1 and compare it with the title for accidental duplication or mismatch.
• Extract robots meta tags and confirm they match the publishing intent.
• Extract JSON-LD and verify that it is valid JSON and relevant to the page type.
• Extract Open Graph fields such as og:title, og:description, and og:url.

For this scenario, raw HTML is often enough unless the CMS injects metadata client-side.

2. Sitewide metadata audit across many URLs

This is where normalization matters most. If you want to scrape title and meta description values across a full site, define your output schema before you start.

A useful schema might include:

• input_url
• final_url
• status_code
• title
• title_length
• meta_description
• meta_description_length
• canonical
• canonical_type such as self, cross-page, missing, relative, duplicate
• h1
• h1_count
• meta_robots
• x_robots_tag if you also inspect headers
• jsonld_count
• jsonld_types
• lang

Then apply simple issue rules, for example:

• Missing or empty title
• Duplicate title across different canonical pages
• Missing description
• Canonical points to a non-200 URL
• Multiple canonicals on one page
• No H1 or multiple H1 elements
• Structured data present but unparsable
• Page marked noindex but linked in a production sitemap

If you plan to compare batches over time, store results in a structured format. How to Store Scraped Data: CSV vs JSON vs SQLite vs Postgres is a practical guide for choosing a storage format.

3. Canonical tag extraction for duplicate-content review

Canonical audits deserve their own workflow because the existence of a canonical tag is not enough. You want to understand whether the canonical is helping or creating ambiguity.

For each page, extract:

• The canonical href exactly as found
• The resolved absolute canonical URL
• Whether the canonical matches the final page URL
• Whether the canonical target returns 200, redirects, or errors
• Whether the page includes more than one canonical tag
• Whether faceted or parameterized URLs canonicalize correctly

This is where a canonical tag extractor becomes more useful than a generic scraper. Your checks should explicitly handle relative URLs, protocol differences, trailing slash variation, uppercase paths, and tracking parameters.

A strong rule of thumb: store both the raw value and the normalized value. That makes debugging much easier when templates or middleware rewrite URLs unexpectedly.

4. Structured data audit

A structured data audit should go beyond “JSON-LD exists.” Pages often include multiple blocks, mixed schema types, or stale data copied from old templates.

Extract and store:

• All application/ld+json blocks
• Detected @type values
• Whether each block parses as valid JSON
• Key fields relevant to the page type, such as headline, name, url, datePublished, image, or breadcrumb items
• Whether the structured data matches visible page signals like title, canonical, or breadcrumb path

For a structured data audit, a frequent source of false confidence is collecting the script block without validating it. Invalid commas, comments, or malformed arrays can break a block even if it looks present in view source.

5. JavaScript-rendered pages

Some frameworks inject titles, meta tags, or JSON-LD after the initial document is delivered. In that case, decide whether your audit should inspect the server response, the rendered DOM, or both.

Checklist:

• Fetch raw HTML and record what is present without rendering.
• Render the page in a headless browser and extract the final DOM values.
• Compare the two outputs for differences in title, robots, canonical, and structured data.
• Wait for a stable page state rather than using arbitrary sleep values.
• Capture console errors if metadata is injected by client-side code.

If selectors are unstable, review your extraction strategy. XPath vs CSS Selectors for Web Scraping can help when choosing the least fragile approach.

6. Competitive or external page review

Sometimes you need to extract page metadata from sites you do not control. In that case, keep the workflow respectful and light.

• Start with low request rates.
• Handle redirects and rate limits cleanly.
• Avoid fetching unnecessary assets.
• Log failures instead of retrying aggressively.
• Record blocked or partial responses separately from valid page data.

If targets are difficult to access or respond inconsistently, it may help to review Common Web Scraping Errors and How to Fix Them, How to Rotate User Agents in Web Scrapers, and Web Scraping Proxy Providers Compared: Residential vs Datacenter vs Mobile. Those topics matter more for large-scale collection than for a small SEO spot check, but they affect data quality when extraction fails silently.

What to double-check

Metadata extraction often looks straightforward until edge cases start polluting your audit sheet. These checks prevent most false positives and missed issues.

Normalize URLs carefully

Resolve relative canonicals against the final page URL, not the input URL if redirects occur. Keep both the original canonical string and the fully resolved version. If your report only stores the normalized version, debugging later becomes harder.

Capture final URL and status code

A title without its response context is incomplete. A redirected page may appear to have clean metadata simply because your scraper followed the redirect and reported the destination page. Store at least the input URL, final URL, redirect count, and final status code.

Differentiate missing from empty

A missing meta description is not the same as an empty description tag. The fix may point to a missing template include in one case and a content generation issue in the other.

Inspect headers when relevant

Some directives appear in response headers rather than HTML. If you are auditing indexability, capture header-level signals such as X-Robots-Tag in addition to <meta name="robots">.

Handle duplicates explicitly

Do not assume one value per field. Pages can contain multiple descriptions, multiple canonicals, or repeated Open Graph tags. Record duplicates and flag them, rather than taking the first value and moving on.

Validate structured data separately

Presence is not validity. Parse each JSON-LD block, capture parse errors, and store a compact summary of schema types so you can scan results quickly.

Keep raw HTML samples for failures

When a batch contains strange output, a saved HTML sample from failing rows can reduce debugging time dramatically. You do not need to archive every page forever, but having examples from broken pages is useful.

Common mistakes

Most metadata audits break for predictable reasons. If your output looks too clean or too chaotic, check for these issues first.

Using only view-source assumptions on dynamic sites

Modern sites may inject metadata after load. If you rely only on raw HTML, you may underreport titles, descriptions, or JSON-LD on JavaScript-heavy pages.

Overwriting useful distinctions during cleanup

Normalization helps analysis, but too much cleanup can hide problems. Trimming whitespace is helpful. Converting every URL to a canonicalized internal format before storing the original is less helpful.

Ignoring redirect chains

A page can appear healthy in the final response while the original URL redirects through several hops. That matters in migration audits and canonical reviews.

Checking only one heading

If you extract the first H1 and stop there, you can miss pages with zero H1 elements or several conflicting headings. Count heading tags and store the primary text.

Treating every mismatch as an error

Not every title and H1 difference is a problem. Not every missing description needs to be rewritten immediately. The point of a metadata audit is to surface review candidates, not to label every variation as a defect.

Skipping storage design

When people ask how to scrape a website for SEO metadata, they often focus on selectors first and data structure second. In practice, poor storage choices create more friction than extraction itself. Decide early whether your output is for a quick CSV, a JSON document pipeline, or a database-backed recurring audit.

When to revisit

A metadata extraction workflow is most valuable when reused. The right time to rerun or refine it is usually tied to change, not just calendar frequency.

Revisit your extraction checklist:

• Before seasonal planning cycles or major content pushes
• Before and after CMS, template, or JavaScript framework changes
• After site migrations, domain changes, or URL rewrites
• When SEO reporting starts showing unexpected drops or inconsistencies
• When your scraper begins missing fields because rendering behavior changed
• When you expand the audit to include Open Graph, hreflang, or new schema types

A practical next step is to turn this article into a lightweight runbook:

1. Define your metadata schema.
2. Choose raw HTML, rendered DOM, or both.
3. Extract a 10-URL sample and validate it by hand.
4. Add normalization and issue flags only after the sample looks correct.
5. Store the output in a format that supports comparison over time.
6. Document edge cases so the next audit is faster than the first.

If your audit later expands into broader page extraction, you may also need workflows for tables, lazy-loaded content, or infinite scroll. In that case, How to Parse HTML Tables into Clean CSV and JSON and How to Scrape Infinite Scroll Pages Without Missing Data are useful companion guides.

The core idea is simple: extract page metadata in a way that is repeatable, reviewable, and tied to decisions. If your checklist helps you compare outputs before and after change, it is doing its job.