Regular expressions can quickly extract structured fragments from scraped page text, but only when the pattern matches the target site's real formats and is tested against realistic edge cases. This reference explains practical regex patterns for emails, prices, dates, URLs, phone numbers, IDs, whitespace, and repeated fields, then shows how to estimate match quality, translate patterns between Python and JavaScript, and decide when a scraper needs a safer parser instead.
Overview
Regex for web scraping is most useful for small, well-defined pieces of text inside content you have already selected. Typical examples include finding an email address in a contact block, extracting a product price from visible text, or identifying repeated reference codes in a page section.
It is not a replacement for an HTML parser. Use a DOM parser such as Beautiful Soup, lxml, Cheerio, or a browser locator to select the relevant element first. Then apply a regular expression to that element's text. This two-stage approach reduces false matches from navigation links, scripts, styles, hidden content, and unrelated page regions.
Before collecting data, check that your use of the site is permitted and that your request rate is appropriate. The web scraping legal and ethical checklist is a useful companion for reviewing robots.txt, terms, personal data, and rate limits.
The patterns below are deliberately practical rather than universal. Email syntax, currency formats, phone numbering, and date conventions vary widely. Treat every pattern as a starting point that must be measured against samples from the specific site.
| Field | Starter pattern | Important assumption |
|---|---|---|
\b[A-Za-z0-9.!#$%&'*+/=?^_`{|}~-]+@[A-Za-z0-9-]+(?:\.[A-Za-z0-9-]+)+\b | Matches common addresses with a dotted domain; it is not full email validation. | |
| ISO date | \b\d{4}-\d{2}-\d{2}\b | Finds the shape YYYY-MM-DD, not whether the date exists. |
| Time | \b(?:[01]\d|2[0-3]):[0-5]\d(?:[:][0-5]\d)?\b | Assumes a 24-hour time with optional seconds. |
| URL | https?://[^\s<>"']+ | Captures a URL-shaped string; trim trailing punctuation and validate separately. |
| Phone | \+?[0-9][0-9() .-]{7,}[0-9] | Broadly supports common separators but does not prove a number is valid. |
| Integer or decimal | \b\d+(?:[.,]\d+)?\b | Does not resolve whether a comma is a decimal or thousands separator. |
| HTML ID | \bid=["']([^"']+)["'] | Use on markup only when a DOM parser is not available. |
| Whitespace | \s+ | Useful for normalizing runs of spaces, tabs, and line breaks. |
How to estimate
A useful way to estimate whether a scraping regex is ready is to measure it against a labeled sample instead of judging it by appearance. Collect representative text from the target pages, including normal matches, missing fields, malformed values, nearby distractors, and formatting variations.
- Define the extraction unit. Decide whether one page should produce one value, several values, or an ordered list. For example, a product page might contain one current price but several historical prices in a comparison table.
- Build a test set. Record examples that should match and examples that must not match. Include punctuation, Unicode characters, line breaks, mobile layouts, localized formats, and empty fields where relevant.
- Count the outcomes. A true positive is a desired match. A false positive is an unwanted match. A false negative is a desired value the pattern missed. If a field is required, also count pages where the selector or source text is absent.
- Calculate simple rates. Precision is
true positives / (true positives + false positives). Recall istrue positives / (true positives + false negatives). These numbers describe the test set, not every future page, so keep the sample representative. - Review the cost of errors. Missing one optional phone number may be tolerable. Assigning a sale price to the wrong product can corrupt an entire dataset. Set stricter acceptance criteria for fields with greater downstream impact.
You can test regex online with a regex tester while developing, but do not paste confidential customer data, authentication tokens, or private page content into an unapproved service. Test the final pattern in the same language and regex engine used by the scraper because flags, Unicode behavior, named groups, and escaping rules can differ.
Inputs and assumptions
Work with text, not raw page noise
Extract visible text from the target element and normalize it before matching. Decode HTML entities, convert non-breaking spaces when appropriate, and preserve meaningful separators. Avoid collapsing all whitespace if line boundaries distinguish one record from another.
When matching raw HTML, characters such as <, >, quotes, and ampersands may be encoded or surrounded by markup. A pattern designed for visible text can fail on source HTML. Prefer a parser, and use regex on the parser's output.
Account for Unicode and locales
The meaning of \w, word boundaries, case-insensitive matching, and character classes depends on the engine and flags. If a field may contain accented letters or non-Latin scripts, test those characters explicitly rather than assuming ASCII behavior. For prices, identify the site's conventions first: 1,234.56, 1.234,56, and 1 234,56 require different normalization rules.
Keep patterns bounded
Prefer explicit character classes, length limits, and clear separators over nested wildcards. A pattern such as .* can cross multiple fields and produce surprising matches. Ambiguous nested quantifiers can also cause excessive backtracking on long or adversarial input. Use bounded repetition, anchor patterns where the field has a known position, and test long strings before deployment.
Regex should not be used to parse arbitrarily nested HTML, JSON, or programming languages. Select those structures with a parser, then use regex only for the small text transformation that remains.
Worked examples
Python: emails and repeated values
import re
text = "Contact sales@example.com or support@example.org."
email_pattern = re.compile(
r"\b[A-Za-z0-9.!#$%&'*+/=?^_`{|}~-]+@[A-Za-z0-9-]+(?:\.[A-Za-z0-9-]+)+\b"
)
emails = email_pattern.findall(text)
codes = re.findall(r"\bSKU-[A-Z0-9]{6}\b", "SKU-AB12CD, SKU-9X88QZ")
Python raw strings, written with an r prefix, reduce confusion around backslashes. findall() returns every non-overlapping match. If a pattern contains capturing groups, Python may return group contents rather than the complete match, so use non-capturing groups such as (?:...) when you do not need the subgroup.
Prices require extraction and normalization
Suppose a page consistently displays prices as a dollar sign followed by two decimal places. A narrower pattern is easier to validate:
price_pattern = re.compile(r"(?
This pattern extracts text but does not convert it to a number. Remove the currency symbol and thousands separators in a separate normalization step, then parse the decimal using rules appropriate to the locale. If the site displays prices without fixed decimals, adjust the pattern and add tests for both forms.
JavaScript: dates and flags
const text = "Published 2026-11-05; revised 2026-12-01.";
const datePattern = /\b\d{4}-\d{2}-\d{2}\b/g;
const dates = text.match(datePattern) || [];
In JavaScript, the g flag requests repeated matches. In Python, repeated matching uses methods such as findall() or finditer() instead. Named groups and Unicode property escapes also have engine-specific syntax, so copy a pattern between languages only after checking its flags and test cases.
For scraper projects, put these examples into automated fixtures. The guide to testing web scrapers with Python covers fixtures, mock pages, golden files, and continuous integration checks that help detect pattern drift.
When to recalculate
Revisit a scraping regex whenever the target site's markup or editorial format changes. A visual redesign may alter the selected text, while a less visible content change can add currency symbols, localized dates, tracking URLs, or new labels. Recalculate precision and recall whenever a sample of newly scraped records shows missing values, duplicates, unexpected matches, or a sharp increase in nulls.
Review patterns on a scheduled basis when the scraper is business-critical, and always after changing the browser automation stack, parser, programming language, regex library, or Unicode-related settings. Re-run the suite after adding a new locale or field type. Keep a small regression set containing every bug that has previously reached production.
A practical maintenance loop is:
- Save representative raw samples before changing the pattern.
- Run the existing tests and record the baseline counts.
- Change one pattern or normalization rule at a time.
- Inspect both matches and non-matches, not only the final dataset.
- Promote the change only when required fields remain accurate and unrelated fields are unaffected.
- Monitor missing-field and validation rates after deployment.
If a pattern keeps growing to accommodate exceptions, stop and reconsider the extraction design. A public JSON endpoint, a stable data attribute, or a structured parser may be more reliable than a larger regex. Related workflows for discovering public JSON endpoints and validating scraped data can help you replace fragile text matching with a more maintainable pipeline.