Regular Expressions Mastery

Regular Expressions — Advanced Patterns and Techniques

Mastering regular expressions for text processing — from advanced syntax through performance optimization, debugging, and cross-language differences.

When to Use

• Parsing or extracting information from unstructured text
• Validating complex input formats (email, URLs, dates)
• Search-and-replace across large codebases
• Building lexers, tokenizers, or simple parsers
• Automating text transformation workflows

Core Patterns Reference

Character Classes

Pattern        Matches
\d             Any digit [0-9]
\w             Word char [a-zA-Z0-9_]
\s             Whitespace [ \t\n\r\f\v]
\b             Word boundary
\D, \W, \S     Negations of above
[aeiou]        Any vowel
[^aeiou]       Any non-vowel (negated class)
[a-f0-9]       Range: a-f or 0-9

Quantifiers

Pattern        Matches       Greedy
a*             Zero or more  Yes (grab all possible)
a+             One or more   Yes
a?             Zero or one   Yes
a*?            Zero or more  No (lazy — as few as possible)
a+?            One or more   No (lazy)
a{3}           Exactly 3
a{2,5}         Between 2 and 5
a{2,}          2 or more

Groups and Backreferences

Pattern               Captures
(abc)                 Capturing group 1
(?:abc)               Non-capturing group
(?P<name>abc)         Named group (Python)
(?P=name)             Backreference to named group
\1                    Backreference to group 1
(?(1)yes|no)          Conditional: if group 1 matched, match "yes", else "no"

Lookahead/Lookbehind (Zero-Width Assertions)

# Lookahead: match X only if followed by Y
# X(?=Y)    — positive lookahead
# X(?!Y)    — negative lookahead

# Lookbehind: match X only if preceded by Y
# (?<=Y)X   — positive lookbehind
# (?<!Y)X   — negative lookbehind

import re

# Match 'foo' only when followed by 'bar'
re.findall(r'foo(?=bar)', 'foobar foobaz')   # → ['foo']

# Match 'foo' only when NOT followed by 'bar'
re.findall(r'foo(?!bar)', 'foobar foobaz')   # → ['foo']

# Match 'bar' only when preceded by 'foo'
re.findall(r'(?<=foo)bar', 'foobar bazbar')  # → ['bar']

# Match USD amounts not preceded by a minus
re.findall(r'(?<![-\$])\$\d+', 'Price: $10, Debt: -$5')  # → ['$10']

Advanced Techniques

Recursive Patterns

# Python's regex doesn't support recursion natively.
# Use regex module (pip install regex) for recursive patterns.
import regex

# Match nested parentheses
pattern = regex.compile(r'\((?:[^()]|(?R))*\)')
pattern.findall('(a(b(c)d)e)')  # → ['(a(b(c)d)e)']

# Match palindromes (the classic recursive challenge)
palindrome = regex.compile(r'(\w)(?:(?R)|(\w?))\1')  # Approximate

Overlapping Matches

# Standard re.findall doesn't find overlapping matches
# Solution: positive lookahead with a capturing group inside
text = 'ababa'
re.findall(r'(?=(aba))', text)  # → ['aba', 'aba']
# Finds both starting at positions 0 and 2

Verbose Mode

# Write readable regex with comments and whitespace
pattern = re.compile(r"""
    ^                    # Start of string
    (?P<protocol>https?) # Protocol (http or https)
    ://
    (?P<domain>          # Domain name
        [\w.-]+          # Subdomain + domain
        \.[a-z]{2,}      # TLD
    )
    (?P<port>:\d+)?      # Optional port
    (?P<path>/[\w/.-]*)? # Optional path
    (?P<query>\?[\w=&]+)?# Optional query string
    $
""", re.VERBOSE | re.IGNORECASE)

Performance Optimization

class RegexOptimizer:
    """Patterns for writing fast regular expressions."""
    
    # 1. Specific over general
    SLOW = r'[a-zA-Z0-9]+@[a-zA-Z0-9]+\.[a-zA-Z]+'
    FAST = r'\w+@\w+\.\w+'  # Same effect, simpler engine
    
    # 2. Avoid catastrophic backtracking
    # BAD: (a|aa)+b  — tries 2^n paths on input 'aaaaaaaaac'
    # FIX: a+b        — linear
    # BAD: (<.*>)+    — catastrophic on nested tags
    # FIX: (<[^>]*>)+ — no backtracking
    
    # 3. Use possessive quantifiers where supported
    # BAD: \d+\b   — backtracks to check boundary
    # FIX: \d++\b  — possessive, no backtracking (Python: no, PCRE: yes)
    
    # 4. Pre-compile, don't recompile
    @staticmethod
    def cached(pattern, flags=0):
        """Cache compiled patterns."""
        if not hasattr(RegexOptimizer, '_cache'):
            RegexOptimizer._cache = {}
        key = (pattern, flags)
        if key not in RegexOptimizer._cache:
            RegexOptimizer._cache[key] = re.compile(pattern, flags)
        return RegexOptimizer._cache[key]

Common Patterns Library

# Email (RFC 5322 simplified)
EMAIL = r"""[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*"""

# URL
URL = r"""https?://(?:[-\w.]|(?:%[\da-fA-F]{2}))+(?::\d+)?(?:/[\w/.~:?#\[\]@!$&'()*+,;=-]*)?/?"""

# IPv4
IPV4 = r"""(?:(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\.){3}(?:25[0-5]|2[0-4]\d|[01]?\d\d?)"""

# ISO 8601 Date
ISO_DATE = r"""\d{4}-(?:0[1-9]|1[0-2])-(?:0[1-9]|[12]\d|3[01])"""

# UUID (all variants)
UUID = r"""[\da-f]{8}-[\da-f]{4}-[1-5][\da-f]{3}-[89ab][\da-f]{3}-[\da-f]{12}"""

# Hex Color
HEX_COLOR = r"""#(?:[0-9a-fA-F]{3}|[0-9a-fA-F]{6})\b"""

# Password: 8+ chars, upper, lower, digit, special
PASSWORD = r"""^(?=.*[a-z])(?=.*[A-Z])(?=.*\d)(?=.*[!@#$%^&*])[A-Za-z\d!@#$%^&*]{8,}$"""

# Code comment extraction
COMMENTS = {
    'python': r'#.*$',
    'javascript': r'//.*$|/\*[\s\S]*?\*/',
    'html': r'<!--[\s\S]*?-->',
}

Debugging Regex

def debug_regex(pattern, text):
    """Debug regex matches step by step."""
    compiled = re.compile(pattern)
    print(f"Pattern: /{pattern}/")
    print(f"Text: {repr(text)}")
    print(f"Match: {compiled.search(text)}")
    print(f"Fullmatch: {compiled.fullmatch(text)}")
    print(f"Findall: {compiled.findall(text)}")
    
    # Show groups
    match = compiled.search(text)
    if match:
        print(f"Groups: {match.groups()}")
        print(f"GroupDict: {match.groupdict()}")
        for i, g in enumerate(match.groups()):
            print(f"  [{i}] = {repr(g)}")

Common Pitfalls

1. Catastrophic backtracking — nested quantifiers on overlapping patterns cause exponential time
2. Failing to escape — . + * ? [ ] ( ) { } \ | ^ $ need \ for literal match
3. Using regex for non-regular languages — regex can't parse HTML, JSON, or nested structures; use a real parser
4. Greedy when lazy needed — (.*) grabs the whole line; (.*?) stops at first match
5. Re.compile in loops — compiling inside a hot loop destroys performance; cache patterns
6. Line boundaries with ^ and $ — default is string boundaries; use re.MULTILINE for line boundaries

Verification Checklist

• Pattern tested against edge cases (empty string, unicode, very long text)
• No catastrophic backtracking risk (test with pathological input)
• Performance measured on expected input size
• Pattern compiled once, not in a loop
• Regex101 or similar tool used during development
• Alternative considered: is a simpler non-regex approach available?

See Also

• pattern-matching-engine — multi-pattern matching at scale
• compiler-interpreter-basics — using regex in lexers
• sql-query-optimization — LIKE/SIMILAR TO patterns vs regex