Requirement Extraction Patterns

Requirement Extraction Patterns

Patterns for parsing requirements from various document formats and sources.

Table of Contents

1. Markdown Formats
2. Issue Tracker Exports
3. Word Documents
4. Excel Spreadsheets
5. PDF Documents

Markdown Formats

Heading-Based Requirements

## REQ-001: User Authentication
The system shall support user authentication via email and password.

### REQ-002: Password Strength
Passwords must be at least 8 characters with mixed case and numbers.

Extraction Pattern:

pattern = r'^#+\s*([A-Z]+-[0-9A-Z-]+):\s*(.+?)$'

Numbered List Requirements

1. **REQ-AUTH-001**: OAuth 2.0 support required
2. **REQ-AUTH-002**: Session timeout after 30 minutes
3. **REQ-DATA-001**: Encrypt sensitive data at rest

Extraction Pattern:

pattern = r'^\d+\.\s*\*\*([A-Z]+-[0-9A-Z-]+)\*\*:\s*(.+)$'

Table-Based Requirements

| ID | Title | Priority | Status |
|----|-------|----------|--------|
| REQ-001 | User Login | High | Approved |
| REQ-002 | Password Reset | Medium | Draft |

Extraction Pattern:

# Skip header rows, parse data rows
pattern = r'\|\s*([A-Z]+-[0-9]+)\s*\|\s*(.+?)\s*\|'

User Story Format

### US-123: Search Products
**As a** customer
**I want to** search for products by name
**So that** I can find items quickly

**Acceptance Criteria:**
- Search box on homepage
- Auto-complete suggestions
- Results in < 1 second

Extraction Pattern:

pattern = r'#+\s*(US-[0-9]+):\s*(.+?)\n\*\*As a\*\*\s+(.+?)\n\*\*I want to\*\*\s+(.+?)\n\*\*So that\*\*\s+(.+?)(?:\n|$)'

Issue Tracker Exports

Jira CSV Export

Issue Key,Summary,Description,Status,Priority
PROJ-123,User Authentication,Implement OAuth login,In Progress,High
PROJ-124,Dashboard UI,Create dashboard view,To Do,Medium

Extraction Script:

import csv

def extract_from_jira_csv(csv_file):
    requirements = []

    with open(csv_file, 'r') as f:
        reader = csv.DictReader(f)
        for row in reader:
            requirements.append({
                'id': row['Issue Key'],
                'title': row['Summary'],
                'description': row.get('Description', ''),
                'status': row.get('Status', ''),
                'priority': row.get('Priority', ''),
                'type': 'requirement',
                'source': csv_file
            })

    return requirements

GitHub Issues JSON Export

[
  {
    "number": 42,
    "title": "Add user authentication",
    "body": "Implement OAuth 2.0 login",
    "labels": ["feature", "authentication"],
    "state": "open"
  }
]

Extraction Script:

import json

def extract_from_github_issues(json_file):
    requirements = []

    with open(json_file, 'r') as f:
        issues = json.load(f)

    for issue in issues:
        # Create ID from issue number
        req_id = f"GH-{issue['number']}"

        requirements.append({
            'id': req_id,
            'title': issue['title'],
            'description': issue.get('body', ''),
            'labels': issue.get('labels', []),
            'status': issue.get('state', ''),
            'type': 'requirement',
            'source': json_file
        })

    return requirements

Word Documents

Using python-docx

from docx import Document

def extract_from_word(docx_file):
    """Extract requirements from Word document."""
    requirements = []
    doc = Document(docx_file)

    for para in doc.paragraphs:
        # Check for requirement ID pattern
        match = re.match(r'^([A-Z]+-[0-9A-Z-]+):\s*(.+)$', para.text)
        if match:
            requirements.append({
                'id': match.group(1),
                'title': match.group(2),
                'source': docx_file,
                'type': 'requirement'
            })

    return requirements

Heading-Based Extraction

def extract_from_word_headings(docx_file):
    """Extract requirements from Word document headings."""
    requirements = []
    doc = Document(docx_file)

    for para in doc.paragraphs:
        # Check if paragraph is a heading
        if para.style.name.startswith('Heading'):
            match = re.match(r'^([A-Z]+-[0-9A-Z-]+):\s*(.+)$', para.text)
            if match:
                requirements.append({
                    'id': match.group(1),
                    'title': match.group(2),
                    'level': para.style.name,
                    'source': docx_file,
                    'type': 'requirement'
                })

    return requirements

Excel Spreadsheets

Using openpyxl

from openpyxl import load_workbook

def extract_from_excel(excel_file, sheet_name='Requirements'):
    """Extract requirements from Excel spreadsheet."""
    requirements = []
    wb = load_workbook(excel_file)
    ws = wb[sheet_name]

    # Assume first row is header
    headers = [cell.value for cell in ws[1]]

    # Find column indices
    id_col = headers.index('ID')
    title_col = headers.index('Title')
    desc_col = headers.index('Description') if 'Description' in headers else None

    # Process data rows
    for row in ws.iter_rows(min_row=2, values_only=True):
        if row[id_col]:  # Skip empty rows
            req = {
                'id': row[id_col],
                'title': row[title_col],
                'source': excel_file,
                'type': 'requirement'
            }

            if desc_col and row[desc_col]:
                req['description'] = row[desc_col]

            requirements.append(req)

    return requirements

PDF Documents

Using PyPDF2

import PyPDF2
import re

def extract_from_pdf(pdf_file):
    """Extract requirements from PDF document."""
    requirements = []

    with open(pdf_file, 'rb') as f:
        reader = PyPDF2.PdfReader(f)

        text = ''
        for page in reader.pages:
            text += page.extract_text()

    # Extract requirements using pattern matching
    pattern = r'([A-Z]+-[0-9A-Z-]+):\s*(.+?)(?=\n[A-Z]+-[0-9A-Z-]+:|\Z)'

    for match in re.finditer(pattern, text, re.DOTALL):
        req_id = match.group(1)
        req_text = match.group(2).strip()

        requirements.append({
            'id': req_id,
            'title': req_text.split('\n')[0],  # First line as title
            'description': req_text,
            'source': pdf_file,
            'type': 'requirement'
        })

    return requirements

Advanced Patterns

Requirements with Metadata

## REQ-AUTH-001: OAuth 2.0 Support
**Priority:** High
**Status:** Approved
**Owner:** Security Team
**Dependencies:** REQ-DATA-001, REQ-NET-003

The system shall support OAuth 2.0 authentication protocol.

Extraction:

def extract_with_metadata(file_path):
    requirements = []

    with open(file_path, 'r') as f:
        content = f.read()

    # Pattern to match requirement block
    pattern = r'#+\s*([A-Z]+-[0-9A-Z-]+):\s*(.+?)\n((?:\*\*.+?\*\*:.+?\n)+)(.+?)(?=\n#+|$)'

    for match in re.finditer(pattern, content, re.DOTALL):
        req_id = match.group(1)
        title = match.group(2)
        metadata_block = match.group(3)
        description = match.group(4).strip()

        # Parse metadata
        metadata = {}
        for meta_line in metadata_block.split('\n'):
            meta_match = re.match(r'\*\*(.+?)\*\*:\s*(.+)', meta_line)
            if meta_match:
                metadata[meta_match.group(1)] = meta_match.group(2)

        requirements.append({
            'id': req_id,
            'title': title,
            'description': description,
            'metadata': metadata,
            'type': 'requirement'
        })

    return requirements

Hierarchical Requirements

# 1. Authentication (REQ-AUTH)
## 1.1 Login (REQ-AUTH-LOGIN)
### 1.1.1 Email Login (REQ-AUTH-LOGIN-001)
### 1.1.2 Social Login (REQ-AUTH-LOGIN-002)
## 1.2 Password Reset (REQ-AUTH-RESET)

Extraction with Hierarchy:

def extract_hierarchical(file_path):
    requirements = []

    with open(file_path, 'r') as f:
        lines = f.readlines()

    hierarchy = []

    for line in lines:
        # Count heading level
        heading_match = re.match(r'^(#+)\s*(.+)$', line)
        if heading_match:
            level = len(heading_match.group(1))
            text = heading_match.group(2)

            # Extract ID if present
            id_match = re.search(r'\(([A-Z]+-[0-9A-Z-]+)\)', text)
            if id_match:
                req_id = id_match.group(1)
                title = re.sub(r'\s*\([A-Z]+-[0-9A-Z-]+\)', '', text).strip()

                # Update hierarchy
                hierarchy = hierarchy[:level-1] + [req_id]

                requirements.append({
                    'id': req_id,
                    'title': title,
                    'level': level,
                    'parent': hierarchy[-2] if len(hierarchy) > 1 else None,
                    'path': ' > '.join(hierarchy),
                    'type': 'requirement'
                })

    return requirements

Validation

Validate Extracted Requirements

def validate_requirements(requirements):
    """Validate extracted requirements."""
    errors = []
    seen_ids = set()

    for req in requirements:
        # Check for required fields
        if not req.get('id'):
            errors.append(f"Missing ID in requirement: {req}")

        if not req.get('title'):
            errors.append(f"Missing title for {req.get('id', 'unknown')}")

        # Check for duplicates
        if req['id'] in seen_ids:
            errors.append(f"Duplicate ID: {req['id']}")
        seen_ids.add(req['id'])

        # Validate ID format
        if not re.match(r'^[A-Z]+-[0-9A-Z-]+$', req['id']):
            errors.append(f"Invalid ID format: {req['id']}")

    return errors

Tips

1. Normalize IDs: Convert all IDs to uppercase for consistency
2. Handle Whitespace: Strip leading/trailing whitespace from extracted text
3. Preserve Context: Keep surrounding text for better understanding
4. Track Source: Always record which file/line the requirement came from
5. Validate Early: Check ID format and uniqueness during extraction
6. Support Multiple Formats: Try multiple patterns if first fails
7. Log Extraction: Keep track of what was extracted vs skipped