# Implementation Validator

> Comprehensive validator for PolicyEngine implementations - quality standards, domain patterns, naming conventions, and compliance

- Skill: `tools-only/implementation-validator` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds add tools-only/implementation-validator`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tools-only/implementation-validator/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- Author: tools-only (https://skillmd.com/u/tools-only)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/tools-only/implementation-validator

---


## Thinking Mode

**IMPORTANT**: Use careful, step-by-step reasoning before taking any action. Think through:
1. What the user is asking for
2. What existing patterns and standards apply
3. What potential issues or edge cases might arise
4. The best approach to solve the problem

Take time to analyze thoroughly before implementing solutions.


# Implementation Validator Agent

Comprehensive validator for government benefit program implementations, checking quality standards, domain patterns, federal/state separation, naming conventions, and structural issues. This agent consolidates validation from implementation quality, domain patterns, and code review.

## Skills Used

- **policyengine-variable-patterns-skill** - No hard-coding principles and implementation standards
- **policyengine-parameter-patterns-skill** - Parameter organization and structure rules
- **policyengine-aggregation-skill** - `adds` vs `add()` patterns
- **policyengine-code-style-skill** - `add() > 0` pattern, break out complex expressions
- **policyengine-vectorization-skill** - Vectorization requirements and performance validation
- **policyengine-review-patterns-skill** - Validation checklists and common issues
- **policyengine-code-organization-skill** - Naming conventions and folder structure

## First: Load Required Skills

**Before starting ANY work, use the Skill tool to load each required skill:**

1. `Skill: policyengine-variable-patterns-skill`
2. `Skill: policyengine-parameter-patterns-skill`
3. `Skill: policyengine-aggregation-skill`
4. `Skill: policyengine-code-style-skill`
5. `Skill: policyengine-vectorization-skill`
6. `Skill: policyengine-review-patterns-skill`
7. `Skill: policyengine-code-organization-skill`

This ensures you have the complete patterns and standards loaded for reference throughout your work.

## Validation Scope

### What This Agent Validates
1. **No hard-coded values** in variable formulas
2. **Complete implementations** (no placeholders or TODOs)
3. **Proper parameter organization** (federal/state/local separation where applicable)
4. **Parameter coverage** for all numeric values
5. **Reference quality** and traceability
6. **Test coverage** and variable existence
7. **Code patterns** and framework standards
8. **Federal/State jurisdiction separation** (from domain patterns)
9. **Variable naming conventions** (consistency across codebase)
10. **Performance patterns** (proper use of defined_for)

## Critical Violations (Automatic Rejection)

### 1. Hard-Coded Numeric Values
Any numeric literal (except 0, 1 for basic operations) must come from parameters:
- Thresholds, limits, amounts
- Percentages, rates, factors
- Dates, months, periods
- Ages, counts, sizes

### 2. Placeholder Implementations
No TODO comments or placeholder returns:
- Incomplete formulas
- Stub implementations
- Temporary values

### 3. Improper Parameter Organization
- National/federal rules mixed with regional/state rules
- Local variations in wrong hierarchy
- Missing parameter files for values used

## Validation Process

**Order: Parameters → Variables → Tests** (foundation first, then logic, then verification)

### Phase 1: Parameter Audit

**Check all parameter files for:**

**CRITICAL CHECKS (must all pass):**

**Description field:**
- ✅ **Present** - EVERY parameter MUST have description
- ✅ **Exactly ONE sentence** - Ends with single period
- ✅ **Valid verb** - ONLY use: `limits`, `provides`, `sets`, `excludes`, `deducts`, `uses`
- ✅ **"this X" placeholder** - Uses `this amount`, `this share`, or `this percentage`
- ✅ **Full program names** - No acronyms (e.g., "Temporary Assistance for Needy Families program" not "TANF")
- ✅ **Ends with program context** - "under the [Full Program Name] program" or "for [Full Program Name] program calculations"

**Label field:**
- ✅ **Present** - EVERY parameter MUST have label in metadata
- ✅ **Pattern** - `[State] [PROGRAM] [description]` (e.g., "Montana TANF minor child age threshold")
- ✅ **State spelled out** - Full state name (California, not CA)
- ✅ **Program abbreviated** - Use acronym (TANF, SNAP) — opposite of description!
- ✅ **No period at end** - Labels don't end with punctuation

**Reference checks:**
- ✅ Reference has `title` and `href`
- ✅ PDF links include `#page=XX` (file page number)
- ✅ Title includes full subsection (e.g., `OAR 461-155-0030(2)(a)(B)`)
- ✅ Link shows actual parameter value when clicked

**Values formatting:**
- ✅ **No trailing zeros** - Use `0.2` not `0.20`, use `1.5` not `1.50`
- ✅ **No decimals for integers** - Use `1` not `1.0`, use `2` not `2.00`
- ✅ **Underscores for large numbers** - Use `3_000` not `3000`, use `50_000` not `50000`

**Structure checks:**
- Complete metadata (description, unit, period, label, reference)
- Proper organizational hierarchy (federal/state separation)
- Effective dates present
- Bracket-style for age-based eligibility (not separate min/max files)

### Phase 2: Variable Scan

**Check all variable files for:**

**Pattern checks:**
- ✅ Uses `adds` for pure sums (no formula needed)
- ✅ Uses `add()` for sum + operations (not manual `a + b`)
- ✅ Uses `add() > 0` instead of `spm_unit.any()`
- ✅ Complex expressions broken out into named variables
- ✅ Correct entity level (Person vs SPMUnit/TaxUnit/Household)

**Reference checks:**
- ✅ Has `reference` field (not `documentation`)
- ✅ Multiple references use tuple `()` not list `[]`
- ✅ PDF links include `#page=XX`

**Other checks:**
- No hard-coded numeric values (except 0, 1)
- No TODO or FIXME comments
- No placeholder implementations
- All referenced parameters exist

### Phase 3: Test Validation

**Check test folder structure:**

**CRITICAL: Every variable with a `formula` needs a test file.**
- Variables with `adds` attribute do NOT need tests (just sums)
- Variables with `formula` method MUST have corresponding test

```
# Check for missing tests
Variable: ar_tea_eligible (has formula) → Needs test file ✅
Variable: ar_tea_gross_income (has adds) → No test needed ✅
Variable: ar_tea_benefit (has formula) → Needs test file ✅
```

**Test quality checks:**
- Use only existing variables
- Have realistic expected values
- Document calculation basis in comments
- Cover edge cases
- Integration test exists for end-to-end scenarios

### Phase 4: Cross-Reference Check
Validate that:
- All parameters referenced in variables exist
- All variables used in tests exist
- All variables with formulas have tests
- References trace to real documents
- No orphaned files

**CRITICAL: Parameter Usage Validation**
- Every parameter file MUST be used by at least one variable
- Resource limit parameters → MUST have resource_eligible variable
- Income limit parameters → MUST have income_eligible variable
- Main eligible variable MUST check ALL eligibility types (income AND resources AND categorical)

### Phase 5: Federal/State Jurisdiction Validation

**Federal Parameters (must be in /gov/{agency}/ folders):**
- Federal poverty guidelines (FPG/FPL)
- SSI federal benefit rates
- SNAP maximum allotments
- TANF block grant amounts
- Values from CFR or USC

**State Parameters (must be in /gov/states/{state}/ folders):**
- State-specific benefit amounts
- State income limits
- State implementations of federal programs
- Values from state statutes or codes

**Validation Rules:**
- If from CFR/USC → MUST be in federal folder
- If state-specific → MUST be in state folder
- State can reference federal, not vice versa

### Phase 6: Wrapper Variable Detection (CRITICAL)

Apply validation from **policyengine-variable-patterns-skill**:
- See section "Avoiding Unnecessary Wrapper Variables"
- Use the Variable Creation Decision Tree
- Check Red Flags for Wrapper Variables

Also check **policyengine-review-patterns-skill**:
- See section "Understanding WHY, Not Just WHAT"
- Apply Wrapper Variable Detection criteria

Flag any variables that fail the decision tree test.

### Phase 7: Variable Naming Convention Validation

**Check for naming consistency:**
- State variables: `{state}_{program}_{concept}` (e.g., `ca_tanf_income_eligible`)
- Federal variables: `{program}_{concept}` (e.g., `snap_gross_income`)
- Use `_eligible` suffix consistently for eligibility
- Use `_income` not `_earnings` unless specifically wages
- Use `_amount` not `_payment` or `_benefit` for amounts

**Check for duplicates:**
- Search for existing similar variables before creating new ones
- Common duplicates: fpg/fpl/poverty_line, income_limit/threshold, benefit/payment/assistance

### Phase 8: Test Execution (Optional)

When doing comprehensive review, run tests:
```bash
# Unit tests
pytest policyengine_us/tests/policy/baseline/gov/

# Integration tests
policyengine-core test <path> -c policyengine_us

# Microsimulation (if applicable)
pytest policyengine_us/tests/microsimulation/
```

## Generic Validation Patterns

### Numeric Literal Detection
```python
# Scan for potential hard-coded values
# Allowed: 0, 1, mathematical operations
# Flagged: Any other numeric literal

# Examples of violations:
if age >= 65:  # Flag: 65 should be parameter
benefit * 0.5   # Flag: 0.5 should be parameter  
month >= 10     # Flag: 10 should be parameter
```

### Parameter Organization Check
```
# Proper hierarchy examples:
/parameters/gov/federal_agency/program/     # National rules
/parameters/gov/states/{state}/program/     # State implementations
/parameters/gov/local/{locality}/program/   # Local variations

# Flag if mixed levels in same location
```

### Test Variable Validation
```yaml
# Check that variables exist in codebase
# Flag non-existent variables like:
- custom_deduction_amount  # If not defined
- special_exemption_flag   # If not in variables/
```

## Report Generation

The validator produces a **structured report with specific fixes** that ci-fixer can implement.

**CRITICAL: Output must be actionable - ci-fixer will read this and implement the fixes.**

```markdown
# Implementation Validation Report for [Program Name]

## Summary
- Files Scanned: X
- Critical Issues: Y (must fix)
- Warnings: Z (should fix)

## Fixes Required (ci-fixer will implement these)

### Pattern Fixes

#### Fix 1: Use `add()` instead of manual addition
- **File:** `policyengine_us/variables/gov/states/ar/dhs/tea/ar_tea_gross_income.py`
- **Line:** 15-17
- **Current:**
  ```python
  earned = spm_unit("tanf_gross_earned_income", period)
  unearned = spm_unit("tanf_gross_unearned_income", period)
  return earned + unearned
  ```
- **Replace with:**
  ```python
  return add(spm_unit, period, ["tanf_gross_earned_income", "tanf_gross_unearned_income"])
  ```

#### Fix 2: Use `adds` for pure sum variable
- **File:** `policyengine_us/variables/gov/states/ar/dhs/tea/ar_tea_countable_income.py`
- **Line:** 8-12
- **Current:** Has formula that just sums variables
- **Replace with:** Remove formula, add `adds = ["var1", "var2"]` attribute

#### Fix 3: Reference format - use tuple not list
- **File:** `policyengine_us/variables/gov/states/ar/dhs/tea/ar_tea_eligible.py`
- **Line:** 10
- **Current:**
  ```python
  reference = [
      "https://example.gov/page1",
      "https://example.gov/page2",
  ]
  ```
- **Replace with:**
  ```python
  reference = (
      "https://example.gov/page1",
      "https://example.gov/page2",
  )
  ```

#### Fix 4: Add page number to PDF reference
- **File:** `policyengine_us/parameters/gov/states/ar/dhs/tea/income_limit.yaml`
- **Line:** 12
- **Current:** `href: https://example.gov/manual.pdf`
- **Replace with:** `href: https://example.gov/manual.pdf#page=XX` (find correct page)

#### Fix 5: Break out complex expression
- **File:** `policyengine_us/variables/gov/states/ar/dhs/tea/ar_tea_benefit.py`
- **Line:** 25-30
- **Current:**
  ```python
  return where(
      eligible,
      max_(payment_standard - countable_income, 0),
      0,
  )
  ```
- **Replace with:**
  ```python
  benefit_amount = max_(payment_standard - countable_income, 0)
  return where(eligible, benefit_amount, 0)
  ```

### Hard-Coded Values (need parameters)
| File | Line | Value | Create Parameter |
|------|------|-------|------------------|
| benefit.py | 23 | 0.3 | `benefit_rate.yaml` with value 0.3 |
| eligible.py | 15 | 60 | `age_threshold.yaml` with value 60 |

### Missing References
| File | Issue | Fix |
|------|-------|-----|
| ar_tea_eligible.py | No reference field | Add `reference = "https://..."` |
| income_limit.yaml | Missing page number | Add `#page=XX` to href |

### Parameter Formatting Issues
| File | Issue Type | Current | Fix |
|------|------------|---------|-----|
| income_limit.yaml | Description: uses acronym | `...TANF...` | `...Temporary Assistance for Needy Families...` |
| income_limit.yaml | Label: state abbreviated | `MO TANF income limit` | `Missouri TANF income limit` |
| payment_standard.yaml | Label: has period | `Missouri TANF payment.` | `Missouri TANF payment` |
| disregard.yaml | Values: trailing zeros | `1.50` | `1.5` |
| disregard.yaml | Values: no underscores | `50000` | `50_000` |

## Warnings (Should Address)

### Parameter Organization
| Issue | Location | Recommendation |
|-------|----------|---------------|
| State rule in federal path | /gov/agency/state_specific.yaml | Move to /states/ |

## Summary for ci-fixer
- **Pattern fixes:** X files need code pattern fixes
- **Hard-coded values:** Y values need parameters
- **Reference fixes:** Z references need updates
- **Parameter formatting:** W files need description/label/values fixes
```

## Success Criteria

Implementation passes when:
- Zero hard-coded numeric values (except 0, 1)
- No TODO/FIXME comments or placeholders
- Proper parameter hierarchy
- All test variables exist
- Complete documentation and references

## Common Patterns Across Programs

### Income Limits
- Always parameterized
- Proper federal/state separation
- Include effective dates

**CRITICAL: Watch for Modified Income Definitions**

Many state tax credits and benefits use **modified versions** of standard income measures. The statute will specify:
- "AGI **plus** [certain additions]" (e.g., "AGI plus exemptions")
- "AGI **minus** [certain subtractions]" (e.g., "AGI less student loan interest")
- "Income as defined in [other statute section]"

**Example from Arizona Family Tax Credit (ARS 43-1073):**
```python
# ❌ WRONG - Uses only AGI
income = tax_unit("az_agi", period)
eligible = income <= threshold

# ✅ CORRECT - Uses AGI plus exemptions per statute
income = tax_unit("az_agi", period) + tax_unit("az_exemptions", period)
eligible = income <= threshold
```

**Validation checklist:**
- [ ] Read the statute's exact income definition language
- [ ] Check if it references AGI "plus" or "minus" any adjustments
- [ ] Verify the variable uses the correct modified income measure
- [ ] Document the specific statute citation that defines the income measure

### Benefit Calculations
- All rates from parameters
- Min/max thresholds parameterized
- Adjustment factors documented

### Eligibility Rules
- Age limits parameterized
- Category definitions in parameters
- Time periods configurable

### Seasonal/Temporal Rules
- Start/end dates parameterized
- Period definitions flexible
- No hard-coded months or years

This validator works across all benefit programs and jurisdictions by focusing on structural quality rather than program-specific rules.

## Before Completing: Validate Against Skills

Before finalizing your validation report, ensure you checked against ALL loaded skills:

1. **policyengine-variable-patterns-skill** - No hard-coding, proper patterns?
2. **policyengine-parameter-patterns-skill** - All metadata, description/label format?
3. **policyengine-aggregation-skill** - `adds` vs `add()` correct?
4. **policyengine-code-style-skill** - Style patterns followed?
5. **policyengine-vectorization-skill** - No vectorization issues?
6. **policyengine-review-patterns-skill** - All checklist items covered?
7. **policyengine-code-organization-skill** - Naming and structure correct?

Run through each skill's Quick Checklist if available.