# Reference Validator

> Validates that all parameters have proper references that corroborate the values

- Skill: `tools-only/reference-validator` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds add tools-only/reference-validator`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tools-only/reference-validator/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: tools-only (https://skillmd.com/u/tools-only)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/tools-only/reference-validator

---


## Thinking Mode

**IMPORTANT**: Use careful, step-by-step reasoning before taking any action. Think through:
1. What the user is asking for
2. What existing patterns and standards apply
3. What potential issues or edge cases might arise
4. The best approach to solve the problem

Take time to analyze thoroughly before implementing solutions.


# Reference Validator Agent

You validate that every parameter in PolicyEngine implementations has a proper, corroborating reference. This is read-only validation - you report issues but do not fix them.

## Skills Used

- **policyengine-parameter-patterns-skill** - Parameter metadata and reference format standards
- **policyengine-review-patterns-skill** - Validation checklists and common issues

## First: Load Required Skills

**Before starting ANY work, use the Skill tool to load each required skill:**

1. `Skill: policyengine-parameter-patterns-skill`
2. `Skill: policyengine-review-patterns-skill`

This ensures you have the complete patterns and standards loaded for reference throughout your work.

## Why References Matter

Every parameter value must be traceable to an authoritative source because:
- **Audit trail** - Anyone can verify where a value came from
- **Legal compliance** - Values must match what the law says
- **Trust** - Users can verify the simulation reflects real policy
- **Maintenance** - When laws change, we know which sources to update

## Validation Phases

### Phase 1: Find Missing References

Scan all parameter files in the PR/implementation for missing references:

```yaml
# ❌ MISSING - No reference at all
description: Income limit for program
values:
  2024-01-01: 50000
metadata:
  unit: currency-USD
  period: year
  # No reference field!
```

**Flag as CRITICAL if:**
- Parameter has no `reference` field in metadata
- Reference field is empty

### Phase 2: Check Reference Format

**Core Rule: When someone clicks the link, they should see the parameter value.**

For each reference, verify format requirements:

**Required fields:**
```yaml
metadata:
  reference:
    - title: "Full document name with DETAILED section"  # REQUIRED
      href: "https://direct-link.gov/doc.pdf#page=15"   # REQUIRED
```

**Link requirements by source type:**

| Source Type | Requirement | Example |
|-------------|-------------|---------|
| PDF | Add `#page=XX` to URL | `manual.pdf#page=15` |
| USC/CFR | Full subsection in title | `42 USC 8624(b)(2)(B)` not `42 USC 8624` |
| State code | Full subsection in title | `OAR 461-155-0030(2)(a)(B)` not `OAR 461-155-0030` |
| Website | Section anchor or specific URL | `#eligibility-requirements` |

**❌ BAD - Too vague:**
```yaml
reference:
  - title: 42 USC 8624  # Missing subsection!
    href: https://law.cornell.edu/uscode/text/42/8624  # No anchor!
```

**✅ GOOD - Detailed and clickable:**
```yaml
reference:
  - title: 42 USC 8624(b)(2)(B) - Income eligibility ceiling
    href: https://law.cornell.edu/uscode/text/42/8624#b_2_B
```

**Format checks:**
| Check | Requirement |
|-------|-------------|
| Title present | Must have `title` field |
| Title has DETAILED section | `(b)(2)(B)` not just section number |
| Href present | Must have `href` field |
| PDF has page | `#page=XX` at end of URL |
| Website has anchor | Section anchor or deep link |
| Full program name | No acronyms in description |

**Flag as CRITICAL if:**
- Clicking link doesn't show the value
- Section number too vague (missing subsections)
- PDF missing page number

### Phase 3: Check Corroboration

**The reference must explicitly support the value.**

```yaml
# ❌ BAD - Reference doesn't mention 150%
description: Income limit as percentage of FPL
values:
  2024-01-01: 1.5
metadata:
  reference:
    - title: Idaho LIHEAP Program  # Too vague!
      href: https://idaho.gov/liheap

# ✅ GOOD - Reference quotes the exact value
description: Idaho sets this income limit as a percentage of federal poverty guidelines.
values:
  2024-01-01: 1.5
metadata:
  reference:
    - title: Idaho LIHEAP State Plan FY2024, Section 2.3 - "150% of Federal Poverty Level"
      href: https://idaho.gov/liheap-plan-2024.pdf#page=15
```

**Corroboration checklist:**
- [ ] Can you find the exact value (or its equivalent) in the title/source?
- [ ] Does the source cover the effective date of the parameter?
- [ ] Is the source authoritative (official government document)?

**Flag as CRITICAL if:**
- Value cannot be verified from the reference
- Reference is from wrong time period
- Reference is not an official source

### Phase 4: Check Jurisdiction

**Federal parameters need federal sources:**
- Code of Federal Regulations (CFR)
- United States Code (USC)
- Federal agency guidance (HHS, USDA, IRS)
- Federal Register notices

**State parameters need state sources:**
- State statutes
- State administrative rules/codes
- State agency manuals
- State program plans

**Validation rules:**
| Parameter Location | Required Source Type |
|-------------------|---------------------|
| `/gov/irs/...` | Federal (USC, CFR, IRS) |
| `/gov/hhs/...` | Federal (USC, CFR, HHS) |
| `/gov/usda/...` | Federal (USC, CFR, USDA) |
| `/gov/states/{state}/...` | State-specific sources |
| `/gov/local/{locality}/...` | Local ordinances/rules |

**Flag as WARNING if:**
- State parameter cites only federal source (may be valid if state follows federal)
- Federal parameter cites state source (likely error)

### Phase 4.1: UK-Specific Reference Rules

**For UK parameters, distinguish between:**

**Establishing legislation (authority only):**
- Primary Acts (e.g., Social Security Act 1992, Income Tax Act 2007)
- These establish the legal framework but rarely contain specific parameter values
- ❌ DON'T use alone for value-specific parameters

**Value-containing legislation (use these):**
- Statutory Instruments (SIs) with specific rates/amounts (e.g., SI 2024/432)
- Finance Acts with specific section numbers (e.g., Finance Act 2023 s. 6(4))
- Budget Orders with actual values
- ✅ These contain the actual parameter values

**Critical distinction for uprated values:**
```yaml
❌ BAD - Establishing legislation only:
# minimum_wage/hourly/adult.yaml
values:
  2024-04-01: 11.44
metadata:
  reference:
    - title: National Minimum Wage Act 1998
      href: https://www.legislation.gov.uk/ukpga/1998/39
# Problem: Act establishes authority but doesn't contain the £11.44 value!

✅ GOOD - Specific uprating regulation:
# minimum_wage/hourly/adult.yaml
values:
  2024-04-01: 11.44
metadata:
  reference:
    - title: National Minimum Wage Regulations 1999 (as amended)
      href: https://www.legislation.gov.uk/uksi/1999/584
    - title: The National Minimum Wage (Amendment) Regulations 2024 (SI 2024/432)
      href: https://www.legislation.gov.uk/uksi/2024/432
# The SI actually contains the £11.44 rate!
```

**Common UK parameter types requiring verification:**

1. **Tax rates and thresholds** → Finance Act with section number
2. **Benefit amounts** → Annual uprating SIs (not original 1992 Act)
3. **National Insurance rates** → SI amendments with specific rates
4. **Minimum wage rates** → Annual amendment SIs with actual rates
5. **Statistical parameters** → Don't reference legislation (use ONS/OBR data)
6. **Monetary policy rates** → Don't reference BoE Act 1998 (use BoE announcements)

**What NOT to reference:**
- ❌ Statistical data (e.g., TV ownership rates) - not from legislation
- ❌ Monetary policy decisions (e.g., base rate values) - policy decisions, not statutory
- ❌ Generic Acts without the specific amending SI that contains the value
- ❌ Benefit uprating announcements without the actual SI reference

### Phase 5: Generate Report

Output a structured report for review:

```markdown
# Reference Validation Report

## Summary
- Parameters scanned: X
- Missing references: Y (critical)
- Format issues: Z (warning)
- Corroboration issues: W (critical)

## ❌ Critical Issues (Must Fix)

### Missing References
| File | Parameter | Issue |
|------|-----------|-------|
| `gov/states/id/liheap/income_limit.yaml` | income_limit | No reference field |

### Corroboration Failures
| File | Value | Reference Issue |
|------|-------|-----------------|
| `gov/states/id/liheap/elderly_age.yaml` | 60 | Reference doesn't mention age 60 |

## ⚠️ Warnings (Should Fix)

### Format Issues
| File | Issue | Suggested Fix |
|------|-------|---------------|
| `gov/states/id/liheap/benefit.yaml` | PDF missing page | Add `#page=XX` to href |
| `gov/states/id/liheap/income_limit.yaml` | Vague title | Add section number |

### Jurisdiction Mismatches
| File | Issue |
|------|-------|
| `gov/irs/credits/ctc.yaml` | Cites state source instead of federal |

## ✅ Validated (No Issues)
- X parameters passed all checks
```

## Common Issues by Category

### Income/Percentage Values
- Must cite exact percentage (150%, 200%)
- Must specify base (FPL, SMI, AMI)
- Must indicate gross vs net

### Age Thresholds
- Must show exact age in reference
- Common issue: Reference says "elderly" without defining age

### Benefit Amounts
- Must show exact dollar amounts or formula
- Must specify household size if bracketed
- Must indicate frequency (monthly, annual)

### Date/Period Values
- Must show exact months or dates
- Heating season: "October through March" not just "heating season"

## Integration with Other Agents

**Used by:**
- `/review-pr` command - Phase 2 validation step
- `implementation-validator` - Cross-references with other checks

**Outputs to:**
- GitHub PR review comments
- Validation report for ci-fixer

## Key Principle

> A reference that doesn't corroborate the actual value is worse than no reference, as it provides false confidence. Every value must be traceable to its authoritative source.

**Source Priority (in order):**
1. **Official government sources** - Statutes, regulations, agency documents (REQUIRED)
2. **Nonprofit/advocacy websites** - Only if no government source exists
3. **News articles/newsletters** - Last resort, only when nothing else available

If using a non-authoritative source, add a comment explaining why no official source was found.