field-selector-quality-audit
Audit a single NVCA field-selector's quality and produce a structured scorecard.
Security model
- This skill operates on local repository files only — no network access required.
- Source document downloads (Tier 2 checks) use
ensureSourceDocx() which fetches from known template source URLs only.
- No credentials or API keys are needed.
Usage
Run the audit for a specific field-selector:
Audit the field-selector: nvca-certificate-of-incorporation
Or audit all field-selectors:
Audit all NVCA field-selectors and update the quality tracker
Checks
Tier 1: Structural (no source download needed)
| # |
Check |
How |
| S1 |
File inventory |
Does field-selector have metadata.yaml, replacements.json, clean.json? Optional: computed.json, normalize.json, selections.json |
| S2 |
Metadata schema valid |
Run existing validateFieldSelectorMetadata() from src/core/metadata.ts |
| S3 |
Field-to-replacement coverage |
For each field in metadata, is there a replacement key referencing {field_name}? |
| S4 |
Ambiguous keys |
Flag replacement keys < 8 chars without context qualifier (e.g., [name], [its]) |
| S5 |
Smart quote coverage |
Keys with apostrophes should have smart-quote variants (or patcher normalizes — check patcher has normalizeQuotes) |
| S6 |
Source SHA present |
source_sha256 in metadata.yaml |
| S7 |
Test fixture exists |
integration-tests/fixtures/{field-selector-id}-*.json exists |
Tier 2: Behavioral (requires source download)
| # |
Check |
How |
| B1 |
Source download + scan |
Count placeholder-shaped matches using the exact regex \[[_A-Z][_A-Z\s]*\] (underscore-fill or all-capital placeholders) |
| B2 |
Replacement coverage ratio |
For the re-audited NVCA selectors, covered unique matches / total unique matches from the exact B1 regex, target >80%. The legacy all-brackets population and >=70% threshold remains only for selectors outside that audited set. |
| B3 |
Unmatched underscore patterns |
[___+] patterns in source not in replacements.json |
| B4 |
Clean effectiveness |
After clean, no footnotes, no "Note to Drafter", no preamble |
Tier 3: Fill quality (requires fill run)
| # |
Check |
How |
| F1 |
Default-only fill |
Fill with defaults, run verifyOutput, count blank placeholders |
| F2 |
Full-values fill |
Fill with all fields from test fixture, assert all verify checks pass |
| F3 |
Formatting anomaly count |
Check 8 from verifier (single-char underlined runs) |
| F4 |
Zero-match replacement keys |
Keys that existed in replacements.json but matched nothing in the source |
Output: Quality Scorecard
{
"field_selector_id": "nvca-voting-agreement",
"maturity": "beta",
"scores": { "structural": "6/7", "behavioral": "3/4", "fill": "0/4", "total": "9/15" },
"checks": [
{ "id": "S1", "name": "File inventory", "passed": true, "details": "metadata.yaml, replacements.json, clean.json present" },
{ "id": "S7", "name": "Test fixture exists", "passed": false, "details": "No fixture matching integration-tests/fixtures/nvca-voting-agreement-*.json" }
],
"field_coverage": { "metadata_fields": 14, "replacement_refs": 10, "uncovered": 4 },
"recommendations": [
"Add test fixture for fill testing (S7)",
"Add replacement keys for 4 uncovered fields (S3)"
]
}
Maturity Tiers
- scaffold: metadata-only, can't fill
- beta: has replacements + clean, score < 11/15 OR no test fixture
- production: score >= 11/15 AND has test fixture AND has computed.json (if conditional sections exist in source)
Workflow
When running the audit:
- Read
templates/nvca-free-non-redistributable/QUALITY_TRACKER.md for current state
- Run Tier 1 checks (always possible)
- Run Tier 2 checks if source document is available (use
ensureSourceDocx())
- Run Tier 3 checks if a test fixture exists
- Compute scorecard and maturity tier
- Output the scorecard as JSON
- Update the quality tracker if requested
Implementation Notes
- Use
validateFieldSelectorMetadata() from src/core/metadata.ts for S2
- Use
ensureSourceDocx() from src/core/field-selector/downloader.ts for B1-B4
- Use
runFieldSelector() from src/core/field-selector/index.ts for F1-F4
- Bracket pattern detection uses a two-or-more-character
\[[_A-Z][_A-Z\s]+\] shape for re-audited NVCA selectors to avoid counting citations, legal references, and single-letter exhibit references
- Zero-match keys come from
PatchResult.zeroMatchKeys returned by the patcher
- Cross-reference zero-match keys with
cleanConfig.removeRanges and cleanConfig.removeParagraphPatterns to suppress expected zero-matches
Corrected 2026-09-02. The first placeholder-only pattern admitted one-letter
exhibit references such as [E], depressing coverage even though they are not
fill sites. Re-audited NVCA selectors now require at least two characters inside
the bracket; underscore blanks such as [__] remain in scope.
1---2name: field-selector-quality-audit3description: Audit NVCA field-selector quality: check file inventory, metadata schema, field-to-replacement coverage, ambiguous keys, smart quotes, test fixtures, and fill quality. Produces a structured scorecard per field-selector with maturity tier classification. Use when user says "audit field-selector quality," "check field-selector coverage," "field-selector scorecard," or "NVCA field-selector quality."4license: Apache-2.05---67# field-selector-quality-audit89Audit a single NVCA field-selector's quality and produce a structured scorecard.1011## Security model1213- This skill operates on **local repository files only** — no network access required.14- Source document downloads (Tier 2 checks) use `ensureSourceDocx()` which fetches from known template source URLs only.15- No credentials or API keys are needed.1617## Usage1819Run the audit for a specific field-selector:20```21Audit the field-selector: nvca-certificate-of-incorporation22```2324Or audit all field-selectors:25```26Audit all NVCA field-selectors and update the quality tracker27```2829## Checks3031### Tier 1: Structural (no source download needed)3233| # | Check | How |34|---|-------|-----|35| S1 | File inventory | Does field-selector have metadata.yaml, replacements.json, clean.json? Optional: computed.json, normalize.json, selections.json |36| S2 | Metadata schema valid | Run existing `validateFieldSelectorMetadata()` from `src/core/metadata.ts` |37| S3 | Field-to-replacement coverage | For each field in metadata, is there a replacement key referencing `{field_name}`? |38| S4 | Ambiguous keys | Flag replacement keys < 8 chars without context qualifier (e.g., `[name]`, `[its]`) |39| S5 | Smart quote coverage | Keys with apostrophes should have smart-quote variants (or patcher normalizes — check patcher has normalizeQuotes) |40| S6 | Source SHA present | `source_sha256` in metadata.yaml |41| S7 | Test fixture exists | `integration-tests/fixtures/{field-selector-id}-*.json` exists |4243### Tier 2: Behavioral (requires source download)4445| # | Check | How |46|---|-------|-----|47| B1 | Source download + scan | Count placeholder-shaped matches using the exact regex `\[[_A-Z][_A-Z\s]*\]` (underscore-fill or all-capital placeholders) |48| B2 | Replacement coverage ratio | For the re-audited NVCA selectors, covered unique matches / total unique matches from the exact B1 regex, target >80%. The legacy all-brackets population and >=70% threshold remains only for selectors outside that audited set. |49| B3 | Unmatched underscore patterns | `[___+]` patterns in source not in replacements.json |50| B4 | Clean effectiveness | After clean, no footnotes, no "Note to Drafter", no preamble |5152### Tier 3: Fill quality (requires fill run)5354| # | Check | How |55|---|-------|-----|56| F1 | Default-only fill | Fill with defaults, run verifyOutput, count blank placeholders |57| F2 | Full-values fill | Fill with all fields from test fixture, assert all verify checks pass |58| F3 | Formatting anomaly count | Check 8 from verifier (single-char underlined runs) |59| F4 | Zero-match replacement keys | Keys that existed in replacements.json but matched nothing in the source |6061## Output: Quality Scorecard6263```json64{65 "field_selector_id": "nvca-voting-agreement",66 "maturity": "beta",67 "scores": { "structural": "6/7", "behavioral": "3/4", "fill": "0/4", "total": "9/15" },68 "checks": [69 { "id": "S1", "name": "File inventory", "passed": true, "details": "metadata.yaml, replacements.json, clean.json present" },70 { "id": "S7", "name": "Test fixture exists", "passed": false, "details": "No fixture matching integration-tests/fixtures/nvca-voting-agreement-*.json" }71 ],72 "field_coverage": { "metadata_fields": 14, "replacement_refs": 10, "uncovered": 4 },73 "recommendations": [74 "Add test fixture for fill testing (S7)",75 "Add replacement keys for 4 uncovered fields (S3)"76 ]77}78```7980## Maturity Tiers8182- **scaffold**: metadata-only, can't fill83- **beta**: has replacements + clean, score < 11/15 OR no test fixture84- **production**: score >= 11/15 AND has test fixture AND has computed.json (if conditional sections exist in source)8586## Workflow8788When running the audit:89901. Read `templates/nvca-free-non-redistributable/QUALITY_TRACKER.md` for current state912. Run Tier 1 checks (always possible)923. Run Tier 2 checks if source document is available (use `ensureSourceDocx()`)934. Run Tier 3 checks if a test fixture exists945. Compute scorecard and maturity tier956. Output the scorecard as JSON967. Update the quality tracker if requested9798## Implementation Notes99100- Use `validateFieldSelectorMetadata()` from `src/core/metadata.ts` for S2101- Use `ensureSourceDocx()` from `src/core/field-selector/downloader.ts` for B1-B4102- Use `runFieldSelector()` from `src/core/field-selector/index.ts` for F1-F4103- Bracket pattern detection uses a two-or-more-character `\[[_A-Z][_A-Z\s]+\]` shape for re-audited NVCA selectors to avoid counting citations, legal references, and single-letter exhibit references104- Zero-match keys come from `PatchResult.zeroMatchKeys` returned by the patcher105- Cross-reference zero-match keys with `cleanConfig.removeRanges` and `cleanConfig.removeParagraphPatterns` to suppress expected zero-matches106107> **Corrected 2026-09-02.** The first placeholder-only pattern admitted one-letter108> exhibit references such as `[E]`, depressing coverage even though they are not109> fill sites. Re-audited NVCA selectors now require at least two characters inside110> the bracket; underscore blanks such as `[__]` remain in scope.
open-agreements