# Holdout Validation

> Structural separation of test authorship from code authorship. Produces behavioral acceptance scenarios that implementation agents cannot see, evaluated independently by Guardian. Use when designing acceptance criteria, validating implementations against intent, or enforcing holdout-set discipline. DO NOT USE FOR: unit test writing (use guardian), implementation (use implementer), verifying task completion (use verification-before-completion), or any other test strategies beyond holdout validation, including integration and performance testing.

- Skill: `karim-bhalwani/holdout-validation` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add karim-bhalwani/holdout-validation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/karim-bhalwani/holdout-validation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- License: MIT
- Author: karim-bhalwani (https://skillmd.com/u/karim-bhalwani)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/karim-bhalwani/holdout-validation

---


# Holdout Validation Skill - Structural Test Separation

> Version: 9.0 | Updated: 01-July-2026 | Architect: Karim Bhalwani | Deps: architect, guardian, verification-before-completion

## Dependencies

Load the following via `read_file` before using this skill. Skills marked ★ have `disable-model-invocation: true` and cannot self-invoke - they **must** be loaded explicitly.

- `~/.copilot/skills/architect/SKILL.md` - spec and holdout scenario authoring patterns (required for the Architect role in this workflow)
- `~/.copilot/skills/guardian/SKILL.md` - holdout evaluation is a Guardian-owned phase; QA patterns apply
- `~/.copilot/skills/verification-before-completion/SKILL.md` ★ - completion gate; confirm all holdout scenarios are evaluated before declaring review complete

## When to Load This Skill

Load this skill when:

- Architect is producing a specification and needs to write holdout acceptance scenarios
- Guardian is reviewing an implementation that has associated holdout files in `.copilot/holdout/`
- Validating that structural blindness between implementation and evaluation is maintained
- Setting up a new project's holdout directory structure

## Overview

When the entity writing the code can also read the tests, the separation between implementation and validation is no longer optional. This skill enforces **holdout-set discipline**: behavioral acceptance scenarios are authored during design, stored separately from the codebase, and evaluated by a structurally independent reviewer (Guardian).

This addresses the documented pattern where reasoning models engage in test gaming - hardcoding return values, rewriting tests to match buggy code, and optimizing for "tests pass" rather than "software works."

## Core Principle

**A model that can see the answer key will use it. Design the system so it cannot.**

## Agent Access Rules

These rules are **structural constraints**, not behavioral suggestions. Reasoning models will use available information regardless of instructions.

### Quick Reference

| Agent Role                                          | Access     |
| --------------------------------------------------- | ---------- |
| **Architect**                                       | WRITE only |
| **Implementation agents** (dev, data, data-sci, ai) | **NONE**   |
| **Guardian**                                        | READ only  |
| **Context-engineer**                                | READ only  |

### Detailed Rules by Role

**Architect** – WRITE only

- Authors holdout scenarios during specification
- Spec references the holdout file but does NOT include scenarios inline

**Implementation agents** (`senior-developer`, `data-engineer`, `data-scientist`, `ai-engineer`) – **NONE**

- MUST NOT read files in `.copilot/holdout/`
- If an implementation agent is given access to or asked to read a file in `.copilot/holdout/`, the agent MUST respond: "Access denied: holdout files are reserved for Guardian evaluation. Proceeding with spec-based testing only." and continue implementation using only the specification.

**Guardian** – READ only

- Loads holdout scenarios during review
- Evaluates implementation against them

**Context-engineer** – READ only

- Tracks holdout pass rates in retrospective documents

### Exception: Missing Holdouts

If holdout scenarios don't exist:

- Implementation agents proceed normally with spec-based testing
- Guardian notes "No holdout scenarios found" in the review
- Guardian recommends Architect provide them for future iterations

## Holdout Scenario Structure

### What Is a Holdout Scenario?

A holdout scenario is an **intent-level validation statement**, not a unit test. It describes what must be true from the perspective of a real user, not what a function should return.

| Unit Test (Instruction)              | Holdout Scenario (Intent)                                                                            |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------- |
| `assert calculate_tax(100) == 7.5`   | A customer in Ontario adding a $100 item to cart sees $107.50 at checkout                            |
| `assert response.status_code == 200` | A logged-in user requesting their profile receives their data within 2 seconds                       |
| `assert len(results) > 0`            | A compliance officer searching for "GDPR violations" finds all flagged records from the last 90 days |

### Scenario Format

```markdown
## Holdout Scenario: [ID]

**Actor:** [Who is performing the action]
**Intent:** [What they are trying to accomplish]
**Preconditions:** [What must be true before the scenario starts]
**Action:** [What the actor does]
**Success Criteria:**

- [Observable outcome 1]
- [Observable outcome 2]
  **Failure Modes:**
- [What should NOT happen]
  **Priority:** [Critical | High | Medium]
```

## Workflow

### During Design (Architect Produces)

1. After completing the specification, the Architect writes 3–10 holdout scenarios per specification document (i.e., per discrete unit of work that has its own spec file)
2. Scenarios focus on **user-observable outcomes**, not implementation details
3. Scenarios are written to `.copilot/holdout/` (or a user-specified directory)
4. Each scenario file is named `HOLDOUT-<feature>.md`, matching the naming convention in the File Layout section
5. The spec references the holdout file but does NOT include the scenarios inline

### During Implementation (Developer Is Blind)

1. Implementation agents receive the spec but **never** receive the holdout scenarios
2. Implementation agents write their own unit and integration tests based on the spec
3. The structural separation ensures tests written by developers validate the spec, not the holdouts
4. If an implementation agent requests access to holdout files, the request is denied

### During Review (Guardian Evaluates)

1. Guardian loads the holdout scenarios from `.copilot/holdout/`
2. For each scenario, Guardian evaluates whether the implementation satisfies the intent
3. Guardian produces a **Holdout Evaluation Report** as part of its review
4. Holdout failures are rated as High severity (they indicate spec-to-intent gaps)
5. When one or more holdout scenarios fail, Guardian MUST block the review from passing and raise a finding requiring the implementation agent to address the gap before completion is declared. Guardian should reference the verification-before-completion skill to confirm all failures are resolved.

### Holdout Evaluation Report Format

```markdown
### Holdout Evaluation

| ID    | Actor    | Intent                            | Status | Evidence                                    |
| ----- | -------- | --------------------------------- | ------ | ------------------------------------------- |
| H-001 | Customer | Add item to cart with correct tax | PASS   | Tax calculation verified in checkout flow   |
| H-002 | Admin    | Export audit log for compliance   | FAIL   | Export function exists but omits timestamps |

**Holdout Pass Rate:** X/Y scenarios passed
**Findings:** [Specific gaps between implementation and user intent]
```

## File Layout

```text
.copilot/holdout/
  HOLDOUT-<feature-1>.md    -- Scenarios for feature 1
  HOLDOUT-<feature-2>.md    -- Scenarios for feature 2
  README.md                 -- Explains holdout directory purpose and access rules
```

The holdout `README.md` should contain:

```markdown
# Holdout Validation Scenarios

**Access Rule:** Implementation agents (senior-developer, data-engineer, data-scientist, ai-engineer)
MUST NOT read files in this directory. These scenarios are exclusively for
Guardian evaluation during review.

**Authored by:** Architect agent during specification phase.
**Evaluated by:** Guardian agent during review phase.
```

## When to Use

- Architect is producing a specification for a non-trivial feature
- Guardian is reviewing an implementation that has an associated spec
- Any time the same agent (or agent pipeline) is writing both code and tests
- When validating that software works for real users, not just for the test suite

## Definition of Done

- [ ] Holdout scenarios written for every feature with a specification
- [ ] Scenarios stored in `.copilot/holdout/`, not in the main codebase
- [ ] Implementation agents confirmed to have no access to holdout files
- [ ] Guardian evaluation report produced with pass/fail per scenario
- [ ] Holdout failures tracked as High severity findings

## Constraints

- Does NOT replace unit or integration tests (those are still written by implementation agents)
- Does NOT contain implementation details or code snippets
- Does NOT get loaded by implementation agents (structural blindness is the mechanism)
- Does NOT override Guardian's existing review process (it extends it)

## Common Pitfalls

- **Writing unit tests as holdouts**: Holdouts are intent-level, not instruction-level. "Function returns 200" is a unit test. "User can log in and see their dashboard" is a holdout.
- **Leaking holdouts into specs**: The spec should reference that holdouts exist, but never include them inline. If the developer can read the spec and infer the holdouts, the separation is weakened.
- **Skipping holdouts for "simple" features**: Simple features still have user intent. Even a config change has a holdout: "The system behaves differently after this config is changed."
- **Guardian skipping holdout evaluation**: If holdout files exist in `.copilot/holdout/`, Guardian MUST evaluate them. Skipping is a review gap.

## Integration Points

| Phase          | Input From                     | Output To                                                               | Context                                            |
| -------------- | ------------------------------ | ----------------------------------------------------------------------- | -------------------------------------------------- |
| Design         | `architect`                    | Holdout scenario files                                                  | Architect writes scenarios after spec              |
| Implementation | Spec (no holdouts)             | `senior-developer` / `data-engineer` / `data-scientist` / `ai-engineer` | Developers work from spec only                     |
| Review         | Holdout files + implementation | `guardian`                                                              | Guardian evaluates implementation against holdouts |
| Tracking       | Holdout evaluation report      | `context-engineer`                                                      | Track holdout pass rates over time                 |

## References

- [Example Holdout Scenario](./references/example-holdout-scenario.md) - Complete worked example showing 3 holdout scenarios for a user authentication feature