# Apastra Scaffold

> Generate new prompt specs, datasets, evaluators, and suites from templates. Creates correctly-formatted files that pass schema validation.

- Skill: `bintzgavin/apastra-scaffold` (Agent Skill)
- Install (CLI): `npx skillmds@latest add bintzgavin/apastra-scaffold`
- Raw SKILL.md: https://api.skillmd.com/api/skills/bintzgavin/apastra-scaffold/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: bintzgavin (https://skillmd.com/u/bintzgavin)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/bintzgavin/apastra-scaffold

---


# Apastra Scaffold

Quickly generate new PromptOps files. All generated files follow the apastra schemas and will pass validation.

## When to Use

Use this skill when you want to:
- Create a new prompt spec for a new use case
- Add test cases for an existing prompt
- Create an evaluator for a new scoring rule
- Set up a new suite tying everything together

## Before Scaffolding Eval Files

If the user has not already agreed on an eval design, pause and invoke **`apastra-writing-evals`** or ask for the missing design brief. Do not silently choose behavior, cases, and thresholds from a vague request.

Minimum design brief before creating eval files:
- target source or prompt behavior
- real or realistic failure mode
- primary eval surface: outcome, step, or trace
- two starter cases
- grader type and threshold
- what is intentionally out of scope

When you scaffold from a completed design:
- keep the first pass narrow unless the user explicitly chooses broader coverage
- encode deterministic or executable checks before judge rubrics when they fit
- preserve traceability with stable IDs, suite descriptions, tags, and dataset case `metadata.source_relpath` when cases map to instruction files
- use exact tool-call order only when the requirement depends on order; otherwise prefer required/forbidden tools, any-order, subset, or superset-style expectations

## Scaffolding a Prompt Spec

When asked to create a new prompt (e.g., "scaffold a prompt for email classification"):

Create `promptops/prompts/<id>.yaml`:

```yaml
id: <kebab-case-id>
variables:
  <var_name>:
    type: string
template: |
  <The actual prompt text with {{var_name}} placeholders>
output_contract:
  type: object
  properties:
    <output_field>:
      type: string
metadata:
  author: <user or team name>
  intent: <what this prompt does>
  tags:
    - <relevant-tags>
```

### Rules for Prompt Specs
- `id` is **required** and must be unique across all prompt specs
- `id` should be kebab-case and include a version suffix (e.g., `classify-email-v1`)
- `variables` is **required** — defines the input schema as a map of variable names to JSON Schema type objects
- `template` is **required** — the prompt text with `{{variable}}` placeholders
- `output_contract` is optional but recommended — defines expected output structure
- `metadata` is optional — use for organization and discovery

## Scaffolding a Dataset

When asked to create test cases (e.g., "create test cases for the email classifier"):

Create `promptops/datasets/<id>.jsonl` — one JSON object per line:

```jsonl
{"case_id": "<unique-case-id>", "inputs": {"<var>": "<value>"}, "expected_outputs": {"<field>": "<expected>"}, "metadata": {"tags": ["<tag>"]}}
```

### Rules for Datasets
- Use `.jsonl` format (one JSON object per line, NOT a JSON array)
- `case_id` is **required** and must be unique within the dataset
- `inputs` is **required** — keys must match the prompt spec's `variables`
- `expected_outputs` is optional — used by evaluators for checking
- `metadata` is optional — useful for tagging difficulty, domain, etc.
- For onboarding, start with two sharp cases: one happy path and one edge/adversarial/negative-control case
- Aim for 5-10 cases in a smoke dataset after the first two show signal, and 20-50+ in a regression dataset once the behavior is stable
- Include edge cases that match the failure mode: empty inputs, very long inputs, adversarial inputs, ambiguous inputs, noisy context, prior regressions

## Scaffolding an Evaluator

When asked to create a scoring rule (e.g., "create an evaluator that checks for JSON output"):

Create `promptops/evaluators/<id>.yaml`:

```yaml
id: <evaluator-id>
type: <deterministic | schema | judge>
metrics:
  - <metric-name>
description: <what this evaluator checks>
config:
  <evaluator-specific configuration>
```

### Evaluator Types

**deterministic** — rule-based checks:
```yaml
id: keyword-check
type: deterministic
metrics:
  - keyword_recall
description: Checks if output contains expected keywords.
config:
  match_field: should_contain
  case_sensitive: false
```

**schema** — validates output structure:
```yaml
id: json-output-valid
type: schema
metrics:
  - schema_valid
description: Validates that model output is valid JSON matching the output contract.
config:
  schema:
    type: object
    required: ["category", "confidence"]
    properties:
      category:
        type: string
      confidence:
        type: number
```

**judge** — AI-graded evaluation:
```yaml
id: quality-judge
type: judge
metrics:
  - coherence
  - relevance
description: Uses AI judgment to score output quality.
config:
  rubric: |
    Score the output on two dimensions (0-1 each):
    - coherence: Is the text well-structured and readable?
    - relevance: Does the output address the input query?
  model: default
```

### Rules for Evaluators
- `id` is **required** and must be unique
- `type` is **required** — must be one of: `deterministic`, `schema`, `judge`
- `metrics` is **required** — array of metric names this evaluator produces (minimum 1)
- For `judge` evaluators: always version the rubric — changing it changes what the metric means
- For subjective judges: prefer narrow labels or anchored 1/3/5 scores over broad 1-10 scales
- For tool/agent behavior: prefer deterministic trace checks when tool names, arguments, required artifacts, or forbidden calls are observable

## Scaffolding a Suite

When asked to create a test suite (e.g., "create a smoke suite for the email classifier"):

Create `promptops/suites/<id>.yaml`:

```yaml
id: <suite-id>
name: <Human Readable Name>
description: <what this suite tests>
datasets:
  - <dataset-id>
evaluators:
  - <evaluator-id>
model_matrix:
  - default
trials: 1
thresholds:
  <metric>: <minimum-score>
```

### Suite Tiers (Recommended)

| Tier | When to Run | Cases | Trials |
|---|---|---|---|
| Smoke | Every prompt edit | 5-10 | 1 |
| Regression | Before merging | 20-50 | 3 |
| Full | Nightly / on-demand | 50+ | 5 |
| Release | Before shipping | 100+ | 5 |

### Rules for Suites
- `id` is **required** and must be unique
- `name` is **required** — human-readable
- `datasets` is **required** — at least one dataset reference
- `evaluators` is **required** — at least one evaluator reference
- `model_matrix` is **required** — at least one model identifier
- Use `"default"` in `model_matrix` to test against the current IDE agent's model

## Full Scaffold Example

When asked something like "scaffold a complete PromptOps setup for sentiment analysis," create all four files:

1. `promptops/prompts/sentiment-v1.yaml`
2. `promptops/datasets/sentiment-smoke.jsonl`
3. `promptops/evaluators/sentiment-accuracy.yaml`
4. `promptops/suites/sentiment-smoke.yaml`

## Scaffolding a Quick Eval (Single File)

For rapid iteration, scaffold a single quick eval file instead of four separate files.

When asked to create a quick eval (e.g., "scaffold a quick eval for agent-run readiness"):

Create `promptops/evals/<id>.yaml`:

```yaml
id: agent-run-readiness
prompt: |
  Evaluate a completed AI-agent implementation run for release readiness.
  Separate final outcome evidence, critical step evidence, and trace evidence.
  Respond with JSON only:
  {
    "decision": "pass|needs-work",
    "outcome_evidence": [{"id": "...", "verdict": "pass|fail", "reason": "..."}],
    "step_evidence": [{"id": "...", "verdict": "pass|fail", "reason": "..."}],
    "trace_evidence": [{"id": "...", "verdict": "pass|fail", "reason": "..."}],
    "risks": [],
    "next_action": "accept|rerun-validation|inspect-trace|revise-output"
  }

  Final response: {{final_response}}
  Outcome evidence: {{outcome_evidence}}
  Step evidence: {{step_evidence}}
  Trace evidence: {{trace_evidence}}
cases:
  - case_id: validated-change
    metadata:
      evidence_surfaces: [outcome, step, trace]
    inputs:
      final_response: "Updated the quick eval and validation passed."
      outcome_evidence: "[QE-OUTCOME-001] Changed the requested eval file."
      step_evidence: "[QE-STEP-001] Ran quick-eval validation after editing promptops/evals/."
      trace_evidence: "[QE-TRACE-001] validation_status=passed before final response."
    assert:
      - type: is-valid-json-schema
        value:
          type: object
          required: [decision, outcome_evidence, step_evidence, trace_evidence]
      - type: contains-all
        value: ['"outcome_evidence"', '"step_evidence"', '"trace_evidence"', QE-OUTCOME-001, QE-STEP-001, QE-TRACE-001]
thresholds:
  pass_rate: 1.0
```

### When to Use Quick Eval vs Full Suite
- **Quick eval**: 1-5 test cases, simple assertions, rapid iteration, or early evidence gathering
- **Full suite**: reusable evaluators, shared datasets, baseline tracking, regression detection, multiple models/trials, or CI readiness

## Scaffolding a Dataset with Inline Assertions

When the user wants inline assertions on their test cases (skipping the evaluator file), include `assert` arrays directly in the JSONL:

```jsonl
{"case_id": "case-1", "inputs": {"text": "Hello"}, "assert": [{"type": "contains", "value": "Bonjour"}, {"type": "not-contains", "value": "error"}]}
{"case_id": "case-2", "inputs": {"text": ""}, "assert": [{"type": "regex", "value": ".*"}]}
```

Inline assertions work alongside separate evaluator files — they complement each other. Use inline assertions for per-case checks and evaluator files for suite-wide scoring rules.

### Available Assertion Types

Deterministic: `equals`, `contains`, `icontains`, `contains-any`, `contains-all`, `regex`, `starts-with`, `is-json`, `contains-json`, `is-valid-json-schema`.

Model-assisted: `similar`, `llm-rubric`, `factuality`, `answer-relevance`.

Performance: `latency`, `cost`.

Negate any type with `not-` prefix (e.g., `not-contains`, `not-is-json`).