# Apastra Getting Started

> Quick setup guide for apastra PromptOps. Create your first prompt spec, dataset, evaluator, and suite in 5 minutes.

- Skill: `bintzgavin/apastra-getting-started` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add bintzgavin/apastra-getting-started`
- Raw SKILL.md: https://api.skillmd.com/api/skills/bintzgavin/apastra-getting-started/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: bintzgavin (https://skillmd.com/u/bintzgavin)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/bintzgavin/apastra-getting-started

---


# Apastra Getting Started

Set up prompt versioning and evaluation in any project. No CI, no cloud, no framework — just files and your IDE agent.

## Onboarding playbook

Use [`onboarding.md`](onboarding.md) when adopting Apastra in **another repository** with **no README context**. It phases install → one flagship eval (**`apastra-writing-evals`**, **`apastra-scaffold`**) → optional baselines / CI. The playbook lives **only** in that file (**not duplicated** in the root README).

Then iterate coverage with [`writing-evals`](../writing-evals) and scaffold `promptops/README.md` from [`templates/promptops-README.md`](templates/promptops-README.md).

During onboarding, offer the opt-in provider request logger after installation. If the user chooses it, use the bundled wizard so they—not the agent—select adapters, save location, activation mode, and retention:

```bash
.agent/bin/apastra request-log configure
```

Request logging is never a prerequisite for evals and remains disabled when skipped. See [`docs/guides/provider-request-logging.md`](../docs/guides/provider-request-logging.md).

## What Is Apastra?

Apastra treats AI prompts as versioned software assets. Prompts, test cases, scoring rules, and trace evidence are files or file references in your repo. Your IDE agent runs evaluations, compares results against baselines, and catches regressions — all locally. Codex and Claude Code hooks give the agent a better surface for context, validation feedback, and trace evidence while it works.

## Quick Setup

### 1. Create the promptops directory

```bash
mkdir -p promptops/prompts promptops/datasets promptops/evaluators promptops/suites promptops/schemas promptops/policies derived-index/baselines derived-index/regressions
```

### 2. Create your first prompt spec

Create `promptops/prompts/summarize.yaml`:

```yaml
id: summarize-v1
variables:
  text:
    type: string
  max_length:
    type: string
template: |
  Summarize the following text in {{max_length}} or fewer words.
  Be concise and capture the key points.

  Text: {{text}}
output_contract:
  type: object
  properties:
    summary:
      type: string
metadata:
  author: your-name
  intent: text-summarization
  tags:
    - summarization
    - core
```

A prompt spec has:
- **id**: stable identifier (never rename, create a new version instead)
- **variables**: the inputs the prompt template expects, with JSON Schema types
- **template**: the actual prompt text with `{{variable}}` placeholders
- **output_contract** (optional): JSON Schema describing expected output structure
- **metadata** (optional): tags, author, intent for organization

### 3. Create test cases

Create `promptops/datasets/summarize-smoke.jsonl` — one JSON object per line:

```jsonl
{"case_id": "incident-note", "inputs": {"text": "During the staging deploy, the checkout API returned 500 errors for 12 minutes after a feature flag was enabled. The team rolled back the flag and added a pre-deploy validation check.", "max_length": "25"}, "expected_outputs": {"should_contain": ["checkout", "rollback"]}}
{"case_id": "technical-paragraph", "inputs": {"text": "Machine learning is a subset of artificial intelligence that enables systems to learn and improve from experience without being explicitly programmed. It focuses on developing algorithms that can access data and use it to learn for themselves.", "max_length": "30"}, "expected_outputs": {"should_contain": ["machine learning", "algorithms"]}}
{"case_id": "empty-edge-case", "inputs": {"text": "", "max_length": "10"}, "expected_outputs": {"should_contain": []}}
{"case_id": "long-document", "inputs": {"text": "Climate change refers to long-term shifts in temperatures and weather patterns. These shifts may be natural, but since the 1800s, human activities have been the main driver of climate change, primarily due to the burning of fossil fuels like coal, oil, and gas, which produces heat-trapping gases.", "max_length": "25"}, "expected_outputs": {"should_contain": ["climate"]}}
{"case_id": "multi-topic", "inputs": {"text": "Python is a programming language. JavaScript runs in browsers. Rust focuses on memory safety. Go was created at Google for systems programming.", "max_length": "20"}, "expected_outputs": {"should_contain": ["programming"]}}
```

Each case has:
- **case_id**: stable identifier for tracking results across runs
- **inputs**: values for the prompt template variables
- **expected_outputs** (optional): values for evaluators to check against

### 4. Create an evaluator

Create `promptops/evaluators/contains-keywords.yaml`:

```yaml
id: contains-keywords
type: deterministic
metrics:
  - keyword_recall
description: Checks if the model output contains expected keywords from the test case.
config:
  match_field: should_contain
  case_sensitive: false
```

Evaluator types:
- **deterministic**: exact match, substring check, regex, keyword recall
- **schema**: validates output against a JSON Schema
- **judge**: uses another AI model to grade output (version the judge prompt!)

### 5. Create a smoke suite

Create `promptops/suites/summarize-smoke.yaml`:

```yaml
id: summarize-smoke
name: Summarize Smoke Suite
description: Quick sanity check for the summarization prompt.
datasets:
  - summarize-smoke
evaluators:
  - contains-keywords
model_matrix:
  - default
trials: 1
thresholds:
  keyword_recall: 0.6
```

A suite ties everything together:
- **datasets**: which test case files to use
- **evaluators**: which scoring rules to apply
- **model_matrix**: which models to test against (use "default" for your IDE agent's model)
- **thresholds**: minimum scores to pass

### 6. Run your first evaluation

Use the **eval** skill:
> Tell your agent: "Use the eval skill to run the summarize-smoke suite"

Or if you have the eval skill installed, your agent already knows how.

### Alternative: Quick Eval (Single File)

If you want to skip creating 4 separate files, use a quick eval instead. A strong first quick eval should still separate what happened, which critical step mattered, and what trace evidence proves the path. Create `promptops/evals/agent-run-readiness.yaml`:

```yaml
id: agent-run-readiness
prompt: |
  Evaluate a completed AI-agent implementation run for release readiness.
  Return JSON with decision, outcome_evidence, step_evidence, trace_evidence, risks, and next_action.

  Final response: {{final_response}}
  Outcome evidence: {{outcome_evidence}}
  Step evidence: {{step_evidence}}
  Trace evidence: {{trace_evidence}}
cases:
  - case_id: validated-promptops-change
    metadata:
      evidence_surfaces: [outcome, step, trace]
    inputs:
      final_response: "Updated the quick eval and validation passed."
      outcome_evidence: "[QE-OUTCOME-001] Changed promptops/evals/agent-run-readiness.yaml."
      step_evidence: "[QE-STEP-001] Ran quick-eval validation after editing promptops/evals/."
      trace_evidence: "[QE-TRACE-001] validation_status=passed before final response."
    assert:
      - type: is-valid-json-schema
        value: { type: object, required: [decision, outcome_evidence, step_evidence, trace_evidence] }
      - type: contains-all
        value: ['"outcome_evidence"', '"step_evidence"', '"trace_evidence"', QE-OUTCOME-001, QE-STEP-001, QE-TRACE-001]
thresholds:
  pass_rate: 1.0
```

Then tell your agent: "Run the agent-run-readiness eval". This is the fastest way to test a prompt while keeping outcome, step, and trace evidence visible.

## File Structure

After setup, your project should look like:

```
promptops/
├── prompts/
│   └── summarize.yaml          # Prompt specs (source of truth)
├── datasets/
│   └── summarize-smoke.jsonl   # Test cases (with optional inline assertions)
├── evaluators/
│   └── contains-keywords.yaml  # Scoring rules (optional if using inline assertions)
├── evals/
│   └── agent-run-readiness.yaml # Quick eval files (prompt + cases + assertions)
├── suites/
│   └── summarize-smoke.yaml    # Test configurations
├── runs/                       # Run artifacts and trace references
├── schemas/                    # JSON schemas (from apastra)
└── policies/                   # Regression policies
derived-index/
├── baselines/                  # Known-good scorecards
└── regressions/                # Regression reports
```

The deterministic `runtime/`, `runs/`, and `validators/` directories are shipped under `.agent/scripts/apastra/`. The skills call those scripts — you don't need to write or modify them.

## Next Steps

1. Use the **eval** skill to run your first evaluation
2. Use the **trace** skill when a real agent failure or hook event should become eval evidence
3. Use the **baseline** skill to establish your first baseline
4. Use the **scaffold** skill to quickly generate new prompt specs
5. Use the **validate** skill to check file formatting
6. **Upgrade to CI**: Use the **setup-ci** skill to add GitHub Actions for PR gating and releases

## Checklist

- [ ] Created `promptops/` directory structure
- [ ] Created at least one prompt spec in `promptops/prompts/`
- [ ] Created at least one dataset in `promptops/datasets/`
- [ ] Created at least one evaluator in `promptops/evaluators/`
- [ ] Created at least one suite in `promptops/suites/`
- [ ] Ran first evaluation using the eval skill