Apastra Getting Started

Apastra Getting Started

Set up prompt versioning and evaluation in any project. No CI, no cloud, no framework — just files and your IDE agent.

Onboarding playbook

Use onboarding.md when adopting Apastra in another repository with no README context. It phases install → one flagship eval (apastra-writing-evals, apastra-scaffold) → optional baselines / CI. The playbook lives only in that file (not duplicated in the root README).

Then iterate coverage with writing-evals and scaffold promptops/README.md from templates/promptops-README.md.

During onboarding, offer the opt-in provider request logger after installation. If the user chooses it, use the bundled wizard so they—not the agent—select adapters, save location, activation mode, and retention:

.agent/bin/apastra request-log configure

Request logging is never a prerequisite for evals and remains disabled when skipped. See docs/guides/provider-request-logging.md.

What Is Apastra?

Apastra treats AI prompts as versioned software assets. Prompts, test cases, scoring rules, and trace evidence are files or file references in your repo. Your IDE agent runs evaluations, compares results against baselines, and catches regressions — all locally. Codex and Claude Code hooks give the agent a better surface for context, validation feedback, and trace evidence while it works.

Quick Setup

1. Create the promptops directory

mkdir -p promptops/prompts promptops/datasets promptops/evaluators promptops/suites promptops/schemas promptops/policies derived-index/baselines derived-index/regressions

2. Create your first prompt spec

Create promptops/prompts/summarize.yaml:

id: summarize-v1
variables:
  text:
    type: string
  max_length:
    type: string
template: |
  Summarize the following text in {{max_length}} or fewer words.
  Be concise and capture the key points.

  Text: {{text}}
output_contract:
  type: object
  properties:
    summary:
      type: string
metadata:
  author: your-name
  intent: text-summarization
  tags:
    - summarization
    - core

A prompt spec has:

• id: stable identifier (never rename, create a new version instead)
• variables: the inputs the prompt template expects, with JSON Schema types
• template: the actual prompt text with {{variable}} placeholders
• output_contract (optional): JSON Schema describing expected output structure
• metadata (optional): tags, author, intent for organization

3. Create test cases

Create promptops/datasets/summarize-smoke.jsonl — one JSON object per line:

{"case_id": "incident-note", "inputs": {"text": "During the staging deploy, the checkout API returned 500 errors for 12 minutes after a feature flag was enabled. The team rolled back the flag and added a pre-deploy validation check.", "max_length": "25"}, "expected_outputs": {"should_contain": ["checkout", "rollback"]}}
{"case_id": "technical-paragraph", "inputs": {"text": "Machine learning is a subset of artificial intelligence that enables systems to learn and improve from experience without being explicitly programmed. It focuses on developing algorithms that can access data and use it to learn for themselves.", "max_length": "30"}, "expected_outputs": {"should_contain": ["machine learning", "algorithms"]}}
{"case_id": "empty-edge-case", "inputs": {"text": "", "max_length": "10"}, "expected_outputs": {"should_contain": []}}
{"case_id": "long-document", "inputs": {"text": "Climate change refers to long-term shifts in temperatures and weather patterns. These shifts may be natural, but since the 1800s, human activities have been the main driver of climate change, primarily due to the burning of fossil fuels like coal, oil, and gas, which produces heat-trapping gases.", "max_length": "25"}, "expected_outputs": {"should_contain": ["climate"]}}
{"case_id": "multi-topic", "inputs": {"text": "Python is a programming language. JavaScript runs in browsers. Rust focuses on memory safety. Go was created at Google for systems programming.", "max_length": "20"}, "expected_outputs": {"should_contain": ["programming"]}}

Each case has:

• case_id: stable identifier for tracking results across runs
• inputs: values for the prompt template variables
• expected_outputs (optional): values for evaluators to check against

4. Create an evaluator

Create promptops/evaluators/contains-keywords.yaml:

id: contains-keywords
type: deterministic
metrics:
  - keyword_recall
description: Checks if the model output contains expected keywords from the test case.
config:
  match_field: should_contain
  case_sensitive: false

Evaluator types:

• deterministic: exact match, substring check, regex, keyword recall
• schema: validates output against a JSON Schema
• judge: uses another AI model to grade output (version the judge prompt!)

5. Create a smoke suite

Create promptops/suites/summarize-smoke.yaml:

id: summarize-smoke
name: Summarize Smoke Suite
description: Quick sanity check for the summarization prompt.
datasets:
  - summarize-smoke
evaluators:
  - contains-keywords
model_matrix:
  - default
trials: 1
thresholds:
  keyword_recall: 0.6

A suite ties everything together:

• datasets: which test case files to use
• evaluators: which scoring rules to apply
• model_matrix: which models to test against (use "default" for your IDE agent's model)
• thresholds: minimum scores to pass

6. Run your first evaluation

Use the eval skill:

Tell your agent: "Use the eval skill to run the summarize-smoke suite"

Or if you have the eval skill installed, your agent already knows how.

Alternative: Quick Eval (Single File)

If you want to skip creating 4 separate files, use a quick eval instead. A strong first quick eval should still separate what happened, which critical step mattered, and what trace evidence proves the path. Create promptops/evals/agent-run-readiness.yaml:

id: agent-run-readiness
prompt: |
  Evaluate a completed AI-agent implementation run for release readiness.
  Return JSON with decision, outcome_evidence, step_evidence, trace_evidence, risks, and next_action.

  Final response: {{final_response}}
  Outcome evidence: {{outcome_evidence}}
  Step evidence: {{step_evidence}}
  Trace evidence: {{trace_evidence}}
cases:
  - case_id: validated-promptops-change
    metadata:
      evidence_surfaces: [outcome, step, trace]
    inputs:
      final_response: "Updated the quick eval and validation passed."
      outcome_evidence: "[QE-OUTCOME-001] Changed promptops/evals/agent-run-readiness.yaml."
      step_evidence: "[QE-STEP-001] Ran quick-eval validation after editing promptops/evals/."
      trace_evidence: "[QE-TRACE-001] validation_status=passed before final response."
    assert:
      - type: is-valid-json-schema
        value: { type: object, required: [decision, outcome_evidence, step_evidence, trace_evidence] }
      - type: contains-all
        value: ['"outcome_evidence"', '"step_evidence"', '"trace_evidence"', QE-OUTCOME-001, QE-STEP-001, QE-TRACE-001]
thresholds:
  pass_rate: 1.0

Then tell your agent: "Run the agent-run-readiness eval". This is the fastest way to test a prompt while keeping outcome, step, and trace evidence visible.

File Structure

After setup, your project should look like:

promptops/
├── prompts/
│   └── summarize.yaml          # Prompt specs (source of truth)
├── datasets/
│   └── summarize-smoke.jsonl   # Test cases (with optional inline assertions)
├── evaluators/
│   └── contains-keywords.yaml  # Scoring rules (optional if using inline assertions)
├── evals/
│   └── agent-run-readiness.yaml # Quick eval files (prompt + cases + assertions)
├── suites/
│   └── summarize-smoke.yaml    # Test configurations
├── runs/                       # Run artifacts and trace references
├── schemas/                    # JSON schemas (from apastra)
└── policies/                   # Regression policies
derived-index/
├── baselines/                  # Known-good scorecards
└── regressions/                # Regression reports

The deterministic runtime/, runs/, and validators/ directories are shipped under .agent/scripts/apastra/. The skills call those scripts — you don't need to write or modify them.

Next Steps

1. Use the eval skill to run your first evaluation
2. Use the trace skill when a real agent failure or hook event should become eval evidence
3. Use the baseline skill to establish your first baseline
4. Use the scaffold skill to quickly generate new prompt specs
5. Use the validate skill to check file formatting
6. Upgrade to CI: Use the setup-ci skill to add GitHub Actions for PR gating and releases

Checklist

• Created promptops/ directory structure
• Created at least one prompt spec in promptops/prompts/
• Created at least one dataset in promptops/datasets/
• Created at least one evaluator in promptops/evaluators/
• Created at least one suite in promptops/suites/
• Ran first evaluation using the eval skill