Agent Health

Agent Health - AI Assistant Instructions

Use these instructions to evaluate and improve your agent using the agent-health CLI and server APIs.

Setup By AI Assistant

Copy this file into your project so your AI coding assistant auto-loads it — either from a local checkout or straight from GitHub.

Claude Code

mkdir -p .claude/skills
curl -o .claude/skills/agent-health.md \
  https://raw.githubusercontent.com/opensearch-project/agent-health/main/docs/skills/AGENT_HEALTH.md
# or, from a checkout: cp docs/skills/AGENT_HEALTH.md .claude/skills/

Auto-discovered as a skill.

Kiro

mkdir -p .kiro/steering
curl -o .kiro/steering/agent-health.md \
  https://raw.githubusercontent.com/opensearch-project/agent-health/main/docs/skills/AGENT_HEALTH.md

Auto-loaded as a steering file.

Cursor / Windsurf / Others

Copy to project root, then tell your assistant: "Read AGENT_HEALTH.md and follow it"

Quick start — what to ask your assistant

Once this file is loaded, just ask:

"Evaluate my agent using agent-health and fix any failures"

Or step by step:

"Run an agent-health benchmark against my agent"
"What failed and why?"
"Fix the high-priority issues"
"Re-run to verify"

What happens under the hood:

1. Assistant runs: npx agent-health benchmark ...
        ↓
2. Gets JSON with failures + judge feedback (matcherResults + improvementStrategies)
        ↓
3. Reads your agent code
        ↓
4. Makes fixes based on the improvement strategies
        ↓
5. Re-runs the benchmark to verify

Prerequisites

Storage (file-based by default; OpenSearch optional)

Agent Health uses file-based storage by default — results are written to a local .agent-health/data/ directory and run / benchmark work with no OpenSearch cluster required. Point at OpenSearch only when you want shared, production-grade persistence (multiple machines / a team dashboard).

To use OpenSearch instead of file storage, set:

SigV4 auth (AWS profile or the default credential chain; es domains or aoss Serverless collections), none (no auth), and basic auth are all supported, configurable via env vars, agent-health.config.json, or the Settings UI — see docs/CONFIGURATION.md.

AWS Credentials for LLM Judge (Required)

The Bedrock LLM judge scores evaluation results and needs AWS credentials to call Bedrock.

.env File

The CLI automatically loads a .env file from the current working directory (the folder where commands are run). Create a .env file with all required variables:

# OpenSearch Storage
OPENSEARCH_STORAGE_ENDPOINT=https://search-my-cluster.us-west-2.es.amazonaws.com
OPENSEARCH_STORAGE_USERNAME=admin
OPENSEARCH_STORAGE_PASSWORD=your-password

# AWS / Bedrock
AWS_PROFILE=your-aws-profile
AWS_REGION=us-west-2
# BEDROCK_MODEL_ID=us.anthropic.claude-sonnet-4-5-20250929-v1:0  # optional, this is the default

Alternatively, pass --env-file <path> to load a .env file from a different location.

Doctor Check

After configuring your environment, verify everything is set up correctly:

npx @opensearch-project/agent-health doctor

This checks OpenSearch connectivity, AWS credentials, and Bedrock access. Fix any reported issues before proceeding.

Commands

# Verify setup
npx @opensearch-project/agent-health doctor --output json

# Generate config files
npx @opensearch-project/agent-health init

# List available resources
npx @opensearch-project/agent-health list agents --output json
npx @opensearch-project/agent-health list test-cases --output json
npx @opensearch-project/agent-health list benchmarks --output json

# Run single test case
npx @opensearch-project/agent-health run -t <test-case-id> -a <agent-key> --output json

# Run full benchmark with export
npx @opensearch-project/agent-health benchmark -n <benchmark-name> -a <agent-key> --export results.json

# Import test cases from JSON file and run benchmark
npx @opensearch-project/agent-health benchmark -f ./test-cases.json -a <agent-key>

# Import with a custom benchmark name
npx @opensearch-project/agent-health benchmark -f ./test-cases.json -n "My Benchmark" -a <agent-key>

# Export test cases from a benchmark (produces import-compatible JSON)
npx @opensearch-project/agent-health export -b <benchmark-name> -o test-cases.json

Code-based test SDK (write tests as files)

Besides JSON test cases, Agent Health has an experimental code-based SDK for writing tests as .eval.js / .eval.ts files (Playwright-style test() / expect() / fixtures / judge() / lifecycle hooks). Run them the same way you run a JSON file:

# `-f` accepts BOTH JSON test-case files and code SDK (.eval.js / .eval.ts) files
npx @opensearch-project/agent-health benchmark -f ./examples/eval-files/demo.eval.js -a my-agent

They produce per-matcher results (matcherResults[]) instead of a single pass/fail. Full guide: docs/SDK.md.

Improvement Workflow

Step 0: Verify Setup

npx @opensearch-project/agent-health doctor --output json

Confirm all checks pass before proceeding. Fix any issues reported.

Step 1: Baseline Evaluation

npx @opensearch-project/agent-health benchmark -n "My Benchmark" -a my-agent --export baseline.json

Step 2: Analyze Failures

Read baseline.json and find entries where passFailStatus: "failed".

Key fields to examine:

• matcherResults[*].reasoning - the canonical judge verdict + reasoning per matcher (llmJudgeReasoning is a legacy, deprecated shim carrying the same text)
• improvementStrategies - Specific recommendations with priority
• trajectory - Step-by-step agent execution

Distinguish failed (the agent answered, judge scored it below threshold) from errored (metricsStatus: "error" — the evaluator itself could not run, e.g. a judge validation error). Errored runs are excluded from pass-rate aggregation; don't treat them as agent misses.

Step 3: Fix Based on Strategies

Focus on priority: "high" issues first:

{
  "category": "Tool Usage",
  "issue": "Agent called search without time filter",
  "recommendation": "Always include start_time and end_time parameters",
  "priority": "high"
}

Read the agent's code and implement the recommendation.

Step 4: Verify Fix

npx @opensearch-project/agent-health benchmark -n "My Benchmark" -a my-agent --export after-fix.json

Compare passRate between baseline and after-fix.

Step 5: Iterate

Repeat until all high-priority issues are resolved.

Output Reference

Benchmark Export Structure

{
  "benchmark": { "id": "...", "name": "...", "testCaseCount": 10 },
  "runs": [{
    "agent": { "key": "my-agent", "name": "My Agent" },
    "passed": 7,
    "failed": 3,
    "passRate": 70,
    "reports": [{
      "testCaseId": "tc-001",
      "passFailStatus": "failed",
      "metricsStatus": "ready",
      "metrics": { "accuracy": 45 },
      "matcherResults": [
        { "method": "llm-judge", "description": "identifies the root cause", "pass": false, "score": 0.45, "reasoning": "The agent failed because..." }
      ],
      "llmJudgeReasoning": "The agent failed because...",
      "improvementStrategies": [{
        "category": "Tool Usage | Reasoning | Completeness",
        "issue": "What went wrong",
        "recommendation": "How to fix it",
        "priority": "high | medium | low"
      }],
      "trajectory": [
        { "type": "thinking", "content": "Agent's reasoning..." },
        { "type": "action", "toolName": "search", "toolArgs": {} },
        { "type": "tool_result", "content": "...", "status": "SUCCESS" },
        { "type": "response", "content": "Final answer..." }
      ]
    }]
  }]
}

Metrics, scoring, and run status

• Metrics are evaluator-defined and heterogeneous. Only the RCA Default evaluator emits a metric named accuracy. Other evaluators emit their own metric names (tool_selection_accuracy, reasoning_coherence, bias_detection, …) and custom evaluators emit whatever their scoringConfig defines. Don't assume metrics.accuracy exists.
• A run's overall score is the rounded mean of whatever numeric metrics the run's evaluator emitted (null/— when none), not a single "accuracy".
• matcherResults[] is the canonical judge surface. Read the judge verdict
  • reasoning from matcherResults entries (method: 'llm-judge'). Each entry is a MatcherResult: { description, pass (boolean), score? (0–1), method, reasoning?, durationMs? }. llmJudgeReasoning is still populated as a backward-compatible shim but is deprecated.
• Run status: passed / failed come from passFailStatus; a separate errored state (metricsStatus: 'error') means the evaluator could not run and is excluded from pass-rate denominators.

Tips

1. Always use --output json for reliable parsing
2. Use --export to get full reports with improvement strategies
3. Fix high-priority issues first - they cause actual failures
4. Compare trajectories between passing and failing cases
5. Make incremental changes - one fix, then re-test
6. Don't over-engineer - fix the specific issue identified

Troubleshooting

Server API Reference

The Agent Health server runs on port 4001 and exposes the following REST APIs. All endpoints return JSON unless noted (SSE endpoints return text/event-stream).

Health & Configuration

Agent Execution & Evaluation

Judge

On /api/judge, modelId is the judge model — the trajectory is already captured, so there is no agent invocation here, and /api/judge does not take a judgeModelId. The agent-vs-judge model distinction (where modelId is the agent's LLM and judgeModelId is the judge's) applies to the run endpoints — POST /api/evaluate and POST /api/storage/evaluation-runs — and the CLI (-m / --judge-model). Agentic-provider judges (pi / agent / agentic / claude-code) pick their own model and ignore the judge model id. The run-level matcherResults[] are assembled by the runner from the judge's response (see Output Reference); the raw /api/judge primitive returns llmJudgeReasoning.

Traces & Metrics

Logs

Observability

Storage: Test Cases

Storage: Benchmarks

Storage: Runs (TestCaseRun)

Storage: Analytics

Storage: Reports

Storage: Evaluators

Storage: Evaluation Runs (code-based SDK)

Comparison

Storage: Admin

Assistant (NEW)

UI Pages

The evaluation surface lives under /evaluations/*. The app uses BrowserRouter — plain paths, no # prefix.

Common Tasks

Ask about a benchmark's results

"What's the pass rate for benchmark bench-xxx? Which test cases are failing and why?" → The assistant will query /api/storage/benchmarks/:id and /api/storage/runs/by-benchmark/:id

Interpret judge reasoning

"Why did test case tc-xxx fail in run run-xxx? What should I fix?" → The assistant reads llmJudgeReasoning and improvementStrategies from the run

Write a test case

"Help me write a test case for testing log search with time filters" → The assistant creates a test case with prompt, context, expectedOutcomes, and labels

Analyze traces

"What are the most expensive LLM calls in run run-xxx?" → The assistant queries /api/traces and /api/metrics/:runId to find token-heavy spans

Compare runs

"Compare the results of run A vs run B in benchmark bench-xxx" → The assistant fetches both runs and diffs pass/fail status, accuracy, and strategies

Diagnose eval-vs-reality gaps

"My agent passes 8/9 in evals but fails in real life — why?" → The assistant investigates: fetches the runs and test cases, examines judge reasoning and expectedOutcomes, checks whether the evaluator is appropriate for the domain, and recommends tightening outcomes or creating a custom evaluator

Evaluator System

How Evaluation Works

1. Agent runs against a test case → produces a trajectory (thinking, action, tool_result, response steps)
2. The judge (LLM) receives the trajectory + expectedOutcomes and scores each outcome
3. Each outcome: Fully achieved = 1.0, Partially = 0.5, Not achieved = 0.0
4. accuracy = (sum of scores / total outcomes) × 100
5. Pass/Fail determined by evaluator's passThreshold + critical failure checks

Built-in Evaluators

Custom Evaluators

Create custom evaluators via the UI or API for domain-specific scoring:

{
  "name": "My Domain Evaluator",
  "systemPrompt": "You are evaluating an agent for [domain]. CRITICAL CRITERIA: ...",
  "scoringConfig": {
    "metrics": [
      { "name": "routing_accuracy", "weight": 0.4, "scale": 100 },
      { "name": "tool_correctness", "weight": 0.4, "scale": 100 },
      { "name": "diagnostic_completeness", "weight": 0.2, "scale": 100 }
    ],
    "passThreshold": 85,
    "scale": 100
  }
}

Assign to a benchmark run via evaluatorId in the run config.

When to Recommend Custom Evaluators

• Domain has hard correctness rules: specific CLI commands, valid syntax, routing tables, API formats
• Default judge is too charitable: agents pass evals but fail in real-world usage (eval-vs-reality gap)
• Multiple independent dimensions: routing correctness AND tool syntax AND diagnostic completeness all matter
• 70% threshold is too low: critical domains (oncall, security, compliance) need 85%+

Common Eval-vs-Reality Gap Causes

1. expectedOutcomes describe results, not procedures: Judge rewards "got the right answer" without verifying "used the right method"
2. 70% threshold too permissive: Missing 30% of steps may be catastrophic in practice
3. No tool-correctness validation: Generic judge can't verify domain-specific tool syntax
4. Trajectory compaction hides details: toolOutput truncated to 1000 chars, content to 500 chars — wrong commands may be invisible to judge
5. Test cases cleaner than reality: Hand-authored prompts include hints that real incidents don't
6. Single-turn evaluation: Real work is iterative; test cases may compress multi-step workflows

Fix Strategies (in order)

1. Tighten expectedOutcomes: Add negative constraints ("do NOT use X"), require specific tool names, specify correct syntax
2. Raise passThreshold: Set to 85%+ for critical domains
3. Use system-tool-usage as secondary evaluator: Catches redundant/wrong tool calls
4. Create domain-specific custom evaluator: Include explicit rubric criteria the generic judge can't verify