Eval Analyzer

Eval Analyzer

Analyze AILANG evaluation baseline results to identify failure patterns, compare model performance, and generate actionable insights.

Quick Start

Most common usage:

# User says: "Analyze the v0.3.24 eval results"
# This skill will:
# 1. Run eval-analyze to categorize failures (standard eval)
# 2. Run agent KPIs to analyze efficiency (agent eval)
# 3. Generate summary with jq queries
# 4. Identify top failing benchmarks
# 5. Show model performance comparison
# 6. Provide optimization recommendations

For agent evaluation analysis (NEW - optimization focus):

# Step 1: Get efficiency metrics (turns, tokens, cost)
.claude/skills/eval-analyzer/scripts/agent_kpis.sh eval_results/baselines/v0.3.24

# Step 2: Investigate expensive benchmarks
.claude/skills/eval-analyzer/scripts/agent_transcripts.sh eval_results/baselines/v0.3.24 simple_print

# Step 3: Compare Python vs AILANG
./tools/compare_agents.sh eval_results/baselines/v0.3.24

See resources/agent_optimization_guide.md for complete optimization strategies.

When to Use This Skill

Invoke this skill when:

• User asks to "analyze eval results", "check benchmarks", "what's failing"
• After running an eval baseline
• When investigating why benchmark performance changed
• User wants to understand failure patterns or model performance
• Comparing two versions of AILANG

Key Eval Commands

All commands work on baseline directories like eval_results/baselines/v0.3.16/.

1. Quick Overview - eval-matrix

Shows comprehensive statistics with model/language breakdowns.

ailang eval-matrix eval_results/baselines/v0.3.16 0.3.16 | head -60

Shows: Overall stats, per-model performance, per-language breakdown, top error codes.

2. Detailed Analysis - eval-analyze

Categorizes failures and can generate design docs for issues.

# Dry run (no design docs, just analysis)
ailang eval-analyze -results eval_results/baselines/v0.3.16 -dry-run

# Full analysis with design doc generation
ailang eval-analyze -results eval_results/baselines/v0.3.16

⚠️ CRITICAL: Must use -results flag, NOT positional argument!

Output: Categorized failures (compile_error, logic_error, runtime_error) with frequency, affected benchmarks, models, and sample errors.

3. Query-Friendly Summary - eval-summary

Generates JSONL for easy querying with jq.

ailang eval-summary eval_results/baselines/v0.3.16

Output: eval_results/baselines/v0.3.16/summary.jsonl

4. Compare Versions - eval-compare

Shows what changed between two versions.

ailang eval-compare eval_results/baselines/v0.3.15 eval_results/baselines/v0.3.16

5. Fair Comparison (RECOMMENDED) - fair_comparison.py

Use this for accurate version comparisons! The eval-compare command may include duplicates or different model sets. This script normalizes data for apple-to-apples comparison.

.claude/skills/eval-analyzer/scripts/fair_comparison.py

What it does:

• Deduplicates runs (keeps last run per benchmark+model)
• Filters to dev models only (gpt5-mini, claude-haiku-4-5, gemini-2-5-flash)
• AILANG only (ignores Python results)
• Shows net fixes vs regressions
• Per-model breakdown

Output:

v0.4.0: 56/123 = 45.5%
v0.4.2: 59/123 = 48.0%
Delta:  +3 (+2.4pp)

✅ Fixed:   11 benchmarks
❌ Broken:  8 benchmarks
NET:       +3 benchmarks

When to use: Before making decisions based on eval results (e.g., reverting changes, merging PRs).

6. Validate Results - validate_eval_results.py

Check for output corruption and race conditions in eval results.

python3 tools/validate_eval_results.py eval_results/baselines/v0.4.2

Checks:

• Output corruption (fibonacci outputting "All results equal", etc.)
• Duplicate runs for same benchmark+model
• Code hash validation (if available)
• Success rate statistics

When to use: After running eval baselines, especially if results look suspicious.

Agent Analysis Scripts (NEW!)

For agent-based evaluation results (Python vs AILANG comparisons with Claude Code):

1. Agent KPIs - Minimize Tokens & Turns

Shows efficiency metrics for agent runs - key for optimizing language and prompts.

.claude/skills/eval-analyzer/scripts/agent_kpis.sh eval_results/WITH_ALL_FIXES

Output:

• Average turns, tokens, cost by language (Python vs AILANG)
• Most expensive benchmarks (by turns) - candidates for optimization
• Most efficient benchmarks - learn from these
• Success rates and performance comparison

Goal: Minimize agent turns and tokens → indicates clearer prompts and simpler language.

2. Agent Transcripts - View AILANG Conversations

View full agent conversation logs to understand what happened.

# View all transcripts
.claude/skills/eval-analyzer/scripts/agent_transcripts.sh eval_results/WITH_ALL_FIXES

# View only failures
.claude/skills/eval-analyzer/scripts/agent_transcripts.sh eval_results/WITH_ALL_FIXES --failed-only

# View specific benchmark
.claude/skills/eval-analyzer/scripts/agent_transcripts.sh eval_results/WITH_ALL_FIXES fizzbuzz

Output:

• Turn-by-turn conversation showing agent's thought process
• Metrics: turns, tokens, duration
• Success/failure status with error category
• First 100 lines of transcript (with hint to view full)

Use for: Understanding why AILANG solutions fail or take many turns.

3. Python vs AILANG Comparison

Use the existing tools/compare_agents.sh script for side-by-side comparison:

./tools/compare_agents.sh eval_results/WITH_ALL_FIXES

Output:

• Side-by-side metrics table
• Solution code comparison
• Transcripts for failed solutions (automatic)
• Winner indicators for each metric

Standard Eval Workflow (Non-Agent)

Step 1: Get High-Level Overview

# Show overall statistics
ailang eval-matrix eval_results/baselines/v0.3.16 0.3.16 | head -60

Look for:

• Overall success rate (target: >60%)
• AILANG vs Python gap (current: ~54%)
• Model performance variance
• Top error codes

Step 2: Identify Problem Areas

# Categorize all failures
ailang eval-analyze -results eval_results/baselines/v0.3.16 -dry-run

Key metrics:

• compile_error frequency (parse/syntax issues)
• logic_error frequency (wrong output)
• runtime_error frequency (crashes)
• Which benchmarks fail most

Step 3: Query with jq (Custom Analysis)

Use jq queries on summary.jsonl for custom analysis:

# Ensure summary exists
ailang eval-summary eval_results/baselines/v0.3.20

# AILANG-only success rate (all models)
jq -s 'map(select(.lang == "ailang")) |
  {total: length, success: (map(select(.stdout_ok == true)) | length),
   rate: ((map(select(.stdout_ok == true)) | length) * 100.0 / length)}' \
  eval_results/baselines/v0.3.20/summary.jsonl

# Dev models only (useful for prompt testing)
jq -s 'map(select(.lang == "ailang" and
  (.model == "gpt5-mini" or .model == "claude-haiku-4-5" or .model == "gemini-2-5-flash"))) |
  {total: length, success: (map(select(.stdout_ok == true)) | length),
   rate: ((map(select(.stdout_ok == true)) | length) * 100.0 / length)}' \
  eval_results/baselines/v0.3.20/summary.jsonl

# Check specific benchmark across all models
jq -s 'map(select(.benchmark == "explicit_state_threading" and .lang == "ailang")) |
  map({model, success: .stdout_ok, error: .error_category})' \
  eval_results/baselines/v0.3.20/summary.jsonl

# Compare two versions (dev models AILANG-only)
jq -s 'map(select(.lang == "ailang" and
  (.model == "gpt5-mini" or .model == "claude-haiku-4-5" or .model == "gemini-2-5-flash"))) |
  {total: length, success: (map(select(.stdout_ok == true)) | length),
   rate: ((map(select(.stdout_ok == true)) | length) * 100.0 / length)}' \
  eval_results/baselines/v0.3.20/summary.jsonl \
  eval_results/baselines/v0.3.21/summary.jsonl

For more jq patterns, see resources/jq_queries.md

Step 4: Deep Dive with Helper Scripts

Use the provided helper scripts for detailed code inspection:

# Failure analysis with error categorization
.claude/skills/eval-analyzer/scripts/analyze_failures.sh eval_results/baselines/v0.3.16

# Model performance comparison
.claude/skills/eval-analyzer/scripts/compare_models.sh eval_results/baselines/v0.3.16

# Examine specific benchmark failures
.claude/skills/eval-analyzer/scripts/examine_code.sh eval_results/baselines/v0.3.16 api_call_json

Step 4: Compare with Previous Version

# Show regressions and improvements
ailang eval-compare eval_results/baselines/v0.3.15 eval_results/baselines/v0.3.16

Step 5: Generate Insights

Based on the data, identify:

1. Systemic Issues: Categories with >50 failures
2. Model Patterns: Which models struggle with which features
3. Benchmark Hotspots: Benchmarks with 100% failure rate
4. Cost Efficiency: Which models give best success/cost ratio
5. Trends: Improvements or regressions vs previous version

Key Metrics to Track

1. Overall Success Rate: AILANG vs Python gap (target: reduce below 50%)
2. Error Code Distribution:
   • PAR_001 (parse errors) - indicates prompt/syntax issues
   • WRONG_LANG - models writing Python instead of AILANG
   • IMPERATIVE - models using imperative patterns
3. Model Performance: Which models work best with AILANG
4. Benchmark-Level: Which benchmarks consistently fail
5. Cost Efficiency: Success rate per dollar spent
6. Repair Success: Is self-repair helping? (currently low)

Common Issues

Issue 1: "Total Runs: 6" instead of 408

Symptom: eval-analyze only finds 6 results

Cause: Used positional argument instead of -results flag

Solution:

# ❌ WRONG
ailang eval-analyze eval_results/baselines/v0.3.16

# ✅ CORRECT
ailang eval-analyze -results eval_results/baselines/v0.3.16

Issue 2: Summary file not found

Symptom: jq queries fail with "file not found"

Cause: Need to run eval-summary first

Solution:

ailang eval-summary eval_results/baselines/v0.3.16

Issue 3: Design docs not generated

Symptom: eval-analyze shows issues but doesn't create docs

Cause: Using -dry-run flag

Solution: Run without -dry-run to generate design docs

Helper Scripts

The skill includes helper scripts in scripts/ directory:

quick_summary.sh

Fast overview using eval-matrix.

.claude/skills/eval-analyzer/scripts/quick_summary.sh eval_results/baselines/v0.3.16

Output: Overall stats, model performance, language breakdown, top error codes.

analyze_failures.sh

Detailed failure analysis with error categorization.

.claude/skills/eval-analyzer/scripts/analyze_failures.sh eval_results/baselines/v0.3.16 ailang

Output: Overall statistics, error categories, top failing benchmarks, model performance, error codes.

compare_models.sh

Model-by-model performance comparison.

.claude/skills/eval-analyzer/scripts/compare_models.sh eval_results/baselines/v0.3.16

Output: Success rates, first-attempt vs final, cost analysis, token usage, best model per benchmark.

examine_code.sh

Inspect generated code from specific benchmarks.

.claude/skills/eval-analyzer/scripts/examine_code.sh eval_results/baselines/v0.3.16 api_call_json
.claude/skills/eval-analyzer/scripts/examine_code.sh eval_results/baselines/v0.3.16 api_call_json gpt5

Output: Generated code, compiler errors, success status, error codes for each model run.

examine_prompts.sh

View prompts used for specific benchmarks.

.claude/skills/eval-analyzer/scripts/examine_prompts.sh eval_results/baselines/v0.3.16 api_call_json

Output: System prompt, user prompt, success status for benchmark runs.

verify_prompt_accuracy.sh

Check if prompt documentation matches actual implementation.

.claude/skills/eval-analyzer/scripts/verify_prompt_accuracy.sh v0.3.16

Output: Reports false limitations, undocumented features, and prompt-code mismatches.

Use this: After creating new prompt versions to catch documentation bugs!

Resources

Analysis Documents

• resources/failure_analysis_v0.3.16.md - Comprehensive analysis of v0.3.16 eval results with root cause analysis

Common jq Patterns

See resources/jq_queries.md for more query examples and patterns.

Progressive Disclosure

This skill loads information progressively:

1. Always loaded: This SKILL.md file (workflow + commands + scripts)
2. Execute as needed: ailang eval-* commands and helper scripts
3. Load on demand: resources/jq_queries.md, analysis documents

Notes

• All eval commands work offline (no API calls for analysis)
• eval-analyze generates design docs using LLM (default: gpt5)
• Summary JSONL format is stable and queryable
• Use -dry-run to preview before generating design docs
• baseline directories typically at eval_results/baselines/vX.X.X/
• This skill complements post-release skill (which runs baselines)