Query DB

Answer questions about evaluation data by querying data/evaluations.db.

Database schema

evaluation_results (key columns)

evaluation_runs

Critical rules

1. Always filter by judge_model — runs can have rows from multiple judges. Mixing judges gives wrong results.
2. Primary score column is tutor_first_turn_score (NOT overall_score, NOT base_score).
3. Use LIKE '%partial-id%' for run ID matching.
4. For effect sizes: Cohen's d = (M1 - M2) / pooled_SD.
5. Always check for NULLs: WHERE tutor_first_turn_score IS NOT NULL.
6. NEVER run DELETE with LIKE patterns — enumerate specific IDs. Wildcard deletes have destroyed live runs.
7. SQLite has no STDEV — use: SQRT(AVG(x*x) - AVG(x)*AVG(x)) for standard deviation.
8. Legacy cell names (cell_1) coexist with canonical names (cell_1_base_single_unified). Use LIKE 'cell_1%' for cross-run queries.

Common query patterns

-- Run summary (tutor scores by cell and judge)
SELECT profile_name, judge_model, COUNT(*) n,
  ROUND(AVG(tutor_first_turn_score),1) mean,
  ROUND(SQRT(AVG(tutor_first_turn_score*tutor_first_turn_score) - AVG(tutor_first_turn_score)*AVG(tutor_first_turn_score)),1) sd
FROM evaluation_results WHERE run_id LIKE '<id>%' AND tutor_first_turn_score IS NOT NULL
GROUP BY profile_name, judge_model;

-- Recent runs
SELECT id, description, total_tests, status, created_at
FROM evaluation_runs ORDER BY created_at DESC LIMIT 20;

-- Cross-judge comparison
SELECT judge_model, COUNT(*) n, ROUND(AVG(tutor_first_turn_score),1) mean
FROM evaluation_results WHERE run_id LIKE '<id>%' AND tutor_first_turn_score IS NOT NULL
GROUP BY judge_model;

-- Multi-turn summary (first vs last turn)
SELECT profile_name,
  ROUND(AVG(tutor_first_turn_score),1) t0,
  ROUND(AVG(tutor_last_turn_score),1) last,
  ROUND(AVG(tutor_development_score),1) dev
FROM evaluation_results WHERE run_id LIKE '<id>%' AND tutor_last_turn_score IS NOT NULL
GROUP BY profile_name;

-- Learner + dialogue quality
SELECT profile_name,
  ROUND(AVG(learner_overall_score),1) learner,
  ROUND(AVG(dialogue_quality_score),1) dq
FROM evaluation_results WHERE run_id LIKE '<id>%' AND learner_overall_score IS NOT NULL
GROUP BY profile_name;

Now answer the user's question: $ARGUMENTS

Converted and distributed by TomeVault — claim your Tome and manage your conversions.