Analyze Run

Analyze evaluation run $ARGUMENTS.

Steps

1. Get run overview — row counts, judge models, cell profiles:

   sqlite3 -header -column data/evaluations.db "SELECT profile_name, judge_model, scenario_type, COUNT(*) n, ROUND(AVG(tutor_first_turn_score),1) t0_mean, ROUND(AVG(tutor_last_turn_score),1) last_mean, ROUND(AVG(tutor_development_score),1) dev FROM evaluation_results WHERE run_id LIKE '$ARGUMENTS%' AND tutor_first_turn_score IS NOT NULL GROUP BY profile_name, judge_model, scenario_type ORDER BY profile_name"
   

   Note: SQLite has no STDEV — compute variance as AVG(x²) - AVG(x)², then take sqrt for SD.

2. Check for multi-turn data — if tutor_last_turn_score is populated, report both Turn 0 and last-turn scores:

   sqlite3 -header -column data/evaluations.db "SELECT profile_name, COUNT(tutor_last_turn_score) has_last_turn, COUNT(learner_overall_score) has_learner, COUNT(dialogue_quality_score) has_dq FROM evaluation_results WHERE run_id LIKE '$ARGUMENTS%' GROUP BY profile_name"
   

3. Show summary: N scored, judge model(s), cell means, recognition delta if applicable.

4. If there are both base and recognition cells, compute:

   • Mean difference (recognition - base)
   • Effect size estimate (Cohen's d using pooled SD)
   • Note any ceiling effects (means > 90)

5. Check learner scores if present:

   sqlite3 -header -column data/evaluations.db "SELECT profile_name, ROUND(AVG(learner_overall_score),1) learner_mean, ROUND(AVG(dialogue_quality_score),1) dq_mean FROM evaluation_results WHERE run_id LIKE '$ARGUMENTS%' AND learner_overall_score IS NOT NULL GROUP BY profile_name"
   

6. Flag issues:

   • Mixed judge models (must filter by judge_model for valid comparisons)
   • Low N per cell (< 20 is underpowered)
   • NULL scores (incomplete judging):

     sqlite3 data/evaluations.db "SELECT COUNT(*) incomplete FROM evaluation_results WHERE run_id LIKE '$ARGUMENTS%' AND tutor_first_turn_score IS NULL"
     

   • Missing last-turn scores on multi-turn rows (may need evaluate --multiturn-only)

Score columns reference

Important

• Always filter by judge_model — runs can have rows from multiple judges
• tutor_first_turn_score is the primary tutor score column (NOT overall_score)
• Run IDs can be partial — use LIKE with % suffix
• Legacy cell names (e.g. cell_1) may coexist with canonical names (cell_1_base_single_unified)

Converted and distributed by TomeVault — claim your Tome and manage your conversions.