Eval — Weekly Markdown Report
Pull the trailing N-day window from eval/store.sqlite and emit a Markdown digest suitable for publishing to Notion / Teams / Slack, or for committing under logs/eval-reports/.
How to invoke
make eval-report D=YYYY-MM-DD W=7 # 7-day window to stdout
make eval-report D=YYYY-MM-DD W=14 OUT=logs/eval-week.md # write to file
Direct invocation:
python3 eval/report.py --as-of YYYY-MM-DD --window 7 --out logs/eval-week.md
Behavior
- Fetch all
eval_runsrows in[as_of - window + 1, as_of]. - Pick the latest run per
card_date(so re-runs don't double-count). - Compute coverage (
N / windowdays judged), composite min/max/median, and per-axis medians. - Emit:
- H1 header with the date range
- Coverage + composite stats
- Axis medians table
- Per-day detail table (date, composite, F/N/D/S/C, judge model, truncated notes)
When to use
- Monday morning summary across the previous 7 days.
- Investigating a drift alert (
make eval-drift) — the per-day table shows which axes are dragging. - After a re-baselining workflow, to confirm the new judge/prompt produces consistent quality.
What to tell the user
If OUT is set, mention the file path and a quick cat or preview command. If piping to stdout, show the digest verbatim (the user can scroll the terminal). For Notion/Teams publishing, point at scripts/notify-teams.sh or the Notion MCP — the harness intentionally does not publish on its own.