Creating Online Evaluations

Author continuously-running online evaluations in Flind AI observability, grounded in real failure modes you've identified. Use when the user wants evaluations that automatically score new generations or whole traces going forward — "create an eval to catch X", "continuously check that responses do Y", "turn these failures into evals", or after reading traces finds a recurring failure. Covers letting the explored data decide how many evals to create, proposing that set for the user to pick, choosing the target and eval type (hog / llm_judge / sentiment), configuring an llm_judge's provider and model, scoping which generations trigger it via conditions, creating disabled, verifying scope, enabling, and backfilling past data. Proposes a sentiment eval when no failure mode is worth catching. Finding and ranking the failure modes worth evaluating is its own job — use exploring-ai-failures first. To debug or manage evaluations that already exist, use exploring-llm-evaluations.

TheHalfMoon Updated

File contents

TheHalfMoon/Flind commit 5bac0f3ba8

Frequently asked questions

npx get-skill add thehalfmoon/creating-online-evaluations