ai-evals-course
- 7 skills
- 0 followers
- 14 hours ago last updated
- ▌ Error Analysis · ai-evals-course bundleRun agent-assisted error analysis on a trace store. Help a human build a review interface over Langfuse, organize human observations into failure modes, validate one LLM judge per selected subjective mode, use DocETL to apply judges to trace batches, and emit a corrected failure report.
- ▌ Synthetic Trace Generation · ai-evals-courseGenerate grounded, diverse user interactions and run them against an instrumented application to create a verified trace dataset for error analysis.
- ▌ Agent Profiler · ai-evals-courseReview Cartwheel model call costs and propose measured cost reductions. Use for Homework 10 after profile/results/calls.csv and profile/results/profile.json have been created.
- ▌ Eval Audit · ai-evals-course bundleAudit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).
- ▌ Evals Start · ai-evals-course bundleEntry point for evals. Use when the user asks for help with evals, does not know where to begin, or asks for something no other skill in this plugin matches. Do NOT use when a more specific skill in this plugin already matches; load that skill directly.
- ▌ Error Discovery · ai-evals-course bundleRun error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes.
- ▌ Build Review Interface · ai-evals-course bundleBuild a custom browser-based annotation interface tailored to your data for reviewing LLM traces and collecting structured feedback. Use when you need to build an annotation tool, review traces, or collect human labels.