build-with-dhiraj Evaluating Agent Behavior
Behavioral evaluations (evals) validate the agent's decision-making, such as tool choice and steerability.
Workflow Decision Tree
- Does it need validation?: If a prompt or tool change affects decision-making, it requires an eval.
- Choosing the Rig:
- appEvalTest (AppRig): For UI-heavy or complex interactive flows.
- evalTest (TestRig): For logic-focused behavior.
- Policy Setting:
- USUALLY_PASSES: For new tests.
- ALWAYS_PASSES: For established tests to lock in regressions.
Checklist
- Setup Workspace: Initialize the test environment.
- Write Assertions: Define the expected tool calls or behavioral markers.
- Verify: Run the eval suite and analyze failures.