Agent Platform Autorater Customization

Customizes an autorater (evaluation metric) for an AI agent so its grades match the user's own judgment, starting from the agent's real trajectories. Use when a user wants a metric tailored to their agent, wants to turn failing trajectories into rubrics, says an existing autorater grades their agent wrong, or wants an autorater to agree with human labels (autorater hillclimbing, judge calibration). Don't use for quick evals with built-in metrics or for improving the agent itself (use agent-platform-eval-flywheel).

Google Updated 24.3k repo stars

File contents

google/skills commit 57015399df

Frequently asked questions

npx get-skill add google/agent-platform-autorater-customization