Results for “evaluator”

Use when a bounded textual artifact (prompt, rubric, tool description, extraction instruction) keeps underperforming and success can be measured with an evaluator, dataset, or trace set. GEPA proposes evaluator-backed candidate rewrites through a normal PR/proposal adoption gate. Do not use for vague behavior changes, governance/persona/core-memory edits, fake metrics, or problems whose first honest task is defining the evaluator or collecting data.

Uses a multi-agent generator-evaluator feedback loop to build high-quality applications from a single prompt, inspired by GANs and Anthropic's harness design.

Set up a new autoresearch experiment interactively. Collects domain, target file, eval command, metric, direction, and evaluator.

Generates deep links to the Arize UI for traces, spans, sessions, datasets, labeling queues, evaluators, and annotation configs. Produces clickable URLs for sharing Arize resources with team members. Use when the user wants to link to or open a trace, span, session, dataset, evaluator, or annotation config in the Arize UI.

Debug, evaluate, and monitor LLM applications with tracing, datasets, and built-in evaluators.

Evaluates execution transcripts and output files against a list of expectations, assigning pass/fail verdicts with cited evidence and critiquing the assertions themselves.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session. Use when the user runs /hub:eval or asks to score, compare, or pick a winner among completed AgentHub agents.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session.

Orchestrator for the eval-output skill suite — evaluate LLM and agent outputs for quality, accuracy, helpfulness, and safety using structured rubrics and LLM-as-judge techniques. Load when the user says "evaluate this output", "score this response", "run an eval", "LLM as judge", "evaluate agent output", "how good is this response", "rate this answer", "eval this", or provides an LLM output that should be assessed for quality. Single entry point for all output evaluation workflows.

Validate process decomposition and architecture design quality before execution begins. Load when the setup-evaluator agent fires (automatic for agent-chain tasks), or when user says "evaluate this setup", "check the decomposition", "validate the architecture", "is this plan sound", "review the agent design". Catches structural errors, missing knowledge, unrealistic step ordering, and topology mismatches. Does NOT modify — only evaluates.

Routes between five assessment pathways — formative, rubric/criteria, authentic/performance, peer/self, and diagnostic — with validity and equity checks. Use when a teacher needs help choosing how to assess.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session. Use when the user runs /hub:eval or asks to score, compare, or pick a winner among completed AgentHub agents.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session.

Build evaluation frameworks for agent systems with deterministic checks, regression suites, multi-dimensional rubrics, quality gates, production monitoring, and outcome measurement.

Create, run, and analyze evaluation suites for Microsoft 365 Copilot declarative agents using the @microsoft/m365-copilot-eval CLI.

Runs regression evaluations comparing agent edits against human-approved golden projects, and registers new goldens after human approval.

Build evaluation frameworks for agent systems, covering rubric design, test set creation, and automated evaluation pipelines.

GAN-inspired Generator-Evaluator agent harness for building high-quality applications autonomously. Based on Anthropic's March 2026 harness design paper.

Design structured evaluation rubrics for scoring LLM and agent outputs — defining quality dimensions, scoring scales, hard gates, score descriptions, and edge cases. Load when the user asks to create an eval rubric, define evaluation criteria, design scoring dimensions, write an eval spec, or says "what should I evaluate", "design a rubric", "create eval criteria", "define quality dimensions", "evaluation rubric for", "how do I measure quality of". Sub-skill of eval-output orchestrator.

Build evaluation frameworks for agent systems. Use when testing agent performance systematically, validating context engineering choices, or measuring improvements over time.

Build evaluation frameworks for agent systems. Use when testing agent performance systematically, validating context engineering choices, or measuring improvements over time.

The learner teaches the concept to the AI, which plays a curious novice peer and identifies gaps through authentic questions. Use when the learner wants to test their understanding — teaching forces a different kind of organisation than studying.

Write robust, maintainable end-to-end tests with Playwright using Page Object Model, proper selectors, auto-waiting, and debugging workflows.

Evaluate LLM outputs systematically — benchmarks, automated metrics, human preference, and regression tracking

Two-stage review engine with four-layer quality model for the WinDAGs meta-DAG. Receives completed node outputs and produces ReviewResult containing QualityVector. Stage 1 (Haiku) checks Floor + Wall on every node. Stage 2 (Sonnet) runs Ceiling evaluation conditionally using economic escalation formula. Enforces BC-EVAL-001 through BC-EVAL-006. Activate when operating as the Evaluator role in the meta-DAG, when reviewing node outputs, when computing quality vectors, or when deciding Stage 2 escalation.

Unified Agent Teams orchestrator for Diverga v12.0.0. Manages Agent Teams creation, VS Arena debate, and subagent dispatch. Single entry point for all parallel/debate workflows. Replaces research-orchestrator and vs-arena skills. Triggers: orchestrator, agent team, create team, parallel agents, debate, competing, collaborate, VS Arena

Agent skill for reviewer - invoke with $agent-reviewer

`analysis-agent`/`task-agent`/`review-agent`: use for input authority, parsing, canonicalization, bounds, state checks, writable fields, rejection, or external-response changes.

Designs autonomous agent harnesses with locked evaluators, editable surfaces, durable logging, novelty gates, pruning, rollback, and human approval boundaries.