Results for “evaluation-metrics”

Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks Use when: agent testing, agent evaluation, benchmark agents, agent reliability, test agent.

Compare coding agents head-to-head on reproducible tasks with pass rate, cost, time, and consistency metrics.

Computes the InfoLM metric from torchmetrics for evaluating text generation against ground truth, with configurable information measures and sentence-level scoring.

Evaluate model quality with task-appropriate metrics and systematic error analysis. Use when: (1) comparing models, (2) analyzing failures, (3) setting go/no-go thresholds. NOT for: production monitoring implementation.

Designs and implements RAG pipelines, covering document chunking, embedding strategies, hybrid search, answer synthesis with source attribution, and evaluation using RAGAS metrics.

Evaluates RAG pipelines using a filesystem-based benchmark with corpus/ and train.json, running evaluate_rag.py to tune retrieval and generation flags and interpret RAGAS metrics.

Every metric encodes an opinion about which mistake hurts.

Evaluate LLM outputs systematically — benchmarks, automated metrics, human preference, and regression tracking

Build evaluation frameworks for agent systems with deterministic checks, regression suites, multi-dimensional rubrics, quality gates, production monitoring, and outcome measurement.

Compute the Equal Error Rate (EER) metric using torchmetrics for binary, multiclass, or multilabel classification tasks, with reference signatures and usage examples.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session. Use when the user runs /hub:eval or asks to score, compare, or pick a winner among completed AgentHub agents.

Design structured evaluation rubrics for scoring LLM and agent outputs — defining quality dimensions, scoring scales, hard gates, score descriptions, and edge cases. Load when the user asks to create an eval rubric, define evaluation criteria, design scoring dimensions, write an eval spec, or says "what should I evaluate", "design a rubric", "create eval criteria", "define quality dimensions", "evaluation rubric for", "how do I measure quality of". Sub-skill of eval-output orchestrator.

Build evaluation frameworks for agent systems, covering rubric design, test set creation, and automated evaluation pipelines.

Formal evaluation framework for Codex sessions implementing eval-driven development (EDD) principles

Build and improve AI or probabilistic software through evaluation-driven development. Use for LLM applications, agents, prompts, RAG, tool use, classifiers, model migrations, quality regressions, golden datasets, LLM-as-judge rubrics, benchmarks, or requests to add evals and measurable release gates. Pair with TDD for deterministic code; do not use as the primary guide for ordinary unit testing without model behavior.

Evaluates LLM performance using BLEU, ROUGE metrics and LLM-as-judge. Use for model testing.

Build the funnel metrics that actually get trusted. Stage-by-stage conversion, velocity, win rate, and the single biggest leak, with every definition pinned so nobody relitigates the numbers in the meeting. Built for B2B RevOps teams, customizable to your CRM and your stage model. Trigger on "build my funnel metrics", "what's my conversion by stage", "where's the leak", "what's our win rate", "how fast do deals move", or any funnel diagnostic.

Validates the Emotional Attitude Score (EAS) metric by measuring its consistency with human judgment on word-level sentiment polarity, using the AmbGIMT dataset and pairwise score comparisons.

Build evaluation frameworks for agent systems. Use when testing agent performance systematically, validating context engineering choices, or measuring improvements over time.

Build evaluation frameworks for agent systems. Use when testing agent performance systematically, validating context engineering choices, or measuring improvements over time.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) using standardized prompts and metrics. Supports HuggingFace, vLLM, and API backends.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session. Use when the user runs /hub:eval or asks to score, compare, or pick a winner among completed AgentHub agents.

Build evaluation frameworks for agent systems

Benchmarks long-context language models across 20 sub-tasks spanning 3k–200k tokens, covering retrieval, reasoning, summarization, and instruction understanding, with exact-match accuracy as the primary metric.

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.