Results for “eval-gate”

Design reproducible evaluations for AI agents with representative task sets, explicit rubrics, appropriate graders, baselines, regression gates, and failure analysis. Use when defining agent quality, comparing prompts or models, validating a release, measuring tool-use reliability, investigating regressions, or deciding whether an agent is ready for production.

Turns model work into a production ML system with data contracts, reproducible training, quality gates, deployable artifacts, and monitoring.

Build and improve AI or probabilistic software through evaluation-driven development. Use for LLM applications, agents, prompts, RAG, tool use, classifiers, model migrations, quality regressions, golden datasets, LLM-as-judge rubrics, benchmarks, or requests to add evals and measurable release gates. Pair with TDD for deterministic code; do not use as the primary guide for ordinary unit testing without model behavior.

Evaluate everything the PA agent manages — tasks, skills, PA network health, billing, calendar connections, and memory quality. Use when: owner asks for an evaluation, wants to know what's working and what isn't, or requests a performance report. Combines supervisor status with quality scoring.

Use `analysis-agent` to map acceptance to validation, `task-agent` to add or run bounded tests, and `review-agent` to assess proof coverage. Skip work with no material change or already-fresh complete validation.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session. Use when the user runs /hub:eval or asks to score, compare, or pick a winner among completed AgentHub agents.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session. Use when the user runs /hub:eval or asks to score, compare, or pick a winner among completed AgentHub agents.

Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology, including dataset creation, metric selection, failure analysis, and iterative improvement.

Read-only EVM client: wallets, tokens, gas across 8 chains.

Read-only EVM client: wallets, tokens, gas across 8 chains.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session.

Launches long-running evaluation batches in isolated tmux sessions with pre-flight verification, monitoring, and post-flight analysis for unattended runs.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session.

Score LLM and agent outputs using LLM-as-judge techniques — direct scoring against rubrics or pairwise comparison between two outputs. Includes built-in bias mitigation for position bias, length bias, and self-enhancement bias. Load when the user asks to score an output, judge a response, evaluate against a rubric, compare two outputs, do direct scoring, run pairwise comparison, or says "rate this", "which response is better", "score this against the rubric", "judge this output", "LLM as judge this". Sub-skill of eval-output orchestrator.

实验判决门：Review LLM 独立评判实验结果 → 4 种判决路径 → 自动更新 claims confidence、ideas status、graph edges

Pre-ingestion verification for epistemic quality in RAG systems with 9-point verification and Two-Round HITL workflow

Agent Vault — fetching secrets from Infisical at runtime

Use when a bounded textual artifact (prompt, rubric, tool description, extraction instruction) keeps underperforming and success can be measured with an evaluator, dataset, or trace set. GEPA proposes evaluator-backed candidate rewrites through a normal PR/proposal adoption gate. Do not use for vague behavior changes, governance/persona/core-memory edits, fake metrics, or problems whose first honest task is defining the evaluator or collecting data.

Provides a formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles to define pass/fail criteria, measure reliability with pass@k metrics, and create regression test suites.

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles

Launches a model evaluation batch with parameter collection, pre-flight checks, execution, and post-run analysis for interactive or foreground runs.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session.

Use `analysis-agent` to analyze permissions, secrets, sensitive data, trust boundaries, and injection; `task-agent` to implement controls; and `review-agent` to assess evidence. Skip self-review and no-trust-impact work.

Head-to-head comparison of coding agents (Claude Code, Aider, Codex, etc.) on custom tasks with pass rate, cost, time, and consistency metrics

Turn model work into a production ML system with data contracts, repeatable training, measurable quality gates, deployable artifacts, and operational monitoring.

Turn model work into a production ML system with data contracts, reproducible training, quality gates, deployable artifacts, and monitoring.

4-stage funnel that screens all 500+ Hyperliquid perps down to the top trading opportunities. Scores setups 0-400 across smart money, market structure, technicals, and funding. BTC macro filter, hourly trend gate (counter-trend = hard skip), cross-scan momentum tracking. Near-zero LLM tokens — all computation in Python. Use when scanning for new trading opportunities on Hyperliquid, evaluating setups, or checking market conditions.

Security audit and vulnerability scanner for AI agent skills before installation. Use when: (1) evaluating a skill from an untrusted source, (2) auditing a skill directory or git repo URL for malicious code, (3) pre-install security gate for Claude Code plugins, OpenClaw skills, or Codex skills, (4) scanning Python scripts for dangerous patterns like os.system, eval, subprocess, network exfiltration, (5) detecting prompt injection in SKILL.md files, (6) checking dependency supply chain risks, (7) verifying file system access stays within skill boundaries. Triggers: "audit this skill", "is this skill safe", "scan skill for security", "check skill before install", "skill security check", "skill vulnerability scan".

Drive Soup (`soup-cli`), a CLI-first tool for fine-tuning and post-training LLMs with one YAML config and one command — SFT, DPO/GRPO/ORPO/SimPO/KTO, QLoRA/DoRA/LoRA+, layer streaming for 4-8 GB GPUs, eval-gated training, and serving. Use when the user wants to `soup init`/`soup train` a model, pick a training method or quantization scheme, estimate cost/memory before training, fine-tune on a small local GPU, migrate a config from Axolotl/LLaMA-Factory/Unsloth, or serve/merge/push a trained adapter. Triggers on: "soup-cli", "soup train", "soup init", "fine-tune an LLM locally", "QLoRA on a laptop GPU", "layer streaming", "soup advise", "soup autopilot", "DPO/GRPO/ORPO training", "merge LoRA adapter".