Results for “eval”

Build type-safe AI agents and graph-based workflows with PydanticAI and PydanticGraph. Agent creation, function tools, capabilities, dependency injection, structured output, streaming, multi-agent patterns, testing, evals, and graph state machines. Use whenever you are building agents, tool-using LLM workflows, or graph-based state machines in Python.

Quantifies how sensitive a language model benchmark's reliability and ranking stability are to specific design choices, such as the selection of scenarios, subscenarios, examples, and few-shot prompts. Use when the user has predictions and gold and needs to compute DIoR.

Run Karpathy-style autonomous ML search on a real training repo: choose the right mode (setup, program.md, bounded loop, results interpretation, or constrained-hardware adaptation), preserve the immutable prepare.py / 300-second / val_bpb contract, and route prompt/skill eval work away to LangSmith, Promptfoo, Braintrust, or skill-autoresearch.

Conducts Data Protection Impact Assessments for AI and ML systems per EDPB Guidelines 04/2025 on AI processing. Covers training data lawfulness evaluation, model risk assessment, automated decision triggers, and AI-specific DPIA methodology. Keywords: AI DPIA, machine learning impact assessment, EDPB AI guidelines, model risk, training data.

Use when experiments complete to judge what claims the results support, what they don't, and what evidence is still missing. Codex MCP evaluates results against intended claims and routes to next action (pivot, supplement, or confirm). Use after experiments finish — before writing the paper or running ablations.

Perform survival analysis and time-to-event modeling in Python using scikit-survival, including Cox models, random survival forests, gradient boosting, survival SVMs, and evaluation metrics like concordance index and Brier score.

Build multi-agent AI systems with LangGraph — the low-level orchestration framework for stateful, graph-based agent workflows. Covers supervisor, swarm, and hierarchical multi-agent patterns; subgraph composition; state management (checkpointers/stores); persistence; evals; and production debugging. Reach for this when designing agent architectures that need cycles, conditional branching, parallel execution, or human-in-the-loop patterns.

WorkorAI talent marketplace skill: candidate job search and employer hiring with white-box match explanations via the WorkorAI MCP server (https://workorai.com/mcp). Use when the user asks to find a job, apply to jobs, respond to employer invitations, or when an employer wants to post jobs, search and evaluate candidates, invite them, and review applicants.

Generate a MoonSpec implementation plan and design artifacts from a single-story spec. Use when the user asks to run or reproduce `/moonspec.plan`, create or update `plan.md`, produce `research.md`, `data-model.md`, `contracts/`, or `quickstart.md`, evaluate repo principles, define separate unit and integration test strategies, and perform repo-aware gap analysis before `/moonspec.tasks`.

Use when the user wants to train, compare, or export machine learning models using Classification Learner or Regression Learner — including opening the app, loading data, training models, evaluating metrics, comparing results, visualizing plots, testing on held-out data, exploring model interpretability, and exporting trained models. Programmatic access to Classification Learner and Regression Learner apps via AppController.

Remove refusal behaviors from open-weight LLMs using OBLITERATUS — mechanistic interpretability techniques (diff-in-means, SVD, whitened SVD, LEACE, SAE decomposition, etc.) to excise guardrails while preserving reasoning. 9 CLI methods, 28 analysis modules, 116 model presets across 5 compute tiers, tournament evaluation, and telemetry-driven recommendations. Use when a user wants to uncensor, abliterate, or remove refusal from an LLM.

Evaluates the correlation between a zero-cost NAS metric (epsilon) and actual training accuracy across different neural architecture search spaces, testing the metric's ability to rank architectures without training. It probes whether output dispersion from constant weight initializations can serve as a reliable.

Multi-agent collaboration plugin that spawns N parallel subagents competing on the same task via git worktree isolation. Agents work independently, results are evaluated by metric or LLM judge, and the best branch is merged. Use when: user wants multiple approaches tried in parallel — code optimization, content variation, research exploration, or any task that benefits from parallel competition. Requires: a git repo.

Multi-agent collaboration plugin that spawns N parallel subagents competing on the same task via git worktree isolation. Agents work independently, results are evaluated by metric or LLM judge, and the best branch is merged. Use when: user wants multiple approaches tried in parallel — code optimization, content variation, research exploration, or any task that benefits from parallel competition. Requires: a git repo.

Multi-agent collaboration plugin that spawns N parallel subagents competing on the same task via git worktree isolation. Agents work independently, results are evaluated by metric or LLM judge, and the best branch is merged. Use when: user wants multiple approaches tried in parallel — code optimization, content variation, research exploration, or any task that benefits from parallel competition. Requires: a git repo.

Multi-agent collaboration plugin that spawns N parallel subagents competing on the same task via git worktree isolation. Agents work independently, results are evaluated by metric or LLM judge, and the best branch is merged. Use when: user wants multiple approaches tried in parallel — code optimization, content variation, research exploration, or any task that benefits from parallel competition. Requires: a git repo.

Multi-agent collaboration plugin that spawns N parallel subagents competing on the same task via git worktree isolation. Agents work independently, results are evaluated by metric or LLM judge, and the best branch is merged. Use when: user wants multiple approaches tried in parallel — code optimization, content variation, research exploration, or any task that benefits from parallel competition. Requires: a git repo.

Multi-agent collaboration plugin that spawns N parallel subagents competing on the same task via git worktree isolation. Agents work independently, results are evaluated by metric or LLM judge, and the best branch is merged. Use when: user wants multiple approaches tried in parallel — code optimization, content variation, research exploration, or any task that benefits from parallel competition. Requires: a git repo.

Use to audit QA coverage and scorecard health in a Rulebase workspace via the Rulebase MCP server. Trigger for "audit our QA coverage", "which agents or channels aren't being evaluated", "are our QA scores meaningful", "is our scorecard working", QA blind spots, score distribution or ceiling effects, and checking whether QA scores relate to SLA or complaint outcomes.

Fine-tunes and evaluates OpenVLA-OFT and OpenVLA-OFT+ policies for robot action generation with continuous action heads, LoRA adaptation, and FiLM conditioning on LIBERO simulation and ALOHA real-world setups. Use when reproducing OpenVLA-OFT paper results, training custom VLA action heads (L1 or diffusion), deploying server-client inference for ALOHA, or debugging normalization, LoRA merge, and cross-GPU issues.

Give your agent a budget, a target, and a deadline — it does the rest. Orchestrates DSL + Opportunity Scanner + Emerging Movers into a full autonomous trading loop on Hyperliquid. Race condition prevention, conviction collapse cuts, cross-margin buffer math, speed filter. 3 risk profiles: conservative, moderate, aggressive. Use when setting up autonomous trading, creating a trading strategy, or running a scan-evaluate-trade-protect loop.

4-stage funnel that screens all 500+ Hyperliquid perps down to the top trading opportunities. Scores setups 0-400 across smart money, market structure, technicals, and funding. BTC macro filter, hourly trend gate (counter-trend = hard skip), cross-scan momentum tracking. Near-zero LLM tokens — all computation in Python. Use when scanning for new trading opportunities on Hyperliquid, evaluating setups, or checking market conditions.

Use for Codex models/pricing, scheduled tasks, skills, settings, setup, troubleshooting, customization, automations, and self-knowledge—including 'you,' 'your,' 'this app,' or 'this coding agent' when they refer to Codex—and for OpenAI APIs/products and ChatGPT Work. Also use for model choice/migration, prompting, SDKs, Responses, Realtime, agents, evals, and Chat/Work/Codex comparisons. Do not use for generic app/software tasks that merely mention Codex.

Use to assemble a fair, evidence-backed coaching pack for a support agent's one-to-one from QA evaluations and conversation history. Trigger for "prepare a coaching session for X", "what areas does X need to improve", "areas of markdown for this agent", "what coaching opportunities stand out", "build a coaching agenda from these tickets", or preparing a weekly or monthly agent review.

Audit agent skills, plugins, prompts, manifests, scripts, dependencies, and bundled assets for provenance, prompt-injection, permission, execution, exfiltration, persistence, and update risk. Use when evaluating a third-party skill before installing, enabling, updating, publishing, or distributing it; reviewing an untrusted SKILL.md, agent configuration, MCP integration, archive, or repository; comparing a package with a known-good version; or investigating unexpected tool, network, credential, or filesystem behavior.

Seed minimal agent harness v0 — manifest, eval checks stub, governance. AUTO-INVOKED after project-setup or retroactive-project-setup when docs/harness/manifest.json is missing. Also triggers on: generate harness, scaffold agents, agent bootstrap, first time agents in this repo, new project agent setup, set up agent harness, agent onboarding files, missing agent configuration, agent instructions setup, make agents read project rules, agent reliability setup, agents not configured. Pairs with project-setup. Evolution is harness-evolution.

Debug LLM applications using the Phoenix CLI. Fetch traces, analyze errors, structure trace review with open coding and axial coding, inspect datasets, review experiments, query annotation configs, and use the GraphQL API. Use whenever the user is analyzing traces or spans, investigating LLM/agent failures, deciding what to do after instrumenting an app, building failure taxonomies, choosing what evals to write, or asking "what's going wrong", "what kinds of mistakes", or "where do I focus" — even without naming a technique.

Design habit-forming product loops using the Hook Model (Trigger, Action, Variable Reward, Investment). Use when the user mentions "users arent coming back", "engagement loops", "habit formation", "push notifications", "variable rewards", "daily active users", "habit zone", or "user retention loops". Also trigger when designing notification strategies, building streaks or progress systems, or analyzing why users stop using a product after initial signup. Covers ethics evaluation and onboarding for habits. For friction reduction and B=MAP, see improve-retention. For viral sharing, see contagious.

Improve agent reliability over time — diagnose why agents fail and fix the setup. Triggers on: agent keeps failing, same mistake again, agent not improving, make agent smarter, agent quality plateau, agents ignore skills, agent skips tests, fix agent behavior, agent unreliable, improve agent setup, self-improving harness, agents worse over time, tune agent instructions, agent going in circles, agent ignores AGENTS.md, repeated agent errors. Requires harness v0 and eval harness. AUTO-ROUTED from harness-engineering on symptoms. Not first setup — harness-generation first.

Read-only diagnostic that scans Claude Code session transcripts to surface which MCP servers and tools you actually use, ranked by call frequency, with zero-use servers flagged for removal. Use when planning an MCP cleanup, evaluating whether to keep a newly-added server, deciding which servers warrant token cost in the catalog, or before authoring an MCP-removal PR. Outputs a markdown report (last N days) — does not modify any settings. Pair with the manual `claude mcp remove <name>` step once findings are reviewed.

Guardian Angel gives AI agents a moral conscience rooted in Thomistic virtue ethics. Rather than relying solely on rule lists, it cultivates stable virtuous dispositions— prudence, justice, fortitude, temperance—that guide every interaction. The foundation is caritas: willing the good of the person you serve. From this flow the cardinal virtues as practical habits of right action and sound judgment. v3.0 introduced virtue-based disposition as the primary evaluation layer, providing deeper coherence than checklists alone. The agent's character becomes the safeguard. v3.1 adds: Plugin enforcement layer with before_tool_call hooks, approval workflows for ambiguous cases, and protections for sensitive infrastructure actions.

Drive Soup (`soup-cli`), a CLI-first tool for fine-tuning and post-training LLMs with one YAML config and one command — SFT, DPO/GRPO/ORPO/SimPO/KTO, QLoRA/DoRA/LoRA+, layer streaming for 4-8 GB GPUs, eval-gated training, and serving. Use when the user wants to `soup init`/`soup train` a model, pick a training method or quantization scheme, estimate cost/memory before training, fine-tune on a small local GPU, migrate a config from Axolotl/LLaMA-Factory/Unsloth, or serve/merge/push a trained adapter. Triggers on: "soup-cli", "soup train", "soup init", "fine-tune an LLM locally", "QLoRA on a laptop GPU", "layer streaming", "soup advise", "soup autopilot", "DPO/GRPO/ORPO training", "merge LoRA adapter".

Instrument a shipped product's AI agents with tracing and observability so you can see what they did, why outputs happened, and what each run cost. Plain-language primer plus free-tier-first backend selection (Langfuse, Phoenix, LangSmith, Braintrust) and OpenTelemetry/OpenInference instrumentation. Load when the user asks to add observability, add tracing, instrument my agents, see what my agent is doing in production, set up Langfuse or Phoenix or LangSmith, debug why my agent gave a bad answer, or track LLM cost per request. Also fires when agent-system-architecture or setup-evaluation requires an observability plan for an agent-chain product. NOT for tracing the coding agent itself — that is run-trace. Precondition for runtime-learning-loop.

Use this skill to classify tabular data end-to-end in MATLAB — load a dataset, prepare and clean it, select promising classifiers, train them, and compare accuracies with cross-validation, holdout, or hyperparameter optimization plus statistical tests. TRIGGER when: user asks to classify tabular data, pick classifiers for a dataset, compare classifier accuracy, run cross-validation or a holdout evaluation, or find the best model with statistical uncertainty. DO NOT TRIGGER when: user has non-tabular inputs (images, sequences, time series), wants a regression model, is training a specific neural network architecture (use matlab-train-network), or wants cost-sensitive learning or an arbitrary class-prior vector (this skill only supports the built-in uniform-prior toggle for imbalanced data).

Run the Ouroboros specification-first development loop: reduce ambiguity with a Socratic interview grounded in live git data (commits, churn, contributors), freeze an immutable seed/spec, render the execution plan through spec-kit (/speckit.plan → /speckit.tasks), execute against that contract through cli-anything agent-native CLI harnesses (cli-hub, --json evidence), verify before claiming success, and keep looping until completion is actually verified. Use when the user wants spec-first clarification, git-aware interviews, immutable requirements, drift-aware implementation, harness-driven execution, or a persistent completion loop that should keep going until tests / checks / acceptance criteria pass. Triggers on: ooo, ouroboros, interview, seed, run workflow, evaluate, evolve, ooo ralph, specification first, socratic interview, git-aware interview, ambiguity reduction, execution plan, cli harness execute, persistent completion.

Use BEFORE generating, refactoring, reviewing, or debugging code. Trigger phrases include "write a function/script/class for X", "review this code/diff/PR", "refactor this", "debug this error", "is this implementation correct", "what's wrong with this code", "improve this code", "translate from X to Y", or any prompt with a code block the user wants you to act on. Also fires when planning architectural changes, picking algorithms or data structures, or evaluating dependency upgrades. Calls the code MCP tool to retrieve an engineering scaffold (failure pattern, procedure, correct-pattern example, verification step) before generating. Catches hallucinated APIs, lost edge cases, premature algorithm commitment, silent contract violations, refactors that change behavior masked by passing tests. Do NOT trigger for pure code reading with no action requested, simple syntax questions, file...