Results for “evaluation”

Validate skill outputs against assertion manifests to detect regressions, enforce quality gates, and bootstrap starter manifests from historical runs.

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.

Runs a five-phase pipeline—evaluation, specification, implementation, review, and QA—to build a feature end-to-end with specialized agents.

Evaluates the naturalness, speaker similarity, and real-time synthesis speed of a Mandarin speech cloning system across diverse practical application scenarios.

Computes CIDEr and related metrics to score how well generated image descriptions align with human consensus, using reference sentences and triplet annotations.

Computes the SQuAD metric using torchmetrics, given predictions and ground truth. Use when evaluating question-answering outputs with exact match and F1 scores.

Adversarial robustness and ML safety specialist. Use when the user asks to talk to Viper, requests the adversary, or needs failure mode analysis, attack surface review, and robustness evaluation.

Use when assessing new ideas, technologies, or approaches before committing resources to them. This skill provides a framework for evaluating innovation opportunities systematically, separating genuine breakthroughs from hype.

Evaluate model quality with task-appropriate metrics and systematic error analysis. Use when: (1) comparing models, (2) analyzing failures, (3) setting go/no-go thresholds. NOT for: production monitoring implementation.

Fine-tune a TAO model with standard supervised training, evaluation, and export, with AutoML bypass and platform-specific credential intake.

Build and evaluate machine learning models using scikit-learn for classification, regression, clustering, dimensionality reduction, and preprocessing.

Fine-tune LLMs using LoRA, QLoRA, and PEFT with Hugging Face, including dataset preparation, hyperparameter tuning, evaluation, and deployment.

Designs autonomous agent harnesses with locked evaluators, editable surfaces, durable logging, novelty gates, pruning, rollback, and human approval boundaries.

Turns model work into a production ML system with data contracts, reproducible training, quality gates, deployable artifacts, and monitoring.

Evaluates how well internal model representations (hidden states) predict token-level information importance in summarization tasks, using NDCG@10 and Spearman's rank correlation.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production.

Agent-driven RAG patterns. Self-RAG, Corrective RAG (CRAG) with web fallback, Adaptive RAG with routing classifier, ReAct with retrieval tool, multi-hop retrieval, plan-and-execute, LangGraph state machines for RAG. USE WHEN: user mentions "agentic RAG", "Self-RAG", "Corrective RAG", "CRAG", "Adaptive RAG", "multi-hop retrieval", "LangGraph RAG", "ReAct RAG", "plan and execute RAG" DO NOT USE FOR: static retrieval pipelines - use `rag-architecture`; query rewriting only - use `query-transformations`; evaluation - use `rag-evaluation`

Use when querying a Rulebase workspace with SQL through the MCP server's query tool — writing queries that finish inside the statement timeout, and avoiding the join fan-outs that silently inflate QA evaluation counts. Trigger for "query my Rulebase data", "the query timed out", "statement timeout", "canceling statement due to statement timeout", counts that don't reconcile between two Rulebase queries, criterion counts exceeding team counts, or any multi-step analysis over Rulebase conversations and evaluations.

Honestly evaluate AI work quality using a two-axis scoring system with mandatory devil's advocate reasoning and cross-session anti-inflation detection.

Manage Azure AI Foundry projects with .NET SDK: create and run agents, manage connections, datasets, deployments, evaluations, and indexes.

Apply foundational design principles—affordances, signifiers, constraints, mappings, and feedback—to evaluate and improve product usability, bridging the gulfs of execution and evaluation.

Optimizes long-running agent sessions with structured context compression, summarization, and durable handoff summaries that preserve decisions, files, risks, and next actions.

Authorized OT/ICS security assessment covering Purdue model zoning, PLC/SCADA exposure, industrial protocol discovery, and safe passive-first evaluation.

Designs and implements RAG pipelines, covering document chunking, embedding strategies, hybrid search, answer synthesis with source attribution, and evaluation using RAGAS metrics.

Provides guidance on production AI operations including RAG pipelines, vector databases, embeddings, fine-tuning, prompt engineering, cost estimation, and quality evaluation.

Expert in designing effective prompts for LLM-powered applications. Masters prompt structure, context management, output formatting, and prompt evaluation. Use when: prompt engineering, system prompt, few-shot, chain of thought, prompt design.

Design multi-agent system architectures, generate tool schemas for Anthropic and OpenAI formats, and evaluate execution logs for cost, latency, and failure bottlenecks.

Turn model work into a production ML system with data contracts, repeatable training, measurable quality gates, deployable artifacts, and operational monitoring.

Turns a natural-language request into one or more coder-eval task YAML files with minimal prompts and weighted success criteria, validated against the coder-eval CLI.

Turn model work into a production ML system with data contracts, reproducible training, quality gates, deployable artifacts, and monitoring.

Evaluates whether a lawsuit is worth pursuing, explains the litigation process from filing to resolution, and guides case preparation, settlement negotiations, and small claims alternatives.