Results for “framework-evaluation”

Build evaluation frameworks for agent systems with deterministic checks, regression suites, multi-dimensional rubrics, quality gates, production monitoring, and outcome measurement.

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles

VS-Enhanced Theoretical Framework Architect with Critique & Visualization Full VS 5-Phase process: Modal theory avoidance, Long-tail exploration, differentiated framework presentation Absorbed A3 (Devil's Advocate) critique and A6 (Conceptual Framework Visualizer) capabilities Use when: building theoretical foundations, designing conceptual models, deriving hypotheses, critiquing frameworks, visualizing models Triggers: theoretical framework, 이론적 프레임워크, conceptual model, 개념적 모형, hypothesis derivation, critique, devil's advocate, 반론, visualization, diagram

通用跨模型审查：Review LLM 对任意研究制品进行独立评审，输出结构化评分、wiki 实体映射与改进建议

Project estimation and time management for agent-orchestrated builds.

Decision-theoretic framework for agents embedded within the environments they model and act upon

Architectural decision-making framework.

Every metric encodes an opinion about which mistake hurts.

Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks Use when: agent testing, agent evaluation, benchmark agents, agent reliability, test agent.

Evaluate model quality with task-appropriate metrics and systematic error analysis. Use when: (1) comparing models, (2) analyzing failures, (3) setting go/no-go thresholds. NOT for: production monitoring implementation.

Launches a model evaluation batch with parameter collection, pre-flight checks, execution, and post-run analysis for interactive or foreground runs.

DSPy declarative framework for automatic prompt optimization treating prompts as code with systematic evaluation and compilers

Evaluate and rank agent results by metric or LLM judge for an AgentHub session. Use when the user runs /hub:eval or asks to score, compare, or pick a winner among completed AgentHub agents.

Designs, optimizes, and evaluates prompts for LLMs, including structured outputs, chain-of-thought, and evaluation frameworks.

Expert skill for building LLM applications with the LlamaIndex framework — RAG pipelines, multi-agent orchestration, event-driven workflows, knowledge graph construction, production deployment, and evaluation. Use when working with LlamaIndex or comparing RAG and agent orchestration frameworks.

Designing and evaluating AI/ML systems: prompt engineering, RAG design, LLM application patterns, AI safety, evaluation frameworks, MLOps, cost optimization. Use for AI pipelines or eval harnesses.

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.

Provides a formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles to define pass/fail criteria, measure reliability with pass@k metrics, and create regression test suites.

Route Firebase AI feature work into either direct app/client Firebase AI Logic SDK integration or a server-owned Genkit workflow. Use when a web, mobile, backend, or full-stack feature needs model calls, typed outputs, reusable flows, tools, retrieval, prompt files, evals, observability, or deployment. Choose client-ai-logic, flow-foundation, tool-and-agent, retrieval-and-prompt, evaluation-and-observability, deployment-runtime, or comparison-or-fallback; route Firebase platform/operator work to `firebase-cli` and broad framework comparisons to `survey`.

AI/LLM red-team assessment using the OWASP LLM Top 10 (2025) + OWASP AI Testing Guide (AITG v1, Nov 2025) frameworks, plus OWASP MCP Top 10 runtime testing for agentic/MCP targets. Tests prompt injection, jailbreaks, system prompt leakage, sensitive data extraction, excessive agency, improper output handling, model extraction, content bias, evasion, membership inference, MCP token exposure, MCP command injection, and more. Uses four tools in combination: FuzzyAI (single-turn jailbreak fuzzing), PyRIT (multi-turn orchestrated attacks), Garak (probe-based vulnerability scanning), and promptfoo (plugin-based red-team evaluation). Each tool covers different OWASP categories; running them together gives systematic coverage. Includes a conditional MCP reconnaissance phase and a post-access AI infrastructure phase (chained from /post-exploit). Produces: OWASP LLM Top 10 + AITG + MCP coverage matrix, findings per category, architecture diagram of the AI system, PoCs for confirmed exploits. Chains into /gh-export for