Results for “eval”

AI/LLM red-team assessment using the OWASP LLM Top 10 (2025) + OWASP AI Testing Guide (AITG v1, Nov 2025) frameworks, plus OWASP MCP Top 10 runtime testing for agentic/MCP targets. Tests prompt injection, jailbreaks, system prompt leakage, sensitive data extraction, excessive agency, improper output handling, model extraction, content bias, evasion, membership inference, MCP token exposure, MCP command injection, and more. Uses four tools in combination: FuzzyAI (single-turn jailbreak fuzzing), PyRIT (multi-turn orchestrated attacks), Garak (probe-based vulnerability scanning), and promptfoo (plugin-based red-team evaluation). Each tool covers different OWASP categories; running them together gives systematic coverage. Includes a conditional MCP reconnaissance phase and a post-access AI infrastructure phase (chained from /post-exploit). Produces: OWASP LLM Top 10 + AITG + MCP coverage matrix, findings per category, architecture diagram of the AI system, PoCs for confirmed exploits. Chains into /gh-export for

Expert strategy advisor for products built on LLMs or agents — not general product strategy (see `product-strategy` for that). Use this — proactively and without waiting to be asked — whenever choosing where to apply AI in a product, deciding between RAG and fine-tuning, designing how much autonomy an AI feature should have, evaluating whether an AI feature is actually defensible, or deciding whether to add AI to a feature at all. Also triggers for: "should this be an agent or a simple LLM call", "how much autonomy should this feature have", "RAG vs fine-tuning", "is this AI feature defensible", "our AI feature keeps hallucinating and users don't trust it", "should we build this with AI or just ship it deterministic", "AI product wedge", "what happens to this feature when the models get better", "human-in-the-loop design for AI features". Produces a decision-focused brief: the wedge, the architecture choice, the autonomy level, and the defensibility bet — each with an explicit trade-off.