Results for “eval”

Drive Soup (`soup-cli`), a CLI-first tool for fine-tuning and post-training LLMs with one YAML config and one command — SFT, DPO/GRPO/ORPO/SimPO/KTO, QLoRA/DoRA/LoRA+, layer streaming for 4-8 GB GPUs, eval-gated training, and serving. Use when the user wants to `soup init`/`soup train` a model, pick a training method or quantization scheme, estimate cost/memory before training, fine-tune on a small local GPU, migrate a config from Axolotl/LLaMA-Factory/Unsloth, or serve/merge/push a trained adapter. Triggers on: "soup-cli", "soup train", "soup init", "fine-tune an LLM locally", "QLoRA on a laptop GPU", "layer streaming", "soup advise", "soup autopilot", "DPO/GRPO/ORPO training", "merge LoRA adapter".

Instrument a shipped product's AI agents with tracing and observability so you can see what they did, why outputs happened, and what each run cost. Plain-language primer plus free-tier-first backend selection (Langfuse, Phoenix, LangSmith, Braintrust) and OpenTelemetry/OpenInference instrumentation. Load when the user asks to add observability, add tracing, instrument my agents, see what my agent is doing in production, set up Langfuse or Phoenix or LangSmith, debug why my agent gave a bad answer, or track LLM cost per request. Also fires when agent-system-architecture or setup-evaluation requires an observability plan for an agent-chain product. NOT for tracing the coding agent itself — that is run-trace. Precondition for runtime-learning-loop.

Produce a standalone test plan by analyzing code for test coverage gaps and edge cases. Use when you need to create, generate, or draft a test plan for a branch, need to analyze test coverage, or need to identify what tests to write for specific files or directories. Does not produce a plain-language plan for a person to run tests by hand — use manual-test-planning for that. Does not write test code — use tdd to implement behavior test-first. Does not refine existing plans — use iterative-plan-review. Does not review code quality, security, or style — use code-review for full code review. Does not evaluate architectural testability or structural coupling — use architectural-analysis for architectural assessment.

Produces an attorney-ready memo from a corpus of legal documents supplied by the user. Use when a user shows up with a folder, zip, or vault of case documents and asks for a case summary, case evaluation, litigation package, intake memo, matter overview, or "can you summarize this case for me." The skill ingests the corpus into a searchable index, OCRs anything non-searchable, inventories and diagnoses the practice area, loads the appropriate practice-area playbook module(s) (PI/tort, commercial litigation, IP infringement, or user-authored extensions), iteratively searches the corpus across eight core dimensions plus any module-specific dimensions, defers specialized document clusters (depositions, medical records, discovery, liens) to dedicated sibling skills, and synthesizes a cited memo.

Pick the right business-model canvas (Lean Canvas, Business Model Canvas, or Value Proposition Canvas) for the stage and fill it with specifics — one segment, one primary canvas, top-3 assumptions, no fluff in the moat or channel boxes. Load when the user asks to fill a business model canvas, lean canvas, value proposition canvas, model this business, map the business model, says "fill the BMC", "make a Lean Canvas", "Value Proposition Canvas for this", "model this idea", "what's the business model", "design the business model". Sub-skill of `venture-exploration`. Hard-bans "everyone" segments, generic channels ("SEO/social/content/ads"), and "unfair advantage = AI/data/network effects" with no concrete asset. Does NOT score viability — for that use `idea-evaluation`.

Use this skill to classify tabular data end-to-end in MATLAB — load a dataset, prepare and clean it, select promising classifiers, train them, and compare accuracies with cross-validation, holdout, or hyperparameter optimization plus statistical tests. TRIGGER when: user asks to classify tabular data, pick classifiers for a dataset, compare classifier accuracy, run cross-validation or a holdout evaluation, or find the best model with statistical uncertainty. DO NOT TRIGGER when: user has non-tabular inputs (images, sequences, time series), wants a regression model, is training a specific neural network architecture (use matlab-train-network), or wants cost-sensitive learning or an arbitrary class-prior vector (this skill only supports the built-in uniform-prior toggle for imbalanced data).

Legal-specific Argument Ranking Engine. Hodnotí právní argumenty 17-sloupcovým scoringem (síla, bezpečnost, dopad, riziko + 10 legal-specific kritérií: Compliance s novelou 2026, Evidence backing, Time-sensitivity, per-document mapping, R-static/R-reversal split, Tom-weight bonus, C-XX/M-XX/N-XX/D-XX integrace). Output: priorizovaný seznam argumentů s kategoriemi CORE/SUPPORT/CONTEXT/EXCLUDED/SUMMARY a per-document bundles (PR/§909/40_06/195). Použití: pre-prioritizace argumentů před F11.x review, Phase 2 Verify input, DÁVKA 3, výživné L04, AT podání. VŽDY použij tento skill, když Tom (nebo legal/strat) zmíní: /lare, lare, argument ranking, ARE matrix, score arguments, prioritize arguments, argument bundle, CORE/SUPPORT/CONTEXT/EXCLUDED, ARE_F11, LARE_F11, legal argument evaluation, argument scoring, compliance scoring, Tom-weight.

Perform a structured audit of a codebase covering security, code quality, performance, dependencies, architecture, and testing hygiene, then produce a prioritized findings report. Use this skill whenever the user asks for a code review, code audit, security review, codebase assessment, "look over this repo", "what's wrong with this codebase", legacy-code triage, pre-acquisition technical due diligence, or any request to systematically evaluate the health of a project. Trigger even when the user is casual ("can you eyeball my repo?") — this skill imposes the structure that ad-hoc review misses. This skill audits a whole repository at a point in time — for reviewing a diff or PR use the built-in code-review skill; for security checks on pending changes use security-review.

Run the Ouroboros specification-first development loop: reduce ambiguity with a Socratic interview grounded in live git data (commits, churn, contributors), freeze an immutable seed/spec, render the execution plan through spec-kit (/speckit.plan → /speckit.tasks), execute against that contract through cli-anything agent-native CLI harnesses (cli-hub, --json evidence), verify before claiming success, and keep looping until completion is actually verified. Use when the user wants spec-first clarification, git-aware interviews, immutable requirements, drift-aware implementation, harness-driven execution, or a persistent completion loop that should keep going until tests / checks / acceptance criteria pass. Triggers on: ooo, ouroboros, interview, seed, run workflow, evaluate, evolve, ooo ralph, specification first, socratic interview, git-aware interview, ambiguity reduction, execution plan, cli harness execute, persistent completion.

Use BEFORE generating, refactoring, reviewing, or debugging code. Trigger phrases include "write a function/script/class for X", "review this code/diff/PR", "refactor this", "debug this error", "is this implementation correct", "what's wrong with this code", "improve this code", "translate from X to Y", or any prompt with a code block the user wants you to act on. Also fires when planning architectural changes, picking algorithms or data structures, or evaluating dependency upgrades. Calls the code MCP tool to retrieve an engineering scaffold (failure pattern, procedure, correct-pattern example, verification step) before generating. Catches hallucinated APIs, lost edge cases, premature algorithm commitment, silent contract violations, refactors that change behavior masked by passing tests. Do NOT trigger for pure code reading with no action requested, simple syntax questions, file...

>- Analyze KiCad projects and PDF schematics: schematics, PCB layouts, Gerbers, footprints, symbols, netlists, and design rules. Reviews designs for bugs, traces nets, cross-references schematic to PCB, extracts BOM data, checks DRC/ERC, DFM, power trees, and regulator circuits. Every finding carries a confidence label and evidence source with trust_summary rollup. Analyzes PDF schematics from dev boards, reference designs, eval kits, and datasheets. Supports KiCad 5–10. Use whenever the user mentions .kicad_sch, .kicad_pcb, .kicad_pro, PCB design review, schematic analysis, PDF schematics, reference designs, Gerber files, DRC/ERC, netlist issues, BOM extraction, signal tracing, power budget, DFM, or wants to understand, debug, compare, or review any hardware design. Also for "check my board", "review before fab", "what's wrong with my schematic", "is this ready to order", "check my...

Systematic strategic planning framework for philosophy and interdisciplinary academic papers targeting preprint platforms (PhilArchive, arXiv, PhilSci-Archive). Use when users want to: (1) plan a paper on a specific topic, (2) identify research gaps and assess originality, (3) develop optimized paper outlines, (4) prepare for preprint submission, or (5) understand platform requirements and writing standards. Triggered by phrases like 'plan a paper on,' 'help me design a paper about,' 'identify research gaps in,' 'is this idea original,' or when users need structured research planning. The skill guides through three phases: Platform Analysis (identifying target venue and studying sample papers), Theoretical Framework (AI-driven literature search and gap identification), and Outline Optimization (structured design with reviewer-perspective self-assessment). Each phase includes quality evaluation standards and validation checkpoints. Output: optimized detailed outline ready for systematic writing (use with acade

Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, artifacts, posters, or applications (examples include websites, landing pages, dashboards, React components, HTML/CSS layouts, or when styling/beautifying any web UI). Generates creative, polished code and UI design that avoids generic AI aesthetics. Trigger scenarios — use when the user mentions any of: - UI design, frontend design, web design, interface design, 界面设计, 前端设计 - responsive layout, mobile adaptation, breakpoints, 响应式, 自适应, 适配 - animation, motion, micro-interaction, transitions, 动画, 动效, 微交互 - UX copy, microcopy, error messages, labels, UX 文案, 文案优化 - performance optimization, bundle size, rendering, 性能优化, 渲染, 加载速度 - accessibility audit, a11y, WCAG, 无障碍, 可访问性 - design review, design critique, UX evaluation, 设计评审, 设计审查 - typography, fonts, type hierarchy, 字体, 排版, 字号 - color palette, color scheme, theming, 配色, 色彩, 主题 - layout, spacing, visual rhythm, grid

AI/LLM red-team assessment using the OWASP LLM Top 10 (2025) + OWASP AI Testing Guide (AITG v1, Nov 2025) frameworks, plus OWASP MCP Top 10 runtime testing for agentic/MCP targets. Tests prompt injection, jailbreaks, system prompt leakage, sensitive data extraction, excessive agency, improper output handling, model extraction, content bias, evasion, membership inference, MCP token exposure, MCP command injection, and more. Uses four tools in combination: FuzzyAI (single-turn jailbreak fuzzing), PyRIT (multi-turn orchestrated attacks), Garak (probe-based vulnerability scanning), and promptfoo (plugin-based red-team evaluation). Each tool covers different OWASP categories; running them together gives systematic coverage. Includes a conditional MCP reconnaissance phase and a post-access AI infrastructure phase (chained from /post-exploit). Produces: OWASP LLM Top 10 + AITG + MCP coverage matrix, findings per category, architecture diagram of the AI system, PoCs for confirmed exploits. Chains into /gh-export for

Federal acquisition compliance auditor for the active Theseus workspace, backed by live FAR/DFARS text via the vendored `ecfr` MCP. USE WHEN the user asks to audit FAR/DFARS clause coverage, validate that cited clauses actually exist in eCFR (catch fabricated or typo'd numbers), check whether a cited clause has been amended since the solicitation issued, validate regulatory references (NIST SP, DAFI, MIL-STD), check that every "shall" requirement has a deliverable, find missing compliance artifacts, audit proposal_instruction ↔ evaluation_factor coverage (UCF Section L↔M or non-UCF equivalent — FAR 16 task orders, FOPRs, BPA calls, OTAs), or "are we compliant with the proposal instructions?". Cross-references the workspace's clause / regulatory_reference / requirement / deliverable / compliance_artifact entities against live eCFR and flags gaps with severity. Format-agnostic. DO NOT USE FOR drafting compliant prose (use proposal-generator) or extracting clauses (Theseus pipeline does that automatically).

Write the context brief a first-time tester gets before testing an app, tool, codebase, or product — it explains WHAT the thing is and why it exists, while deliberately withholding HOW anything works. The companion to the fresh-eyes skill; the brief it produces is the "starting information" handed to a fresh-eyes tester. Use this whenever the user is preparing a fresh-eyes or first-time-user test and needs the setup material, or says things like "describe my app without giving anything away", "write the context card for the tester", "explain what it is but not how to use it", "set up the newcomer test", "what would the tester be told going in?", or wants a spoiler-free description of their product. Also use it when someone asks for the briefing/intro that a new tester, new hire, or evaluator should receive before first contact with the thing being tested.

Expert strategy advisor for products built on LLMs or agents — not general product strategy (see `product-strategy` for that). Use this — proactively and without waiting to be asked — whenever choosing where to apply AI in a product, deciding between RAG and fine-tuning, designing how much autonomy an AI feature should have, evaluating whether an AI feature is actually defensible, or deciding whether to add AI to a feature at all. Also triggers for: "should this be an agent or a simple LLM call", "how much autonomy should this feature have", "RAG vs fine-tuning", "is this AI feature defensible", "our AI feature keeps hallucinating and users don't trust it", "should we build this with AI or just ship it deterministic", "AI product wedge", "what happens to this feature when the models get better", "human-in-the-loop design for AI features". Produces a decision-focused brief: the wedge, the architecture choice, the autonomy level, and the defensibility bet — each with an explicit trade-off.