Results for “test-generation”

Evaluate RAG pipelines with Ragas — measure faithfulness, answer relevancy, context precision/recall, and noise sensitivity using LLM-as-judge metrics; run automated test suite generation with TestsetGenerator; integrate with LangChain, LlamaIndex, and CI pipelines.

Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.

Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.

Generate a complete Rust Model Context Protocol server project with tools, prompts, resources, and tests using the official rmcp SDK.

Measures whether coding agents actually follow skills, rules, or agent definitions by generating test scenarios, running agents, and classifying tool calls to report compliance rates.

Analyzes production code to find gaps in existing tests by reasoning about hypothetical mutations and checking whether tests would catch them.

Simulates meiotic recombination to produce offspring genomes from parent pairs, modeling Mendelian segregation, de novo mutation, sex determination, trait inference, and clinical evaluation against a disease registry.

Seed minimal agent harness v0 — manifest, eval checks stub, governance. AUTO-INVOKED after project-setup or retroactive-project-setup when docs/harness/manifest.json is missing. Also triggers on: generate harness, scaffold agents, agent bootstrap, first time agents in this repo, new project agent setup, set up agent harness, agent onboarding files, missing agent configuration, agent instructions setup, make agents read project rules, agent reliability setup, agents not configured. Pairs with project-setup. Evolution is harness-evolution.

Casting personas: rapid generation from diverse inputs, registry-based persistence and lifecycle, data-driven evolution, inter-agent sync. Not for UI walkthroughs (Echo) or user research (Field).

Generate retrieval practice questions at varied difficulty levels for a topic or concept. Use when creating quiz starters, revision activities, or low-stakes testing materials.

Set up and run mutation testing (StrykerJS / mutmut) to measure whether tests assert behavior, not just execute lines. Use when "add mutation testing", "are our tests real", "can our test suite be gamed", or after an agent bulk-generated tests. Coverage plan → plan-test-coverage. Writing tests → test-unit.

Control LLM output with regex and grammars, guarantee valid JSON/XML/code generation, enforce structured formats, and build multi-step workflows with Guidance - Microsoft Research's constrained generation framework

Mantém a coerência visual entre peças geradas por IA usando modelo fixo, prompt base, seed e referência de estilo, com teste de coerência e biblioteca de prompts.

Builds, refines, tests, or validates a CLI-Anything harness for a GUI application or source repository, producing a Python-based Click CLI with REPL mode and JSON output.

Generates AGENTS.md files for repository folders to provide coding agents with project-specific context including build commands, testing instructions, code style, project structure, and operational boundaries, only where AGENTS.md is missing.

Run NVIDIA garak probe suites against an LLM endpoint to test for jailbreaks, prompt injection, data leakage, and toxic generation, then interpret the hit-rate report for triage and reporting.

Advanced data analysis, pattern detection, and insight generation from structured and unstructured datasets. Use when the user wants to analyze data, perform statistical analysis, find insights, detect patterns, identify anomalies, compare segments, test hypotheses, or generate data-driven recommendations. Triggers on phrases like 'analyze data', 'data analysis', 'find insights', 'analyze dataset', 'statistical analysis', 'find patterns', 'compare groups', 'test hypothesis', 'correlation analysis', or 'trend analysis'.

Patterns and techniques for evaluating and improving AI agent outputs. Use this skill when: - Implementing self-critique and reflection loops - Building evaluator-optimizer pipelines for quality-critical generation - Creating test-driven code refinement workflows - Designing rubric-based or LLM-as-judge evaluation systems - Adding iterative improvement to agent outputs (code, reports, analysis) - Measuring and improving agent response quality

Improve agent reliability over time — diagnose why agents fail and fix the setup. Triggers on: agent keeps failing, same mistake again, agent not improving, make agent smarter, agent quality plateau, agents ignore skills, agent skips tests, fix agent behavior, agent unreliable, improve agent setup, self-improving harness, agents worse over time, tune agent instructions, agent going in circles, agent ignores AGENTS.md, repeated agent errors. Requires harness v0 and eval harness. AUTO-ROUTED from harness-engineering on symptoms. Not first setup — harness-generation first.

Drive Palmier Pro, an open source AI-native macOS video editor (Swift, SwiftUI/AppKit, AVFoundation) that exposes its timeline as an MCP server at `http://127.0.0.1:19789/mcp` so Claude Code/Desktop, Cursor, or Codex can read and edit a project's tracks, clips, media, transcript, captions, color/effects, and trigger generative AI (video/image/audio) requests side-by-side with a human editor. Use when the user wants to connect an agent to Palmier Pro's MCP server, call its timeline/clip/media/generation tools (`get_timeline`, `add_clips`, `move_clips`, `generate_video`, ...), build/run/test the Swift app from source, or debug the MCP tool surface in `ToolDefinitions.swift`/`ToolExecutor+*.swift`. Triggers on: "palmier pro", "palmier-pro", "AI video editor MCP", "connect Claude to my video editor", "palmier MCP server", "edit my timeline with an agent", "swift build PalmierPro", "palmier-pro mcpb", "manage_project"/"get_timeline"/"add_clips" tool.