Results for “evaluation”

EU MDR 2017/745 compliance specialist for medical device classification, technical documentation, clinical evidence, and post-market surveillance. Covers Annex VIII classification rules, Annex II/III technical files, Annex XIV clinical evaluation, and EUDAMED integration.

Agent profile for evaluate RAG quality, chunking, retrieval, citations, hallucination risks, freshness, and regression sets. Use when Codex needs a specialist agent perspective for planning, implementation, review, debugging, validation, or handoff in this domain.

Expert skill for building LLM applications with the LlamaIndex framework — RAG pipelines, multi-agent orchestration, event-driven workflows, knowledge graph construction, production deployment, and evaluation. Use when working with LlamaIndex or comparing RAG and agent orchestration frameworks.

Route academic-figure work into the lightest workable PaperBanana mode instead of jumping straight to a full multi-agent generation run. Use when the user needs publication-quality methodology diagrams, statistical plots, figure evaluation, polishing an existing figure, batch/sweep generation, or a full-paper figure package from text or PDF. Even if the user does not say "paperbanana" — also triggers on: academic figure, methodology diagram, publication figure, generate diagram from paper, statistical plot from CSV, figure evaluation, polish figure, NeurIPS/ICML figure, arxiv illustration, plan-then-refine diagram pipeline.

Guides the creation of high-quality MCP servers that enable LLMs to interact with external services through well-designed tools, covering planning, implementation, testing, and evaluation.

Bootstraps a clinical ASR evaluation environment by verifying NVIDIA_API_KEY, installing Python dependencies, and running a smoke test against hosted TTS/ASR services.

Create a comprehensive company research brief with executive quotes, product strategy, and organizational context for competitive analysis, partnership evaluation, interview preparation, or market entry decisions.

Guides authorized Wi-Fi security assessments including handshake capture, PMKID collection, rogue AP detection, and offline password policy evaluation using aircrack-ng, hashcat, and Wireshark.

Orchestrates cross-domain design work by dispatching to specialized design sub-skills and applying behavioral science frameworks for audits, critiques, and strategy.

Validates the Emotional Attitude Score (EAS) metric by measuring its consistency with human judgment on word-level sentiment polarity, using the AmbGIMT dataset and pairwise score comparisons.

Evaluates the correlation between automatic text generation scores and human quality ratings, including robustness to domain and quality drift, using metrics like Kendall's Tau and Pearson correlation.

Provides guidance for automatically evolving and optimizing AI agents across any domain using LLM-driven evolution algorithms. Use when building self-improving agents, optimizing agent prompts and skills against benchmarks, or implementing automated agent evaluation loops.

Recommends against performing dynamic GnRH stimulation testing in the evaluation of hypogonadism, as it provides no additional diagnostic information beyond baseline gonadotropins and sex steroids. Consider this recommendation when evaluating hypogonadism and contemplating GnRH testing.

Estimate market size using TAM, SAM, and SOM with top-down and bottom-up approaches for market opportunity assessment, investor pitches, or market entry evaluation.

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution on local Docker, Slurm HPC, or cloud platforms.

Evaluates machine learning classifiers on their ability to distinguish signal from background in particle physics simulations, measuring how well algorithms rank signal events above background ones using the AUC metric.

Evaluates speaker verification models by computing true positive rate at fixed false positive rate thresholds, probing embedding space separation of same-speaker versus different-speaker pairs.

Benchmarks large language models on symbolic music understanding and instruction following using text-based ABC notation, covering syntax parsing, error detection, segment-level reasoning, and sequence-level musical analysis.

Route research prompts to evidence, retrieval, scraping, data, market, or scientific research skills. Use when prompts mention deep research, current facts, citations, Exa, iterative retrieval, search-first, scraping, data pipelines, PubMed, USPTO, gget, literature review, or scholar evaluation.

PASS/FAIL eval rubrics and alignment loops for Sid Orchestra. Use when the user says sid evals, @sid-evals, grade this, eval gate, alignment score, or wants to stop AI slop with evaluation gates.

Plan, configure, and chain Nemotron model customization steps into single-step or multi-step pipelines for curation, translation, fine-tuning, RL alignment, benchmarking, checkpoint conversion, optimization, and evaluation.

Access AI-ready drug discovery datasets and benchmarks from Therapeutics Data Commons, covering ADME, toxicity, drug-target interactions, and molecular generation with standardized splits and evaluation metrics.

Guides the creation of high-quality MCP servers that let LLMs interact with external services through well-designed tools, covering planning, implementation, testing, and evaluation.

Retrieval-Augmented Generation pipelines — ingestion, chunking, embedding, vector stores, retrieval, evaluation. Use when building a RAG pipeline, choosing chunking strategies or embedding models, debugging retrieval quality or hallucinations, evaluating an existing RAG system, or scaling/migrating vector stores.

Conducts end-to-end user research, from study design through behavioral insight synthesis, using frameworks like Nielsen's Heuristics, SUS, and User Journey Mapping.

Benchmarks long-context language models across 20 sub-tasks spanning 3k–200k tokens, covering retrieval, reasoning, summarization, and instruction understanding, with exact-match accuracy as the primary metric.

Use when symbolic mathematics must cross into NumPy vector evaluation or SciPy numerical algorithms: lambdify contracts, domains, dtypes, parameters, residuals, tolerances, and symbolic-versus-numeric verification. Do not use for work confined entirely to one of those libraries.

Route AI agent engineering prompts to architecture, orchestration, evaluation, safety, debugging, context, prompt, MCP, persona, local AI, and Compound Engineering skills. Use when prompts mention agents, agent harnesses, agentic workflows, orchestration, evals, context management, MCP servers, or compound engineering.

Execute QA test plans on Holoscan Sensor Bridge hardware by reading a test document, filtering tests by setup, running automatable tests with pass/fail evaluation, and producing a structured report.

Simulates a NeurIPS/SC/ICSE-style peer review with five reviewer personas (HPC, ML, Stats, Reproducibility, Devil's Advocate) that verify every claim against actual result data before submission.

Evaluates the faithfulness of abstractive summaries by generating questions from summary sentences and verifying if the answers can be extracted from the source document, reporting Pearson and Spearman correlations with human judgments.

Benchmarks systems on the Abstraction and Reasoning Corpus (ARC) by requiring inference of abstract transformation rules from few input-output grid demonstrations and application to novel test cases, reporting the fraction of tasks solved.

Use when planning a PLDI campaign across its annual clock — backward-planning from the November deadline through winter reviewing, the February response window, March notification, post-acceptance artifact evaluation, PACMPL production, and the June conference, with owners for each deliverable.

Use when the superpowers-evaluator returned REWORK on a batch, when fixing rework items from an evaluation report, or when receiving any code review feedback on superpowers output. Requires technical rigor and verification instead of performative agreement or blind implementation.

Guides the creation of high-quality MCP servers that enable LLMs to interact with external services through well-designed tools, covering planning, implementation, testing, and evaluation across multiple programming languages.

Train, evaluate, export, quantize, and run inference for a Deformable DETR 2D object detection model using TAO, with deformable attention for efficient multi-scale feature processing.