Plugins

12 plugins

Run Agent Evaluation

Sets up evaluation framework, runs benchmarks, and produces comparative analysis of agent performance.

9 skills · plugin

Microsoft 365 Agents Toolkit

Toolkit for building and evaluating Microsoft 365 Copilot declarative agents — scaffolding, JSON manifest development, capability configuration, and eval workflows.

6 skills · plugin

ML Model Lifecycle

Train, evaluate, and deploy a production ML system with monitoring.

10 skills · plugin

Design Pricing Strategy

Design a pricing strategy by analyzing market, evaluating financial impact, and recommending pricing models.

6 skills · plugin

Monetization to Billing Pipeline

For product managers and finance ops: brainstorm strategies, evaluate financial impact, then execute billing workflows.

4 skills · plugin

Analyze and Prioritize Feature Requests

Install this pack to categorize, evaluate, and prioritize customer feature requests against product goals.

3 skills · plugin

Prototyping Testing

Prototyping and testing skills: wireframe specs, usability heuristics, heuristic evaluations, accessibility audits, A/B test design, and benchmark analysis.

8 skills · plugin

Google Cloud Well-Architected

For architects evaluating Google Cloud workloads against the Well-Architected Framework pillars: reliability, cost optimization, and operational excellence.

6 skills · plugin

Visual Critique

Visual critique skills: hierarchy analysis, brand consistency checks against mood/voice/tokens, composition evaluation, and typography audits — with a /critique-screen command that compiles a prioritised fix list.

7 skills · plugin

@alirezarezvani

14 regulatory affairs & quality management skills for HealthTech/MedTech: ISO 13485 QMS, MDR 2017/745, FDA 510(k)/PMA, GDPR/DSGVO, ISO 27001 ISMS, CAPA management, risk management, clinical evaluation, SOC 2 compliance.

10 skills · plugin

@alirezarezvani

Multi-agent collaboration — spawn N parallel subagents that compete on code optimization, content drafts, research approaches, or any task that benefits from diverse solutions. 7 slash commands (/hub:init, /hub:spawn, /hub:status, /hub:eval, /hub:merge, /hub:board, /hub:run), agent templates, DAG-based orchestration, LLM judge mode, message board coordination.

8 skills · plugin

@alirezarezvani

Engineering Team

32 engineering skills: architecture, frontend, backend, fullstack, QA, DevOps, security, AI/ML, data engineering, Playwright (9 sub-skills), self-improving agent, Stripe integration, TDD guide, tech stack evaluator, Google Workspace CLI, a11y audit (WCAG 2.2), Azure cloud architect, GCP cloud architect, security pen testing, Snowflake development, adversarial-reviewer, ai-security, cloud-security,

16 skills · plugin

Results for “eval”

108 skills

Agent Platform Eval Flywheel

Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology, including dataset creation, metric selection, failure analysis, and iterative improvement.

14.4k · bundle

Nemo Evaluator Plugin

Run evaluation tasks against a NeMo Platform server using the Evaluator plugin CLI and Python SDK.

Launches a model evaluation batch with parameter collection, pre-flight checks, execution, and post-run analysis for interactive or foreground runs.

orchestra-research

Nemo Evaluator Sdk

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution on local Docker, Slurm HPC, or cloud platforms.

10.4k · bundle

Huggingface Community Evals

Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware, with backend selection between vLLM, Transformers, and accelerate.

10.8k · bundle

Launches long-running evaluation batches in isolated tmux sessions with pre-flight verification, monitoring, and post-flight analysis for unattended runs.

More results

orchestra-research

Evaluating Llms Harness

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) using standardized prompts and metrics. Supports HuggingFace, vLLM, and API backends.

10.4k · bundle

Evaluates the robustness and alignment with human judgment of reference-based and reference-free evaluation metrics for machine translation and summarization, particularly under adversarial conditions.

Evaluates open-ended generation quality of multilingual LLMs across brainstorming, planning, and long-form tasks, using AYA and DOLLY datasets with qualitative fluency and quality scoring.

Evaluates social bias in question-answering models using the BBQ benchmark, measuring accuracy and a bias score across ambiguous and disambiguated contexts to reveal reliance on stereotypes.

Evaluates speech language models on beyond-semantic speech attributes such as dialect comprehension, multi-turn context memory, emotion perception, age-aware response generation, and non-verbal cue handling, reporting accuracy and judge-based scores.

Benchmarks long-context language models across 20 sub-tasks spanning 3k–200k tokens, covering retrieval, reasoning, summarization, and instruction understanding, with exact-match accuracy as the primary metric.

Computes the Anderson-Darling test statistic and p-value using scipy.stats.anderson for evaluating predictions against ground truth.

Enforces rigorous ML modeling, feature engineering, training, and evaluation standards at principal-engineer level.

Instrument LLM applications with Langfuse for tracing, prompt versioning, evaluation, and dataset management across Python and JavaScript SDKs.

orchestra-research

Evaluating Code Models

Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality.

10.4k · bundle

Nemotron Retrieval Recipes

Plan, debug, tune, evaluate, export, or deploy public Nemotron embedding and reranking retrieval recipes using the current checkout.

Computes the InfoLM metric from torchmetrics for evaluating text generation against ground truth, with configurable information measures and sentence-level scoring.

Evaluates machine-generated log summaries without human-written references, using LLM judgment and dense embeddings to score relevance, informativeness, and coherence.

Tao Train Rtdetr

Train, evaluate, distill, quantize, export, and run inference for RT-DETR object detection models using NVIDIA TAO.

Evaluates the naturalness, speaker similarity, and real-time synthesis speed of a Mandarin speech cloning system across diverse practical application scenarios.

Computes the SQuAD metric using torchmetrics, given predictions and ground truth. Use when evaluating question-answering outputs with exact match and F1 scores.

Phoenix Observability

Self-hosted observability platform for LLM applications, providing tracing, evaluation, datasets, experiments, and real-time monitoring to debug and improve AI systems.

Evaluates text-to-image generation models by decomposing assessment into five specialized skills (object presence, count, spatial relations, scale, and text rendering) and open-ended prompts, producing interpretable binary scores with visual and textual explanations.

Evaluates how well internal model representations (hidden states) predict token-level information importance in summarization tasks, using NDCG@10 and Spearman's rank correlation.

Evaluates geometric consistency in text-to-video generation by measuring structural and motion coherence across camera trajectories, detecting deformation and occlusion artifacts in static scenes.

Microsoft Foundry

Deploy, evaluate, fine-tune, and manage Microsoft Foundry agents end-to-end using Azure Developer CLI and MCP tools.

Fine-tune models on Azure AI Foundry using SFT, DPO, or RFT, covering dataset preparation, training job submission, deployment, and evaluation.

Evaluates text-to-speech systems by measuring distributional distance between synthetic and real speech across five factors, producing a scalar score without subjective MOS ratings.

Simulates meiotic recombination to produce offspring genomes from parent pairs, modeling Mendelian segregation, de novo mutation, sex determination, trait inference, and clinical evaluation against a disease registry.

Benchmarks LLM bias by scoring responses to automatically generated open-ended questions across sensitive attributes, producing a composite fitness score from 0 to 5.

Computes the HARE Score, an entity- and relation-centric metric for evaluating machine-generated histopathology reports against ground truth, using GatorTronS+SapBERT embeddings and relation F1.

Runs an autonomous optimization loop that iteratively improves an artifact against an objective and evaluator using Hypothesis Tree Refinement, with subagent executors in isolated git worktrees.

Evaluates text-to-image models on spatial relationship accuracy using the VISOR metric, separating object detection from spatial correctness to reveal biases like object priority and merging.

Evaluates spatial realism and temporal flow consistency of AI-generated videos using embedding spaces and Fourier transforms, producing bounded STREAM-S and STREAM-T scores.

Scores generated image captions against reference captions and source images using the Polos metric, which is trained to align with human judgments and probes hallucination robustness and open-vocabulary evaluation.