Results for “benchmark”

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Evaluate LLM outputs systematically — benchmarks, automated metrics, human preference, and regression tracking

Benchmarks long-context language models across 20 sub-tasks spanning 3k–200k tokens, covering retrieval, reasoning, summarization, and instruction understanding, with exact-match accuracy as the primary metric.

Import object detection models from HuggingFace or NVIDIA NGC into a DeepStream pipeline with automated ONNX download, TensorRT engine build, custom parser, multi-stream benchmark, and PDF report generation.

Benchmarks mobile GUI agents on multi-step tasks across 20 Android apps, measuring task completion and essential-state navigation with Task Success Rate and Essential State Achieved Rate.

Benchmarks systems on the Abstraction and Reasoning Corpus (ARC) by requiring inference of abstract transformation rules from few input-output grid demonstrations and application to novel test cases, reporting the fraction of tasks solved.

Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.

Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.

Benchmarks LLM agents fine-tuned with the Agent Data Protocol across software engineering, web browsing, OS/database tool use, and reasoning tasks, reporting unit test pass rates and task success rates.

Quantifies how sensitive a language model benchmark's reliability and ranking stability are to specific design choices, such as the selection of scenarios, subscenarios, examples, and few-shot prompts. Use when the user has predictions and gold and needs to compute DIoR.

Local LLM operations with Ollama on Apple Silicon, including setup, model pulls, chat launchers, benchmarks, and diagnostics.

Use the model-recommender skill, Workflow C (Roster Refresh), to research and update the model roster from live benchmarks.

Run evaluation tasks against a NeMo Platform server using the Evaluator plugin CLI and Python SDK.

Creates, runs, and analyzes Arize experiments for evaluating and comparing model performance using the ax CLI.

Compares coding agents head-to-head on reproducible tasks, measuring pass rate, cost, time, and consistency.

Build, deploy, and manage continuous AI agents using a visual workflow editor or development toolkit.

Launches long-running evaluation batches in isolated tmux sessions with pre-flight verification, monitoring, and post-flight analysis for unattended runs.

Compare coding agents head-to-head on reproducible tasks with pass rate, cost, time, and consistency metrics.

Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.

Launches a model evaluation batch with parameter collection, pre-flight checks, execution, and post-run analysis for interactive or foreground runs.

Scaffolds a program.md research program for autoresearch by auto-detecting codebase signals and interviewing for missing details.

Dataset analysis and data quality specialist. Use when the user asks to talk to Cypher, requests the data detective, or needs dataset assessment, bias analysis, and benchmark evaluation.

Run autonomous optimization loops that iteratively improve artifacts against evaluators using hypothesis tree refinement, without overfitting.

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.

Recommend model families and validation strategy based on data, constraints, and objective. Use when: (1) choosing algorithms, (2) balancing bias/variance, (3) planning benchmark baselines. NOT for: final legal/compliance sign-off.

Curates clinical-specialty term lists, generates IPA-tagged synthetic audio via TTS, and produces NeMo-format manifests for ASR benchmark evaluation.

Predict molecular properties, train graph neural networks, and run drug discovery workflows using DeepChem's featurizers, models, and MoleculeNet benchmarks.

Reference guide for evaluating LLM-generated translations between CUDA and OpenMP, covering memory model mapping, kernel launch patterns, shared memory, atomics, and common failure modes.

Quantifies audio source separation quality by computing the signal-to-distortion ratio (SDR) between ground-truth and estimated stems, with per-stem and record-level averaging.

Benchmarks automatic speech translation and recognition on English-French and English-Romanian datasets, reporting BLEU and WER on tokenized outputs.

Evaluates text-to-image models on spatial relationship accuracy using the VISOR metric, separating object detection from spatial correctness to reveal biases like object priority and merging.

Evaluates open-ended generation quality of multilingual LLMs across brainstorming, planning, and long-form tasks, using AYA and DOLLY datasets with qualitative fluency and quality scoring.

Featurizes molecules for machine learning with molfeat (100+ featurizers) — ECFP/MACCS/MAP4 fingerprints, RDKit and Mordred physicochemical descriptors, and pretrained embeddings (ChemBERTa, ChemGPT, GIN) exposed as scikit-learn transformers that convert SMILES into feature vectors. Use when turning molecules into ML-ready feature matrices for QSAR/QSPR or virtual screening, or benchmarking fingerprint against descriptor and embedding representations; for training models and MoleculeNet benchmarks on those features prefer alterlab-deepchem, and for low-level fingerprint or descriptor primitives prefer alterlab-rdkit. Part of the AlterLab Academic Skills suite.

Benchmarks automatic post-editing (APE) models on WMT'18 SMT, SubEdits, and MLQE-PE datasets, reporting BLEU, ChrF, and TER scores computed with SacreBLEU and TERCOM.

Provides guidance for automatically evolving and optimizing AI agents across any domain using LLM-driven evolution algorithms. Use when building self-improving agents, optimizing agent prompts and skills against benchmarks, or implementing automated agent evaluation loops.

Extract structured review data from Google Maps search results using the BrowserAct API, enabling local business analysis, reputation monitoring, and competitive benchmarking.