Results for “benchmark”

Benchmarks a text-to-text transformer across GLUE, SuperGLUE, CNN/Daily Mail, SQuAD, and WMT, reporting GLUE average, BLEU, ROUGE-2-F, and Exact Match scores.

Queries Hugging Face benchmark leaderboards to find the best AI models for a task, filters by device constraints, and returns a ranked comparison table with scores.

Benchmarks medical AI agents on synthetic EHR tasks, measuring success rates for data retrieval, temporal aggregation, and threshold-based conditional logic with exact-match scoring.

Reduce per-token latency on Jetson vLLM servers by appending speculative decoding configuration, with guidance on when to enable and how to benchmark the improvement.

Benchmarks LLM bias by scoring responses to automatically generated open-ended questions across sensitive attributes, producing a composite fitness score from 0 to 5.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) using standardized prompts and metrics. Supports HuggingFace, vLLM, and API backends.

Evaluates social bias in question-answering models using the BBQ benchmark, measuring accuracy and a bias score across ambiguous and disambiguated contexts to reveal reliance on stereotypes.

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution on local Docker, Slurm HPC, or cloud platforms.

Solves single- and multi-objective optimization problems with NSGA-II/III, MOEA/D, and other evolutionary algorithms, including constraint handling, Pareto front analysis, and benchmark problems.

Benchmarks large language models on symbolic music understanding and instruction following using text-based ABC notation, covering syntax parsing, error detection, segment-level reasoning, and sequence-level musical analysis.

Plan, configure, and chain Nemotron model customization steps into single-step or multi-step pipelines for curation, translation, fine-tuning, RL alignment, benchmarking, checkpoint conversion, optimization, and evaluation.

Solve single and multi-objective optimization problems using NSGA-II/III, MOEA/D, and other evolutionary algorithms with customizable operators, constraint handling, and benchmark problems.

Quantifies the minimum accuracy gap needed between two models for a sampled micro-benchmark to reliably preserve their ranking, using the MDAD metric from Yauney et al. (2025).

Benchmarks large audio-language models against adversarial audio attacks using the CAA dataset, computing WER, ROUGE-L, cosine similarity, and coherence scores to assess robustness in conversational settings.

Import object detection models from HuggingFace or NVIDIA NGC into a DeepStream pipeline with automated ONNX download, TensorRT engine build, custom parser, multi-stream benchmark, and PDF report generation.

Benchmarks long-context language models across 20 sub-tasks spanning 3k–200k tokens, covering retrieval, reasoning, summarization, and instruction understanding, with exact-match accuracy as the primary metric.

Benchmarks systems on the Abstraction and Reasoning Corpus (ARC) by requiring inference of abstract transformation rules from few input-output grid demonstrations and application to novel test cases, reporting the fraction of tasks solved.

Quantifies how sensitive a language model benchmark's reliability and ranking stability are to specific design choices, such as the selection of scenarios, subscenarios, examples, and few-shot prompts. Use when the user has predictions and gold and needs to compute DIoR.

Benchmarks LLM agents fine-tuned with the Agent Data Protocol across software engineering, web browsing, OS/database tool use, and reasoning tasks, reporting unit test pass rates and task success rates.

Run evaluation tasks against a NeMo Platform server using the Evaluator plugin CLI and Python SDK.

Launches long-running evaluation batches in isolated tmux sessions with pre-flight verification, monitoring, and post-flight analysis for unattended runs.

Launches a model evaluation batch with parameter collection, pre-flight checks, execution, and post-run analysis for interactive or foreground runs.

Predict molecular properties, train graph neural networks, and run drug discovery workflows using DeepChem's featurizers, models, and MoleculeNet benchmarks.

Reference guide for evaluating LLM-generated translations between CUDA and OpenMP, covering memory model mapping, kernel launch patterns, shared memory, atomics, and common failure modes.

Quantifies audio source separation quality by computing the signal-to-distortion ratio (SDR) between ground-truth and estimated stems, with per-stem and record-level averaging.

Evaluates text-to-image models on spatial relationship accuracy using the VISOR metric, separating object detection from spatial correctness to reveal biases like object priority and merging.

Evaluates open-ended generation quality of multilingual LLMs across brainstorming, planning, and long-form tasks, using AYA and DOLLY datasets with qualitative fluency and quality scoring.

Benchmarks automatic post-editing (APE) models on WMT'18 SMT, SubEdits, and MLQE-PE datasets, reporting BLEU, ChrF, and TER scores computed with SacreBLEU and TERCOM.

Solve single- and multi-objective optimization problems with NSGA-II/III, MOEA/D, and other evolutionary algorithms, including Pareto front analysis, constraint handling, and benchmarking on standard test problems.

Measures inference latency of binarized, 8-bit, and 32-bit convolutional layers on edge devices to evaluate the efficiency and speedup of the Larq Compute Engine framework compared to standard implementations.