Results for “metric”

Computes the InfoLM metric from torchmetrics for evaluating text generation against ground truth, with configurable information measures and sentence-level scoring.

Compute the Equal Error Rate (EER) metric using torchmetrics for binary, multiclass, or multilabel classification tasks, with reference signatures and usage examples.

Computes CIDEr and related metrics to score how well generated image descriptions align with human consensus, using reference sentences and triplet annotations.

Evaluates the correlation between a zero-cost NAS metric (epsilon) and actual training accuracy across different neural architecture search spaces, testing the metric's ability to rank architectures without training. It probes whether output dispersion from constant weight initializations can serve as a reliable.

Scores generated image captions against reference captions and source images using the Polos metric, which is trained to align with human judgments and probes hallucination robustness and open-vocabulary evaluation.

Evaluates an AI judge system's pairwise ranking accuracy on generated commit messages against a heuristic ground truth from five automatic text metrics, using the MCMD dataset.

Compute the F1Score metric using torchmetrics when predictions and ground-truth labels are available.

Run evaluation tasks against a NeMo Platform server using the Evaluator plugin CLI and Python SDK.

Computes the Receiver Operating Characteristic (ROC) metric using torchmetrics, supporting binary, multiclass, and multilabel tasks.

Measures distributional similarity between original GAN-generated images and their semantically manipulated counterparts using the Fréchet Inception Distance (FID) metric.

Computes Theil's U (uncertainty coefficient) between predictions and ground truth using the torchmetrics implementation, handling categorical data and NaN strategies.

Filters and prioritizes compound libraries in drug discovery using drug-likeness rules, structural alerts, complexity metrics, and a query language.

Computes the SQuAD metric using torchmetrics, given predictions and ground truth. Use when evaluating question-answering outputs with exact match and F1 scores.

Analyze Neuropixels extracellular recordings end-to-end with SpikeInterface, covering loading, preprocessing, drift correction, spike sorting, quality metrics, and unit curation.

Designs and implements RAG pipelines, covering document chunking, embedding strategies, hybrid search, answer synthesis with source attribution, and evaluation using RAGAS metrics.

Evaluates geometric consistency in text-to-video generation by measuring structural and motion coherence across camera trajectories, detecting deformation and occlusion artifacts in static scenes.

Track ML experiments with automatic logging, visualize training in real-time, optimize hyperparameters with sweeps, and manage model registry with W&B.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) using standardized prompts and metrics. Supports HuggingFace, vLLM, and API backends.

Evaluates the robustness and alignment with human judgment of reference-based and reference-free evaluation metrics for machine translation and summarization, particularly under adversarial conditions.

Computes the HARE Score, an entity- and relation-centric metric for evaluating machine-generated histopathology reports against ground truth, using GatorTronS+SapBERT embeddings and relation F1.

Evaluates text-to-image models on spatial relationship accuracy using the VISOR metric, separating object detection from spatial correctness to reveal biases like object priority and merging.

Validates the Emotional Attitude Score (EAS) metric by measuring its consistency with human judgment on word-level sentiment polarity, using the AmbGIMT dataset and pairwise score comparisons.

Evaluates spatial realism and temporal flow consistency of AI-generated videos using embedding spaces and Fourier transforms, producing bounded STREAM-S and STREAM-T scores.

Evaluates the correlation between automatic text generation scores and human quality ratings, including robustness to domain and quality drift, using metrics like Kendall's Tau and Pearson correlation.

Evaluates speaker verification models by computing true positive rate at fixed false positive rate thresholds, probing embedding space separation of same-speaker versus different-speaker pairs.

Evaluates machine learning classifiers on their ability to distinguish signal from background in particle physics simulations, measuring how well algorithms rank signal events above background ones using the AUC metric.

Train, evaluate, export, and run inference for monocular depth estimation models using Metric Depth Anything v2 or Relative Depth Anything architectures via the TAO toolkit.

Build clinical deep-learning pipelines with PyHealth: load EHR, signal, and imaging datasets, define prediction tasks, instantiate models, train with the PyHealth Trainer, and compute clinical metrics.

Quantifies the minimum accuracy gap needed between two models for a sampled micro-benchmark to reliably preserve their ranking, using the MDAD metric from Yauney et al. (2025).

Benchmarks long-context language models across 20 sub-tasks spanning 3k–200k tokens, covering retrieval, reasoning, summarization, and instruction understanding, with exact-match accuracy as the primary metric.