Model Training & Fine-tuning Agent Skills

Evaluates automated metrics and vision-language models on identifying granular errors in detailed image descriptions and ranking paired descriptions against human judgments, using macro F1, pairwise accuracy, Spearman rank ρ, and Kendall's τ.

Train quantum circuits with automatic differentiation and build hybrid quantum-classical models using PennyLane, including VQE, QAOA, and integration with PyTorch, JAX, and TensorFlow.

Computes the AUROC metric using torchmetrics, handling binary, multiclass, and multilabel tasks with configurable thresholds and averaging.

Evaluates the robustness and alignment with human judgment of reference-based and reference-free evaluation metrics for machine translation and summarization, particularly under adversarial conditions.

Scores generated image captions against reference captions and source images using the Polos metric, which is trained to align with human judgments and probes hallucination robustness and open-vocabulary evaluation.

Audits medical LLM benchmarks across five lifecycle phases using 46 medically tailored criteria to assess clinical relevance, data integrity, safety-critical capabilities, validity, and governance.

Evaluates text-to-speech systems by measuring distributional distance between synthetic and real speech across five factors, producing a scalar score without subjective MOS ratings.

Evaluates text-to-image models on spatial relationship accuracy using the VISOR metric, separating object detection from spatial correctness to reveal biases like object priority and merging.

Reduce high-dimensional data with UMAP for visualization, clustering preprocessing, and supervised or semi-supervised learning, including parameter tuning guidance.

Evaluates the correlation between automatic text generation scores and human quality ratings, including robustness to domain and quality drift, using metrics like Kendall's Tau and Pearson correlation.

Computes the InfoLM metric from torchmetrics for evaluating text generation against ground truth, with configurable information measures and sentence-level scoring.

Computes the LogAUC metric using the torchmetrics implementation for binary, multiclass, or multilabel classification tasks.

Computes the Recall metric using torchmetrics, including configuration for binary, multiclass, and multilabel tasks.

Evaluates machine-generated log summaries without human-written references, using LLM judgment and dense embeddings to score relevance, informativeness, and coherence.

Visualize training metrics, debug models with histograms, compare experiments, visualize model graphs, and profile performance with TensorBoard.

Evaluates the correlation between a zero-cost NAS metric (epsilon) and actual training accuracy across different neural architecture search spaces, testing the metric's ability to rank architectures without training. It probes whether output dispersion from constant weight initializations can serve as a reliable.

Compute the F1Score metric using torchmetrics when predictions and ground-truth labels are available.

Computes the R2Score metric using torchmetrics, handling single and multi-output predictions with options for adjusted and variance-weighted scores.

Benchmarks inference latency and computational runtime of transformer models and MLX operations across Apple Silicon and NVIDIA GPU backends, with configurable input lengths and batch sizes.

Benchmarks a text-to-text transformer across GLUE, SuperGLUE, CNN/Daily Mail, SQuAD, and WMT, reporting GLUE average, BLEU, ROUGE-2-F, and Exact Match scores.

Benchmarks large language models on symbolic music understanding and instruction following using text-based ABC notation, covering syntax parsing, error detection, segment-level reasoning, and sequence-level musical analysis.

Computes the Anderson-Darling test statistic and p-value using scipy.stats.anderson for evaluating predictions against ground truth.

Benchmarks automatic post-editing (APE) models on WMT'18 SMT, SubEdits, and MLQE-PE datasets, reporting BLEU, ChrF, and TER scores computed with SacreBLEU and TERCOM.

Benchmarks systems on the Abstraction and Reasoning Corpus (ARC) by requiring inference of abstract transformation rules from few input-output grid demonstrations and application to novel test cases, reporting the fraction of tasks solved.

Evaluates open-ended generation quality of multilingual LLMs across brainstorming, planning, and long-form tasks, using AYA and DOLLY datasets with qualitative fluency and quality scoring.

Benchmarks zero-shot in-context learning on BIG-Bench Hard multiple-choice tasks, comparing self-generated demonstrations against direct prompting and chain-of-thought baselines, and reports accuracy.

Evaluates social bias in question-answering models using the BBQ benchmark, measuring accuracy and a bias score across ambiguous and disambiguated contexts to reveal reliance on stereotypes.

Benchmarks large audio-language models against adversarial audio attacks using the CAA dataset, computing WER, ROUGE-L, cosine similarity, and coherence scores to assess robustness in conversational settings.

Benchmarks LLM bias by scoring responses to automatically generated open-ended questions across sensitive attributes, producing a composite fitness score from 0 to 5.

Write, debug, and test NumPyro probabilistic programs on JAX with correct shapes, PRNG keys, and inference choice.

Fine-tune open models or Gemini models using Agent Platform infrastructure, from environment setup through data preparation, job configuration, monitoring, and deployment.

Train, evaluate, export, quantize, and run inference on Mask2Former models for panoptic, instance, and semantic segmentation using NVIDIA TAO.

Finetune the NV-Generate-CTMR MAISI VAE/autoencoder on user-supplied CT or MRI NIfTI volumes using a staged config and datalist workflow.

Train, evaluate, export, quantize, and run inference for a Deformable DETR 2D object detection model using TAO, with deformable attention for efficient multi-scale feature processing.

Validates NVIDIA TAO DAFT datasets for structure, schema, and cross-reference errors using the `tao-daft validate` CLI tool.

Convert single-node PyTorch distributed scripts into multi-node Slurm sbatch jobs and debug common multi-node failures, covering srun-native and torch.distributed approaches, container setup, NCCL timeouts, and interactive allocation.