Results for “evaluation”

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debug...

Create and modify NeMo AutoModel training and evaluation recipes, including YAML structure, builders, and execution flow.

Creates and manages AI agent teams for complex engineering tasks, with model routing, cost optimization, and performance evaluation.

Computes the Anderson-Darling test statistic and p-value using scipy.stats.anderson for evaluating predictions against ground truth.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debuggin

Provides guidance on using the Aeon library for time series forecasting, covering model selection, implementation, and evaluation.

Build high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services. Covers design principles, architecture, implementation patterns, and evaluation-driven development.

Designing and evaluating AI/ML systems: prompt engineering, RAG design, LLM application patterns, AI safety, evaluation frameworks, MLOps, cost optimization. Use for AI pipelines or eval harnesses.

Evaluates execution transcripts and output files against a list of expectations, assigning pass/fail verdicts with cited evidence and critiquing the assertions themselves.

Build a library of metacognitive prompts targeting planning, monitoring, or evaluation for a specific task. Use when developing students' thinking-about-thinking during independent work.

Implement OpenFeature feature flags in software projects. Use when adding feature flags with OpenFeature SDKs, configuring providers, setting up evaluation context, or integrating the OpenFeature MCP Server.

Plan, debug, tune, evaluate, export, or deploy public Nemotron embedding and reranking retrieval recipes using the current checkout.

Compare coding agents head-to-head on reproducible tasks with pass rate, cost, time, and consistency metrics.

Builds Retrieval-Augmented Generation systems with document chunking, embedding generation, vector storage, and retrieval pipelines, including evaluation and optimization.

Instrument LLM applications with Langfuse to trace, score, and monitor cost, quality, and latency across OpenAI and LangChain integrations.

Guides the creation of high-quality MCP servers, covering design, implementation, testing, and evaluation for Python and TypeScript.

Measures distributional similarity between original GAN-generated images and their semantically manipulated counterparts using the Fréchet Inception Distance (FID) metric.

Computes the InfoLM metric from torchmetrics for evaluating text generation against ground truth, with configurable information measures and sentence-level scoring.

Scores generated text for morphosyntactic well-formedness by measuring how closely it adheres to language-specific dependency rules extracted from treebanks.

Evaluates machine-generated log summaries without human-written references, using LLM judgment and dense embeddings to score relevance, informativeness, and coherence.

Production machine-learning engineering workflow for data contracts, reproducible training, model evaluation, deployment, monitoring, and rollback. Use when building, reviewing, or hardening ML systems beyond one-off notebooks.

Production machine-learning engineering workflow for data contracts, reproducible training, model evaluation, deployment, monitoring, and rollback. Use when building, reviewing, or hardening ML systems beyond one-off notebooks.

Train, evaluate, distill, quantize, export, and run inference for RT-DETR object detection models using NVIDIA TAO.

Dataset analysis and data quality specialist. Use when the user asks to talk to Cypher, requests the data detective, or needs dataset assessment, bias analysis, and benchmark evaluation.

LLM observability platform for tracing, evaluation, and monitoring. Use when debugging LLM applications, evaluating model outputs against datasets, monitoring production systems, or building systematic testing pipelines for AI applications.

LLM observability platform for tracing, evaluation, and monitoring. Use when debugging LLM applications, evaluating model outputs against datasets, monitoring production systems, or building systematic testing pipelines for AI applications.

Spawns multiple parallel AI agents that compete on the same task using isolated git worktrees, evaluates results, and merges the best solution.

Build AI applications using the Azure AI Projects SDK for TypeScript, managing agents, connections, deployments, datasets, indexes, and evaluations.

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.

Evaluates the naturalness, speaker similarity, and real-time synthesis speed of a Mandarin speech cloning system across diverse practical application scenarios.

Computes CIDEr and related metrics to score how well generated image descriptions align with human consensus, using reference sentences and triplet annotations.

Computes the SQuAD metric using torchmetrics, given predictions and ground truth. Use when evaluating question-answering outputs with exact match and F1 scores.

Evaluate model quality with task-appropriate metrics and systematic error analysis. Use when: (1) comparing models, (2) analyzing failures, (3) setting go/no-go thresholds. NOT for: production monitoring implementation.

Fine-tune a TAO model with standard supervised training, evaluation, and export, with AutoML bypass and platform-specific credential intake.

Build and evaluate machine learning models using scikit-learn for classification, regression, clustering, dimensionality reduction, and preprocessing.

Fine-tune LLMs using LoRA, QLoRA, and PEFT with Hugging Face, including dataset preparation, hyperparameter tuning, evaluation, and deployment.