Results for “evaluation”

Designs autonomous agent harnesses with locked evaluators, editable surfaces, durable logging, novelty gates, pruning, rollback, and human approval boundaries.

Turns model work into a production ML system with data contracts, reproducible training, quality gates, deployable artifacts, and monitoring.

Evaluates how well internal model representations (hidden states) predict token-level information importance in summarization tasks, using NDCG@10 and Spearman's rank correlation.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production.

Manage Azure AI Foundry projects with .NET SDK: create and run agents, manage connections, datasets, deployments, evaluations, and indexes.

Apply foundational design principles—affordances, signifiers, constraints, mappings, and feedback—to evaluate and improve product usability, bridging the gulfs of execution and evaluation.

Optimizes long-running agent sessions with structured context compression, summarization, and durable handoff summaries that preserve decisions, files, risks, and next actions.

Designs and implements RAG pipelines, covering document chunking, embedding strategies, hybrid search, answer synthesis with source attribution, and evaluation using RAGAS metrics.

Provides guidance on production AI operations including RAG pipelines, vector databases, embeddings, fine-tuning, prompt engineering, cost estimation, and quality evaluation.

Expert in designing effective prompts for LLM-powered applications. Masters prompt structure, context management, output formatting, and prompt evaluation. Use when: prompt engineering, system prompt, few-shot, chain of thought, prompt design.

Design multi-agent system architectures, generate tool schemas for Anthropic and OpenAI formats, and evaluate execution logs for cost, latency, and failure bottlenecks.

Turn model work into a production ML system with data contracts, repeatable training, measurable quality gates, deployable artifacts, and operational monitoring.

Turn model work into a production ML system with data contracts, reproducible training, quality gates, deployable artifacts, and monitoring.

Guides the creation of high-quality MCP servers, covering design, implementation, testing, and evaluation for integrating external services with LLMs.

Measures whether coding agents actually follow skills, rules, or agent definitions by generating test scenarios, running agents, and classifying tool calls to report compliance rates.

Evaluates geometric consistency in text-to-video generation by measuring structural and motion coherence across camera trajectories, detecting deformation and occlusion artifacts in static scenes.

Autonomous experiment loop that optimizes any file by a measurable metric. Inspired by Karpathy's autoresearch. The agent edits a target file, runs a fixed evaluation, keeps improvements (git commit), discards failures (git reset), and loops indefinitely. Use when: user wants to optimize code speed, reduce bundle/image size, improve test pass rate, optimize prompts, improve content quality (headlines, copy, CTR), or run any measurable improvement loop. Requires: a target file, an evaluation command that outputs a metric, and a git repo.

Autonomous experiment loop that optimizes any file by a measurable metric. Inspired by Karpathy's autoresearch. The agent edits a target file, runs a fixed evaluation, keeps improvements (git commit), discards failures (git reset), and loops indefinitely. Use when: user wants to optimize code speed, reduce bundle/image size, improve test pass rate, optimize prompts, improve content quality (headlines, copy, CTR), or run any measurable improvement loop. Requires: a target file, an evaluation command that outputs a metric, and a git repo.

Deploy, evaluate, fine-tune, and manage Microsoft Foundry agents end-to-end using Azure Developer CLI and MCP tools.

Evaluates RAG pipelines using a filesystem-based benchmark with corpus/ and train.json, running evaluate_rag.py to tune retrieval and generation flags and interpret RAGAS metrics.

Curates clinical-specialty term lists, generates IPA-tagged synthetic audio via TTS, and produces NeMo-format manifests for ASR benchmark evaluation.

Evaluates text-to-speech systems by measuring distributional distance between synthetic and real speech across five factors, producing a scalar score without subjective MOS ratings.

Provides guidance and code for production AI workflows including RAG pipelines, vector databases, embedding indexing, prompt engineering, cost estimation, semantic caching, and quality evaluation.

Creates and manages annotation configs and annotation queues on Arize, and applies human annotations to project spans via the Python SDK.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) using standardized prompts and metrics. Supports HuggingFace, vLLM, and API backends.

Entrena modelos de machine learning con Scikit-learn, LightGBM y XGBoost, desde un baseline hasta un modelo productivo con validación robusta y explicabilidad.

Quantifies audio source separation quality by computing the signal-to-distortion ratio (SDR) between ground-truth and estimated stems, with per-stem and record-level averaging.

Computes the HARE Score, an entity- and relation-centric metric for evaluating machine-generated histopathology reports against ground truth, using GatorTronS+SapBERT embeddings and relation F1.

Audits medical LLM benchmarks across five lifecycle phases using 46 medically tailored criteria to assess clinical relevance, data integrity, safety-critical capabilities, validity, and governance.

Benchmarks LLM bias by scoring responses to automatically generated open-ended questions across sensitive attributes, producing a composite fitness score from 0 to 5.

You are an expert in LLM observability and evaluation. You think in terms of traces, spans, and metrics. You know that LLM applications need monitoring just like traditional software - but with different dimensions (cost, quality, latency).