Results for “eval”

LLM output evaluation — automated metrics, LLM-as-judge, A/B testing, regression testing. Use when measuring LLM output quality, comparing prompt or model versions, building an automated eval pipeline, setting up regression tests for prompt changes, or evaluating RAG systems and bias/safety.

Diagnose and improve MSBuild project evaluation performance by analyzing phases, glob patterns, import chains, and property functions.

Trace, evaluate, and monitor LLM applications with an open-source observability platform.

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.

Compares coding agents head-to-head on reproducible tasks, measuring pass rate, cost, time, and consistency.

Execute the full AgentHub competition lifecycle in a single command: initialize, capture baseline, spawn agents, evaluate results, and merge the winner.

Manage Azure AI Foundry projects, connections, datasets, indexes, and evaluations using the Java SDK.

Instrument LLM applications with Langfuse for tracing, prompt versioning, evaluation, and dataset management across Python and JavaScript SDKs.

Compare coding agents head-to-head on reproducible tasks with pass rate, cost, time, and consistency metrics.

Provides expertise in Langfuse for LLM observability, including tracing, prompt management, evaluation, and integration with LangChain, LlamaIndex, and OpenAI.

Instruments LLM applications with Langfuse for tracing, observability, and evaluation, covering setup, OpenAI and LangChain integrations, and best practices.

Design high-level system architecture, create Architecture Decision Records (ADRs), evaluate technology trade-offs, and plan for scalability.

Instrument LLM applications with Langfuse for tracing, prompt management, evaluation, and cost tracking, including integrations with OpenAI, LangChain, and LlamaIndex.

Spawns multiple parallel AI agents that compete on the same task using isolated git worktrees, evaluates results, and merges the best solution.

Run autonomous optimization loops that iteratively improve artifacts against evaluators using hypothesis tree refinement, without overfitting.

Self-hosted observability platform for LLM applications, providing tracing, evaluation, datasets, experiments, and real-time monitoring to debug and improve AI systems.

Scaffolds a coder-eval task suite in a repository by scanning for testable components, writing one real task, and validating it.

Design multi-agent system architectures, generate tool schemas for Anthropic and OpenAI formats, and evaluate execution logs for cost, latency, and failure bottlenecks.

Deploy, evaluate, fine-tune, and manage Microsoft Foundry agents end-to-end using Azure Developer CLI and MCP tools.

Fine-tune models on Azure AI Foundry using SFT, DPO, or RFT, covering dataset preparation, training job submission, deployment, and evaluation.

Validate enterprise workloads against Google Cloud best practices using public client libraries and the REST API to manage evaluations, rules, scanned resources, and validation results.

Build AI applications on Microsoft Foundry using the Azure AI Projects Python SDK, including agent creation, tool integration, thread management, and evaluation.

DeepEval — LLM evaluation framework, RAG metrics, hallucination detection, red-teaming, CI/CD integration

Retrieval-Augmented Generation pipelines — ingestion, chunking, embedding, vector stores, retrieval, evaluation. Use when building a RAG pipeline, choosing chunking strategies or embedding models, debugging retrieval quality or hallucinations, evaluating an existing RAG system, or scaling/migrating vector stores.

Connect to an existing Chrome browser via Chrome DevTools Protocol to navigate, click, type, screenshot, evaluate JavaScript, read the accessibility tree, and monitor console or network events.

Holistic evaluation framework for language models measuring accuracy, calibration, robustness, and fairness

Coordinates quality assurance workflows by routing testing tasks to specialized sub-skills for API testing, performance benchmarking, test analysis, tool evaluation, and process optimization.

Automates detection engineering workflows in Google SecOps by extracting threat intelligence, generating detection opportunities, simulating attacker behavior with synthetic events, evaluating rule coverage, and creating new YARA-L 2.0 rules to close gaps.

Design and apply QA methodology for software teams: test strategy, regression testing, CI failure triage, test automation, quality gates and metrics, risk-based testing, exploratory testing, test design techniques, AI code quality gates (independent verification, acceptance-criteria testability review for agentic Spec-Driven Development), mutation-guided test hardening and review evidence (surviving mutants, weak assertions, diff-aware mutation testing), agentic eval design (dataset test design, judge-as-system-under-test, flaky-eval discipline), QA career levels (Senior/Staff/Principal), and SDET engineering (test infrastructure, gTAA, CI/CD integration). Do not use for root-cause debugging of production incidents, security implementation or threat modeling, or evaluation framework governance and statistical analysis — route those to systematic-debugging, secure-software-engineering, and agent-evals-and-observability respectively.