Results for “evaluation”

Guides the creation of high-quality MCP servers, covering design, implementation, testing, and evaluation for integrating external services with LLMs.

Measures whether coding agents actually follow skills, rules, or agent definitions by generating test scenarios, running agents, and classifying tool calls to report compliance rates.

Evaluates geometric consistency in text-to-video generation by measuring structural and motion coherence across camera trajectories, detecting deformation and occlusion artifacts in static scenes.

Use this skill when selecting or replacing an ELD provider. Covers the FMCSA Registered ELD list, evaluation criteria, common provider removals from the registered list, and the 60-day transition rule.

Use this skill when referring a driver to SAP after a positive test or refusal. Covers SAP selection, the evaluation/treatment cycle, return-to-duty test, and follow-up testing schedule.

Autonomous experiment loop that optimizes any file by a measurable metric. Inspired by Karpathy's autoresearch. The agent edits a target file, runs a fixed evaluation, keeps improvements (git commit), discards failures (git reset), and loops indefinitely. Use when: user wants to optimize code speed, reduce bundle/image size, improve test pass rate, optimize prompts, improve content quality (headlines, copy, CTR), or run any measurable improvement loop. Requires: a target file, an evaluation command that outputs a metric, and a git repo.

Autonomous experiment loop that optimizes any file by a measurable metric. Inspired by Karpathy's autoresearch. The agent edits a target file, runs a fixed evaluation, keeps improvements (git commit), discards failures (git reset), and loops indefinitely. Use when: user wants to optimize code speed, reduce bundle/image size, improve test pass rate, optimize prompts, improve content quality (headlines, copy, CTR), or run any measurable improvement loop. Requires: a target file, an evaluation command that outputs a metric, and a git repo.

Deploy, evaluate, fine-tune, and manage Microsoft Foundry agents end-to-end using Azure Developer CLI and MCP tools.

Manage Arize AI Observability and Evaluation resources on Azure using the .NET SDK. Create, read, update, and delete Arize organizations via Azure Marketplace.

Evaluates RAG pipelines using a filesystem-based benchmark with corpus/ and train.json, running evaluate_rag.py to tune retrieval and generation flags and interpret RAGAS metrics.

Curates clinical-specialty term lists, generates IPA-tagged synthetic audio via TTS, and produces NeMo-format manifests for ASR benchmark evaluation.

Evaluates text-to-speech systems by measuring distributional distance between synthetic and real speech across five factors, producing a scalar score without subjective MOS ratings.

Guardian of V.A.I.R.E. quality standards (Value/Agency/Identity/Resilience/Echo). Handles pre-release evaluation, scorecard assessment, and pass/fail determination. Use when UX quality gates are needed. Does not write code.

Provides guidance and code for production AI workflows including RAG pipelines, vector databases, embedding indexing, prompt engineering, cost estimation, semantic caching, and quality evaluation.

Creates and manages annotation configs and annotation queues on Arize, and applies human annotations to project spans via the Python SDK.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) using standardized prompts and metrics. Supports HuggingFace, vLLM, and API backends.

Entrena modelos de machine learning con Scikit-learn, LightGBM y XGBoost, desde un baseline hasta un modelo productivo con validación robusta y explicabilidad.

Quantifies audio source separation quality by computing the signal-to-distortion ratio (SDR) between ground-truth and estimated stems, with per-stem and record-level averaging.

Computes the HARE Score, an entity- and relation-centric metric for evaluating machine-generated histopathology reports against ground truth, using GatorTronS+SapBERT embeddings and relation F1.

Audits medical LLM benchmarks across five lifecycle phases using 46 medically tailored criteria to assess clinical relevance, data integrity, safety-critical capabilities, validity, and governance.

Benchmarks LLM bias by scoring responses to automatically generated open-ended questions across sensitive attributes, producing a composite fitness score from 0 to 5.

Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB da

You are an expert in LLM observability and evaluation. You think in terms of traces, spans, and metrics. You know that LLM applications need monitoring just like traditional software - but with different dimensions (cost, quality, latency).

Use this skill for search indexing, embeddings, RAG chunking, freshness, retrieval evaluation, source citations. Trigger when the task involves ai engineering work related to Search Indexing RAG, implementation, audits, debugging, strategy, or validation.

Submit and manage TAO training, evaluation, and inference jobs on SLURM GPU clusters over SSH with sbatch/srun, Pyxis/Enroot containers, and Lustre-backed storage.

Collects launch inputs and runs preflight checks before executing TAO workflows such as AutoML, training, evaluation, inference, export, TensorRT engine generation, or DEFT jobs on supported platforms.

Identify and analyze 3-5 distinct customer segments with demographics, jobs-to-be-done, pain points, and product fit analysis for market opportunity evaluation.

Designs and implements production-grade RAG systems by chunking documents, generating embeddings, configuring vector stores, building hybrid search pipelines, applying reranking, and evaluating retrieval quality.

Evaluate acquisition channels using unit economics, customer quality, and scalability to decide whether to scale, test, or kill a growth channel.

Tracks bidirectional confidence scores across five domains (TECH, OPS, JUDGMENT, COMMS, ORCH) for agents and users, with triggers, anti-patterns, and pool scoring per runtime.

Runs an autonomous optimization loop that iteratively improves an artifact against an objective and evaluator using Hypothesis Tree Refinement, with subagent executors in isolated git worktrees.

Evaluates text-to-image models on spatial relationship accuracy using the VISOR metric, separating object detection from spatial correctness to reveal biases like object priority and merging.

Evaluates spatial realism and temporal flow consistency of AI-generated videos using embedding spaces and Fourier transforms, producing bounded STREAM-S and STREAM-T scores.

Evaluates open-ended generation quality of multilingual LLMs across brainstorming, planning, and long-form tasks, using AYA and DOLLY datasets with qualitative fluency and quality scoring.

Evaluates social bias in question-answering models using the BBQ benchmark, measuring accuracy and a bias score across ambiguous and disambiguated contexts to reveal reliance on stereotypes.

You are an expert in LLM observability and evaluation. You think in terms of traces, spans, and metrics. You know that LLM applications need monitoring just like traditional software - but with different dimensions (cost, quality, latency).