Results for “eval”

Run Comet's Opik — open-source LLM observability, evaluation, and optimization — from one routing-first skill: install the Python/TypeScript SDK, stand up a server (Comet.com cloud, Docker Compose via `./opik.sh`, or Kubernetes/Helm), wire tracing through `@opik.track` or one of 50+ framework integrations (OpenAI, Anthropic, LangChain, LangGraph, LlamaIndex, CrewAI, DSPy, Ollama, Bedrock, Vercel AI SDK, …), score outputs with LLM-as-a-judge metrics (Hallucination, Moderation, Answer Relevance, Context Precision), and run Datasets/Experiments evaluations including PyTest CI gates. Use when the user wants LLM tracing, prompt evaluation, production LLM monitoring, agent optimization, or guardrails with Opik. Triggers on: opik, comet opik, opik configure, opik.sh, llm observability, llm tracing, llm as a judge, hallucination metric, prompt evaluation, opik dashboard, opik guardrails, agent optimizer.

Improve CLI build performance by enabling MSBuild Server for persistent caching of evaluation results across sequential builds.

Manage NVIDIA Brev GPU instances for TAO training, evaluation, and inference using the Brev CLI and Docker.

Provides technical leadership frameworks for architecture decisions, engineering team scaling, technology strategy, and technical debt assessment.

Instrument LLM applications with Langfuse to trace, score, and monitor cost, quality, and latency across OpenAI and LangChain integrations.

Build AI applications using the Azure AI Projects SDK for TypeScript, managing agents, connections, deployments, datasets, indexes, and evaluations.

Runs a five-phase pipeline—evaluation, specification, implementation, review, and QA—to build a feature end-to-end with specialized agents.

Turns model work into a production ML system with data contracts, reproducible training, quality gates, deployable artifacts, and monitoring.

Evaluates a containerized framework for deploying distributed big data workloads, measuring execution time and cloud cost scaling from four to eight nodes.

Evaluates Google Cloud workloads against the Sustainability pillar of the Well-Architected Framework, providing actionable recommendations to minimize environmental impact.

Manage Azure AI Foundry projects with .NET SDK: create and run agents, manage connections, datasets, deployments, evaluations, and indexes.

Train, evaluate, export, prune, and run inference for PointPillars 3D object detection models from LiDAR point clouds using NVIDIA TAO.

Turn model work into a production ML system with data contracts, reproducible training, quality gates, deployable artifacts, and monitoring.

Measures whether coding agents actually follow skills, rules, or agent definitions by generating test scenarios, running agents, and classifying tool calls to report compliance rates.

Manage Arize AI Observability and Evaluation resources on Azure using the .NET SDK. Create, read, update, and delete Arize organizations via Azure Marketplace.

Implements production LLM operations: RAG pipelines, embeddings, vector databases, fine-tuning, advanced prompt engineering, cost estimation, quality evals, semantic caching, streaming, and agents.

Autonomously improve a real artifact (code, training recipe, agent harness, data pipeline, prompt) against an objective and an evaluator, using Hypothesis Tree Refinement (HTR) from the Arbor paper. Use this whenever someone wants to iteratively optimize something over many experiments without overfitting — e.g. "get my model's eval score up", "improve this agent/harness", "tune this pipeline", "beat the baseline on this benchmark", "run a search over approaches and keep the best", "do an MLE-bench / Kaggle-style optimization", or any long-horizon "make this artifact better and don't just memorize the dev set" task. Trigger it even when the user doesn't say "Arbor" or "hypothesis tree" but describes repeated experiment-and-evaluate loops, branching exploration of competing ideas, or worries about a dev/test gap. Runs Claude itself as the coordinator with subagent executors in...

Submit and manage TAO training, evaluation, and inference jobs on SLURM GPU clusters over SSH with sbatch/srun, Pyxis/Enroot containers, and Lustre-backed storage.

Collects launch inputs and runs preflight checks before executing TAO workflows such as AutoML, training, evaluation, inference, export, TensorRT engine generation, or DEFT jobs on supported platforms.

Assess the security posture of Kubernetes etcd clusters by evaluating encryption at rest, TLS configuration, access controls, backup encryption, and network isolation.

Evaluates Google Cloud workloads against the Reliability pillar of the Well-Architected Framework, providing actionable recommendations for building, deploying, and managing reliable systems.

Bootstraps a clinical ASR evaluation environment by verifying NVIDIA_API_KEY, installing Python dependencies, and running a smoke test against hosted TTS/ASR services.

Analyzes npm packages by fetching registry metadata, evaluating bundle size via the bundlephobia API, checking security advisories with npm audit, and mapping dependency trees using arborist.

Build, deploy, evaluate, debug, and measure latency for a GPU-accelerated FastAPI service that detects whether operators perform assembly-line steps in order via event boundary detection and VLM classification.

This skill should be used when the user asks to "start an LLM project", "design batch pipeline", "evaluate task-model fit", "structure agent project", or mentions pipeline architecture, agent-assisted development, cost estimation, or choosing between LLM and traditional approaches.

Sets up Google's agents-cli toolkit and guides through the full agent development lifecycle: scaffold, build, evaluate, deploy, publish, and monitor on Gemini Enterprise Agent Platform.

Evaluates Google Cloud workloads against the Performance Optimization pillar of the Well-Architected Framework, providing actionable recommendations for resource allocation, modular design, elasticity, and monitoring.

Execute QA test plans on Holoscan Sensor Bridge hardware by reading a test document, filtering tests by setup, running automatable tests with pass/fail evaluation, and producing a structured report.

Systematically evaluates Kubernetes cluster security by simulating attacker techniques against the API server, kubelet, etcd, pods, RBAC, network policies, and secrets using tools like kube-hunter, Kubescape, and kube-bench.

Audit Google Cloud Platform environments for security misconfigurations using Forseti, Security Command Center, and gcloud CLI to evaluate IAM policies, firewall rules, storage permissions, and CIS compliance.

Design, implement, and validate reproducible machine-learning pipelines spanning data preparation, training, evaluation, registry, and deployment gates. Use when the user requests an ML pipeline, needs to turn model scripts into an orchestrated workflow, or provides pipeline components that must be connected safely.

This skill should be used when the user asks to "perform vulnerability scanning", "scan networks for open ports", "assess web application security", "scan wireless networks", "detect malware", "check cloud security", or "evaluate system compliance". It provides comprehensive guidance on security scanning tools and methodologies.

This skill should be used when the user asks to "perform vulnerability scanning", "scan networks for open ports", "assess web application security", "scan wireless networks", "detect malware", "check cloud security", or "evaluate system compliance". It provides comprehensive guidance on security scanning tools and methodologies.

Evaluates Google Cloud workloads against the Cost Optimization pillar of the Well-Architected Framework, identifies cost requirements and constraints, and provides actionable recommendations for building, deploying, and managing workloads cost-efficiently.

Evaluates computational throughput and real-time efficiency of embedded CPU and GPU platforms by measuring peak FLOPS via a matrix rotation kernel and assessing inference latency and power consumption on a robotic vision pipeline.

Test a pipeline stage's skill file by running the stage WITH and WITHOUT the skill on the same input, comparing outputs, and proposing skill edits. Ryan Law principle 3 — recursive self-improvement. Run after any board complaint about a stage, and monthly per core stage.