Results for “mmlu”

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Run Megatron-LM (MLM) and Megatron Bridge training with mock or real data, covering correlation testing, available recipes, and multi-GPU examples.

Guides machine learning development with experiment tracking, hyperparameter optimization, model registry, and MLOps pipeline integration.

Provides a persistent memory CLI for LLM agents, installed via npx.

Use this skill whenever a developer wants to turn an idea into a complete, production-ready, end-to-end system plan BEFORE writing any code. Trigger on 'plan this system', 'design the architecture for', 'help me blueprint', 'deep plan for X', 'break this idea into components', 'expand into action points', 'full implementation plan', or when the user pastes a project idea wanting architecture, components, pipelines, and file-level execution mapped out. Casual phrasing also triggers: 'help me think this through end-to-end', 'plan before coding'. Also covers living-workspace patterns: self-improving knowledge bases, reflection loops with auditor agents, four-agent teams, schema-as-code, wiki health scoring. **Resume triggers**: 'where did we leave off', 'continue this plan', 'resume my blueprint' - rehydrates state from disk via SESSION.md/NEXT.md/decisions/. Web research is mandatory every invocation.

Design and implement ML operations — model registry, serving patterns, deployment strategies (shadow/canary/blue-green), drift detection, feature stores, retraining triggers, and prediction monitoring. Use when asked to "deploy a model", "model registry", "MLflow", "feature store", "drift detection", "retrain trigger", "shadow mode", "model versioning", "serving infrastructure", or "ML pipeline". Do NOT use for: prompt engineering or RAG pipelines — see prompt-engineering and rag-architect skills. Do NOT use for: general API deployment without an ML component.

Launch distributed Megatron-LM training jobs on a SLURM cluster with a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules, container conventions, monitoring, and per-rank failure diagnosis.

Manages the machine learning lifecycle with experiment tracking, model versioning, reproducible runs, and deployment through the MLflow platform.

Persistent memory CLI for LLM agents. Store facts, recall past knowledge, link related memories, manage lifecycle.

Production machine-learning engineering workflow for data contracts, reproducible training, model evaluation, deployment, monitoring, and rollback. Use when building, reviewing, or hardening ML systems beyond one-off notebooks.

Build production ML systems with PyTorch 2.x, TensorFlow, and modern ML frameworks, including model serving, feature engineering, A/B testing, and monitoring.

GPU-accelerated data curation for LLM training, supporting text, image, video, and audio with fuzzy deduplication, quality filtering, semantic deduplication, PII redaction, and NSFW detection.

Manage and use Lambda Labs GPU cloud instances for ML training and inference with SSH access, persistent filesystems, and multi-node clusters.

Designing and evaluating AI/ML systems: prompt engineering, RAG design, LLM application patterns, AI safety, evaluation frameworks, MLOps, cost optimization. Use for AI pipelines or eval harnesses.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production.

MLflow, model versioning, experiment tracking, model registry, and production ML systems

Turn model work into a production ML system with data contracts, reproducible training, quality gates, deployable artifacts, and monitoring.

Turns model work into a production ML system with data contracts, reproducible training, quality gates, deployable artifacts, and monitoring.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debuggin

生产级机器学习工程工作流——数据契约、可复现训练、模型评估、服务部署、监控与回滚

Orchestrates AI/ML workflows for building LLM applications, RAG systems, AI agents, and ML pipelines, covering design, integration, observability, and security.

Reserved and on-demand GPU cloud instances for ML training and inference. Use when you need dedicated GPU instances with simple SSH access, persistent filesystems, or high-performance multi-node clusters for large-scale training.

AI and machine learning workflow covering LLM application development, RAG implementation, agent architecture, ML pipelines, and AI-powered features.

Build and maintain a persistent markdown wiki that an LLM updates on the user's behalf, usually inside an Obsidian vault or git-tracked notes repo. Use when raw sources such as web articles, papers, meeting notes, transcripts, screenshots, or past analyses need to be turned into an interlinked knowledge base with immutable source files, LLM-written wiki pages, `index.md`, `log.md`, schema rules in `AGENTS.md` or `CLAUDE.md`, source summaries, query notes, and recurring lint passes. Triggers on: llm-wiki, personal wiki, obsidian wiki, research vault, knowledge base, source ingest, persistent notes, wiki maintenance, source summaries, query filing.

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.

Add programmable safety guardrails to LLM applications at runtime, including jailbreak detection, input/output validation, fact-checking, hallucination detection, PII filtering, and toxicity detection.

Evaluates LLM performance using BLEU, ROUGE metrics and LLM-as-judge. Use for model testing.

Orchestrates AI/ML workflows for building LLM applications, RAG systems, AI agents, and ML pipelines, covering design, integration, observability, and security.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production.

Write publication-ready ML/AI papers for NeurIPS, ICML, ICLR, ACL, AAAI, COLM. Use when drafting papers from research repos, structuring arguments, verifying citations, or preparing camera-ready submissions. For systems venues (OSDI, NSDI, ASPLOS, SOSP), use systems-paper-writing instead.

Track ML experiments, manage model registry with versioning, deploy models to production, and reproduce experiments with MLflow.