Results for “llm-judgment”

Evaluates LLM performance using BLEU, ROUGE metrics and LLM-as-judge. Use for model testing.

Provides production-grade techniques for evaluating LLM outputs using LLMs as judges, covering direct scoring, pairwise comparison, bias mitigation, rubric generation, and confidence calibration.

Score LLM and agent outputs using LLM-as-judge techniques — direct scoring against rubrics or pairwise comparison between two outputs. Includes built-in bias mitigation for position bias, length bias, and self-enhancement bias. Load when the user asks to score an output, judge a response, evaluate against a rubric, compare two outputs, do direct scoring, run pairwise comparison, or says "rate this", "which response is better", "score this against the rubric", "judge this output", "LLM as judge this". Sub-skill of eval-output orchestrator.

实验判决门：Review LLM 独立评判实验结果 → 4 种判决路径 → 自动更新 claims confidence、ideas status、graph edges

LLM output evaluation — automated metrics, LLM-as-judge, A/B testing, regression testing. Use when measuring LLM output quality, comparing prompt or model versions, building an automated eval pipeline, setting up regression tests for prompt changes, or evaluating RAG systems and bias/safety.

This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.

選択肢(A/B/C)の深掘り比較→淘汰→推奨で判断負担を下げ判断の質を上げるスキル。5段階（L1固定3点/L1.5案拡張Diverge・自動/L2評価軸マトリクス/L3複数LLM弁証論/L4過去判断照合）。 「比較して」「深掘りして」「メリデメ教えて」「お勧めは？」「徹底的に」「過去の判断と照合」「前にどう決めたっけ」「/sentaku」等で発火。teian(浅)の深掘り要求を受け取り、brainstorming(深:設計全体)と棲み分け。

通用跨模型审查：Review LLM 对任意研究制品进行独立评审，输出结构化评分、wiki 实体映射与改进建议

Push the LLM to reconsider, refine, and improve its recent output. Use when user asks for deeper critique or mentions a known deeper critique method, e.g. socratic, first principles, pre-mortem, red team.

multi-llm-review

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) using standardized prompts and metrics. Supports HuggingFace, vLLM, and API backends.

LLaVA-Critic: Learning to Evaluate Multimodal Models

Push the LLM to reconsider, refine, and improve its recent output. Use when user asks for deeper critique or mentions a known deeper critique method, e.g. socratic, first principles, pre-mortem, red team.

Push the LLM to reconsider, refine, and improve its recent output. Use when user asks for deeper critique or mentions a known deeper critique method, e.g. socratic, first principles, pre-mortem, red team.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Provides expertise in Langfuse for LLM observability, including tracing, prompt management, evaluation, and integration with LangChain, LlamaIndex, and OpenAI.

Build and evaluate LLM prompts. Use when crafting system prompts, comparing variants, estimating tokens, or managing prompt templates.

DeepEval — LLM evaluation framework, RAG metrics, hallucination detection, red-teaming, CI/CD integration

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Meta's 7-8B specialized moderation model for LLM input/output filtering. 6 safety categories - violence/hate, sexual content, weapons, substances, self-harm, criminal planning. 94-95% accuracy. Deploy with vLLM, HuggingFace, Sagemaker. Integrates with NeMo Guardrails.

Karpathy's LLM Wiki — build and maintain a persistent, interlinked markdown knowledge base. Ingest sources, query compiled knowledge, and lint for consistency.

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production. Use when "langfuse, llm observability, llm tracing, prompt management, llm evaluation, monitor llm, debug llm, langfuse, observability, tracing, llm-monitoring, evaluation, prompt-management, debugging, analytics" mentioned.

Audit-ready decision artifacts for LLM outputs — assumptions, risks, recommendation, and review gating (schema-valid JSON).

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency on NVIDIA GPUs (A100/H100).

LLM Wiki — persistent markdown knowledge base that compounds across sessions (Karpathy model)

Meta's 7-8B specialized moderation model for LLM input/output filtering. 6 safety categories - violence/hate, sexual content, weapons, substances, self-harm, criminal planning. 94-95% accuracy. Deploy with vLLM, HuggingFace, Sagemaker. Integrates with NeMo Guardrails.

LIMA: Less Is More for Alignment

Evaluate and rank agent results by metric or LLM judge for an AgentHub session.

Meta's 86M prompt injection and jailbreak detector. Filters malicious prompts and third-party data for LLM apps. 99%+ TPR, <1% FPR. Fast (<2ms GPU). Multilingual (8 languages). Deploy with HuggingFace or batch processing for RAG security.

Selects the optimal LLM model and provider for each task based on complexity, cost budget, and capability requirements. Routes cheap tasks to Haiku/GPT-4o-mini and complex tasks to Sonnet/Opus/o1. Use when deciding which model to call, optimizing LLM costs, or building multi-model agent systems. Activate on "which model", "model selection", "route to model", "LLM cost", "model routing", "cheap vs expensive model". NOT for prompt engineering (use prompt-engineer), model fine-tuning, or training custom models.

Multi-dimensional lead qualification scoring. Evaluates leads against BANT criteria, firmographic fit, behavioral signals, and intent indicators. Outputs qualified/disqualified verdict with detailed reasoning.

Evaluate LLM outputs systematically — benchmarks, automated metrics, human preference, and regression tracking

Produces a machine-validated, auditable JSON decision record with assumptions, risks, recommendation, and review gating for high-stakes decisions.

Acceptance Eval

Probing Multimodal LLMs as World Models for Driving