Results for “llm-as-judge”
64 skills
github Arize Evaluator
Creates and runs LLM-as-judge evaluators on Arize, including managing tasks, column mappings, and continuous monitoring.
36.2k · bundle
ssrjkk LLM Eval
Evaluates LLM performance using BLEU, ROUGE metrics and LLM-as-judge. Use for model testing.
2 · bundle
muratcankoylan Advanced Evaluation
Provides production-grade techniques for evaluating LLM outputs using LLMs as judges, covering direct scoring, pairwise comparison, bias mitigation, rubric generation, and confidence calibration.
16.9k · bundle
dvy1987 Eval Judge
Score LLM and agent outputs using LLM-as-judge techniques — direct scoring against rubrics or pairwise comparison between two outputs. Includes built-in bias mitigation for position bias, length bias, and self-enhancement bias. Load when the user asks to score an output, judge a response, evaluate against a rubric, compare two outputs, do direct scoring, run pairwise comparison, or says "rate this", "which response is better", "score this against the rubric", "judge this output", "LLM as judge this". Sub-skill of eval-output orchestrator.
3 · bundle
github Eval Driven Dev
Build automated evaluation pipelines for Python LLM applications using real LLM calls and structured test datasets.
36.2k · bundle
dvy1987 Eval Output
Orchestrator for the eval-output skill suite — evaluate LLM and agent outputs for quality, accuracy, helpfulness, and safety using structured rubrics and LLM-as-judge techniques. Load when the user says "evaluate this output", "score this response", "run an eval", "LLM as judge", "evaluate agent output", "how good is this response", "rate this answer", "eval this", or provides an LLM output that should be assessed for quality. Single entry point for all output evaluation workflows.
3 · bundle
More results
projectious-work LLM Evaluation
LLM output evaluation — automated metrics, LLM-as-judge, A/B testing, regression testing. Use when measuring LLM output quality, comparing prompt or model versions, building an automated eval pipeline, setting up regression tests for prompt changes, or evaluating RAG systems and bias/safety.
0
kursku Advanced Evaluation
This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.
55 · bundle
antigravity Evaluation
Build evaluation frameworks for agent systems, covering rubric design, test set creation, and automated evaluation pipelines.
42.4k
projectious-work Eval Gate Authoring
Turn observed run outputs into eval-spec Artifacts, paired Gates, and policy bindings. Use when creating or calibrating automated, human, or LLM-as-judge eval gates for processkit workflows.
0 · bundle
akillness Opik
Run Comet's Opik — open-source LLM observability, evaluation, and optimization — from one routing-first skill: install the Python/TypeScript SDK, stand up a server (Comet.com cloud, Docker Compose via `./opik.sh`, or Kubernetes/Helm), wire tracing through `@opik.track` or one of 50+ framework integrations (OpenAI, Anthropic, LangChain, LangGraph, LlamaIndex, CrewAI, DSPy, Ollama, Bedrock, Vercel AI SDK, …), score outputs with LLM-as-a-judge metrics (Hallucination, Moderation, Answer Relevance, Context Precision), and run Datasets/Experiments evaluations including PyTest CI gates. Use when the user wants LLM tracing, prompt evaluation, production LLM monitoring, agent optimization, or guardrails with Opik. Triggers on: opik, comet opik, opik configure, opik.sh, llm observability, llm tracing, llm as a judge, hallucination metric, prompt evaluation, opik dashboard, opik guardrails, agent optimizer.
42 · bundle
sinhoneyy Eval
Evaluate and rank agent results by metric or LLM judge for an AgentHub session. Use when the user runs /hub:eval or asks to score, compare, or pick a winner among completed AgentHub agents.
11
thedixitjain Eval
Evaluate and rank agent results by metric or LLM judge for an AgentHub session. Use when the user runs /hub:eval or asks to score, compare, or pick a winner among completed AgentHub agents.
2
pablolion Bmad Advanced Elicitation
Push the LLM to reconsider, refine, and improve its recent output. Use when user asks for deeper critique or mentions a known deeper critique method, e.g. socratic, first principles, pre-mortem, red team.
12 · bundle
lambenthan Exp Eval
实验判决门：Review LLM 独立评判实验结果 → 4 种判决路径 → 自动更新 claims confidence、ideas status、graph edges
77
fukukei23 Sentaku
選択肢(A/B/C)の深掘り比較→淘汰→推奨で判断負担を下げ判断の質を上げるスキル。5段階（L1固定3点/L1.5案拡張Diverge・自動/L2評価軸マトリクス/L3複数LLM弁証論/L4過去判断照合）。 「比較して」「深掘りして」「メリデメ教えて」「お勧めは？」「徹底的に」「過去の判断と照合」「前にどう決めたっけ」「/sentaku」等で発火。teian(浅)の深掘り要求を受け取り、brainstorming(深:設計全体)と棲み分け。
0
salacoste Bmad Advanced Elicitation
Push the LLM to reconsider, refine, and improve its recent output. Use when user asks for deeper critique or mentions a known deeper critique method, e.g. socratic, first principles, pre-mortem, red team.
1 · bundle
alirezarezvani Eval
Evaluate and rank agent results by metric or LLM judge for an AgentHub session.
20.4k
delorenj Bmad Advanced Elicitation
Push the LLM to reconsider, refine, and improve its recent output. Use when user asks for deeper critique or mentions a known deeper critique method, e.g. socratic, first principles, pre-mortem, red team.
1 · bundle
aaaaqwq Legal
Think through any legal situation like a lawyer. Issue spotting, jurisdiction, risk assessment, actionable conclusions.
1 · bundle
yanacuti1121 Ragas
Evaluate RAG pipelines with Ragas — measure faithfulness, answer relevancy, context precision/recall, and noise sensitivity using LLM-as-judge metrics; run automated test suite generation with TestsetGenerator; integrate with LangChain, LlamaIndex, and CI pipelines.
2
jarbitechture Eval
Evaluate and rank agent results by metric or LLM judge for an AgentHub session.
0
lambenthan Review
通用跨模型审查：Review LLM 对任意研究制品进行独立评审，输出结构化评分、wiki 实体映射与改进建议
77
q2805187159 LLM Wiki
Karpathy's LLM Wiki — build and maintain a persistent, interlinked markdown knowledge base. Ingest sources, query compiled knowledge, and lint for consistency.
3
fukukei23 Multi LLM Review
multi-llm-review
0 · bundle
jiachen-t-wang Probing Multimodal Llms As World Models For Driving Arxiv 24
Probing Multimodal LLMs as World Models for Driving
6
lucassantana-dev Eval
Evaluate LLM outputs systematically — benchmarks, automated metrics, human preference, and regression tracking
1 · bundle
orchestra-research Evaluating Llms Harness
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) using standardized prompts and metrics. Supports HuggingFace, vLLM, and API backends.
10.4k · bundle
jiachen-t-wang Llava Critic Learning To Evaluate Multimodal Models Arxiv 24
LLaVA-Critic: Learning to Evaluate Multimodal Models
6
dylanckawalec Eval
Evaluate and rank agent results by metric or LLM judge for an AgentHub session.
3
antigravity LLM Council
Run Fireworks-hosted open-weight model councils that compare responses and synthesize a final answer.
42.4k · bundle
github Phoenix Evals
Build and run evaluators for AI/LLM applications using Phoenix, covering error analysis, custom evaluators, experiments, and production monitoring.
36.2k · bundle
bytesagain LLM
Build and evaluate LLM prompts. Use when crafting system prompts, comparing variants, estimating tokens, or managing prompt templates.
12 · bundle
affaan-m Loop Design Check
Designs and reviews feedback loops for AI agents to ensure goals are machine-decidable, loops are damped, and human judgment is preserved.
226k
qcmuu Evaluating Llms Harness
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
0 · bundle
delorenj Bmad Checkpoint Preview
LLM-assisted human-in-the-loop review. Make sense of a change, focus attention where it matters, test. Use when the user says "checkpoint", "human review", or "walk me through this change".
1 · bundle