How do I install the LLM Evaluator skill?

Run npx skillmds@latest add johnalbertini14-glitch/llm-evaluator in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

What does the LLM Evaluator skill do?

LLM-as-a-Judge evaluation system with Langfuse integration It is listed under AI & ML on SkillMD.

Is LLM Evaluator safe to use?

This skill has not completed SkillMD's automated safety review yet. Independent scanners report: SkillSpector: PASS, Skill Scanner: PASS. Capability flags: executes scripts, makes network calls. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with LLM Evaluator?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is LLM Evaluator free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published LLM Evaluator?

johnalbertini14-glitch (@johnalbertini14-glitch) published this skill. Their other Agent Skills are listed on their SkillMD profile.

LLM Evaluator by johnalbertini14-glitch

Name: LLM Evaluator
Rating: 4.2 (5 reviews)
Author: johnalbertini14-glitch

File contents

LLM Evaluator ⚖️

LLM-as-a-Judge evaluation system using Langfuse. Score AI outputs on relevance, accuracy, hallucination, and helpfulness. Backfill scoring on historical traces. Uses GPT-5-nano for cost-efficient judging.

Usage

# Test with sample cases
python3 scripts/evaluator.py test

# Score a specific Langfuse trace
python3 scripts/evaluator.py score <trace_id>

# Score with a single evaluator
python3 scripts/evaluator.py score <trace_id> --evaluators relevance

# Backfill scores on recent unscored traces
python3 scripts/evaluator.py backfill --limit 20

Evaluators

relevance (0-1) — How relevant is the response to the query?
accuracy (0-1) — Is the response factually correct?
hallucination (0-1) — Does the response contain fabricated information?
helpfulness (0-1) — How useful is the response?

Requirements

OPENROUTER_API_KEY environment variable (for GPT-5-nano judge)
LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY environment variables
LANGFUSE_HOST — your Langfuse instance URL
Python 3.10+
langfuse, requests packages

Credits

Built by AgxntSix — AI ops agent by M. Abidi 🌐 agxntsix.ai | Part of the AgxntSix Skill Suite for OpenClaw agents

Frequently asked questions

LLM Evaluator