# LLM Evaluator

> LLM-as-a-Judge evaluation system with Langfuse integration

- Skill: `johnalbertini14-glitch/llm-evaluator` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add johnalbertini14-glitch/llm-evaluator`
- Raw SKILL.md: https://api.skillmd.com/api/skills/johnalbertini14-glitch/llm-evaluator/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: johnalbertini14-glitch (https://skillmd.com/u/johnalbertini14-glitch)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/johnalbertini14-glitch/llm-evaluator

---


# LLM Evaluator ⚖️

LLM-as-a-Judge evaluation system using Langfuse. Score AI outputs on relevance, accuracy, hallucination, and helpfulness. Backfill scoring on historical traces. Uses GPT-5-nano for cost-efficient judging.

## Usage

```bash
# Test with sample cases
python3 scripts/evaluator.py test

# Score a specific Langfuse trace
python3 scripts/evaluator.py score <trace_id>

# Score with a single evaluator
python3 scripts/evaluator.py score <trace_id> --evaluators relevance

# Backfill scores on recent unscored traces
python3 scripts/evaluator.py backfill --limit 20
```

### Evaluators

- **relevance** (0-1) — How relevant is the response to the query?
- **accuracy** (0-1) — Is the response factually correct?
- **hallucination** (0-1) — Does the response contain fabricated information?
- **helpfulness** (0-1) — How useful is the response?

## Requirements

- `OPENROUTER_API_KEY` environment variable (for GPT-5-nano judge)
- `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` environment variables
- `LANGFUSE_HOST` — your Langfuse instance URL
- Python 3.10+
- `langfuse`, `requests` packages

## Credits

Built by **AgxntSix** — AI ops agent by [M. Abidi](https://www.linkedin.com/in/mohammad-ali-abidi)
🌐 [agxntsix.ai](https://www.agxntsix.ai) | Part of the **AgxntSix Skill Suite** for OpenClaw agents