# MCP Evaluator

> MCP Evaluator

- Skill: `lunasoft2001/mcp-evaluator` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add lunasoft2001/mcp-evaluator`
- Raw SKILL.md: https://api.skillmd.com/api/skills/lunasoft2001/mcp-evaluator/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: lunasoft2001 (https://skillmd.com/u/lunasoft2001)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/lunasoft2001/mcp-evaluator

---


# MCP Evaluator

Perform a three-phase audit of an MCP server and produce a structured report with scored findings and actionable fixes.

## Preparation

Collect these artifacts from the MCP being evaluated:

1. **Source code** — `src/` or main implementation files (TypeScript, Python, etc.)
2. **Tool definitions** — where tools are registered (e.g., `src/index.ts`, `server.py`)
3. **Configuration** — `mcp.json` or `.env.example` or equivalent
4. **Package manifest** — `package.json` / `requirements.txt` / `pyproject.toml`
5. **Companion skill** — `~/.copilot/skills/<mcp-name>/SKILL.md` if it exists

Load the two reference checklists:
- `references/mcp-rai-checklist.md`
- `references/mcp-quality-checklist.md`

## Evaluation Workflow

### Phase 1 — Security & Privacy (OWASP / RAI)

Evaluate against `references/mcp-rai-checklist.md`.

Sections:
1. Credential & Secret Handling
2. PII and Data Privacy
3. Destructive Operations Safety
4. Injection & SSRF Risks
5. Authentication & Token Security

Score each section: **PASS / PARTIAL / FAIL**
Quote exact file + line number as evidence where possible.

### Phase 2 — Technical Quality

Evaluate against `references/mcp-quality-checklist.md`.

Sections:
1. Tool Design (descriptions, parameters, types)
2. Error Handling
3. Companion Skill existence
4. Dependencies & Supply Chain
5. Logging & Observability

Score each section: **PASS / PARTIAL / FAIL**

### Phase 3 — Accountability

| Check | Score | Evidence |
|-------|-------|----------|
| License field in package manifest | - | - |
| Source code in public/private repo | - | - |
| Changelog or version tracking | - | - |
| Author / maintainer contact | - | - |

## Report Format

```markdown
# MCP Audit Report: <mcp-name>

**Date**: <today>
**Auditor**: GitHub Copilot (mcp-evaluator)
**Source analyzed**: <files read>

---

## Phase 1: Security & Privacy

| Section | Score | Evidence |
|---------|-------|----------|
| Credential & Secret Handling | PASS/PARTIAL/FAIL | <file:line or description> |
| PII and Data Privacy | ... | ... |
| Destructive Operations Safety | ... | ... |
| Injection & SSRF Risks | ... | ... |
| Authentication & Token Security | ... | ... |

**Security Score**: X/5 passing

---

## Phase 2: Technical Quality

| Section | Score | Evidence |
|---------|-------|----------|
| Tool Design | PASS/PARTIAL/FAIL | ... |
| Error Handling | ... | ... |
| Companion Skill | ... | ... |
| Dependencies & Supply Chain | ... | ... |
| Logging & Observability | ... | ... |

**Quality Score**: X/5 passing

---

## Phase 3: Accountability

| Check | Score | Evidence |
|-------|-------|----------|
| License | PASS/PARTIAL/FAIL | ... |
| Source repo | ... | ... |
| Version tracking | ... | ... |
| Maintainer | ... | ... |

---

## Overall Verdict

**SAFE TO USE** / **NEEDS FIXES** / **SECURITY RISK**

---

## Blockers (must fix before deploying)

<FAIL findings only>

## Recommendations (non-blocking)

<PARTIAL findings and improvement suggestions>
```

## Grading Rules

- **PASS**: Criterion fully met with clear evidence.
- **PARTIAL**: Partially met — issue present but not severe.
- **FAIL**: Clear violation or complete absence of a required security control.

**Overall verdict logic**:
- Any FAIL in Phase 1 → **SECURITY RISK** (stop, fix before any use)
- Any FAIL in Phase 2 → **NEEDS FIXES**
- All PASS or PARTIAL → **SAFE TO USE**

## Notes

- When reading source files, search for: `token`, `key`, `secret`, `password`, `credential`, `apiKey`, `Bearer` to detect potential secret exposure.
- For TypeScript MCPs, the tool registration is typically in `src/index.ts` or `src/tools/`.
- If no companion SKILL.md exists and the MCP performs consequential actions (publishing, sending, deleting), flag as PARTIAL under both Destructive Operations Safety and Companion Skill.
- Do not penalize MCPs that correctly use environment variables (`process.env.TOKEN`) for credentials — that is the correct pattern.