Rubric Evaluator

Rubric Evaluator

Evaluate a skill directory with a 6-section, 31-item rubric. Use deterministic scripts for structural and safety checks, then apply semantic model checks from the bundled rubric reference. If the runtime or installation path is incomplete, use the fallback playbook and label the result as provisional.

Script paths

The bundled scripts live in this skill's own scripts/ directory. On Claude Code, reference them through the plugin root so they resolve after installation: ${CLAUDE_PLUGIN_ROOT}/skills/rubric-evaluator/scripts/<name>. If ${CLAUDE_PLUGIN_ROOT} is unset — a plain checkout or the Codex host — drop the prefix and run scripts/<name> from this skill's directory. The commands below use the ${CLAUDE_PLUGIN_ROOT} form.

Workflow

1. Confirm the target is a directory containing SKILL.md. If the path is missing or ambiguous, ask for the exact skill directory.

2. Run the deterministic checks with the portable wrapper:

   sh "${CLAUDE_PLUGIN_ROOT}/skills/rubric-evaluator/scripts/run_checks.sh" <target-skill-dir>
   

   The wrapper tries python3, then python, then the Windows py -3 launcher. If the wrapper cannot run, read references/fallbacks.md and continue in the documented degraded mode.

3. If the wrapper is unavailable but a Python interpreter is known, run the script directly:

   python3 "${CLAUDE_PLUGIN_ROOT}/skills/rubric-evaluator/scripts/check_rules.py" <target-skill-dir>
   

   The script returns JSON with findings, a provisional rule-only grade, coverage, and provenance (rubric version/hash and target hash). Rule-only output is always partial: 14 semantic checks remain.

4. Read references/model-rubric.md after the rule output is available. Apply all 14 model checks and produce findings with the same schema as the rule findings.

5. Combine rule and model findings into the rule result's findings array, preserving its provenance fields. Keep every finding, including pass and na, so the final grade is auditable. If rule finding 6.1 is na, it lists suspected credential lines that need review: read them, set 6.1 to fail for a real secret or pass for a documented example value, and change its checker to model. Replace 6.1 rather than appending a duplicate. Evaluate the same target snapshot throughout; if files change, rerun the evaluation.

6. Validate and refresh the combined result with scripts/check_rules.py <combined-findings.json> --finalize, saving stdout to a different file. Add --require-complete when all 31 checks must be judged. A result is complete only when all IDs are present and none is na; otherwise label the grade provisional. The script computes the grade from failed findings:

   • Any failed BLOCKER means F.
   • No failed BLOCKER and no failed MAJOR means S.
   • No failed BLOCKER and 1-2 failed MAJOR means A.
   • No failed BLOCKER and 3-4 failed MAJOR means B.
   • No failed BLOCKER and 5 or more failed MAJOR means C.
   • Failed MINOR findings do not affect the grade.

7. Render the report. If you have written combined findings to a JSON file, run:

   python3 "${CLAUDE_PLUGIN_ROOT}/skills/rubric-evaluator/scripts/render_report.py" <combined-findings.json> --skill-name <skill-name>
   

   If no file is written, mirror the same report structure manually.

Fallbacks

Read references/fallbacks.md when scripts/run_checks.sh fails, Python is unavailable, the skill is being used from an unpacked repo instead of an installed skill, the target path is outside the current workspace, or the user needs a clear next-hardening plan.

When fallback mode is used, state it in the TL;DR. Do not pretend a manual scan is equivalent to the deterministic rule script. If deterministic rule checks cannot run, mark the report as provisional, keep any uncertain rule findings as na, and list the missing runtime or installation condition as residual risk.

Finding Schema

Use this schema for every rule and model finding:

{
  "id": "3.4",
  "section": "trigger",
  "item": "body-only trigger anti-pattern absent",
  "severity": "BLOCKER",
  "status": "fail",
  "checker": "model",
  "why": "The description does not include the invocation condition.",
  "how_to_fix": "Move both the skill capability and trigger condition into the description."
}

Allowed values:

• severity: BLOCKER, MAJOR, MINOR
• status: pass, fail, na
• checker: rule, model

Report Rules

Lead with the grade and failed finding counts. Put failed BLOCKER findings first, failed MAJOR findings second, and failed MINOR findings last. For each failed finding, include both:

• why the issue matters
• how to fix it

Keep passed findings collapsed into section summaries unless the user asks for the full matrix. State evaluation coverage beside the grade. Never present missing or na checks as PASS. Empty or invalid findings are input errors, not an S grade.

Resource Use

• Use scripts/check_rules.py for deterministic checks, rule-only grading, and JSON output.
• Use scripts/run_checks.sh as the first-choice launcher for deterministic checks across common Python command names.
• Use scripts/render_report.py when combined findings are available as JSON and a markdown report is useful.
• Use scripts/compare_results.py to compare saved before/after results; only a failed check that now passes is resolved.
• scripts/results.py provides shared schema validation, coverage, provenance, and quality gates for these commands; it is imported, not run directly.
• Read references/automation.md when comparing evaluations, evaluating multiple skills, using CI gates, or interpreting provenance.
• Read references/model-rubric.md only after running deterministic checks or when authoring model findings; it contains the 14 semantic checks, pass/fail criteria, examples, and wording guidance.
• Read references/fallbacks.md only when runtime, installation, path, or platform constraints prevent the normal workflow, or when the user asks for follow-up hardening work.