wuyoscar Evaluate outputs against evidence and criteria
Use this skill for judgments about existing outputs or observed behavior, not for
finding source locations (jev-documents) or assigning routine business labels (jev-triage).
Pick the evaluation mode
- Code review: read diff and test-evidence review and adapt the code-review template. Return review leads with source IDs and verification steps, not merge approval.
- Other output review: define the user's rubric, supply the actual output and supporting evidence, and ask independent outcome/evidence questions. Let the host write task-specific integration and tests when requested.
- Authorized safety evaluation: continue to the safety workflow below; read only the relevant batch, multi-turn or team protocol. A researcher supplies cases, an authorized harness invokes targets, and an independent checker validates outcomes. Jev does not generate attacks or grant scope.
Use safely
Choose the service once and keep that choice. If unset, ask A: real Jev via
OpenRouter (OPENROUTER_API_KEY) or TypeSafe (TYPESAFE_API_KEY), or B: simulation
with this agent or an explicitly chosen available model such as DeepSeek. Wait for
consent; errors do not authorize switching. Check key presence only, never values.
Real calls send evidence and cost money; get approval before sending private data.
For B, skip CLI/API calls. Mark agent_simulation or model_simulation, identify
the actual model when available, set jev_called: false, probability: null and
confidence: null. Return a value, evidence-based reason and needs_review; use
null/review when evidence is missing. Do not invent Jev output or probabilities.
Choice uses supplied labels, Noul uses booleans, Score uses integer rubric indices.
For A, use the existing jev-decide CLI with the chosen --provider openrouter
or --provider typesafe. If absent, explain the dependency; do not silently install.
--dry-run is offline validation, not a judgment. Exit 0 means selected/scored,
2 means review, 1 means error. Read each value: false Noul remains false. Selection
is not permission, and confidence is not accuracy. Keep unknown/review paths.
First request
Adapt the example. The shared CLI needs Python 3.10+;
no sibling skill is needed. Host tools still own collection and actions.
Resolve <skill-dir> to this installed folder:
jev-decide decide <skill-dir>/assets/example.json --dry-run
# After approval, send the edited request with the selected provider:
jev-decide decide /path/to/request.json --provider openrouter
Context and checks
Jev does not inherit the agent's history. Give each judgment enough context: the criteria, actual output, source evidence and missing facts. Put independent outcome and evidence questions in the same request. Use bounded concurrency for independent requests only; the host schedules them. Wait for new observations before dependent checks. Review leads are not merge approval or proof of intent.
For captured safety-test transcripts, read the safety workflow only when needed. The host owns target authorization and execution; this skill judges supplied evidence and does not expand the test scope.
Examples
Completion evidence check · Detect unsupported success language · Plan versus action
More tasks and local templates. Open only the matching row; there is no need to read the full README before a judgment.