Skill Battle
Default behavior
- State one narrow claim the target skill should improve.
- Choose representative tasks that measure that claim, not generic model quality.
- Run identical tasks with and without the skill. Use multiple trials for stochastic tasks.
- Prefer deterministic graders; use blind LLM judging only where necessary.
- Track regressions, cost/time, token usage, and user burden alongside quality.
- Keep full prompts, outputs, traces, and local artifacts local by default.
- Use a train/held-out split if the skill or rubric is tuned during the experiment.
- Never publish demo/synthetic numbers as benchmark results.
- For multi-turn skills, simulate the same hidden user state in both conditions rather than flattening the workflow into one prompt.
- Use the repository's sibling
skill-eval-loop when available. It already enforces blind paired runs, skill-load checks, deterministic verification, and strict-majority adoption.
Publication
Run locally first. Do not imply a cloud submission happened unless the configured Edge endpoint returned a canonical result identifier.
The default public record is Results Only: scores, deltas, exact model, skill and method hashes, counts, cost, latency and token aggregates, plus technical provenance. It excludes prompts, outputs, transcripts, private files, secrets, and proprietary instructions.
Before the first upload, disclose the destination and fields. Then offer optional sharing:
insights: derived failure, improvement, and regression tags plus redacted findingsevidence: selected raw or redacted benchmark inputs, outputs, grader traces, or transcripts
Never silently escalate disclosure.
Trust
Treat trust and disclosure separately:
- Community → Reproduced → Attested → Edge Verified
- Results Only → Insights → Evidence
A Reproduced Results Only result is valid social evidence even when private prompts never leave the runner's machine.
Never invent a Battle URL. If no Edge publishing endpoint is configured, return the local evidence path and state that hosting remains pending.
1---2name: skill-battle3description: Run a credible with-skill versus without-skill evaluation for an Agent Skill, preserve evidence, and prepare a public-safe result. Use when the user asks whether a skill actually works, wants to benchmark a skill, compare a baseline with a skill, or generate proof for a skill claim.4---56# Skill Battle78## Default behavior9101. State one narrow claim the target skill should improve.112. Choose representative tasks that measure that claim, not generic model quality.123. Run identical tasks with and without the skill. Use multiple trials for stochastic tasks.134. Prefer deterministic graders; use blind LLM judging only where necessary.145. Track regressions, cost/time, token usage, and user burden alongside quality.156. Keep full prompts, outputs, traces, and local artifacts **local by default**.167. Use a train/held-out split if the skill or rubric is tuned during the experiment.178. Never publish demo/synthetic numbers as benchmark results.189. For multi-turn skills, simulate the same hidden user state in both conditions rather than flattening the workflow into one prompt.1910. Use the repository's sibling `skill-eval-loop` when available. It already enforces blind paired runs, skill-load checks, deterministic verification, and strict-majority adoption.2021## Publication2223Run locally first. Do not imply a cloud submission happened unless the configured Edge endpoint returned a canonical result identifier.2425The default public record is **Results Only**: scores, deltas, exact model, skill and method hashes, counts, cost, latency and token aggregates, plus technical provenance. It excludes prompts, outputs, transcripts, private files, secrets, and proprietary instructions.2627Before the first upload, disclose the destination and fields. Then offer optional sharing:2829- `insights`: derived failure, improvement, and regression tags plus redacted findings30- `evidence`: selected raw or redacted benchmark inputs, outputs, grader traces, or transcripts3132Never silently escalate disclosure.3334## Trust3536Treat trust and disclosure separately:3738- Community → Reproduced → Attested → Edge Verified39- Results Only → Insights → Evidence4041A Reproduced Results Only result is valid social evidence even when private prompts never leave the runner's machine.4243Never invent a Battle URL. If no Edge publishing endpoint is configured, return the local evidence path and state that hosting remains pending.
getedgehq