Native Agent Skill Validation
Establish Ownership
Use the runtime-provided skill-creator skill for skill creation, editing, package structure, and basic validation. Begin this workflow only after a candidate skill exists.
Load native-subagents-first before using native agents. Follow it as the sole lifecycle, model, budget, resume, and direct-tool contract.
Define The Forward Test
- State one behavior the skill should cause.
- Define pass, fail, and inconclusive criteria before testing.
- Choose an observable that distinguishes the intended mechanism from plausible alternatives. If every alternative can pass, the test is inconclusive.
- Build a realistic user-style task from raw artifacts.
- Withhold the intended answer, suspected defect, proposed fix, and prior conclusions.
- Keep permissions and side effects no broader than the real task requires.
Hold Conditions Fixed
Compare treatment and control with:
- the same Astra model and reasoning effort;
- the same role, tool surface, prompt, permissions, and artifact snapshot;
- a fresh child and uncontaminated workspace state for every trial;
- neutral model-visible trial identities generated before condition assignment;
- randomized or counterbalanced run order;
- only the candidate skill condition changed.
Do not reuse children across treatment and control. Do not leave prior outputs, patches, filenames, or evaluator notes where later trials can discover them.
Mark any model, effort, prompt, tool, permission, artifact, or policy-read mismatch as inconclusive. Do not attribute the outcome to skill text until rerun under matching conditions.
Run And Verify
- Run the basic
quick_validate.py check from skill-creator.
- Choose the claim:
- Natural trigger: leave candidate metadata available and use a realistic prompt that does not name the skill. Audit loaded, read, announced, and applied separately. This tests retrieval and usability, not revision effect.
- Revision effect: compare isolated prior-revision and candidate-revision conditions under the same prompt; verify which body each condition read and applied.
- Treat explicit invocation as invoked-usability evidence, not natural-trigger or revision-effect evidence. Never label a run "without the candidate" when it could read or apply it.
- Predeclare the expected directional result and score each independently testable branch; do not promote an untested branch.
- Before spawning, persist exact prompt and policy bytes and hashes, a direct policy diff, neutral run IDs, assignment/order, isolated workspace manifests, and the rubric outside the test workspace.
- Wait for and preserve each complete final answer plus a bounded lossless structural extract; never export a whole rollout. Follow the rollout-audit evidence boundary.
- Score complete answers against the rubric without revealing condition, then verify cited files, outputs, commands, and mutations in the parent.
- Treat status as lifecycle evidence only; never treat a completed status as proof the task succeeded.
- Repeat independent isolated comparisons before promoting a behavior claim.
- Send instruction defects back to
skill-creator for editing, then start a fresh validation cycle.
Use at least two independent matched trials per condition for a behavior-promotion claim. Use a single matched pair only as exploratory evidence and label it non-promotable.
Judge
Check whether:
- frontmatter triggered for the right request;
- the body and directly linked references supplied enough guidance;
- behavior improved under the fixed worker condition;
- the result generalized beyond wording copied from the skill;
- final evidence survived parent verification;
- no contamination or unintended mutation occurred.
Report
- Candidate skill and behavior claim.
- Fixed Astra worker condition.
- Control and treatment prompts.
- Trial count and fresh-state method.
- Exact skill availability/revisions, prompt and policy bytes/hashes/diff, neutral assignment, run order, workspace manifests, and rollout IDs.
- Complete child answers, structural records, blind per-run classification, and parent verification.
- Pass, fail, or inconclusive result.
- Confounds, exact behavior branches supported, and whether each result is exploratory or promotable.
1---2name: native-agent-skill-validation3description: Forward-test an existing or changed Codex skill with fresh, contamination-resistant native-agent trials. Use to test whether instructions trigger, generalize, and change behavior under a fixed Astra worker condition; do not use to create or update the skill itself.4---56# Native Agent Skill Validation78## Establish Ownership910Use the runtime-provided `skill-creator` skill for skill creation, editing, package structure, and basic validation. Begin this workflow only after a candidate skill exists.1112Load [native-subagents-first](../native-subagents-first/SKILL.md) before using native agents. Follow it as the sole lifecycle, model, budget, resume, and direct-tool contract.1314## Define The Forward Test15161. State one behavior the skill should cause.172. Define pass, fail, and inconclusive criteria before testing.183. Choose an observable that distinguishes the intended mechanism from plausible alternatives. If every alternative can pass, the test is inconclusive.194. Build a realistic user-style task from raw artifacts.205. Withhold the intended answer, suspected defect, proposed fix, and prior conclusions.216. Keep permissions and side effects no broader than the real task requires.2223## Hold Conditions Fixed2425Compare treatment and control with:2627- the same Astra model and reasoning effort;28- the same role, tool surface, prompt, permissions, and artifact snapshot;29- a fresh child and uncontaminated workspace state for every trial;30- neutral model-visible trial identities generated before condition assignment;31- randomized or counterbalanced run order;32- only the candidate skill condition changed.3334Do not reuse children across treatment and control. Do not leave prior outputs, patches, filenames, or evaluator notes where later trials can discover them.3536Mark any model, effort, prompt, tool, permission, artifact, or policy-read mismatch as inconclusive. Do not attribute the outcome to skill text until rerun under matching conditions.3738## Run And Verify39401. Run the basic `quick_validate.py` check from `skill-creator`.412. Choose the claim:42 - **Natural trigger:** leave candidate metadata available and use a realistic prompt that does not name the skill. Audit loaded, read, announced, and applied separately. This tests retrieval and usability, not revision effect.43 - **Revision effect:** compare isolated prior-revision and candidate-revision conditions under the same prompt; verify which body each condition read and applied.443. Treat explicit invocation as invoked-usability evidence, not natural-trigger or revision-effect evidence. Never label a run "without the candidate" when it could read or apply it.454. Predeclare the expected directional result and score each independently testable branch; do not promote an untested branch.465. Before spawning, persist exact prompt and policy bytes and hashes, a direct policy diff, neutral run IDs, assignment/order, isolated workspace manifests, and the rubric outside the test workspace.476. Wait for and preserve each complete final answer plus a bounded lossless structural extract; never export a whole rollout. Follow [the rollout-audit evidence boundary](../native-agent-evals/references/rollout-audit.md#export-bounded-evidence).487. Score complete answers against the rubric without revealing condition, then verify cited files, outputs, commands, and mutations in the parent.498. Treat status as lifecycle evidence only; never treat a completed status as proof the task succeeded.509. Repeat independent isolated comparisons before promoting a behavior claim.5110. Send instruction defects back to `skill-creator` for editing, then start a fresh validation cycle.5253Use at least two independent matched trials per condition for a behavior-promotion claim. Use a single matched pair only as exploratory evidence and label it non-promotable.5455## Judge5657Check whether:5859- frontmatter triggered for the right request;60- the body and directly linked references supplied enough guidance;61- behavior improved under the fixed worker condition;62- the result generalized beyond wording copied from the skill;63- final evidence survived parent verification;64- no contamination or unintended mutation occurred.6566## Report6768- Candidate skill and behavior claim.69- Fixed Astra worker condition.70- Control and treatment prompts.71- Trial count and fresh-state method.72- Exact skill availability/revisions, prompt and policy bytes/hashes/diff, neutral assignment, run order, workspace manifests, and rollout IDs.73- Complete child answers, structural records, blind per-run classification, and parent verification.74- Pass, fail, or inconclusive result.75- Confounds, exact behavior branches supported, and whether each result is exploratory or promotable.
Kbediako