Severity Testing Experimental Design Lens
Philosophical Mode: Falsificationist
Primary Question: "Would this design have caught the error?"
Focus: Adversarial Cases, Negative Controls, Falsification Tests, Easy-Pass Detection, Confirmatory Theater
Arguments
/autoskillit:exp-lens-severity-testing [context_path] [experiment_plan_path]
- context_path (optional positional arg 1) — Absolute path to a lens context file
containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.
If provided, read this file before beginning analysis to obtain structured context.
If omitted, discover context by exploring the CWD.
- experiment_plan_path (optional positional arg 2) — Absolute path to the full
experiment plan. If provided, read for complete experimental methodology and design.
If omitted, locate the experiment plan by exploring the CWD.
When to Use
- Evaluating whether positive results are meaningful or trivially achievable
- Checking for adversarial robustness of experimental conclusions
- User invokes
/autoskillit:exp-lens-severity-testing or /autoskillit:make-experiment-diag severity
Critical Constraints
NEVER:
- Modify any source code files
- Accept a "pass" result without asking what a false result would have looked like under this design
- Create files outside
{{AUTOSKILLIT_TEMP}}/exp-lens-severity-testing/
- Run subagents in the background (
run_in_background: true is prohibited)
ALWAYS:
For every positive claim, identify what error the test was capable of detecting
Inventory negative controls and sanity checks explicitly — their absence is a finding
Rate severity before reporting conclusions, not after
Flag confirmatory theater: experiments designed to confirm rather than risk refutation
BEFORE creating any diagram, LOAD the /autoskillit:mermaid skill using the Skill tool - this is MANDATORY
If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.
Write output to {{AUTOSKILLIT_TEMP}}/exp-lens-severity-testing/exp_diag_severity_testing_{YYYY-MM-DD_HHMMSS}.md
After writing the file, emit the structured output token as literal plain text with no
markdown formatting on the token name (the adjudicator performs a regex match):
diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-severity-testing/exp_diag_severity_testing_{...}.md
Analysis Workflow
Step 0: Parse optional arguments
If positional arg 1 (context_path) is provided and the file exists, read it to obtain
IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional
arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full
methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD
exploration for these fields if the context file supplies them.
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Positive Results Claimed
- Find all conclusions and positive claims in the experiment
- Look for: demonstrates, improves, outperforms, achieves, shows, confirms, validates
Negative Controls & Sanity Checks
- Find negative controls, baselines, and sanity check tests
- Look for: negative_control, sanity, ablation, degenerate, trivial, null, random
Adversarial Conditions
- Find adversarial or stress-test conditions applied
- Look for: adversarial, attack, stress, perturbation, corruption, noise, edge_case
Alternative Explanations Tested
- Find whether alternative explanations were examined
- Look for: alternative, confound, artifact, spurious, coincidence, luck
Prediction Specificity
- Find how specific the predictions were before seeing data
- Look for: prediction, hypothesis, preregistered, expected, prior
Step 2: Assess Severity for Each Claim
For each claim:
- What error was the test capable of detecting?
- What would a false positive result have looked like under this design?
- Were negative controls or sanity checks included?
- Were adversarial conditions tested?
- Is the test informative (would a bad result look different from a good result)?
Step 3: Rate Severity and Identify Gaps
Severity ratings: HIGH / MEDIUM / LOW
Flag confirmatory theater when design is structured to confirm rather than risk refutation.
Step 4: Create Optional Severity-Flow Diagram
Show Claims → HIGH/MEDIUM/LOW severity tests → Severity verdicts.
Step 5: Write Output
Write the analysis to: {{AUTOSKILLIT_TEMP}}/exp-lens-severity-testing/exp_diag_severity_testing_{YYYY-MM-DD_HHMMSS}.md (relative to the current working directory)
Pre-Diagram Checklist
Before creating the diagram, verify:
Related Skills
/autoskillit:make-experiment-diag - Parent skill
/autoskillit:mermaid - MUST BE LOADED before creating diagram
/autoskillit:exp-lens-error-budget
/autoskillit:exp-lens-validity-threats
1---2name: exp-lens-severity-testing3categories: [exp-lens]4activate_deps: [mermaid]5description: Analyze severity of experimental tests — adversarial cases, negative controls, falsification tests, easy-pass detection, and confirmatory theater. Falsificationist lens answering "Would this design have caught the error?"6hooks:7 PreToolUse:8 - matcher: "*"9 hooks:10 - type: command11 command: "echo 'Severity Testing Lens - Analyzing adversarial robustness of experimental conclusions...'"12 once: true13metadata:14 source: https://skillmd.com/skills/talont-org/exp-lens-severity-testing15---1617# Severity Testing Experimental Design Lens1819**Philosophical Mode:** Falsificationist20**Primary Question:** "Would this design have caught the error?"21**Focus:** Adversarial Cases, Negative Controls, Falsification Tests, Easy-Pass Detection, Confirmatory Theater2223## Arguments2425`/autoskillit:exp-lens-severity-testing [context_path] [experiment_plan_path]`2627- **context_path** (optional positional arg 1) — Absolute path to a lens context file28 containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.29 If provided, read this file before beginning analysis to obtain structured context.30 If omitted, discover context by exploring the CWD.31- **experiment_plan_path** (optional positional arg 2) — Absolute path to the full32 experiment plan. If provided, read for complete experimental methodology and design.33 If omitted, locate the experiment plan by exploring the CWD.3435## When to Use3637- Evaluating whether positive results are meaningful or trivially achievable38- Checking for adversarial robustness of experimental conclusions39- User invokes `/autoskillit:exp-lens-severity-testing` or `/autoskillit:make-experiment-diag severity`4041## Critical Constraints4243**NEVER:**44- Modify any source code files45- Accept a "pass" result without asking what a false result would have looked like under this design46- Create files outside `{{AUTOSKILLIT_TEMP}}/exp-lens-severity-testing/`47- Run subagents in the background (`run_in_background: true` is prohibited)4849**ALWAYS:**50- For every positive claim, identify what error the test was capable of detecting51- Inventory negative controls and sanity checks explicitly — their absence is a finding52- Rate severity before reporting conclusions, not after53- Flag confirmatory theater: experiments designed to confirm rather than risk refutation54- BEFORE creating any diagram, LOAD the `/autoskillit:mermaid` skill using the Skill tool - this is MANDATORY55- If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.56- Write output to `{{AUTOSKILLIT_TEMP}}/exp-lens-severity-testing/exp_diag_severity_testing_{YYYY-MM-DD_HHMMSS}.md`57- After writing the file, emit the structured output token as **literal plain text** with no58 markdown formatting on the token name (the adjudicator performs a regex match):5960 ```61 diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-severity-testing/exp_diag_severity_testing_{...}.md62 ```6364---6566## Analysis Workflow6768### Step 0: Parse optional arguments6970If positional arg 1 (context_path) is provided and the file exists, read it to obtain71IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional72arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full73methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD74exploration for these fields if the context file supplies them.7576### Step 1: Launch Parallel Exploration Subagents7778Spawn Explore subagents to investigate:7980**Positive Results Claimed**81- Find all conclusions and positive claims in the experiment82- Look for: demonstrates, improves, outperforms, achieves, shows, confirms, validates8384**Negative Controls & Sanity Checks**85- Find negative controls, baselines, and sanity check tests86- Look for: negative_control, sanity, ablation, degenerate, trivial, null, random8788**Adversarial Conditions**89- Find adversarial or stress-test conditions applied90- Look for: adversarial, attack, stress, perturbation, corruption, noise, edge_case9192**Alternative Explanations Tested**93- Find whether alternative explanations were examined94- Look for: alternative, confound, artifact, spurious, coincidence, luck9596**Prediction Specificity**97- Find how specific the predictions were before seeing data98- Look for: prediction, hypothesis, preregistered, expected, prior99100### Step 2: Assess Severity for Each Claim101102For each claim:1031. What error was the test capable of detecting?1042. What would a false positive result have looked like under this design?1053. Were negative controls or sanity checks included?1064. Were adversarial conditions tested?1075. Is the test informative (would a bad result look different from a good result)?108109### Step 3: Rate Severity and Identify Gaps110111Severity ratings: HIGH / MEDIUM / LOW112Flag **confirmatory theater** when design is structured to confirm rather than risk refutation.113114### Step 4: Create Optional Severity-Flow Diagram115116Show Claims → HIGH/MEDIUM/LOW severity tests → Severity verdicts.117118### Step 5: Write Output119120Write the analysis to: `{{AUTOSKILLIT_TEMP}}/exp-lens-severity-testing/exp_diag_severity_testing_{YYYY-MM-DD_HHMMSS}.md` (relative to the current working directory)121122---123124## Pre-Diagram Checklist125126Before creating the diagram, verify:127128- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool129- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)130- [ ] Diagram will include a color legend table131132---133134## Related Skills135136- `/autoskillit:make-experiment-diag` - Parent skill137- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram138- `/autoskillit:exp-lens-error-budget`139- `/autoskillit:exp-lens-validity-threats`