# Interpret Results > Analyzes evaluation results by requiring a stated hypothesis before examining data, then compares expectations to actual result files to prevent post-hoc rationalization. - Skill: `samyakjhaveri/interpret-results` (Agent Skill) - Install (CLI): `npx skillmds add samyakjhaveri/interpret-results` - Raw SKILL.md: https://api.skillmd.com/api/skills/samyakjhaveri/interpret-results/raw - Safety review: PASS (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Research & Search - Tags: Evaluation Results, Hypothesis Testing, Json Analysis, Result Interpretation, Scientific Method - Author: SamyakJhaveri (https://skillmd.com/u/samyakjhaveri) - Updated: 2026-08-22 - Page: https://skillmd.com/skills/samyakjhaveri/interpret-results --- # Hypothesis-First Result Interpreter Use when analyzing evaluation or augmentation results. Prevents post-hoc rationalization by requiring the user to state their expectations BEFORE seeing any data. Every interpretation must be grounded in actual result files, not approximations. **Trigger:** When user types `/interpret-results` with an optional result scope. ## Iron Law ``` NO INTERPRETATION WITHOUT A PRIOR HYPOTHESIS ``` ## Arguments - `$ARGUMENTS` — optional scope for which results to analyze: - `` — e.g., `claude-sonnet`, `gemini-2.5-flash-lite` - `` — e.g., `cuda-to-omp`, `omp-to-cuda` - `` — e.g., `backprop`, `hotspot` - ` ` — combined filter - `all` — full cross-model comparison - Omit to be prompted interactively ## Anti-Rationalization Table | Excuse | Reality | |--------|---------| | "I just want to see the numbers" | Numbers without hypotheses lead to p-hacking and cherry-picked narratives | | "My hypothesis is obvious" | State it anyway — obvious hypotheses are often wrong (backprop tier inversion was "obvious") | | "This is exploratory" | Exploratory still needs a framework — define what would surprise you | | "I'll form a hypothesis after I see the data" | That is called post-hoc rationalization. It is the #1 threat to scientific credibility | | "The pattern is clear" | Clear patterns in small samples are often noise. State the null hypothesis and test it | ## Red Flags — STOP and Restart Process - Presenting an interpretation before the user has stated their hypothesis - Cherry-picking results that confirm expectations while ignoring contradictions - Citing a number without a file path reference to the actual result JSON - Using `speedup_ratio` from wall-clock timing as evidence (known unreliable) - Claiming statistical significance without a test (we have small N — be honest about it) - Generalizing from a single kernel or model to "LLMs" broadly If any red flag triggers: STOP. Return to Phase 1. Do not proceed until the user has stated their prior hypothesis. ## Project Context - **Result directories:** - `results/evaluation/together-qwen-3.5-397b-a17b/` — Qwen 3.5 397B (Phase 3 canonical + ablation) - Pre-Phase-3 data (claude-sonnet, gemini, groq-llama) was purged 2026-04-20 - **Result JSON structure:** Each file contains `overall_status` (authoritative verdict), `attempts[]` array, `build_error_snippet`, `run_stderr_snippet`, `timing_method`, `thinking_enabled`, `num_samples`, `seed`, `top_p`, `augment_level`, `sample_id` - **Failure taxonomy:** PASS, BUILD_FAIL, RUN_FAIL, VERIFY_FAIL, EXTRACTION_FAIL, TIMEOUT - **Known timing limitation:** All results use `wall_time`. `translated_cpu_time_seconds` and `translated_kernel_time_seconds` are null. Do NOT use `speedup_ratio` for claims. - **Qwen canonical (Phase 3, 2026-04-24):** - 708 total results: 504 canonical (168 pairs × 3 samples) + 204 ablation (51 pairs × L1-L4) - Overall: 243 PASS (34.3%), 290 BUILD_FAIL (41.0%), 131 RUN_FAIL (18.5%), 43 VERIFY_FAIL (6.1%) - Zero-shot (max_retries=1, no self-repair). Temperature=0.7. Thinking=on. - 146 clean pairs + 22 KNOWN_FAIL-involved (excluded by analyze_eval.py) - **Pipeline audit findings (2026-04-24):** See `docs/eval-findings/2026-04-24-qwen-pipeline-audit.md` - 18 BUILD_FAILs are "header confusion" (prompt shows headers, model #includes instead of inlining) - 8 PASS results from KNOWN_FAIL pairs (LLM "fixed" broken source during translation) - Ablation pass rate declines monotonically: L1=74.5% → L4=54.9% ## Known Confounding Variables (from pipeline audit) When interpreting results, always consider these confounds: 1. **Header confusion BUILD_FAILs (~18/290):** Prompt shows source headers and says "inline." Some models #include instead → BUILD_FAIL. This is partially prompt design, not purely model quality. 2. **Zero-shot only:** No self-repair means BUILD_FAILs that a retry could fix are counted as failures. 3. **KNOWN_FAIL pollution:** 22 pairs involving KNOWN_FAIL specs ran. Excluded by analysis, but 8 PASS results suggest LLMs can recover from broken source code — interesting signal. 4. **nvc++ strictness:** OpenMP target translations compiled with nvc++ (NVIDIA HPC SDK 24.3) which is stricter about pragma syntax than GCC. Some OMP BUILD_FAILs may pass on GCC. 5. **Verification mode asymmetry:** 527 results use cross_api_combined_pattern, 181 use kernel_only_target_pattern. Pass rates differ by verification mode — compare within mode. ## Workflow ### Phase 1: Collect Prior Hypothesis (MANDATORY — cannot be skipped) Before reading ANY result files, ask the user to state: ``` Before I show you results, I need three things: 1. EXPECTATION: What do you expect to see? (e.g., "Claude should outperform Gemini on CUDA-to-OMP because...") 2. NULL HYPOTHESIS: What is the default/boring explanation? (e.g., "All models perform equally — differences are noise") 3. FALSIFICATION: What specific observation would make you abandon your expectation? (e.g., "If Gemini outperforms Claude on >50% of kernels, my expectation is wrong") ``` **Verification gate:** User must provide all three answers. If they try to skip with "just show me the numbers" or similar, redirect them to the Anti-Rationalization Table above and ask again. Do NOT proceed without a stated hypothesis. Record the user's responses — they will be referenced in the analysis output. ### Phase 2: Read Actual Results Only after Phase 1 is complete, read the result files matching the scope: ```bash # Example: list all result files for a model ls results/evaluation//*.json | head -50 # Example: read a specific result # Use Read tool on each JSON, extract overall_status ``` For each result file, extract: - `overall_status` (the ONLY authoritative verdict — not top-level `run_status`) - `attempts[]` length (number of retries) - `build_error_snippet` (for BUILD_FAIL) - `run_stderr_snippet` (for RUN_FAIL) - Kernel name and direction (from filename) Build a structured data table from actual file contents. **Verification gate:** Every number reported must have a source file path. If a result file is missing or unreadable, note it explicitly — do not estimate or skip. ### Phase 3: Compare to Stated Expectations Structure the analysis in this exact order — do NOT rearrange: #### 3a. Observed Results (raw data) Present the raw data table. No interpretation yet. ``` === OBSERVED RESULTS: === Kernel Direction Model Status Attempts backprop cuda-to-omp claude-sonnet PASS 1 backprop cuda-to-omp gemini-flash PASS 2 backprop cuda-to-omp groq-llama BUILD_FAIL 3 ... Summary: PASS, BUILD_FAIL, RUN_FAIL, VERIFY_FAIL Source: results/evaluation// ``` #### 3b. Comparison to User's Expectation Directly reference what the user said in Phase 1: ``` Your expectation was: "" Observation: ``` Be specific about which results match and which contradict. Do not soften contradictions. #### 3c. Null Hypothesis Assessment ``` Your null hypothesis was: "" Assessment: Can we reject it? ``` With small sample sizes (typical for per-kernel analysis), be honest about statistical power. "We cannot reject the null with N=3 models" is a valid and important conclusion. #### 3d. Alternative Explanations (minimum 2) Generate at least 2 alternative explanations for the observed pattern that the user did NOT suggest. These must be plausible and grounded in the data: - Training data distribution differences between models - Kernel-specific code patterns (reductions, stencils, control flow) matching model strengths - Prompt sensitivity (same code, different tokenization) - Compiler/build environment effects (not model effects) - Sample size limitations masking true performance #### 3e. Confounding Variables List variables that could explain the results but are not controlled: - Model temperature and sampling strategy - Tokenizer differences across models - Prompt format differences (if any) - Time-of-day API performance variance - Retry logic recovering from transient failures vs. systematic failures #### 3f. Recommended Follow-Up Experiments Propose 2-3 concrete next experiments that would strengthen or weaken the hypothesis: ``` Recommended experiments: 1. [Experiment]: [Tests]: [Command]: 2. [Experiment]: ... ``` **Verification gate:** Every recommended experiment must be actionable with the current project infrastructure. Do not suggest experiments that require new tools or data sources not already in the project. ### Phase 4: Summary Verdict ``` === INTERPRETATION SUMMARY === Hypothesis: Confidence: Key finding: Biggest surprise: Paper-ready: Next action: ``` **Verification gate:** The verdict must be consistent with the data presented in Phase 3. If the data is INCONCLUSIVE, say so — do not force a conclusion.