Mechinterp Investigator

MechInterp Investigator

This skill guides a systematic investigation of SAE features to arrive at meaningful, non-trivial labels. It orchestrates the other mechinterp skills into a coherent research workflow.

Phase 0: Triage (ALWAYS START HERE)

Goal: Quickly filter out weak/auxiliary features that don't warrant deep investigation.

Time: 1-2 minutes

Many SAE features have minimal influence on model outputs. Triage identifies these early so you can skip expensive analysis.

Step 0.1: Check Decoder Weight Percentile

import torch

sae_path = '/mnt/e/dev_spillover/SplatNLP/sae_runs/run_20250704_191557/sae_model_final.pth'
sae_checkpoint = torch.load(sae_path, map_location='cpu', weights_only=True)
decoder_weight = sae_checkpoint['decoder.weight']  # [512, 24576]

# Get this feature's max absolute decoder weight
feature_decoder = decoder_weight[:, FEATURE_ID]
max_abs = torch.abs(feature_decoder).max().item()

# Compare to all features
all_max_abs = torch.abs(decoder_weight).max(dim=0).values
percentile = (all_max_abs < max_abs).float().mean() * 100

print(f"Feature {FEATURE_ID} decoder weight percentile: {percentile:.1f}%")

Step 0.2: Quick Overview Check (if <10%)

If decoder percentile < 10%, run a quick overview:

poetry run python -m splatnlp.mechinterp.cli.overview_cli \
    --feature-id {FEATURE_ID} --model ultra --top-k 10

Signs of clear structure (proceed to Phase 1):

• One family dominates (>40% of breakdown)
• Strong weapon concentration (>50% one weapon)
• Clear binary ability pattern
• Top PageRank token has score > 0.20

Signs of no structure (label as weak):

• Family breakdown is flat (all <15%)
• Weapons are diverse
• Top PageRank score < 0.10
• High sparsity (>99%) with no clear pattern

Triage Decision

Decoder percentile < 10% AND no clear structure in overview?
  │
  Yes → Label as "Weak/Aux Feature {ID}" and STOP
  │
  No → Proceed to Phase 1 (Overview)

Weak Feature Label Format

{
  "dashboard_name": "Weak/Aux Feature {ID}",
  "dashboard_category": "auxiliary",
  "dashboard_notes": "TRIAGE: Decoder weight {X}th percentile, no clear structure in overview. Skipped deep dive.",
  "hypothesis_confidence": 0.0,
  "source": "claude code (triage)"
}

When to Override Triage

Even with low decoder weights, proceed if:

• The feature is part of a cluster you're investigating
• You have external reason to believe it's important
• You're doing exhaustive analysis of a subset

⚠️ Deep Dive Basics

A proper deep dive requires experiments, not just reading overview data. The overview shows correlations; experiments reveal causation.

Minimum Requirements for a Deep Dive

Binary Abilities Need Special Handling

Binary abilities (you have them or you don't) show delta=0 in 1D sweeps because there's no scaling. This does NOT mean they're unimportant.

To evaluate binary abilities:

1. Check PageRank score (correlation strength)
2. Check presence rate: What % of high-activation examples contain it?
3. Compare mean activation WITH vs WITHOUT the binary token
4. Run 2D heatmap: scaling_ability × binary_ability to see conditional effect

Binary Ability Analysis Protocol (CRITICAL)

Binary abilities can have strong conditional effects that ONLY show up in 2D analysis. Here's the exact methodology:

Step 1: Check presence rate enrichment

from splatnlp.mechinterp.skill_helpers import load_context
import polars as pl

ctx = load_context('ultra')
df = ctx.db.get_all_feature_activations_for_pagerank(FEATURE_ID)

# Find binary token ID
binary_id = None
for tok_id, tok_name in ctx.inv_vocab.items():
    if tok_name == 'comeback':  # or stealth_jump, etc.
        binary_id = tok_id
        break

# Calculate enrichment
threshold = df['activation'].quantile(0.90)  # Top 10%
high_df = df.filter(pl.col('activation') >= threshold)

with_binary_all = df.filter(pl.col('ability_input_tokens').list.contains(binary_id))
with_binary_high = high_df.filter(pl.col('ability_input_tokens').list.contains(binary_id))

baseline_rate = len(with_binary_all) / len(df)
high_rate = len(with_binary_high) / len(high_df)
enrichment = high_rate / baseline_rate

print(f"Baseline presence: {baseline_rate:.1%}")
print(f"High-activation presence: {high_rate:.1%}")
print(f"Enrichment ratio: {enrichment:.2f}x")
# Enrichment > 1.5x suggests binary ability is characteristic

Step 2: Check mean activation WITH vs WITHOUT

with_binary = df.filter(pl.col('ability_input_tokens').list.contains(binary_id))
without_binary = df.filter(~pl.col('ability_input_tokens').list.contains(binary_id))

mean_with = with_binary['activation'].mean()
mean_without = without_binary['activation'].mean()
delta = mean_with - mean_without

print(f"Mean WITH: {mean_with:.4f}")
print(f"Mean WITHOUT: {mean_without:.4f}")
print(f"Delta: {delta:+.4f}")
# Delta > 0.03 suggests meaningful effect

Step 3: Run 2D heatmap (MOST IMPORTANT)

Binary abilities can have conditional effects that vary by the scaling ability level:

# Manual 2D analysis for binary abilities
# (The built-in 2D heatmap may not handle binary tokens correctly)

scaling_ids = {3: 48, 6: 49, 12: 50, 21: 53, 29: 80}  # ISM example
binary_id = 27  # Comeback

print("Scaling | No Binary | With Binary | Delta")
print("-" * 50)

for level, tok_id in scaling_ids.items():
    level_df = df.filter(pl.col('ability_input_tokens').list.contains(tok_id))

    with_binary = level_df.filter(pl.col('ability_input_tokens').list.contains(binary_id))
    without_binary = level_df.filter(~pl.col('ability_input_tokens').list.contains(binary_id))

    mean_with = with_binary['activation'].mean() if len(with_binary) > 0 else 0
    mean_without = without_binary['activation'].mean() if len(without_binary) > 0 else 0
    delta = mean_with - mean_without

    print(f"{level:>7} | {mean_without:>9.4f} | {mean_with:>11.4f} | {delta:>+.4f}")

Example (Feature 13352):

ISM × Comeback 2D Analysis:
ISM | No CB  | With CB | Delta
  0 | 0.066  | 0.117   | +0.051
  3 | 0.122  | 0.261   | +0.139
  6 | 0.147  | 0.352   | +0.205  ← PEAK INTERACTION
 12 | 0.094  | 0.163   | +0.069
 21 | 0.094  | 0.129   | +0.035

Interpretation: Comeback has STRONG conditional effect at ISM 3-6.
The +0.205 delta at ISM_6 means Comeback DOUBLES the activation!
1D sweep showed delta=0 because most examples have ISM=0 (low baseline).

Step 4: Test combinations of binary abilities together

# Test multiple binary abilities together
binary_id_1 = 27  # e.g., comeback
binary_id_2 = 1   # e.g., stealth_jump

both = df.filter(
    pl.col('ability_input_tokens').list.contains(binary_id_1) &
    pl.col('ability_input_tokens').list.contains(binary_id_2)
)
neither = df.filter(
    ~pl.col('ability_input_tokens').list.contains(binary_id_1) &
    ~pl.col('ability_input_tokens').list.contains(binary_id_2)
)

# Then do 2D analysis at each scaling level
# Combinations can have stronger effects than individual abilities!

Key Insight: Binary abilities may have stronger effects when combined. Always test combinations, not just individual tokens.

Additional Learnings

1. Conditional effects can be much stronger than marginal effects: A feature might show ISM with only 0.069 max_delta in 1D sweeps, but a binary ability combination at moderate ISM could produce +0.335 delta - the interaction effect can be 5x stronger than the marginal effect. 1D sweeps can dramatically underestimate a feature's true behavior.

2. Depletion is informative: If a binary ability shows enrichment < 1.0 (e.g., 0.72x), the feature actively avoids that ability. This is meaningful for interpretation - it tells you what the feature excludes, not just what it includes.

3. Manual 2D analysis required for binary tokens: The Family2DHeatmapRunner uses parse_token() which expects family_name_AP format, but binary abilities appear as just the token name (e.g., comeback not comeback_10). Use manual 2D analysis code for binary abilities (see protocol above).

4. "Weak feature" needs decoder weight check: A feature with weak activation effects (max_delta < 0.03) might still have high influence on outputs. Remember: net influence = activation strength × decoder weight. Before labeling as "weak", check the feature's decoder weights to the output tokens it contributes to. A "weak activation" feature with high decoder weights may actually be important.

5. Watch for error-correction features: If 1D sweeps show small deltas or effects only in unusual rung combinations, the feature may fire when prerequisites are MISSING (OOD detection). Test "explains-away" behavior by comparing activation when low-level evidence is present vs missing. Example: Does feature fire MORE when SCU_3 is absent from a high-SCU build?

6. Beware of flanderization in top activations: The top 100 activations over-emphasize extreme cases. The TRUE concept often lives in the mid-activation range (25-75th percentile). Always compare mid vs top activation regions - if they show different weapon/ability patterns, label the mid-range concept and note the extremes as "super-stimuli".

What Counts as Evidence

Common Mistakes to Avoid

1. Presenting overview as findings - Overview is hypotheses, not conclusions
2. Ignoring binary abilities - Delta=0 doesn't mean unimportant
3. Skipping bottom tokens - Suppressors reveal what feature avoids
4. Only running 1D sweeps - 2D heatmaps needed for interaction effects
5. Not checking weapon patterns - Feature may be weapon-specific, not ability-specific
6. Using only top activations - Top activations (90%+ of max) may be "flanderized" extremes; check core region (25-75% of max)
7. Missing error-correction features - Small deltas in weird rung combos may indicate OOD detection
8. Confusing data sparsity with suppression - Zero examples at a condition ≠ "suppression to 0" (see below)
9. Shallow validation - Just checking if numbers "look right" without running enrichment analysis
10. Semantic contradictions in labels - e.g., "Zombie" (embraces death) + "high SSU" (avoids death) is contradictory
11. Reporting weapon percentages from top-100 - Use top 20-30% instead; top-100 can be 5-10x off (e.g., 78% vs 10%)
12. Not checking meta archetypes - Weapons may cluster by playstyle, not kit; use splatoon3-meta skill
13. Assuming kit-based patterns - Check if weapons share sub/special BEFORE assuming it's kit-related
14. Ignoring flanderization crossover - Note where a "super-stimulus" weapon overtakes the general pattern (usually 90%+ of max activation)

⚠️ CRITICAL: Data Sparsity vs Suppression

This is a common and dangerous mistake. When you see "activation = 0" or "no effect" at some condition, ask: Is this suppression or data sparsity?

Example of the mistake (Feature 1819):

Original claim: "QR is HARD SUPPRESSOR - SSU_57+QR_any=0.000"
Reality: There were ZERO examples with SSU_57 + any QR in the dataset!
         The "0.000" was missing data, not suppression.

How to detect data sparsity:

# ALWAYS check sample sizes when claiming suppression!
at_high_ssu = df.filter(pl.col('ability_input_tokens').list.contains(ssu_57_id))
with_qr = at_high_ssu.filter(pl.col('ability_input_tokens').list.set_intersection(qr_ids).list.len() > 0)

print(f"Examples at SSU_57 with QR: {len(with_qr)}")  # If 0, this is SPARSITY not suppression!

Rule: Never claim "suppression" unless you have ≥20 examples in the suppressed condition. Report sample sizes with all claims.

Philosophy

A meaningful label should capture:

• What concept the feature encodes (not just "detects token X")
• Why the model might have learned this representation
• How it relates to strategic/tactical gameplay

Avoid trivial labels like:

• "SCU Detector" (just describes token presence)
• "High activation feature" (describes statistics, not meaning)

Aim for interpretable labels like:

• "Aggressive Slayer Build" (strategic concept)
• "Special Spam Enabler" (functional role)
• "Backline Support Kit" (playstyle archetype)

Investigation Workflow

Phase 0: Triage

See Phase 0: Triage above. Always start here.

If feature passes triage (decoder weight ≥10% OR has clear structure), proceed to Phase 1.

Phase 1: Initial Assessment

Run the overview and classify the feature type:

poetry run python -m splatnlp.mechinterp.cli.overview_cli \
    --feature-id {FEATURE_ID} --model {MODEL} --top-k 20

Classify based on family breakdown:

CRITICAL: Always check for non-monotonic effects! Higher AP doesn't always mean higher activation.

Phase 1.5: Activation Region Analysis (CRITICAL - Anti-Flanderization)

Don't only examine extreme activations! High activations may be "flanderized" - exaggerated, extreme versions of the true concept that over-emphasize niche cases.

Key insight: The TRUE concept often lives in the core region (25-75% of effective max), not the top examples. Top activations (90%+ of effective max) can mislead you into labeling a niche pattern instead of the general concept.

Why "effective max"? Activation distributions are heavy-tailed. Using effective_max = 99.5th percentile of nonzero activations prevents single outliers from making the core region nearly empty.

Run activation region analysis:

from splatnlp.mechinterp.skill_helpers import load_context
import numpy as np
from collections import Counter

ctx = load_context("{MODEL}")
df = ctx.db.get_all_feature_activations_for_pagerank({FEATURE_ID})

acts = df['activation'].to_numpy()
weapons = df['weapon_id'].to_list()

# Use EFFECTIVE MAX (99.5th percentile) to handle heavy-tailed distributions
# This prevents single outliers from making the core region nearly empty
nonzero_acts = acts[acts > 0]
effective_max = np.percentile(nonzero_acts, 99.5)
true_max = acts.max()
print(f"True max: {true_max:.4f}, Effective max (99.5%ile): {effective_max:.4f}")

# Define activation regions as % of EFFECTIVE max
regions = [
    ('Floor (≤1%)', lambda a: a <= 0.01 * effective_max),
    ('Low (1-10%)', lambda a: 0.01 * effective_max < a <= 0.10 * effective_max),
    ('Below Core (10-25%)', lambda a: 0.10 * effective_max < a <= 0.25 * effective_max),
    ('Core (25-75%) - TRUE CONCEPT', lambda a: 0.25 * effective_max < a <= 0.75 * effective_max),
    ('High (75-90%)', lambda a: 0.75 * effective_max < a <= 0.90 * effective_max),
    ('Flanderization Zone (90%+)', lambda a: a > 0.90 * effective_max),
]

for region_name, filter_fn in regions:
    indices = [i for i, a in enumerate(acts) if filter_fn(a)]
    weps = [weapons[i] for i in indices]
    print(f"\n{region_name} (n={len(indices)}):")
    for wep, count in Counter(weps).most_common(5):
        name = ctx.id_to_weapon_display_name(wep)
        print(f"  {name}: {count}")

Key signals to look for:

Example (Feature 9971):

Core (25-75%): Splattershot (115), Wellstring (65), Sploosh (57)...
Flanderization (90%+): Bloblobber (44), Glooga Deco (39), Range Blaster (28)

Interpretation: Core region shows GENERAL offensive investment.
Flanderization zone shows EXTREME SCU on special-dependent weapons (super-stimuli).
Label the general concept, note the super-stimuli pattern.

CRITICAL: Always check the Bottom Tokens (Suppressors) section! Tokens that rarely appear in high-activation examples can reveal what the feature avoids:

Example: If SCU is enhanced but quick_respawn, special_saver, and comeback are ALL suppressed, the feature doesn't just detect "SCU" - it detects "death-averse SCU builds" (players who stack SCU but don't plan to die).

Phase 1.6: Weapon Distribution Analysis (CRITICAL - Anti-Flanderization)

NEVER report weapon percentages from top-100 samples. Top-100 is severely flanderized and can give wildly misleading weapon distributions.

Example (Feature 14096 - Real Case):

Top 100:     Dark Tetra 78%, Stamper 20%  ← WRONG, flanderized
Top 10%:     Stamper 35%, Dark Tetra 21%  ← Better but still skewed
Top 30%:     Stamper 23%, Dark Tetra 10%  ← TRUE CONCEPT
Full dataset: Stamper 9%, Dark Tetra 3.5% ← Includes noise/floor

Use top 20-30% for weapon characterization:

import polars as pl
import numpy as np
from collections import Counter
from splatnlp.mechinterp.skill_helpers import load_context

ctx = load_context('ultra')
df = ctx.db.get_all_feature_activations_for_pagerank(FEATURE_ID)

# Get percentile thresholds
acts = df['activation'].to_numpy()
thresholds = {p: np.percentile(acts, p) for p in [0, 50, 70, 80, 90, 95, 99]}

# Analyze by region
regions = [
    ("Bottom 50% (noise)", 0, 50),
    ("50-70% (weak)", 50, 70),
    ("Top 30% (TRUE CONCEPT)", 70, 100),
    ("Top 10%", 90, 100),
    ("Top 1% (flanderized)", 99, 100),
]

print("Region | Top Weapons")
print("-" * 60)

for name, p_low, p_high in regions:
    t_low, t_high = thresholds[p_low], thresholds.get(p_high, float('inf'))
    if p_high == 100:
        region_df = df.filter(pl.col('activation') >= t_low)
    else:
        region_df = df.filter((pl.col('activation') >= t_low) & (pl.col('activation') < t_high))

    if len(region_df) == 0:
        continue

    weapon_counts = region_df.group_by('weapon_id').agg(
        pl.col('activation').count().alias('n')
    ).sort('n', descending=True)

    top3 = []
    for row in weapon_counts.head(3).iter_rows(named=True):
        wname = ctx.id_to_weapon_display_name(row['weapon_id'])
        pct = row['n'] / len(region_df) * 100
        top3.append(f"{wname[:12]}({pct:.0f}%)")

    print(f"{name:<25} | {', '.join(top3)}")

Interpretation Guide:

Rule: Report weapon percentages from top 20-30%, note if top-1% differs significantly.

Phase 1.6.5: Ability Flanderization Check (CRITICAL)

The same flanderization that applies to weapons applies to abilities. A binary ability with high tail enrichment but low core coverage is a super-stimulus, not the core concept.

The Rule: If a "dominant" driver has <30% core coverage, it's a tail marker, not the headline concept.

Use the core coverage experiment:

cd /root/dev/SplatNLP

# Direct subcommand (recommended)
poetry run python -m splatnlp.mechinterp.cli.runner_cli coverage \
    --feature-id {FEATURE_ID} --model ultra \
    --tokens respawn_punisher,comeback,stealth_jump \
    --threshold 0.30

Output tables:

• token_coverage: Shows core_coverage_pct, tail_enrichment, is_tail_marker for each token
• weapon_coverage: Shows core vs tail weapon distributions (catches weapon flanderization)

Coverage Interpretation:

Example (Feature 13934):

respawn_punisher: 8.57x tail enrichment, BUT only 12% core coverage
→ RP is a super-stimulus, NOT the core concept
→ Wrong label: "RP Backline Anchor"
→ Right approach: Split core by RP presence to reveal hidden modes

When you find a super-stimulus (<30% coverage):

1. Split the core by presence/absence of the super-stimulus
2. Analyze both modes separately
3. Look for what they have in COMMON (the true concept)
4. Label the commonality, note the super-stimulus as a tail marker

Phase 1.7: Meta-Informed Weapon Analysis (USE AFTER WEAPON SWEEP)

After identifying top weapons, always check if they match a known meta archetype using the splatoon3-meta skill.

Step 1: Look up weapon kits

Check references/weapons.md for each top weapon's sub and special:

# Top weapons from Feature 14096 (top 30%):
kits = {
    "Splatana Stamper": ("Burst Bomb", "Zipcaster"),
    "Dark Tetra Dualies": ("Autobomb", "Reefslider"),
    "Glooga Dualies": ("Splash Wall", "Booyah Bomb"),
    "Dapple Dualies Nouveau": ("Torpedo", "Reefslider"),
    "Splatana Wiper": ("Torpedo", "Ultra Stamp"),
}

# Check for shared subs/specials
from collections import Counter
subs = Counter(k[0] for k in kits.values())
specials = Counter(k[1] for k in kits.values())

# If one sub/special dominates → kit-based feature
# If diverse → playstyle-based feature

Step 2: Check archetype reference

Read references/archetypes.md to see if weapons match a known archetype:

Step 3: Classification decision

Kit Analysis Result:
├─ Shared sub weapon? → Feature may encode SUB PLAYSTYLE
├─ Shared special? → Feature may encode SPECIAL FARMING
├─ No kit pattern + archetype match? → PLAYSTYLE FEATURE (label as archetype)
└─ No kit pattern + no archetype? → WEAPON CLASS feature (check if all dualies, all shooters, etc.)

Example (Feature 14096):

Top 30% weapons: Stamper, Dark Tetra, Glooga, Dapple, Wiper
Kit analysis: Diverse subs (Burst, Auto, Splash Wall, Torpedo), diverse specials
Archetype check: Dark Tetra + Splatana Wiper = "Zombie Slayer" archetype!
Conclusion: PLAYSTYLE feature encoding Zombie Slayer (death-accepting aggressive)
Label: "Zombie Slayer QR (Splatana/Dualies)" - tactical category

When to invoke splatoon3-meta skill:

• After weapon_sweep shows concentrated weapon pattern
• When top weapons seem unrelated by kit but share a playstyle
• To validate that ability patterns match expected meta builds
• To identify if weapons share archetype despite different kits

Phase 1.7.5: Kit Component Analysis (OPTIONAL but Recommended)

When to use: After weapon sweep, check if the core weapons share patterns in ANY kit component: sub weapon, special weapon, or main weapon class. This can reveal WHY certain build philosophies emerge.

Key insight: Weapons may cluster by:

• Sub weapon (Burst Bomb users, Beakon users → explains SPU/ISS builds)
• Special weapon (Aggressive push specials → explains survival builds)
• Main weapon class (All dualies, all chargers → explains mobility/positioning builds)

The feature may be driven by ONE of these - identify which, then determine if it's causal or spurious.

Component 1: Sub Weapon Pattern Analysis

When relevant: If kit_sweep (Phase 1.7/3d) shows sub concentration, investigate further.

from collections import Counter

# Map top weapons to their subs (from weapons.md)
weapon_subs = {
    "Splattershot Jr.": "Splat Bomb",
    "Neo Splash-o-matic": "Suction Bomb",
    "Sploosh-o-matic 7": "Splat Bomb",
    # ... add more as needed
}

# Categorize subs
sub_categories = {
    # Lethal bombs
    "Splat Bomb": "lethal", "Suction Bomb": "lethal", "Burst Bomb": "lethal",
    "Curling Bomb": "lethal", "Autobomb": "lethal", "Torpedo": "lethal",
    "Fizzy Bomb": "lethal", "Ink Mine": "lethal", "Toxic Mist": "lethal",
    # Utility/Support
    "Squid Beakon": "utility", "Splash Wall": "utility", "Sprinkler": "utility",
    "Point Sensor": "utility", "Angle Shooter": "utility",
}

# Count categories
sub_counts = Counter()
for weapon in top_weapons:
    sub = weapon_subs.get(weapon)
    if sub:
        category = sub_categories.get(sub, "other")
        sub_counts[category] += 1

print("Sub Weapon Breakdown:")
for sub, count in Counter(weapon_subs.get(w) for w in top_weapons if weapon_subs.get(w)).most_common():
    print(f"  {sub}: {count}")

Sub pattern implications:

Component 2: Special Weapon Pattern Analysis

When relevant: After weapon sweep, check if core weapons share a special weapon pattern.

from collections import Counter

# Map top weapons to their specials (from weapons.md)
weapon_specials = {
    "Splatana Stamper": "Zipcaster",
    "Sloshing Machine": "Booyah Bomb",
    "Squeezer": "Trizooka",
    # ... add more as needed
}

# Categorize specials
special_categories = {
    # Zoning/Area Denial
    "Ink Storm": "zoning", "Wave Breaker": "zoning", "Tenta Missiles": "zoning",
    "Killer Wail 5.1": "zoning", "Triple Inkstrike": "zoning",
    # Team Support
    "Tacticooler": "team_support", "Big Bubbler": "team_support",
    "Splattercolor Screen": "team_support",
    # Aggression/Push
    "Trizooka": "aggression", "Crab Tank": "aggression", "Ink Jet": "aggression",
    "Ultra Stamp": "aggression", "Booyah Bomb": "aggression", "Reefslider": "aggression",
    "Kraken Royale": "aggression", "Zipcaster": "aggression",
    # Utility/Defense
    "Ink Vac": "utility", "Super Chump": "utility", "Triple Splashdown": "utility",
}

# Count categories
category_counts = Counter()
for weapon in top_weapons:
    special = weapon_specials.get(weapon)
    if special:
        category = special_categories.get(special, "other")
        category_counts[category] += 1

print("Special Category Breakdown:")
for cat, count in category_counts.most_common():
    print(f"  {cat}: {count/sum(category_counts.values())*100:.0f}%")

Special pattern implications:

Component 3: Main Weapon Class Pattern Analysis

When relevant: If weapons seem diverse but may share a class (all shooters, all dualies, all chargers).

# Weapon class mapping (from weapon-vibes.md)
weapon_classes = {
    "Splattershot": "shooter", "Splattershot Jr.": "shooter", "Splattershot Pro": "shooter",
    "Dark Tetra Dualies": "dualie", "Dapple Dualies": "dualie", "Splat Dualies": "dualie",
    "E-liter 4K": "charger", "Splat Charger": "charger", "Goo Tuber": "charger",
    "Luna Blaster": "blaster", "Range Blaster": "blaster", "Rapid Blaster": "blaster",
    "Hydra Splatling": "splatling", "Mini Splatling": "splatling",
    "Splatana Stamper": "splatana", "Splatana Wiper": "splatana",
    # ... add more as needed
}

# Count classes
class_counts = Counter(weapon_classes.get(w, "other") for w in top_weapons)

print("Weapon Class Breakdown:")
for cls, count in class_counts.most_common():
    pct = count / len(top_weapons) * 100
    print(f"  {cls}: {pct:.0f}%")

Class pattern implications:

Step 4: Determine if Pattern is CAUSAL or SPURIOUS

This is the critical step. A strong pattern in ANY component could be causal or spurious.

Questions to determine causality:

1. Does the kit component align with decoder output?

   • Decoder promotes SCU/SS/SPU + aggressive specials → Special farming is likely causal
   • Decoder promotes ISS/SPU + shared sub weapon → Sub spam is likely causal
   • Decoder promotes SSU/QSJ + all dualies → Weapon class mobility is likely causal

2. Do weapons share OTHER traits that better explain the clustering?

   • All dualies with aggressive specials → Is it the CLASS or the SPECIAL?
   • Test: Do other dualies (without aggressive specials) also cluster here?

3. Does the build philosophy make sense for this kit component?

   • Survival builds + aggressive specials → "Stay alive to use push special" (causal)
   • Mobility builds + all dualies → "Dualies need SSU for dodge-roll play" (causal)
   • Survival builds + diverse subs/specials + all chargers → "Chargers can't trade" (class is causal)

Example Analysis (Special-driven):

Feature 14964 special breakdown: 77% aggression (Zipcaster, Booyah Bomb, Trizooka)
Build philosophy: "Balanced utility spread for survival"

Analysis:
- Decoder suppresses death-trading (Comeback, RP) ✓
- Decoder promotes survival abilities (SS, ISM) ✓
- Weapons have LOW-MED death tolerance ✓
- Weapons have aggressive push specials ✓
- Sub weapons are DIVERSE (no pattern)
- Weapon classes are DIVERSE (shooters, slosher, splatana)

Conclusion: CAUSAL - Players build for survival BECAUSE they have aggressive specials
           that require staying alive to deploy effectively.

Note: "Core weapons have aggressive push specials (77%) requiring survival to deploy"

Example Analysis (Class-driven):

Feature shows: 80% dualies (Dark Tetra, Dapple, Dualie Squelchers)
Decoder promotes: SSU, QSJ, RSU (mobility family)

Analysis:
- Specials are DIVERSE (not the driver)
- Subs are DIVERSE (not the driver)
- All weapons are DUALIES with dodge-roll mechanics ✓
- Dualies benefit uniquely from SSU for roll distance/recovery

Conclusion: CAUSAL - Dualies cluster because dodge-roll playstyle needs mobility
           The feature encodes "dualie mobility optimization"

Counter-example (Spurious):

Feature has 70% aggression specials
But: All weapons are CLOSE-range SLAYER with HIGH death tolerance
And: Decoder promotes QR, Comeback (death-trading)

Conclusion: SPURIOUS - Weapons are aggressive slayers who happen to have aggressive specials
           The special type is incidental to the slayer playstyle.
           Primary driver is ROLE (slayer), not KIT.

Step 5: Record findings in notes

If pattern is CAUSAL, add to dashboard_notes:

KIT PATTERN: {component} - {X}% {category/type} ({list top examples}).
INTERPRETATION: [Why this explains the build philosophy]

If pattern is SPURIOUS, note briefly:

KIT PATTERN: Diverse/incidental. Weapons cluster by [range/role/playstyle], not kit.

When to skip this phase:

• Feature is clearly mechanical (single ability stacker like "SCU_57 threshold")
• Weapons are highly diverse with no concentration in any component
• Earlier analysis already identified clear driver (e.g., single weapon dominance)

Phase 1.8: Weapon Range/Role Classification (REQUIRED for Labels)

Before proposing any label, you MUST classify the feature's weapons by range and role. This prevents incorrect role assumptions (e.g., calling Jr./Rapid Blasters "anchors" when they're midrange).

Step 1: Extract properties for top 5-10 core weapons from weapon-vibes.md

Step 2: Find the common pattern

If most weapons share:

• LONG/SNIPER + BACK + ANCHOR → use "Anchor" or "Backline" qualifier
• MID/LONG + MID + SKIRMISH/SUPPORT → use "Midrange" qualifier
• CLOSE/MID + FRONT + SLAYER → use "Slayer" or "Frontline" qualifier
• NO/BAD NS_FIT + LOW DEATH_TOL → "Visible" or "Positional" concept (not stealth, not trading)

Step 3: Record in notes

Always include weapon classification in dashboard_notes:

WEAPON ROLE: Midrange (MID-LONG range, SKIRMISH/SUPPORT jobs, NO/BAD NS fit, LOW death tolerance)

Phase 2: Hypothesis Generation

Based on Phase 1, generate hypotheses about what the feature might encode:

For single-family dominated features:

• H1: Pure token detector (trivial - try to disprove)
• H2: Threshold detector (activates only at high AP)
• H3: Interaction detector (family + something else)
• H4: Weapon-conditional (family matters only for certain weapons)

For multi-family features:

• H1: Synergy detector (families work together)
• H2: Build archetype (strategic loadout pattern)
• H3: Playstyle indicator (aggressive, defensive, support)
• H4: Shared NEED (different builds solving the same tactical problem)

Build NEED Framework (For Multi-Modal/Diffuse Features)

When a feature activates on seemingly different build types, ask: "What NEED do these builds share?"

Features can encode solutions to problems, not just correlations. Different builds may trigger the same feature because they're different answers to the same question.

Step 1: Identify the tactical constraint these builds solve

Step 2: Check weapon properties (use splatoon3-meta)

Compare enriched weapons on these axes from weapon-vibes.md:

• Ink feel: STARVING / HUNGRY / AVERAGE / EFFICIENT / PAINTER
• Range: MELEE / CLOSE / MID / LONG / SNIPER
• Ninja Squid affinity: CORE / GOOD / MEH / BAD / NO
• Death tolerance: HIGH / MED / LOW
• Role: SLAYER / SUPPORT / ANCHOR / SKIRMISH / ASSASSIN

If all enriched weapons share properties (e.g., all HUNGRY ink + NO ninja squid + LOW death tolerance), the feature may encode a need specific to that weapon class.

Step 3: Reframe the modes as "answers to the same question"

Example (Feature 13934):

Mode A (12%): RP anchor builds (E-liter) - "I won't die, make their deaths hurt"
Mode B (88%): Zombie utility builds (DS) - "I will die sometimes, optimize respawns"

Shared NEED: "Death management for non-stealth, low-death-tolerance, midrange+ weapons"
Both modes are VALID ANSWERS to the same tactical question.

Step 4: Label the NEED, not the modes

Instead of: "Mixed: Zombie + RP Anchor" (describes the modes) Label as: "Balanced Utility Axis (Non-Stealth Midline+)" (describes the need)

Key Insight: The model learned that these seemingly different builds share a common requirement. The feature encodes that requirement, and the modes are just different implementations.

For weapon-specific features:

• H1: Weapon class pattern (all shooters, all chargers, etc.)
• H2: Meta build (optimal loadout for that weapon)
• H3: Weapon-ability interaction

Phase 3: Targeted Experiments

Run experiments to test hypotheses. Available experiment types:

⚠️ CRITICAL: Iterative Conditional Test

…(truncated)