Ica Lens

ICA Lens: A Training-Free Interpretability Lens on Model Components

When to Use

Activate this skill when the goal is to characterise the interpretable structure of a component's activation distribution without paying the up-front cost of training a sparse dictionary.

Typical triggers:

• A new model, layer, or hook point needs an initial inventory of candidate interpretable directions before committing to SAE / transcoder training.
• An existing SAE / transcoder feature set needs an independent baseline of interpretable directions to compare against (overlap, coverage, perturbation strength, sparse-probe accuracy under a matched budget).
• A specific concept or behaviour is suspected to live on a low-dimensional, non-Gaussian subspace of a residual / MLP / head-output state, and a fast, gradient-free extractor is needed to expose it.
• A study of non-Gaussianity itself — where interpretable structure concentrates across depth, modality, and model family — is the object of interest.
• Targeted perturbation / steering needs a direction handle whose discovery is decoupled from the model's training signal.
• An interpretability artefact needs to be browsed, annotated, or shared via an explorer UI rather than recomputed each session.

Keywords: ICA, Independent Component Analysis, FastICA, non-Gaussianity, dictionary-free interpretability, interpretable directions, component interpretability, residual-stream directions, MLP-output directions, attention-head directions, sparse probing, targeted probe perturbation, SAE alternative, SAE baseline, ICA explorer.

Method in One Paragraph

The ICA Lens treats each cached activation $\mathbf{a} \in \mathbb{R}^{d_{\text{model}}}$ at a chosen hook point as a sample from an unknown source distribution and fits an unmixing matrix $\mathbf{W}{\mathrm{ICA}}$ such that the components $\mathbf{f}(\mathbf{a}) = \mathbf{W}{\mathrm{ICA}},\mathbf{a}$ are maximally non-Gaussian and mutually independent. The motivating observation is that interpretable, token-selective directions in language models appear systematically less Gaussian than random directions in the same space, so non-Gaussianity is a sufficient surrogate objective to recover candidate monosemantic axes without any reconstruction or sparsity loss. Compared with sparse-autoencoder-family methods, no over-complete dictionary is trained, no per-feature dead-direction / split-feature pathology is incurred, and the recovered directions can be evaluated by exactly the same downstream protocols — sparse probing, targeted probe perturbation, top-activating example mining, and feature dashboards.

When to Prefer ICA Lens Over a Trained Dictionary

The general rule: start with ICA Lens, escalate to a learned dictionary only when ICA's resolution, over-completeness, or circuit-replacement guarantees are demonstrably insufficient.

Quick Reference

• Repository: https://github.com/liusida/ica-lens-paper
• Paper: https://arxiv.org/abs/2606.11722
• Project page: https://liusida.github.io/ica-lens-paper/
• Hugging Face Space (hosted explorer): https://huggingface.co/spaces/EEEAILab/ICAExplorer
• Hugging Face artifacts dataset: https://huggingface.co/datasets/sida/ica-lens-paper
• Docs:
  • docs/quickstart.md
  • docs/fit_one_layer_qwen36_27b.md — worked example for fitting ICA on a new model / layer (use as template)
  • docs/reproduction.md
  • docs/troubleshooting.md
  • docs/artifact_contract.md, docs/artifact_datasets.md, docs/model_and_layer_conventions.md

Installation / Setup

Prerequisites

• Python 3.10+
• PyTorch (CUDA recommended for activation capture and ICA fitting)
• uv package manager
• FastAPI-based explorer

Clone the repository (required first — all commands below assume the repo root as cwd)

git clone https://github.com/liusida/ica-lens-paper.git
cd ica-lens-paper

Minimal install (any task)

uv sync

Extra steps required for any SAEBench-touching evaluation (TPP, sparse probe, SAE-overlap)

git submodule update --init --recursive
bash scripts/setup_saebench_envs.sh

The Four Things You Actually Do With This Skill

1. Browse pre-fitted ICA components in the explorer

The released artifacts ship with non-Gaussianity scores, top-activating contexts, ERF entries, and SAE-overlap annotations for GPT-2 Small, Gemma 2 2B, and Qwen 3.5 2B Base. This is the fastest way to answer interpretability questions on those models without re-fitting anything.

uv sync
uv run python scripts/fetch_artifacts.py --models --databases
uv run python scripts/verify_artifacts.py
uv run python -m server.app --port 8001

Switch to the full database when you need token-coloured context examples:

uv run python scripts/fetch_artifacts.py --models --databases --database-variant full
uv run python scripts/verify_artifacts.py --database-variant full
ICA_EXPLORER_DB_PATH=artifacts/fetched/databases/ica_probe_full.sqlite \
uv run python -m server.app --port 8001

2. Fit ICA on a new component (your own model, layer, or hook point)

This is the central reusable workflow. The recipe is the same regardless of model: capture activations at the target hook point, then fit FastICA on them. Use docs/fit_one_layer_qwen36_27b.md as a template — it documents the only place model-specific config matters.

# 1. Capture activations at the chosen hook point(s)
uv run python workflows/01_capture_activations.py \
    --config configs/activations/<your_model>.toml \
    --output-root results/<your_run>/activations \
    --include-embedding

# 2. Fit FastICA on the captured activations
uv run python workflows/02_fit_ica.py \
    --config configs/fit_ica/<your_model>.toml \
    --activation-root results/<your_run>/activations \
    --output-root results/<your_run>/ica

The TOML configs control:

• which hook points / layers are captured (configs/activations/*.toml)
• the token budget and FastICA hyperparameters (configs/fit_ica/*.toml)
• model-tokenizer wiring (configs/models/*.toml)

Author a new TOML pair by copying the closest released model's pair and editing the hook-point list.

3. Score the fitted components

Once ICA components exist, the same downstream protocols apply regardless of which model produced them.

# Non-Gaussianity diagnostics (per-direction interpretability prior)
uv run python workflows/03_compute_nongaussianity.py \
    --models <your_model> --token-budget 1000000 \
    --activation-root results/<your_run>/activations \
    --ica-root results/<your_run>/ica \
    --output-root results/<your_run>/nongaussianity

# Top-activating examples + explorer DB (for browsing / annotation)
uv run python workflows/04_build_explorer_db.py \
    --models <your_model> --token-budget 1000000 \
    --activation-root results/<your_run>/activations \
    --ica-root results/<your_run>/ica \
    --output-db results/<your_run>/databases/ica_probe.sqlite
uv run python workflows/05_populate_erf.py \
    --models <your_model> --token-budget 1000000 \
    --db-path results/<your_run>/databases/ica_probe.sqlite \
    --ica-root results/<your_run>/ica

# Component-level evaluation against trained SAE features
uv run python workflows/06_compare_ica_sae_overlap.py \
    --models <your_model> --token-budget 1000000 \
    --ica-root results/<your_run>/ica \
    --output-root results/<your_run>/ica_sae_overlap

# Targeted Probe Perturbation (TPP) — the headline SAEBench score
uv run python workflows/07_run_saebench_tpp.py \
    --model <your_model> --token-budget 1000000 \
    --ica-root results/<your_run>/ica

# Sparse probing — recoverability of labelled properties from ICA components
uv run python workflows/08_run_saebench_sparse_probe.py \
    --model <your_model> --token-budget 1000000 \
    --ica-root results/<your_run>/ica --methods all

Use these as interpretability instruments, not just as paper-reproduction commands: the per-direction non-Gaussianity score is a cheap prior on which components to inspect first; SAE overlap localises which learned dictionary atoms a given ICA direction stands in for; TPP and sparse probing quantify the direction's causal handle on labelled behaviour.

4. Inspect your own components in the explorer

The explorer reads any SQLite that follows the artifact contract, so you can point it at a self-fitted run while keeping released artifacts available for cross-comparison:

ICA_EXPLORER_DB_PATH=results/<your_run>/databases/ica_probe.sqlite \
ICA_EXPLORER_ICA_ROOT=results/<your_run>/ica \
uv run python -m server.app --port 8001

For a low-cost end-to-end dry run of the entire pipeline (capture → fit → score → explorer DB), use the demo orchestrator before committing GPU time:

uv run python scripts/reproduce_all.py --mode demo --clean --force --erf-limit 1

Programmatic Surface

Construct the FastAPI app in-process

from server.app import create_app

app = create_app()                       # honours ICA_EXPLORER_* env vars
# mount, test with fastapi.testclient.TestClient, or inspect routes

See bundled scripts/explorer_app_usage.py for a runnable demonstration including OpenAPI inspection.

Programmatic artifact-variant selection

from scripts.fetch_artifacts import _with_database_variant

selected = _with_database_variant(artifact_set, "full")

See bundled scripts/artifact_variant_example.py.

Key APIs / Surfaces

Workflow entry points (the operational interface)

Core server / scripts surface

Environment variables

Released artifacts (use as ready-made components for the three covered models)

For any other model, follow the Qwen3.6-27B doc as the template.

Related Tools

ICA Lens sits inside the feature-dictionary-learning family as the dictionary-free baseline. Pair it with the other family members along a cost–fidelity gradient:

• SAELens (../SAE/) — when over-complete coverage and per-feature monosemanticity are required at the same site.
• Transcoder circuits (../transcoder/) — when an interpretable input → output mapping is needed to slot into circuit-replacement / attribution graphs.
• Crosscoders (../crosscoder/) — when a shared basis across layers or fine-tuning checkpoints is the object of interest.
• SAEBench (submodule, shared infrastructure) — supplies the TPP and sparse-probe evaluators used by workflows/07 and workflows/08; ICA-Lens results report on the same axes that SAE / transcoder evaluations use, so cross-comparison is one-to-one.

Common Patterns & Best Practices

1. Pick the site before the method. ICA's quality depends entirely on the activation distribution at the chosen hook point — residual stream, MLP output, attention-head output, post-LN, etc. Decide where the interpretability claim lives, then capture exactly that hook point.
2. Always start from released artifacts for GPT-2 / Gemma 2 2B / Qwen 3.5 2B Base; only re-fit when extending to a new model, layer, or hook point.
3. Use non-Gaussianity as the first filter when triaging hundreds of components — rank by workflows/03 output before opening the explorer.
4. Score ICA and any candidate SAE on the same protocol (TPP, sparse probe, SAE overlap) before claiming one is "more interpretable" than the other.
5. Mini DB by default for exploration; full DB only when token-coloured contexts are needed.
6. Demo before full reproduction — scripts/reproduce_all.py --mode demo validates the orchestration end-to-end before committing GPU time.
7. Config-driven everything — for a new model / layer, copy and edit a TOML pair (configs/activations/*.toml + configs/fit_ica/*.toml) rather than passing flags.
8. Respect output layout — fetched data under artifacts/fetched/, generated data under results/; the explorer assumes this split.
9. Use the env vars (ICA_EXPLORER_DB_PATH, ICA_EXPLORER_ICA_ROOT) to mix released and self-fitted artifacts in one explorer instance.
10. Don't claim circuit-level mechanism from ICA alone — ICA produces correlated, non-causal directions. Promote a candidate to a causal claim with targeted perturbation (workflows/07) or by switching to a transcoder for replacement-context circuit work.

Demo Scripts

scripts/explorer_app_usage.py

#!/usr/bin/env python3
"""
Programmatic usage example for the ICA Lens FastAPI explorer.

Demonstrates importing `create_app`, building the FastAPI app, inspecting
registered routes, and fetching the generated OpenAPI schema via TestClient.

    uv sync
    uv run python scripts/explorer_app_usage.py
"""

from __future__ import annotations

import json
import sys
from pathlib import Path
from typing import Any

try:
    from fastapi.testclient import TestClient
except ImportError as exc:
    raise SystemExit(
        "fastapi is required to run this example. "
        "Install project dependencies with `uv sync` first."
    ) from exc


def ensure_repo_root_on_path() -> Path:
    repo_root = Path(__file__).resolve().parents[1]
    repo_root_str = str(repo_root)
    if repo_root_str not in sys.path:
        sys.path.insert(0, repo_root_str)
    return repo_root


def build_app():
    from server.app import create_app
    return create_app()


def summarize_routes(app: Any) -> list[dict[str, Any]]:
    route_summaries: list[dict[str, Any]] = []
    for route in getattr(app, "routes", []):
        methods = sorted(getattr(route, "methods", []) or [])
        path = getattr(route, "path", "<unknown>")
        name = getattr(route, "name", "<unnamed>")
        route_summaries.append({"name": name, "path": path, "methods": methods})
    return route_summaries


def fetch_openapi_schema(app: Any) -> dict[str, Any]:
    with TestClient(app) as client:
        response = client.get("/openapi.json")
        response.raise_for_status()
        return response.json()


def main() -> int:
    repo_root = ensure_repo_root_on_path()
    print(f"Repository root: {repo_root}")
    app = build_app()
    print("FastAPI app created successfully.")
    routes = summarize_routes(app)
    print(f"Registered routes: {len(routes)}")
    schema = fetch_openapi_schema(app)
    print(f"OpenAPI path count: {len(schema.get('paths', {}))}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

scripts/artifact_variant_example.py

#!/usr/bin/env python3
"""
Programmatic usage example for ICA Lens artifact variant selection.

Demonstrates `_with_database_variant(artifact_set, variant)` from
`scripts.fetch_artifacts`, used to switch a manifest between the mini and
full SQLite explorer databases without running the full download flow.

    uv sync
    uv run python scripts/artifact_variant_example.py
"""

from __future__ import annotations

import json
import sys
from pathlib import Path
from typing import Any


def ensure_repo_root_on_path() -> Path:
    repo_root = Path(__file__).resolve().parents[1]
    repo_root_str = str(repo_root)
    if repo_root_str not in sys.path:
        sys.path.insert(0, repo_root_str)
    return repo_root


def build_sample_artifact_set() -> dict[str, Any]:
    return {
        "models": [
            {"name": "gpt2", "path": "artifacts/fetched/models/gpt2/"},
            {"name": "gemma2_2b", "path": "artifacts/fetched/models/gemma2_2b/"},
            {"name": "qwen3_5_2b_base", "path": "artifacts/fetched/models/qwen3_5_2b_base/"},
        ],
        "databases": {
            "mini": {"path": "artifacts/fetched/databases/ica_probe_mini.sqlite"},
            "full": {"path": "artifacts/fetched/databases/ica_probe_full.sqlite"},
        },
    }


def main() -> int:
    ensure_repo_root_on_path()
    from scripts.fetch_artifacts import _with_database_variant

    artifact_set = build_sample_artifact_set()
    for variant in ("mini", "full"):
        selected = _with_database_variant(artifact_set, variant)
        print(f"\nVariant: {variant}")
        print(json.dumps(selected, indent=2, sort_keys=True))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Attribution

• Sida Liu — Independent Researcher — me@liusida.com
• Feijiang Han — University of Maryland — feijhan@umd.edu

License: MIT.