exps_logistic - Logistic Regression MI Estimation
Purpose
Measure mutual information (MI) lower bound between CoT rationales and problem parameters using multinomial logistic regression on embeddings. Compares code vs NL representations.
Key Files
| File |
Purpose |
src/exps_logistic/main.py |
Main experiment runner |
src/exps_logistic/config.py |
Configuration, CLI args, kind presets |
src/exps_logistic/data_utils.py |
Data loading, gamma label creation |
src/exps_logistic/classifier.py |
Logistic regression classifier |
src/exps_logistic/featurizer.py |
Embedding extraction (BERT, TF-IDF) |
src/exps_logistic/run.sh |
Dev script for single model |
src/exps_logistic/prod_logistic.sh |
Production script for all models |
src/exps_logistic/notebooks/generate_plots.py |
Plot generation |
Kind Presets (in config.py)
FG_KINDS = {"add", "sub", "mul", "lcs", "knap", "rod", "ilp_assign", "ilp_prod", "ilp_partition"}
CLRS_KINDS = {
"activity_selector", "articulation_points", "bellman_ford", "bfs",
"binary_search", "bridges", "bubble_sort", "dag_shortest_paths",
"dfs", "dijkstra", "find_maximum_subarray_kadane", "floyd_warshall",
"graham_scan", "heapsort", "insertion_sort", "jarvis_march",
"kmp_matcher", "lcs_length", "matrix_chain_order", "minimum",
"mst_kruskal", "mst_prim", "naive_string_matcher", "optimal_bst",
"quickselect", "quicksort", "segments_intersect",
"strongly_connected_components", "task_scheduling", "topological_sort",
}
NPHARD_KINDS = {"edp", "gcp", "ksp", "spp", "tsp"}
EXTENDED_KINDS = FG_KINDS | CLRS_KINDS | NPHARD_KINDS # 44 total kinds
Gamma Labels
Format: {kind}|d{digits}|b{bin} (e.g., knap|d8|b24)
- Fine-grained kinds have parsed bin values based on problem parameters
- CLRS/NP-hard kinds get
bNA (no fine-grained parsing)
CLI Arguments
| Arg |
Description |
--results-dir |
Path to exps_performance results |
--models |
Model filter (e.g., anthropic/claude-haiku-4.5) |
--rep |
Representation: code or nl |
--label |
Label type: kind, theta_new, or gamma |
--kinds-preset |
Kind filter: fg, clrs, nphard, or extended |
--feats |
Featurizer: hf-cls, tfidf, st, openai |
--embed-model |
HuggingFace model for embeddings |
--no-cv |
Disable cross-validation |
--bits |
Report cross-entropy in bits |
Generated Plots (in src/exps_logistic/notebooks/)
| Plot |
Description |
boxplot_extended.png |
Code vs NL MI comparison |
kde_extended.png |
KDE density comparison |
mi_vs_accuracy_extended.png |
MI vs task accuracy correlation |
contrast_extended.png |
Per-model contrast (code - nl) |
label_dist_by_kind.png |
Sample counts by problem kind |
label_dist_by_digits.png |
Fine-grained distribution by digits |
label_dist_heatmap.png |
Kind × Digits heatmap |
label_dist_top30.png |
Top 30 most frequent labels |
label_dist_zipf.png |
Zipf-like frequency distribution |
label_dist_pie.png |
Category breakdown pie chart |
Key Results (Extended Kinds - Jan 9, 2026)
- 28 successful runs across 6 models
- Pearson r = 0.486, p = 0.009 (MI vs accuracy correlation)
- 500 unique labels across 44 problem kinds
- Code representation shows higher MI lower bounds than NL
- Models tested: claude-haiku-4.5, gemini-2.5-flash, gpt-4o-mini, llama-3.1-405b-instruct, ministral-14b-2512, qwen-2.5-coder-32b-instruct
Running Commands
# Dev run (single model)
bash src/exps_logistic/run.sh
# Production run (all models, all seeds)
bash src/exps_logistic/prod_logistic.sh
# Generate plots
uv run --no-sync python src/exps_logistic/notebooks/generate_plots.py
1---2name: exps-logistic3description: Documentation for the logistic regression MI estimation experiment (exps_logistic)4---5
6# exps_logistic - Logistic Regression MI Estimation
7
8## Purpose
9Measure mutual information (MI) lower bound between CoT rationales and problem parameters using multinomial logistic regression on embeddings. Compares code vs NL representations.
10
11## Key Files
12| File | Purpose |
13|------|---------|
14| `src/exps_logistic/main.py` | Main experiment runner |
15| `src/exps_logistic/config.py` | Configuration, CLI args, kind presets |
16| `src/exps_logistic/data_utils.py` | Data loading, gamma label creation |
17| `src/exps_logistic/classifier.py` | Logistic regression classifier |
18| `src/exps_logistic/featurizer.py` | Embedding extraction (BERT, TF-IDF) |
19| `src/exps_logistic/run.sh` | Dev script for single model |
20| `src/exps_logistic/prod_logistic.sh` | Production script for all models |
21| `src/exps_logistic/notebooks/generate_plots.py` | Plot generation |
22
23## Kind Presets (in config.py)
24
25```python
26FG_KINDS = {"add", "sub", "mul", "lcs", "knap", "rod", "ilp_assign", "ilp_prod", "ilp_partition"}
27
28CLRS_KINDS = {
29 "activity_selector", "articulation_points", "bellman_ford", "bfs",
30 "binary_search", "bridges", "bubble_sort", "dag_shortest_paths",
31 "dfs", "dijkstra", "find_maximum_subarray_kadane", "floyd_warshall",
32 "graham_scan", "heapsort", "insertion_sort", "jarvis_march",
33 "kmp_matcher", "lcs_length", "matrix_chain_order", "minimum",
34 "mst_kruskal", "mst_prim", "naive_string_matcher", "optimal_bst",
35 "quickselect", "quicksort", "segments_intersect",
36 "strongly_connected_components", "task_scheduling", "topological_sort",
37}
38
39NPHARD_KINDS = {"edp", "gcp", "ksp", "spp", "tsp"}
40
41EXTENDED_KINDS = FG_KINDS | CLRS_KINDS | NPHARD_KINDS # 44 total kinds
42```
43
44## Gamma Labels
45Format: `{kind}|d{digits}|b{bin}` (e.g., `knap|d8|b24`)
46- Fine-grained kinds have parsed bin values based on problem parameters
47- CLRS/NP-hard kinds get `bNA` (no fine-grained parsing)
48
49## CLI Arguments
50| Arg | Description |
51|-----|-------------|
52| `--results-dir` | Path to exps_performance results |
53| `--models` | Model filter (e.g., `anthropic/claude-haiku-4.5`) |
54| `--rep` | Representation: `code` or `nl` |
55| `--label` | Label type: `kind`, `theta_new`, or `gamma` |
56| `--kinds-preset` | Kind filter: `fg`, `clrs`, `nphard`, or `extended` |
57| `--feats` | Featurizer: `hf-cls`, `tfidf`, `st`, `openai` |
58| `--embed-model` | HuggingFace model for embeddings |
59| `--no-cv` | Disable cross-validation |
60| `--bits` | Report cross-entropy in bits |
61
62## Generated Plots (in `src/exps_logistic/notebooks/`)
63| Plot | Description |
64|------|-------------|
65| `boxplot_extended.png` | Code vs NL MI comparison |
66| `kde_extended.png` | KDE density comparison |
67| `mi_vs_accuracy_extended.png` | MI vs task accuracy correlation |
68| `contrast_extended.png` | Per-model contrast (code - nl) |
69| `label_dist_by_kind.png` | Sample counts by problem kind |
70| `label_dist_by_digits.png` | Fine-grained distribution by digits |
71| `label_dist_heatmap.png` | Kind × Digits heatmap |
72| `label_dist_top30.png` | Top 30 most frequent labels |
73| `label_dist_zipf.png` | Zipf-like frequency distribution |
74| `label_dist_pie.png` | Category breakdown pie chart |
75
76## Key Results (Extended Kinds - Jan 9, 2026)
77- **28 successful runs** across 6 models
78- **Pearson r = 0.486, p = 0.009** (MI vs accuracy correlation)
79- **500 unique labels** across 44 problem kinds
80- Code representation shows higher MI lower bounds than NL
81- Models tested: claude-haiku-4.5, gemini-2.5-flash, gpt-4o-mini, llama-3.1-405b-instruct, ministral-14b-2512, qwen-2.5-coder-32b-instruct
82
83## Running Commands
84```bash
85# Dev run (single model)
86bash src/exps_logistic/run.sh
87
88# Production run (all models, all seeds)
89bash src/exps_logistic/prod_logistic.sh
90
91# Generate plots
92uv run --no-sync python src/exps_logistic/notebooks/generate_plots.py
93```
majiayu000