Organize Ml Workspace

Organize ML Workspace

Where things live, when to create a new file, what each file is allowed to contain.

Next-step pointers — where you go after this skill

Always re-emit the Pre-flight checklist with evidence before declaring the turn done.

Sibling skills — open just-in-time

Don't pre-read every sibling at session start (paralysis). Open each sibling SKILL.md when a step calls for it (e.g. open python-env-manager before G-ENV-MGR; open iterate-ml-experiment before handing off the design-note write). Emit this tracker once per turn:

Sibling skills (just-in-time):
  - data-science-python-stack, python-env-manager, python-api,
    python-code-style, iterate-ml-experiment, explore-ml-data,
    build-ml-pipeline, evaluate-ml-pipeline, test-ml-pipeline,
    smoke-test-ml-pipeline

Stop conditions — read before anything else

• Missing dependency. If import skore raises, STOP. Invoke python-env-manager for the install command. Do not drop skore.Project in favor of mlflow / pickles / "print metrics" — the workspace contract assumes a Project on disk.
• Symbol from memory is forbidden. Any skore.Project / project.put / skore.evaluate signature must come from a python-api call this turn.
• Existing layout wins — detect first. Run the Detection table before scaffolding. Don't rename, relocate, or "tidy up" existing folders.
• Notebooks are not silent. Existing .ipynb files in the experiment folder → surface the convention shift and ask. Don't auto-convert.
• Scratch is read-only against the skore Project. Probes under scratch/<ts>_<short>.py may call project.get(...), project.summarize(), walk an existing report. They MUST NOT call skore.evaluate(...) or project.put(...). When project.get(key) raises KeyError, the fix is the lookup shape: get is by id, not by key. Use summarize() → (key, id) → get(id). Never substitute by re-running evaluate + put.
• Tabular library is asked, not assumed (G-TABULAR). Pandas being importable via skore is not a pick. Invoke data-science-python-stack for the structured ask. Free-text ("quick", "you pick") does NOT resolve. Persisted in JOURNAL.md Status Workspace decisions.
• Package name is asked, not inferred (G-PKG-NAME). Before any pyproject.toml / manifest creation (including pixi init / uv init / poetry init), fire an AskUserQuestion for the src/<pkg>/ import name. Folder name in snake_case is the default. Manifest creation before G-PKG-NAME passes is forbidden — running init first creates a [project] name entry, and reading "name is in the manifest" back is circular. If a manifest exists, confirm via AskUserQuestion — continuity from a prior session is not continuity from a user decision.
• Skore Project mode is asked, not assumed (G-SKORE-MODE). Before any template instantiation containing skore.Project(...), fire an AskUserQuestion for local | hub | mlflow. Default proposal: local. All three are always offered; a detected hub config in the CWD (config.toml / SKH__* / cached login) only influences the default, never prunes the choices. Hub triggers a follow-up for the workspace name (org/team on the hub — distinct from local-mode workspace=); mlflow triggers a follow-up for the MLflow tracking server URI (tracking_uri=) — the agent cannot infer it. Persists as skore mode: (+ skore hub workspace: when hub, + skore mlflow tracking uri: when mlflow). Without it the <SKORE_PROJECT_INIT> substitution has no shape to fill. Details: → references/g_skore_mode.md.
• Switching skore mode mid-project is forbidden by default. Once recorded, do not silently change. A switch orphans every existing report in the prior store — skore has no built-in migration. Requires explicit AskUserQuestion confirmation surfacing the migration burden, then rewrite all <SKORE_PROJECT_INIT> blocks. Procedure: → references/g_skore_mode.md § "Switching mid-project".
• Env manager is asked, not assumed (G-ENV-MGR). Hand off to python-env-manager. Pixi on PATH is detection, not permission. Don't run pixi init / uv init / poetry init until G-ENV-MGR has passed in python-env-manager.
• Harness "no clarifying questions" hints do NOT waive these gates. G-TABULAR, G-PKG-NAME, G-ENV-MGR, G-SKORE-MODE, python-api consultation, new-vs-edit decision are operating-contract gates. "Quick" / "go fast" never waives them.
• Post-hoc audit — required before ending the turn. Walk every pre-flight row; if any Evidence cell is unfilled, surface the non-compliance explicitly. Most common failure: "I scaffolded successfully so everything must be fine".

Forbidden shortcuts

Pre-flight — emit before any code

Each ticked box needs an Evidence line (format spec in iterate-ml-experiment § "Pre-flight evidence requirements"; see also python-env-manager/references/preflight_evidence.md).

Pre-flight (organize-ml-workspace):
- [ ] `Workspace decisions` in `journal/JOURNAL.md` Status checked
      for pre-recorded gates (tabular, env_manager, package, skore mode)
      Evidence: lists each <gate>: <value | not recorded>
                | "n/a — JOURNAL.md does not exist yet (truly fresh)"
- [ ] Tier 1 mandatory libs importable: sklearn, skrub, skore
      Evidence: Write scratch/<ts>_check_tier1.py + `pixi run python …` output.
                **Inline `python -c` is NOT evidence**.
- [ ] Layout detection done: <existing | fresh>
      Evidence: ls/Glob on project root + matched signal from Detection
- [ ] G-TABULAR resolved: pandas | polars
      Evidence: AskUserQuestion id=<id> via data-science-python-stack |
                JOURNAL.md Status (Workspace decisions) | user quote turn N
- [ ] G-ENV-MGR resolved
      Evidence: AskUserQuestion id=<id> via python-env-manager |
                JOURNAL.md Status (Workspace decisions)
- [ ] G-PKG-NAME resolved: <name>
      Evidence: AskUserQuestion id=<id>, answer=<name> |
                JOURNAL.md Status (Workspace decisions) |
                existing manifest's [project].name **confirmed via AskUserQuestion**
                (reading the manifest alone is NOT sufficient)
- [ ] G-SKORE-MODE resolved: local | hub | mlflow
      Evidence: AskUserQuestion id=<id>, answer=<local|hub|mlflow> |
                JOURNAL.md Status (Workspace decisions) `skore mode:` row
      If hub: also captures `skore hub workspace:` row.
      If mlflow: also captures `skore mlflow tracking uri:` row.
- [ ] `pyproject.toml` present at root declaring `src/<pkg>/`;
      editable install wired via `python-env-manager` § Editable workspace
      Evidence: Read pyproject.toml (this turn) + manager's editable-install call
- [ ] python-api consulted for: Project, put, evaluate
      Evidence: Read scratch/api/skore/<v>/{project_local,evaluate}.md
                | Write of the same files (this turn)
                | "n/a — symbols already in workspace cache"
- [ ] Decision: new experiment file vs edit existing
      Evidence: AskUserQuestion id=<id> | user quote turn N |
                "n/a — first experiment in a fresh workspace"
- [ ] `journal/` scaffolded with empty placeholder JOURNAL.md
      Evidence: Write journal/JOURNAL.md (this turn) | "already exists"
- [ ] Pre-flight re-emitted with evidence before final message.
      Evidence: this checklist appears in the end-of-turn summary.

Detection — existing workspace first

Any signal present → glue to existing convention. No renames, no relocates. None present → fresh scaffold (below).

→ next: G-PKG-NAME, then python-env-manager for G-ENV-MGR.

Default layout (fresh workspace)

project/
├── pyproject.toml          # declares src/<pkg>/ as installable
├── <manager manifest>      # pixi.toml / poetry / uv / hatch / environment.yml
├── src/<pkg>/
│   ├── __init__.py         # exposes PROJECT_ROOT
│   ├── data.py             # data loading, splits, split_kwargs
│   ├── features.py         # transformers, encoders, feature fns
│   ├── pipeline.py         # the learner declaration (skrub DataOps)
│   └── evaluate.py         # ONLY: CV strategy + optional metric overrides
├── journal/
│   ├── JOURNAL.md          # session-start log; index of experiments
│   └── 01_baseline.md      # one `.md` per planned experiment
├── experiments/
│   └── 01_baseline.py      # one `# %%` script per experiment
├── audit/
│   └── 01_baseline.py      # body owned by audit-ml-pipeline (read-only)
├── tests/
│   └── smoke/              # body owned by smoke-test-ml-pipeline
├── scratch/                # agent-only (gitignored entirely)
└── reports/                # skore Project lives here

The package is installable. pyproject.toml declares src/<pkg>/; the manager installs in editable mode so from <pkg>.pipeline import build_learner works from any CWD. Wiring per-manager: python-env-manager § Editable workspace.

Runtime deps (sklearn, skrub, skore, tabular) live in the manager's manifest, not in [project.dependencies].

Deliberately absent: no data/ (user-owned), no models/ (out of scope). Add later only on user request — don't pre-empt. The sole writer into data/ is explore-ml-data, and only its own EDA deliverables (data/eda.py / data/eda.md / data/eda_<table>.html); the user's raw data is never modified by any skill.

File-creation rules

Design note first, then code

Before creating experiments/NN_<short_name>.py, the matching journal/NN_<short_name>.md must exist and have been validated by the user. Design-note content is owned by iterate-ml-experiment; this skill only enforces the pairing.

Four-way stem pairing

Every experiment is identified by NN_<short_name> in four places:

journal/NN_<short_name>.md            (design note)
experiments/NN_<short_name>.py        (script)
tests/smoke/test_NN_<short_name>.py   (smoke test)
audit/NN_<short_name>.py              (audit file — read-only)

By the time the experiment flips to done in JOURNAL.md, all four exist. The design note is written first; the script lands on approval; the smoke test body is filled by smoke-test-ml-pipeline; the audit file is placed and executed by audit-ml-pipeline at § 4 record-outcome.

The audit file is read-only against the workspace's skore Project and data — see audit-ml-pipeline § Read-only contract.

New experiment → new file. Iterating → ask first.

Default: new file. 02_text_encoder.py, 03_grouped_cv.py. The numeric prefix preserves iteration order under ls.

When the user says "let's tweak experiment 02", do not assume. Fire AskUserQuestion:

Should this be a new experiment file (e.g. 04_text_encoder_v2.py) or an in-place edit of 02_text_encoder.py?

In-place edits overwrite the prior result in the skore Project if the same key is reused — flag this. In-place also requires revisiting the matching smoke test (→ smoke-test-ml-pipeline).

Decision flow (12 steps — full version in references/scaffold_steps.md)

→ next: iterate-ml-experiment § 0 (bootstrap) for the first design note.

Files in src//

Each has a narrow contract:

• __init__.py — exposes PROJECT_ROOT (absolute, derived from __file__, not CWD). Modules needing project-relative paths import this constant. Requires editable install.
• data.py — loaders, materialization of X, y, any split_kwargs (groups, time, …) at the X marker. Pipeline mechanics in build-ml-pipeline.
• features.py — feature functions and transformers.
• pipeline.py — the learner declaration (a SkrubLearner). build_learner exposes data_dir_preview=None so the experiment script can pass an absolute path from PROJECT_ROOT.
• evaluate.py — only the inputs to skore.evaluate: the cross-validator (splitter = ...), optional metric overrides. Does NOT call skore.evaluate, does NOT open a Project, does NOT persist.

Experiment scripts — experiments/NN_*.py

# %% cell markers, not .ipynb. Template: templates/experiment.py. What the script does:

1. Open / attach to the skore.Project at reports/ (or hub / mlflow server, per skore mode:).
2. Import the learner from <pkg>.pipeline and CV from <pkg>.evaluate.
3. Call skore.evaluate(...).
4. Call project.put("<experiment-key>", report).

Confirm signatures via python-api. Cross-validator choice is evaluate-ml-pipeline.

Project init substitution — the <SKORE_PROJECT_INIT> marker in templates/experiment.py is replaced at scaffold time per the recorded skore mode: decision. Three forms (local / hub / mlflow), side-by-side anatomy, audit-file copy rule: → references/g_skore_mode.md.

Experiment scripts stay clean of agent-only print(...). Inspection lives in scratch/. One exception: a bare report expression — that's a notebook-display side effect.

Experiment key convention — the file's stem (e.g. 01_baseline.py → "01_baseline"). One file → one key → one report.

Companion skills

Templates

• templates/experiment.py — copied per new experiment
• templates/pyproject.toml — declares src/<pkg>/ as installable
• templates/src___init__.py — package init with PROJECT_ROOT
• templates/src_data.py / src_features.py / src_pipeline.py / src_evaluate.py — one-time skeletons
• templates/.gitignore — dropped at scaffold if none exists

Copy, don't rewrite. Section names encode contracts.

References (load on demand)

• references/scaffold_steps.md — full prose elaboration of the 13-step Decision flow with examples and rationale.
• references/g_skore_mode.md — the G-SKORE-MODE gate in detail: the three project init forms (local / hub / mlflow) side-by-side, anatomy of the <SKORE_PROJECT_INIT> substitution, switching mid-project, out-of-scope notes (running the MLflow server, Skore Hub account creation).