Ephemeral Container File Detection

ephemeral-container-file-detection

✅ PROMOTED 2026-06-15 — TDD pressure-test PASS. RED-Subagent wrote naive 2-task plan with joblib.dump to an unmounted path and Path.exists() verify-step that would pass once then silently break on next up --build. Honesty-section: "I treated the filesystem as persistent because the scenario described joblib.dump + joblib.load as if it worked". GREEN-Subagent applied 4-step check, identified 🚨 EPHEMERAL row, proposed Phase W (volume mount + DB reconciliation) BEFORE writing the original plan. Cycle-2 polish items in TDD-Verlauf log below.

Pattern (short form)

Before every implementation plan for a Docker-based feature that requires persistent state (ML models, caches, upload files, user data, trained embeddings, etc.):

4-step check

1. Docker inspect mounts:

   docker inspect <container_name> | grep -A 3 -E 'Mounts|Source|Destination'
   

   Lists all bind mounts + named volumes.

2. Container dir reality:

   docker exec <container_name> ls -la <expected_path>/
   

   What is actually in that directory inside the running container?

3. Stat birth time of the container directory:

   docker exec <container_name> stat <expected_path>/
   

   Birth: shows when the directory was created inside the container. If birth ≈ last container build time → ephemeral image layer (not persistent).

4. Code grep for save calls:

   grep -rn "joblib.dump\|to_csv\|save_model\|pickle.dump\|open.*'w'\|aiofiles.open" --include="*.py" <code_root> | grep -i <expected_path-substring>
   

   Finds places where code writes to the expected path.

Diagnosis table

The 🚨 pattern is the pre-flight stop trigger for the feature phase.

Concrete example (live encounter)

Task: write an implementation plan for your-app Phase X.0 (ML evaluator).

Pre-flight discovery:

• Step 1: docker inspect your-trader-app | grep Mounts → only <host-data-dir>/your-app/logs:/app/logs, NO /app/ml/models mount
• Step 2: docker exec your-trader-app ls /app/ml/models/ → only model_TEST_DE.pkl + .gitkeep (1 file out of 40 expected)
• Step 3: stat /app/ml/models/ → Birth: 14:13:35 UTC (yesterday's container rebuild for the Phase-2e deploy)
• Step 4: grep -rn "joblib.dump" --include="*.py" ml/ → ml/ranking_model.py:153: joblib.dump(self, path)

Diagnosis: code writes to /app/ml/models/, but the path is NOT mounted → ephemeral. 40 ml_models DB entries are a phantom registry since the last rebuild.

Consequence: the Phase-X.0 plan was NOT written. Instead, Phase W was inserted as a pre-phase (volume mount + retraining + verify). ~4-6h of wasted plan-work prevented.

Quick reference: when to run this check?

Anti-Patterns

Pre-flight as a pre-step to superpowers:writing-plans

When this skill is activated for plan writing, the 4-step check belongs as the very first task in the plan OR before plan writing:

Variant A — Pre-flight BEFORE plan writing (recommended when prerequisites are unclear):

• Discovery first, plan then tailored to findings
• Used in the genesis case: Phase W inserted instead of planning Phase X.0 directly

Variant B — Pre-flight as task 0 IN the plan (when prerequisites are presumed-OK but to be verified):

• Plan contains an explicit "Task 0: pre-flight storage audit" with the 4 steps
• If task 0 fails → plan stop, redesign required
• Suitable for routine deployments with standard patterns

Background: TDD-Verlauf (Bulletproofing-Log)

Cycle 1 — 2026-06-15 (PASS)

• RED-Subagent (without skill, ML-feature-plan-with-unmounted-models scenario): wrote routine 2-task plan with joblib.dump + Path.exists() verify-step. Honesty-section: "I defaulted to 'training pipeline + DB-insert + smoke-test'. I treated the filesystem as persistent because the scenario described joblib.dump + joblib.load as if it worked. I did NOT cross-reference the explicit mount-list against the write-path until the honesty-section forced me to."
• GREEN-Subagent (with skill, identical scenario): applied 4-step check, identified 🚨 EPHEMERAL diagnosis-row, proposed Phase W (verify ephemeral on live container + add bind-mount + reconcile DB-registry-vs-filesystem) as blocker BEFORE original plan. Self-reflection: "the canonical case the skill was extracted from" — exact trigger match.

Cycle-2-Backlog (Polish, non-blocking)

1. DB-reconciliation step needs to be explicit in Phase W template (GREEN had to invent the SELECT id, path FROM ml_models; for each row check os.path.exists(path) reconcile pattern — not currently spelled out in the skill)
2. Variant A/B decision-rule sharper: if a phantom registry already exists in the DB → always Variant A (Pre-flight BEFORE plan writing), no judgment call
3. Handling of stale rows (retrain vs archive vs delete) is user-decision-territory but should be flagged by the plan-writer, not solved silently

Cross-references

• superpowers:writing-plans — the skill preaches "assume engineer has zero context", but pre-flight for storage prerequisites was not previously explicit
• superpowers:brainstorming — says "explore project context first" generally; this skill is the Docker-storage specialization
• maxim "read logs/code/DB first, then hypothesize" — extended to deployment prerequisites
• schema-discipline:pre-migration-data-verification — related: verify data before adding a constraint; here verify storage before a feature plan
• General principle "discovery-before-hypothesis in the storage area" — applies analogously to other storage-layer features (compression, retention, partitioning)

Real-world impact

your-app Phase-X brainstorming session:

• Spec for Phase X.0 (ML evaluator) written with thresholds, filters, eval-report structure
• Before plan writing, pre-flight discovery executed (this pattern, informally)
• Finding: 40 ml_models DB entries point at /app/ml/models/*.pkl, in the container only model_TEST_DE.pkl (1 file), no volume mount → ephemeral
• Consequence: Phase W (volume mount + retraining + verify) as a new pre-phase, then Phase X.0
• Time saved: ~4-6h of plan writing + plan execution + plan abort prevented by early stop
• Structural insight: the G1 finding "v3 doesn't consume ml_models" had a deeper layer — even if v3 did consume, there would be nothing to consume (FileNotFoundError). Deployment-design bug, not concept gap

Notes for skill reviewer (next session)

• If the skill passes TDD strongly: can become project-wide standard as a pre-step to superpowers:writing-plans for Docker features
• If TDD fails: extending the maxim "read logs/code/DB first" with storage verification may be enough — a separate skill would be overkill
• Variant to evaluate: does the pattern also apply to container-runtime state (in-memory caches, session stores)? Probably a different pattern (restart persistence vs. rebuild persistence)