Swmm Experiment Audit

SWMM Experiment Audit

Part of Agentic SWMM — install the project first for the executable toolchain (aiswmm CLI, SWMM solver, MCP servers).

What this skill provides

• A standard audit layer for Agentic SWMM runs.
• Consolidation of dispersed manifest.json, QA JSON, logs, metrics, and artifact paths.
• Machine-readable outputs for reproducibility and review.
• Obsidian-compatible Markdown notes for human research records.
• Default local Obsidian export into a clean English audit vault.
• Automatic update of the Obsidian Experiment Audit Index.
• Optional run-to-run comparison for baseline/scenario or before/after parser validation.

This skill records what happened. It does not run SWMM, build models, invent missing artifacts, or replace module-level validation.

When to use this skill

Use this skill after any of these events:

• swmm-end-to-end completes successfully.
• swmm-end-to-end stops or fails after producing partial artifacts.
• A user wants an Obsidian-ready experiment note for an existing run directory.
• A user wants to compare two run directories.
• A run needs evidence for reproducibility, metric provenance, QA status, or paper claims.

Do not use this skill as a substitute for swmm-runner, swmm-builder, or calibration tools. Run the model first, then audit the run directory.

Output contract

For every audited run, write these files into the run's 09_audit/ directory unless explicit output paths are provided:

• experiment_provenance.json — machine-readable provenance (immutable).
• experiment_note.md — human-readable Obsidian digest.
• model_diagnostics.json — deterministic SWMM screening checks.
• comparison.json — run-to-run comparison (only when --compare-to is given).

experiment_provenance.json is the machine-readable source for:

• run identity
• repo state
• tool versions
• command trace
• input hash records
• artifact index
• metrics with source artifacts and source tables
• QA checks
• detected warnings and limitations

comparison.json records differences against another run when --compare-to is provided. If no comparison target is provided, it still records that no comparison was requested.

experiment_note.md is Obsidian-compatible Markdown with YAML frontmatter. It should stay readable in GitHub as plain Markdown.

The direct script (audit_run.py) writes a copy of the audit note to the Obsidian vault by default; pass --no-obsidian to suppress it. The canonical CLI (aiswmm audit) does not export to Obsidian by default; pass --obsidian to enable it.

When Obsidian export is active, the note is written into:

~/Documents/Agentic-SWMM-Obsidian-Vault/20_Audit_Layer/Experiment_Audits

and the index is updated at:

~/Documents/Agentic-SWMM-Obsidian-Vault/20_Audit_Layer/Experiment Audit Index.md

CLI

The canonical CLI is aiswmm audit. It wraps the underlying script with backup of prior audit files, MOC regeneration (runs/INDEX.md), and the M2 audit → memory auto-trigger. The direct script path is also supported and is the primary option for MCP / Mode-0 calls.

Canonical CLI (aiswmm audit)

Obsidian export is opt-in with aiswmm audit — pass --obsidian to copy the note to the default vault. Without --obsidian the note is written only into 09_audit/ inside the run directory.

aiswmm audit --run-dir runs/acceptance/latest

With comparison:

aiswmm audit --run-dir runs/acceptance/codex-check-peakfix \
  --compare-to runs/acceptance/codex-check

With explicit metadata and Obsidian export:

aiswmm audit --run-dir runs/real-todcreek-minimal \
  --case-name "Tod Creek minimal" \
  --workflow-mode "minimal real-data fallback" \
  --objective "Verify real-data SWMM execution and preserve provenance." \
  --obsidian

Skip the M2 memory auto-trigger (useful for acceptance / benchmark runs):

aiswmm audit --run-dir runs/acceptance/latest --no-memory

Run-to-run comparison with the standalone verb:

aiswmm compare --run-a runs/baseline --run-b runs/scenario

Direct script path (python3 scripts/audit_run.py)

Use when calling from MCP or when the full aiswmm install is unavailable. Obsidian export is on by default in the script; disable with --no-obsidian.

Initialize a first-user Obsidian vault:

python3 skills/swmm-experiment-audit/scripts/init_obsidian_vault.py

python3 skills/swmm-experiment-audit/scripts/audit_run.py \
  --run-dir runs/acceptance/latest \
  --no-obsidian

With comparison:

python3 skills/swmm-experiment-audit/scripts/audit_run.py \
  --run-dir runs/acceptance/codex-check-peakfix \
  --compare-to runs/acceptance/codex-check \
  --no-obsidian

With explicit metadata including --case-id (records the case slug in experiment_provenance.json for cross-session memory recall):

python3 skills/swmm-experiment-audit/scripts/audit_run.py \
  --run-dir runs/real-todcreek-minimal \
  --case-id tod-creek \
  --case-name "Tod Creek minimal" \
  --workflow-mode "minimal real-data fallback" \
  --objective "Verify real-data SWMM execution and preserve provenance." \
  --no-obsidian

With an Obsidian vault folder:

python3 skills/swmm-experiment-audit/scripts/audit_run.py \
  --run-dir runs/acceptance/latest \
  --obsidian-dir "/path/to/Obsidian/Agentic SWMM/04_Experiments"

Audit rules

• Always preserve relative paths when the artifact is inside the repository.
• Include absolute paths in JSON only when useful for local traceability.
• Record SHA256 for existing file artifacts when feasible.
• Record artifact role, producer, and downstream use.
• Preserve command return codes, stdout paths, stderr paths, and timings when available.
• Treat failed or partial runs as auditable. Missing artifacts should be recorded as missing, not invented.
• Keep metrics tied to source artifacts and source tables.
• For SWMM peak flow, prefer Node Inflow Summary / Maximum Total Inflow.
• Use Outfall Loading Summary / Max Flow only as fallback for outfalls.
• Do not extract peak flow from Node Depth Summary; that table reports depth and HGL, not flow.

Relationship to swmm-end-to-end

swmm-end-to-end is the executor and orchestrator.

swmm-experiment-audit is the recorder and auditor.

The agent should run this audit skill after every build/run/QA attempt, even when the workflow stops early or fails. The audit output should reference whatever artifacts exist in the run directory and clearly mark missing or incomplete evidence.

Obsidian support

The generated experiment_note.md is designed for Obsidian:

• YAML frontmatter
• stable headings
• tables for QA, metrics, and artifact index
• relative paths for vault portability
• no chat transcript or conversational content

The note is also valid GitHub Markdown, so it can be committed as an example or exported as supplementary evidence if desired.

The default local vault is:

~/Documents/Agentic-SWMM-Obsidian-Vault

It is organized for first-time Obsidian use:

00_Home/
10_Memory_Layer/
20_Audit_Layer/
30_Evidence_Layer/
40_Skill_Evolution/
90_Templates/

The direct script always writes canonical outputs into the run directory and copies the note to the Obsidian vault unless --no-obsidian is given. The canonical CLI (aiswmm audit) writes canonical outputs only; add --obsidian to also copy to the vault.

In both paths, use --obsidian-dir and --obsidian-index to target a vault location other than the default ~/Documents/Agentic-SWMM-Obsidian-Vault.