2438 Refactor Microsim Generator For Token Efficiency Cca15798

Session Log: Refactor MicroSim Generator for Token Efficiency

Date: February 10, 2026 Session Focus: Create batch processing utilities to eliminate ~430K tokens of waste per MicroSim generation run Skills Used: microsim-generator, chapter-content-generator Primary Tools: Python script development, SKILL.md updates

Table of Contents

1. Session Overview
2. Problem Analysis
3. Files Created
4. Files Modified
5. Verification Results
6. sim-status.json Schema
7. Batch Workflow
8. Summary and Results

Session Overview

The batch generation of 31 MicroSims for geometry chapters 1-3 consumed ~1.6M tokens. Analysis showed ~55% of tokens were wasted on repetitive tasks: redundant skill guide reads (25%), boilerplate file generation (15%), mkdocs.yml nav editing (10%), and spec parsing from chapters (5%). This session created five Python utilities to eliminate that waste, plus a sim-status.json tracking file for resuming work across context windows.

Key Achievement: Created a complete pipeline of 5 Python scripts (standard library only) that eliminates ~430K tokens per batch run and enables fast restart after context window fills.

Problem Analysis

Files Created

All new files in /Users/dan/Documents/ws/claude-skills/src/microsim-utils/:

1. shared.py — Common Utilities

Common module imported by all scripts. Contains:

• ANSI color constants: GREEN, RED, YELLOW, BLUE, CYAN, BOLD, DIM, RESET
• Unicode symbols: CHECK (✔), CROSS (✘), WARN (⚠), ARROW (→), BULLET (•)
• find_project_root(start_dir): Walks up from start_dir to find the directory containing mkdocs.yml
• load_mkdocs_config(project_dir): Parses site_url, site_name, docs_dir from mkdocs.yml using regex (no PyYAML dependency)
• kebab_case(title): Converts title strings to kebab-case directory names (e.g., "Point Line and Plane" → "point-line-and-plane")
• parse_yaml_frontmatter(content): Extracts flat key-value YAML frontmatter from markdown content
• detect_library(html_content): Detects JavaScript library from <script> src attributes in main.html (p5.js, vis-network, Chart.js, Mermaid, Plotly, Leaflet, vis-timeline)
• LIBRARY_CDNS: CDN URL mapping for scaffold generation (p5.js 1.11.10, vis-network 9.1.9, Chart.js 4.4.4, Mermaid 10, Plotly 2.35.0, Leaflet 1.9.4)
• LIBRARY_CSS: CSS CDN URLs for libraries that need them (vis-timeline, Leaflet)

2. extract-sim-specs.py (Priority 1, ~80K token savings)

Parses all #### Diagram: / #### Drawing: headers from chapter markdown files. Handles both layout patterns found in the geometry course:

• iframe-before-details (chapters 01-10): iframe appears above the <details> block
• details-before-iframe (chapter 11): <details> block appears first, iframe below

Key features:

• Extracts sim_id from iframe src path, explicit **sim-id:** field, or kebab-case title fallback
• Extracts library from explicit **Library:** field or Implementation line hints
• Infers Bloom level from learning objective text when not explicitly stated
• --status-file flag generates sim-status.json combining spec extraction with filesystem scan

python3 extract-sim-specs.py --project-dir /path/to/project --output specs.json --verbose
python3 extract-sim-specs.py --project-dir /path/to/project --chapter 03-angles-and-relationships --output ch03.json
python3 extract-sim-specs.py --project-dir /path/to/project --status-file sim-status.json --verbose

Output fields per spec: sim_id, title, summary, heading_type, chapter, element_type, bloom_level, library, iframe_src, iframe_height, spec_text, status

3. generate-sim-scaffold.py (Priority 2, ~150K token savings)

Reads specs JSON from extract-sim-specs.py and generates three scaffold files per sim:

• main.html: Correct CDN for detected library, schema meta tag, <main> tag (no id attribute), link back to lesson plan
• index.md: YAML frontmatter (title, description, image paths, quality_score: 0), iframe embed, fullscreen button, placeholder sections for About, How to Use, Lesson Plan, References
• metadata.json: Dublin Core + educational + pedagogical fields with TODO placeholders

The agent then only needs to write the .js file — all boilerplate is pre-generated.

python3 generate-sim-scaffold.py --spec-file specs.json --project-dir /path/to/project --verbose
python3 generate-sim-scaffold.py --spec-file specs.json --sim-id angle-explorer --dry-run
python3 generate-sim-scaffold.py --spec-file specs.json --force  # overwrite existing

4. update-mkdocs-nav.py (Priority 3, ~100K token savings)

Scans docs/sims/ for directories containing index.md, extracts display titles (from frontmatter > # heading > directory name), and replaces the MicroSims nav section in mkdocs.yml with an alphabetically sorted list.

• Keeps "List of MicroSims: sims/index.md" as first entry
• Idempotent — safe to run multiple times
• Finds section boundaries by detecting - MicroSims: start and next top-level nav entry

python3 update-mkdocs-nav.py --project-dir /path/to/project --dry-run --verbose
python3 update-mkdocs-nav.py --project-dir /path/to/project

5. add-iframes-to-chapter.py (Priority 4, ~50K token savings)

Finds #### Diagram: / #### Drawing: entries in chapter markdown that are missing iframe embeds and inserts them before the <details> block.

Additional features:

• --fix-heights: Parses JS files for createCanvas() calls and updates iframe heights to match
• --fix-paths: Normalizes absolute paths (/sims/) to relative (../../sims/), fixes 500xp typos to 500px

python3 add-iframes-to-chapter.py --chapter 11-circles --project-dir /path/to/project --dry-run --verbose
python3 add-iframes-to-chapter.py --all --project-dir /path/to/project --fix-heights --fix-paths

6. validate-sims.py (Priority 5, ~50K token savings)

100-point quality rubric aligned with the existing calculate-quality-score.py in the microsim-utils skill:

Non-p5.js sims get full marks on the p5.js conventions category.

python3 validate-sims.py --project-dir /path/to/project --sim angle-builder --verbose
python3 validate-sims.py --project-dir /path/to/project --min-score 0 --format table
python3 validate-sims.py --project-dir /path/to/project --output scores.json

7. README.md

Full documentation with:

• Overview table of all 5 scripts with token savings
• ASCII workflow diagram showing script chaining
• Per-script documentation with usage examples
• sim-status.json schema documentation
• Design constraints (standard library only, Python 3.7+, idempotent operations)
• Complete batch generation workflow with 6-step CLI commands

Files Modified

8. skills/microsim-generator/SKILL.md

Section added: "Batch Generation Utilities" after the existing "mkdocs.yml Integration" section (around line 325).

Changes:

• Updated "mkdocs.yml Integration" section to reference update-mkdocs-nav.py for batch operations
• Added table of 5 utilities with purpose and token savings
• Added 6-step batch workflow with exact CLI commands
• Added sim-status.json integration documentation explaining the lifecycle states

9. skills/chapter-content-generator/SKILL.md

Updated: The <details markdown="1"> template (around line 315) to include three machine-readable fields:

**sim-id:** [kebab-case-directory-name]<br/>
**Library:** [p5.js | vis-network | Chart.js | Mermaid | Plotly | Leaflet | vis-timeline]<br/>
**Status:** Specified

Added explanatory text describing how these fields enable machine-readable extraction by batch utilities.

10. skills/chapter-content-generator/references/content-element-types.md

Three changes made:

1. Fixed height="500xp" typo → height="500px" — 8 occurrences across all example specifications (Diagrams, Infographics, MicroSims, Charts, Timelines, Maps, Workflows, Graph Models, Details Block Template)

2. Replaced {MICROSIM_NAME} → {sim-id} — all occurrences, with updated note about kebab-case naming

3. Updated Details Block Template (end of file) to include sim-id, Library, Status structured fields with explanatory text

4. Updated CMDB Architecture Diagram example to include the new structured fields as a concrete example

Verification Results

All 5 verification tests passed against the geometry-course repository:

Test 1: extract-sim-specs.py

Total specs found: 169
  01-foundations-of-geometry: 8 specs
  02-logic-and-proof: 11 specs
  03-angles-and-relationships: 17 specs
  04-geometric-constructions: 16 specs
  05-coordinate-geometry: 14 specs
  06-transformations-congruence: 13 specs
  07-triangle-congruence: 15 specs
  08-triangle-centers: 14 specs
  09-polygons-quadrilaterals: 13 specs
  10-similarity-trigonometry: 14 specs
  11-circles: 16 specs
  12-area-volume-applications: 18 specs

• Correctly detected both iframe-before-details and details-before-iframe patterns
• Green checkmarks (✔) for specs with existing iframes, yellow warnings (⚠) for missing iframes

Test 2: generate-sim-scaffold.py --dry-run

[dry-run] Would create docs/sims/chords-in-a-circle/main.html
[dry-run] Would create docs/sims/chords-in-a-circle/index.md
[dry-run] Would create docs/sims/chords-in-a-circle/metadata.json
✔ [dry-run] Scaffolded: 1  Skipped: 0  Total: 1

Test 3: update-mkdocs-nav.py --dry-run

Found 144 MicroSims with index.md
[dry-run] Would replace lines 41-169 (129 lines) with 146 lines
[dry-run] MicroSims nav entries: 144

Test 4: validate-sims.py --sim angle-builder

MicroSim                                 Score  Grade
-----------------------------------------------------
✔ angle-builder                             85     A
      main.html: missing schema meta tag
      metadata.json: missing educational section
      screenshot PNG missing
      p5.js: uses DOM functions (createButton(, createSlider(, createSelect()

Score breakdown: 85/100 (loses 3 for missing schema tag, 5 for missing educational section, 5 for missing screenshot, 2 for DOM functions)

Test 5: extract-sim-specs.py --status-file

sim-status.json summary:
  specified     83
  scaffolded    3
  implemented   30
  validated     51
  deployed      60
  total         227

227 total entries: 169 from chapter specs + 58 additional sims found in docs/sims/ not referenced in chapters.

sim-status.json Schema

Generated by extract-sim-specs.py --status-file. Enables fast restart after context window fills.

Status Lifecycle

specified → scaffolded → implemented → validated → deployed

Entry Schema

{
  "sim_id": "angle-builder",
  "title": "Angle Builder",
  "chapter": "01-foundations-of-geometry",
  "bloom_level": "Apply",
  "library": "p5.js",
  "status": "deployed",
  "has_iframe": true,
  "quality_score": 85
}

Batch Workflow

The complete workflow for generating MicroSims across multiple chapters:

PROJECT=/Users/dan/Documents/ws/geometry-course
UTILS=/Users/dan/Documents/ws/claude-skills/src/microsim-utils

# 1. Extract all specs from chapter markdown
python3 $UTILS/extract-sim-specs.py \
    --project-dir $PROJECT --output /tmp/specs.json \
    --status-file $PROJECT/docs/sims/sim-status.json --verbose

# 2. Scaffold unbuilt sims (main.html, index.md, metadata.json)
python3 $UTILS/generate-sim-scaffold.py \
    --spec-file /tmp/specs.json --project-dir $PROJECT --verbose

# 3. Agent implements .js files (this is where the creative work happens)
#    Read sim-status.json to find sims with status "scaffolded"

# 4. Insert iframes into chapter markdown for newly built sims
python3 $UTILS/add-iframes-to-chapter.py \
    --all --project-dir $PROJECT --fix-heights --fix-paths

# 5. Update mkdocs.yml navigation
python3 $UTILS/update-mkdocs-nav.py --project-dir $PROJECT

# 6. Validate quality scores
python3 $UTILS/validate-sims.py --project-dir $PROJECT --output /tmp/scores.json

Summary and Results

What Was Built

• 5 Python utility scripts + 1 shared module + README.md in src/microsim-utils/
• 3 skill files updated to support machine-readable spec extraction and batch workflows
• 8 typo fixes (500xp → 500px) in content-element-types.md
• sim-status.json lifecycle tracking with 5 status levels

Design Constraints Met

• Standard library only (no pip dependencies)
• Python 3.7+ compatible
• All scripts support --project-dir, --dry-run, --verbose
• Idempotent operations
• if __name__ == '__main__': with argparse

Geometry Course Stats

• 169 specs extracted across 12 chapters
• 227 total sims tracked in sim-status.json
• 83 sims still in "specified" status (ready for scaffolding + implementation)
• 60 sims fully deployed (validated + iframe in chapter)

Token Impact

Estimated savings per batch generation run: ~430K tokens (~27% of the 1.6M baseline), enabling generation of the remaining 83 unbuilt sims in approximately 2 context windows instead of 4-5.