Tlamatini Self Update Inclusion

Tlamatini — Self-Update Inclusion Sweep

Audience: Claude Code working ON the Tlamatini codebase for Angela. Goal: guarantee that every asset a new feature introduces is (1) carried into the release by build.py, and (2) handled correctly by the self-update swap — either preserved (user data) or replaced (app code). Nothing minimal forgotten, ever.

This skill is the safety net for shipping. A feature can be 100% correct in source and still be invisible to every existing user after they click About ▸ Check for updates — because the asset was never bundled, or because the swap deleted/kept the wrong thing. This sweep makes that class of bug impossible to ship silently.

Pipeline files this skill owns

apply_update.ps1 must itself be shipped by build.py (required_file_copies) so a self-updated install carries the next updater. The pipeline is self-hosting — this is invariant #1's most easily-forgotten case.

How the release is assembled — the 7 carrier mechanisms in build.py

Every runtime asset needs at least one of these carriers (some deliberately have two destinations). When you add an asset, ask "which carrier moves it?" If the answer is "none", it will NOT ship.

The forgettable carriers are 2 (new top-level tree), 4, 5, 6, and 7 — anything that lives at the repo root or needs an explicit PyInstaller flag or its own copy block. Mechanisms 1 and 3 are automatic, which is exactly why a new agent needs no build edit but a new root-level script does.

security/ is mandatory. Missing source/copy failures abort the build, and build_runtime_assets.py inventories the tree (excluding evidence/logs) and checks every carried file. Register new runtime trees in SOURCE_TREES as well as their actual copy/add-data carrier; optional copying is unsuitable for required code.

How an update swaps — preserve vs replace

apply_update.ps1 does: validate staged build and backup helpers → stop running app → verify a WAL-aware SQLite backup → agents → agents_backup → delete old install except $Preserve → move in new build except $Preserve → relaunch and migrate.

So every top-level entry is in one of three buckets:

• PRESERVED ($Preserve) — user data / runtime state. Kept across updates. Must equal the shared preserved_user_state.json contract, including runtime-writable dirs.
• SWAP-BACKED — agents only (renamed to agents_backup, then replaced). One backup kept.
• REPLACED — everything else (the exe, python/jre/git, .ps1/.ico, prompt.pmt, Tlamatini.md, README.md, agents_descriptions.md, images, skills_pkg, db.sqlite3).

The FOUR invariants (this is the whole job)

Invariant 1 — CARRY: every new asset reaches the release

For each asset a feature adds, a carrier (table above) moves it into dist/manage/pkg.zip. The high-risk cases: a new repo-root file (→ support_files or *_file_copies), a new top-level tree (→ --add-data or optional_dir_copies), a new dependency (→ requirements.txt + maybe hidden-import / --collect-all), a new bundled runtime.

Invariant 2 — PRESERVE PARITY: the two preserve lists are identical

apply_update.ps1 $Preserve (executed) == self_update.py docstring "Preserved across the swap" list (documented). A drift here means the docs lie about what survives.

Invariant 3 — PRESERVE CORRECTNESS: state preserved, code replaced

$Preserve == the names in preserved_user_state.json == the installer fallback == the updater docstring list (case-insensitive sets). The top-level names from build.py::empty_dirs must be a subset; catalogs, contacts and the separately built uninstaller are also legitimate preserved entries. Add new state to all consumers. uninstall.py intentionally has a different removal policy. Conversely an app-code top-level entry (anything under optional_dir_copies like images/skills_pkg, or python/jre/git) must NOT be preserved, or users get stuck on stale code forever.

Invariant 4 — DB DELIVERY: new migration rows actually reach users

The live database sits inside replaced _internal/, but its data is preserved through DB/ToLoad. After stopping the app, carried Python runs the shipped sqlite_copy.py helper to produce a verified online backup including committed WAL pages. Only success permits the swap and creates post_update_migrate.flag. The next startup restores that database and applies new migrations. Never replace this with a plain copy of db.sqlite3, and never preserve the old DB without the first-launch migration path.

THE SWEEP — run this every time

Step 0 — run the deterministic checker (does 90% of the work)

python .claude/skills/tlamatini-self-update-inclusion/scripts/sweep_self_update.py

It parses the three files and reports [PASS] / [FINDING] for invariants 2, 3, the root-.ps1 census (invariant 1's worst case), app-dir-must-not-be-preserved (invariant 3), and a migrations-since-last-tag count (invariant 4). Exit code is non-zero if any finding — so it's usable as a pre-release gate. Fix every [FINDING] before shipping.

Step 1 — diff since the last release and classify every new path

# what changed since the last shipped tag
git diff --name-status "$(git describe --tags --abbrev=0 --match 'v[0-9]*')"..HEAD
# any brand-new TOP-LEVEL repo entries (the highest-risk for "forgot to ship")
git diff --name-status "$(git describe --tags --abbrev=0 --match 'v[0-9]*')"..HEAD \
  | awk '$1=="A"{print $2}' | awk -F/ '{print $1}' | sort -u

For each new top-level file or dir, run it through the asset taxonomy below and confirm its carrier is wired. New nested files inside agent/static, agent/templates, agent/agents/<x>, agent/skills_pkg, or any .py in a package are auto-carried — note them but they need no edit.

Step 2 — the carrier checks the script can't fully judge (do by eye)

Run these greps and reconcile each hit against build.py:

# (a) New third-party imports in pool agents / app → requirements.txt + maybe hidden-imports/collect-all
grep -rnE "^\s*(import|from)\s+([a-z0-9_]+)" Tlamatini/agent/agents --include=*.py \
  | grep -ivE "import (os|sys|json|re|time|subprocess|socket|threading|pathlib|typing|shutil|logging|urllib|zipfile|tarfile|wave|base64|struct|math|datetime|tempfile|argparse|queue|signal|ctypes|glob|io|collections|functools|itertools)\b"
grep -nE "hiddenimports|--hidden-import|--collect-all|--collect-submodules" build.py
# (b) Any external EXE/runtime a new agent shells out to (like jre/git) → must be bundled
grep -rnE "subprocess|Popen|shutil.which|\.exe\b" Tlamatini/agent/agents --include=*.py | grep -iE "\.exe|which\(" | head
# (c) New requirements vs what build pins
grep -nE "_agent_libs|AGENT_DEP|pip install|requirements" build.py | head

If a new agent imports a new library, it must be in requirements.txt and importable by the carried Python (see bundle_carried_python / the _agent_libs verify list) — otherwise the frozen pool agent crashes at runtime even though the source is correct.

Step 3 — confirm the asset actually lands in pkg.zip (ground truth)

The only 100%-sure check is to look at a real bundle. If a recent dist/manage or pkg.zip exists, list it; otherwise note that a build is required to verify physically:

[ -f pkg.zip ] && python - <<'PY'
import zipfile
names = zipfile.ZipFile("pkg.zip").namelist()
for probe in ("apply_update.ps1","db.sqlite3","agents/blenderer/blenderer.py","Tlamatini.exe"):
    print(("OK  " if any(probe in n for n in names) else "MISS"), probe)
PY

If no bundle exists, do NOT claim it ships — say "verified in source/wiring; physical bundle check needs a python build.py run."

Step 4 — fix every finding, then re-run Step 0 until clean.

Asset taxonomy — type → carrier → preserve?

The single most dangerous omission is the last-but-one row: a new runtime-state dir added to empty_dirs but not to $Preserve + the self_update.py docstring → wiped on every update. The sweep script flags exactly this.

Where to make each fix

• Carry a root file → add to support_files (scripts/icons) or required_file_copies (data) in build.py, with a one-line comment on why it must be next to the exe.
• Carry a new tree → add an --add-data line (if read from the bundle) or an optional_dir_copies entry (if read from the install root).
• Carry a dep → requirements.txt; if PyInstaller can't see it, add a hidden-import or --collect-all; if a pool agent imports it, confirm the carried Python has it (_agent_libs).
• Preserve a new state dir → add the top-level name to apply_update.ps1 $Preserve and shared JSON, installer fallback and self_update.py docs (keep them identical), and ship it empty via build.py empty_dirs.
• Stop preserving stale app code → remove it from $Preserve (+ docstring).
• DB → preserve committed data with verified SQLite online backup into DB/ToLoad, then restore it and run first-launch migrate. Keep db.sqlite3 out of the top-level preserve set because it lives inside replaced _internal/.

Done criteria (all must hold)

PDF and database-helper carrier gate (2026-09-16)

The PDF backend and Image-Interpreter engine run inside the frozen web process. Keep their explicit hidden imports / archive requirements and pyinstaller_hooks/hook-pymupdf.py; the separate carried Python cannot satisfy imports inside Tlamatini.exe. PDF.js assets are carried by the static trees, and context_files/pdf_canvas/ survives through the existing preserved context_files directory.

apply_update.ps1, preserved_user_state.json and standalone sqlite_copy.py are REQUIRED root assets, not warning-only support copies. The in-app updater prefers the incoming swap script so new preservation rules apply immediately. The swapper checks the backup helper/runtime before shutdown, requires verified online backup before changing application files, and uses carried Python with -I rather than a machine-installed interpreter. Keep helper source in the self-modify snapshot and agent.sqlite_copy in the frozen archive too.

Frontend carrier gate (v1.48.13)

Because agent/static is tree-carried, dialog_theme.css, dialog_policy.js, and release_notes_renderer.js ship automatically only if they remain inside that tree and are referenced by both source templates and collected-static output. Verify template load order and the startup-derived static cache stamp after JavaScript/CSS/template changes. Run tests only when the user's instructions permit them. Confirm the updater still uses the shared renderer/policy and that long-operation locks restore data-bs-toggle.

External-MCP runtime/default carrier gate (v1.48.14 target)

runtime_provisioner.py and external_mcp_defaults.py are application code and must ship through the normal frozen/source carriers. The downloaded private runtime and persistent Memory graph live under %LOCALAPPDATA%\Tlamatini, outside the install swap, so they must not be added to installer payloads, empty_dirs, or $Preserve. external_mcps.json is different: it is preserved user state and a gitignored local catalog, replaced with maintained inactive defaults for public builds. Verify public build.py output contains only inactive memory + sequential-thinking, keyed/private builds take the explicit private path, regen_secrets.py handles catalog env secrets, and the public live-secret gate aborts unsafe output.

v1.48.17 updater/parser and public-private build gate

The staged swap must retain Uninstaller.exe; Windows comments that explain this policy stay on standalone PowerShell lines so parser-sensitive continuations remain valid. Verify agent/test_preserved_user_state.py source-derives the code-seeded default catalog, proves the public builder clears TLAMATINI_BUNDLE_EXTERNAL_MCPS, and proves only the explicit private builder supplies it. Do not reintroduce an assertion that the environment variable is absent from build.py: the variable is intentionally read by the shared builder, while release entry points control whether it exists.

v1.50.0 security-toolkit carrier + evidence-carryover gate

security/ is the first asset to use carrier mechanism 7 (a bespoke copytree, not a list), and the first to need a third bucket beyond preserve/replace. Verify all three halves:

1. CARRIED — build.py's security block copies security/ to the install root, skipping security_logs / *.log / __pycache__. It fails CLOSED, with per-file receipt validation as well as the Step-0 source census.
2. REPLACED, deliberately — 'security' is NOT in apply_update.ps1 $Preserve and must stay out. It is application code: a corrected defender has to be able to reach a user running a broken one. Do not "protect" it by preserving it — that would freeze the toolkit forever.
3. EVIDENCE CARRIED OVER — security/security_logs/ (alerts.log, monitor.log, the visible asset-test proof) is the operator's forensic evidence living inside that replaced tree, the same shape of problem as db.sqlite3. apply_update.ps1 step 3c stashes it to a unique Temp/_security_logs_carryover_<id> before deletion and step 5b restores it. A failed stash ABORTS before deletion; failed restore LEAVES the stash, never overwriting other evidence. self_update.py's docstring mirrors this. Never "fix" a failed restore by dropping the stash.

Also verify security_logs remains in SKIP_DIRS in both build_complete_public_release.py and check_private_data.py (kept mirrored): the release scrubber must not rewrite forensic artifacts, and the private-data scanner must not drown in the operator's own usernames / IPs.

v1.50.0 release carrier gate

Verify agent/agents/netspeed_calculator/, migrations 0195-0197, its wrapped-tool wiring, prompt harness, and CSS/JS connector assets ship through the normal agents/static/migrations carriers. Verify agent/sqlite_copy.py is present in the frozen application and both DB menu views plus pre-Django swap import it. Verify Googler's updated googler.py, config.yaml, test_googler_dorks.py, and optional visible dork-hunt harness are carried together so neither the structured builder nor the two-tier plain-HTTP-first/browser-fallback resilience path can ship without its syntax/preset/retry/fallback contract. Verify the four HTTP routes, seven browser routes, explicit-engine Tier-0 bypass, tolerant booleans, and answer-route attribution together. Verify skills_pkg/adding_external_mcp/ and all references ship with the skill tree. Private contact synchronization may create gitignored contacts.private.json for an explicit keyed build, but public output and TlamatiniSourceCode/ must remain contact-empty; preserved runtime user state remains contacts.json, not the private build staging file. Resolve the current annotated release and HEAD using Git; do not reuse historical version/count claims.

1. sweep_self_update.py exits clean (no [FINDING]).
2. Every new top-level repo path from the since-last-tag diff has a wired carrier.
3. Shared JSON, swapper, installer fallback and updater docs agree; all empty_dirs top-level names are covered.
4. No app-code dir is preserved; no runtime-state dir is left unpreserved.
5. New deps are in requirements.txt and importable by the carried Python.
6. The DB story is coherent (replaced + honest docs, OR preserved + first-run migrate).
7. If a physical bundle was available, the probe in Step 3 shows the new assets OK; otherwise you stated a build is needed to physically confirm.
8. python -m ruff check clean on any edited .py; apply_update.ps1 still parses.

Integrity and release-mode gate (2026-09-16)

• Read build_runtime_assets.py, preserved_user_state.json, install.py and both complete-release wrappers as well as the three primary pipeline files.
• The checker parses required_file_copies with AST and support_files; a helper moved from one to the other is not a missing-carrier finding.
• Carry build_runtime_assets.py at the install root AND in the frozen module archive. The installer imports the same checker. runtime-assets.json records every file's size/hash, exact membership, version and boolean self-modify mode.
• Validate ZIP paths, membership and streamed hashes before publication, installer extraction and updater staging. Recheck staged bytes before shutdown. Reject legacy packages without receipts; rebuild them. A receipt is NOT a signature and cannot establish publisher authenticity.
• --self-modify requires both identity locations and the complete snapshot; default/--no-self-modify requires neither. Missing pkg.zip aborts wrappers.
• The PowerShell swapper requires the exact install-local Temp/_update/staging boundary, rejects reparse ancestors, and never falls back to a self-killing taskkill /T. Existing user state wins; absent state receives new defaults.
• Reinstallation retains the live _internal/db.sqlite3 and WAL companions and requests first-launch migration; preserving only top-level DB/ is insufficient.
• Retain the 1,990,000,000-byte final outer ZIP ceiling. Never prune required assets to meet it. Local frontend and PDF.js bytes stay pinned and CDN-free.
• Respect a no-tests request: these sweeps inspect/copy files only. Do not launch Django, rebuild, migrate, install, swap, or execute application test suites. Report source/snapshot evidence separately from a freshly built runtime.

Companion references

• copy_source_assets.py (repo root) — the self-modify snapshot generator; its REQUIRED_SNAPSHOT_FILES completeness check is a sibling guarantee (that the source tree ships), distinct from this skill (that the runnable release ships + survives an update).
• VERSIONING.md — the git-tag version contract (a self-update compares tags via self_update.is_newer).
• docs/claude/architecture.md → Self-Knowledge & Self-Modification for the build flags.

Central model settings runtime gate (2026-09-20)

The model registry must ship as compiled agent.agents.model_settings and as portable agents/model_settings.py. Frozen services import compiled code and resolve portable data with get_agents_root(), never a synthetic __file__ source path inside _internal. Require check_agent_runtimes in the frozen archive and self-modify snapshot. The build executes this command before ZIP creation: every template must prepare, reusable helpers and planner catalogs must refresh, actual model loaders must resolve saved settings, and File-Creator must write exact scratch bytes with the selected runtime Python. File-only inclusion sweeps are not a substitute for this execution check.

The complete operational contract is docs/model_configuration.md; dated executed checks are in docs/model-configuration-verification.md. Preserve saved global values and explicit agent overrides during updates. Absent new keys use registry fallbacks/defaults; existing choices must not be silently replaced. Document model key changes, empty-value semantics, next-load/reconnect behavior and helper carriage in README, BookOfTlamatini, self-knowledge, prompts and agent/MCP creation guides. UTF-8/BOM loader checks must include Windows default-codepage execution. The current gate checks 89 templates and 21 model loaders; derive future counts from source.