tlamatini-self-modify-inclusion

name

tlamatini-self-modify-inclusion

description

Sweep the whole codebase and keep the SELF-MODIFY source snapshot complete — so the TlamatiniSourceCode/ tree that a `--self-modify` build ships carries every source/build asset Tlamatini needs to read, modify, and REBUILD herself, with heavy media/secrets correctly omitted+redacted and omitted binaries listed in the restore manifest. Invoke whenever you add/rename a new agent, source file, top-level dir, dependency, build input, prompt/identity file, migration, static/template asset, or any rebuild input — and ALWAYS after a feature like Blenderer or the self-update capability lands. Owns copy_source_assets.py and audits it against five invariants with a runnable sweep that GENERATES a snapshot and proves a rebuild's inputs all survive. Sibling of tlamatini-self-update-inclusion (which guards the runnable release).

Tlamatini — Self-Modify Inclusion Sweep

Audience: Claude Code working ON the Tlamatini codebase for Angela. Goal: guarantee the source snapshot a self-able-modify build ships (TlamatiniSourceCode/, generated by copy_source_assets.py) carries everything a rebuild needs — so a running Tlamatini can read → modify → regenerate her own Tlamatini.exe without missing the minimal thing — while staying lean (media/secrets out) and safe (no leaked keys).

This is the twin of tlamatini-self-update-inclusion. That one guards the runnable release (what users download). This one guards the SOURCE tree Tlamatini carries of herself. They overlap on one fact: a rebuild from the snapshot must be able to re-run the same build.py — so every input build.py consumes must survive the snapshot.

The acid test (the whole job in one sentence)

Can cd <install>/TlamatiniSourceCode && python build.py --self-modify succeed using only the snapshot plus the documented restore steps? If yes, the snapshot is complete.

---

The file this skill owns

copy_source_assets.py (repo root) — generates the snapshot. Six knobs decide what ships:

Knob	Controls
EXCLUDED_DIR_NAMES	directory names pruned everywhere (.git, node_modules, build, dist, pools, staticfiles, Temp, Templates, agents_backup, updater, TlamatiniSourceCode, …)
EXCLUDED_EXTENSIONS	file types dropped (.pdf/.pptx/.png/.mp4/..., binaries, generated state). .ico/.wav/.svg are deliberately KEPT (build-required)
EXCLUDED_FILE_NAMES	exact files dropped (db.sqlite3, data.keys, settings.local.json, _version.py, …)
EXCLUDED_FILE_GLOBS	patterns dropped (*.version.txt, *_log.*)
RESTORE_FROM_INSTALL	binaries the build NEEDS but the snapshot OMITS → restored from the install at rebuild (jd-cli.jar, XAIHT-Tlamatini.mp4). The rebuild's escape hatch.
REQUIRED_SNAPSHOT_FILES + _wants_redaction()	the completeness floor (raises if a critical file is missing) and the secret-redaction rules (config.json + agent config.yaml)

It also writes _SOURCE_SNAPSHOT_MANIFEST.json (counts/restore list) and _REBUILD_INSTRUCTIONS.md (the restore + rebuild runbook). The walk is generic — any new .py/.js/.css/.yaml/.pmt/... ships automatically. The danger is the exclusions wrongly dropping a rebuild input, an omitted binary missing from the restore manifest, a new always-present capability file not in REQUIRED_SNAPSHOT_FILES, or a secret leaking.

---

The FIVE invariants

Invariant 1 — INCLUDE: every rebuild SOURCE input survives

Every file build.py reads to build (the build scripts, requirements.txt, all --add-data source trees, optional/required_file_copies, optional_dir_copies sources, support_files, the icon/wav/svg, every agent template, every SKILL.md, prompt.pmt, Tlamatini.md, agentic_skill.md, monitoring-prompt.pmt) must appear in the snapshot — or be in RESTORE_FROM_INSTALL. A source file dropped by an exclusion rule = a rebuild that fails or degrades.

Invariant 2 — OMIT-LEAN, but RESTORE what the build needs

Heavy media, dev trees, regenerable state, and secrets stay out (keep it ~10 MB, leak-free). But anything omitted that the build genuinely consumes must be in RESTORE_FROM_INSTALL (restored from the install) or covered by a regeneration step in _REBUILD_INSTRUCTIONS.md (e.g. collectstatic for staticfiles). Optional inputs (the images gallery) may degrade gracefully — decide per asset whether degradation is acceptable or it needs a restore entry.

Invariant 3 — REDACT: no live secret survives

config.json and every agent config.yaml are scrubbed to <KEY goes here> placeholders; data.keys is excluded outright. The snapshot must contain zero live keys.

Invariant 4 — COMPLETENESS: critical files are guaranteed, not incidental

REQUIRED_SNAPSHOT_FILES raises if a build-/run-critical source is missing. When you add a new always-present capability (a new build script, a new identity file, a flagship new agent), add its canonical file here so a future careless exclusion can't silently drop it.

Invariant 5 — REBUILD COHERENCE: the manifest + instructions are true

_SOURCE_SNAPSHOT_MANIFEST.json and _REBUILD_INSTRUCTIONS.md exist, errors == [], and the restore steps name real omitted binaries with correct install-relative sources.

---

THE SWEEP — run this every time

Step 0 — run the deterministic checker (does the heavy lifting)

python .claude/skills/tlamatini-self-modify-inclusion/scripts/sweep_self_modify.py
# add --keep to leave the generated snapshot under Temp/_self_modify_sweep for inspection

It actually generates a snapshot into Temp/ and verifies: it builds error-free with all REQUIRED_SNAPSHOT_FILES; every resolvable build.py input is carried or restorable (distinguishing source-drop FINDINGS from intentionally-omitted media/regenerable NOTES); no live secret survives (re-running copy_source_assets' own redaction logic on the OUTPUT + a config-file value scan); the restore manifest + instructions exist and name real binaries; and no source-like extension is in the exclude set. Exit code = findings (0 == clean). Fix every [FINDING]; read every [note] and confirm the degradation/regeneration is intended.

Step 1 — diff since the last release and classify new paths

git diff --name-status "$(git describe --tags --abbrev=0 --match 'v[0-9]*')"..HEAD
# brand-new TOP-LEVEL repo entries — the highest risk for a wrongly-excluded rebuild input
git diff --name-status "$(git describe --tags --abbrev=0 --match 'v[0-9]*')"..HEAD \
  | awk '$1=="A"{print $2}' | awk -F/ '{print $1}' | sort -u

For each new path ask the taxonomy question below. New .py/.js/.css/.yaml/.pmt/SKILL.md inside already-walked trees auto-ship (note, no action). The risks are: a new binary build input, a new excluded dir/ext/name that hides source, a new secret-bearing config file, or a new always-present file that belongs in REQUIRED_SNAPSHOT_FILES.

Step 2 — the judgment passes the script can't fully make

# (a) New binary build inputs the build reads (icons/fonts/wav/jar/etc.) that EXCLUDED_EXTENSIONS would drop
grep -nE "--add-data=|--icon|optional_dir_copies|required_file_copies|support_files" build.py | grep -iE "\.(ico|wav|svg|png|jpg|ttf|otf|jar|dll|bin|dat)" 
# (b) New config files that may carry secrets but AREN'T covered by _wants_redaction()
grep -rilE "api[_-]?key|token|secret|password|client_secret" Tlamatini --include=*.json --include=*.yaml --include=*.yml \
  | grep -viE "/agents/.*/config.yaml$|agent/config.json$"
# (c) New dependency / runtime added (cross-check the self-update skill too)
git diff "$(git describe --tags --abbrev=0)"..HEAD -- requirements.txt

• (a) Any binary the build needs and the snapshot drops → add to RESTORE_FROM_INSTALL (omit + restore) or to the KEEP exceptions (if small + truly source-required, like the .ico/.wav/.svg already are).
• (b) Any NEW config file that can hold a live key → extend _wants_redaction() to scrub it (and confirm Step 0's redaction check then covers it), or exclude it by name.
• (c) A new dep doesn't affect the snapshot directly (it ships requirements.txt), but the rebuild pip installs it — make sure requirements.txt is current (the self-UPDATE skill owns the bundled side).

Step 3 — fix every finding, re-run Step 0 until clean, eyeball the notes.

---

Asset taxonomy — type → ship in snapshot? → action

New asset	In snapshot by default?	Action
New .py/.js/.css/.html/.yaml/.pmt/.proto/SKILL.md (source)	✅ auto (generic walk)	none
New agent (agent/agents/<x>/ source + config.yaml)	✅ auto; config.yaml auto-redacted	none (verify redaction in Step 0)
New migration (.py)	✅ auto	none
New build script / build input file at repo root	✅ if not excluded	add to REQUIRED_SNAPSHOT_FILES if always-present
New small build-required binary (.ico/.wav/.svg-class)	✅ (those exts KEPT)	if a NEW binary ext, KEEP it (remove from excludes) or restore-map it
New heavy binary the build needs (.jar/.mp4/large media)	❌ omitted	add to RESTORE_FROM_INSTALL (+ it lands in _REBUILD_INSTRUCTIONS.md)
New optional media (gallery image)	❌ omitted	leave omitted if degradation OK; else restore-map
New config file with secrets	✅ shipped — ⚠️ maybe UNREDACTED	extend _wants_redaction() or exclude by name
New runtime-state / generated dir	should be ❌	add its name to EXCLUDED_DIR_NAMES
New flagship capability file (e.g. self_update.py, Tlamatini.md)	✅ auto	add to REQUIRED_SNAPSHOT_FILES so it's guaranteed

The single most dangerous omissions: a source file hidden by an over-broad exclusion (Step 0 flags it as a source-drop FINDING) and a new secret-bearing config that _wants_redaction() doesn't cover (Step 2b — the raw scan in Step 0 catches obvious key shapes, but a novel config path needs the redaction rule extended).

---

Where to make each fix (copy_source_assets.py)

• A source file got excluded → narrow the offending rule in EXCLUDED_EXTENSIONS / EXCLUDED_FILE_NAMES / EXCLUDED_FILE_GLOBS / EXCLUDED_DIR_NAMES, or add a KEEP carve-out.
• An omitted binary the build needs → add "snapshot/rel/path": "install/rel/source" to RESTORE_FROM_INSTALL (it auto-renders into _REBUILD_INSTRUCTIONS.md).
• Guarantee a critical file → add its snapshot-relative path to REQUIRED_SNAPSHOT_FILES.
• A new secret-bearing config → extend _wants_redaction() to return 'json'/'yaml' for it (and add the key suffix to _SECRET_KEY_RE if it's a novel secret-key name).
• A new state/output dir leaking in → add its name to EXCLUDED_DIR_NAMES.

---

Done criteria (all must hold)

1. sweep_self_modify.py exits clean (no [FINDING]); every [note] is reviewed + intended.
2. Every new top-level repo path from the since-last-tag diff is classified and handled.
3. The generated snapshot builds error-free and contains every REQUIRED_SNAPSHOT_FILES entry.
4. Every build.py SOURCE input is in the snapshot or in RESTORE_FROM_INSTALL.
5. Zero live secrets in the snapshot (redaction check + value scan pass).
6. The manifest + _REBUILD_INSTRUCTIONS.md are accurate; restore targets are real.
7. python -m ruff check copy_source_assets.py is clean.
8. (Ideal) you eyeballed a --keep snapshot and spot-confirmed the newest feature's source files are physically present.

---

Companion references

• tlamatini-self-update-inclusion — the sibling skill guarding the runnable release (build.py carriers + the update swap). Run BOTH after any feature: one proves users can receive the feature, this one proves Tlamatini can rebuild herself with it.
• docs/claude/architecture.md → Self-Knowledge & Self-Modification and docs/claude/gotchas.md → Building & Packaging for the --self-modify flag mechanics.
• VERSIONING.md — the rebuild produces the next version from the git tag.