lumi-migrate-legacy

name	lumi-migrate-legacy
description	Backfill missing schema fields (provenance, confidence) on legacy wiki entries after a Lumina version upgrade. Use proactively when /lumi-check reports L01/L11 findings on multiple entries, when manifest shows legacyMigrationNeeded:true, or when user mentions "lint do", "upgrade", "missing provenance", or "schema gap" after running install.
allowed-tools	["Bash","Read"]

/lumi-migrate-legacy

Read README.md at the project root before this SKILL.md.

Role

You are the wiki's schema migration agent. After a Lumina version bump that introduces new required or optional frontmatter fields, existing entries may be missing those fields. Your job is to read the CHANGELOG, find the exact fields each version introduced, and backfill them with inferred values — not blind defaults. Every field you set should reflect what you actually know about the entry.

This skill is CHANGELOG-driven, not hardcoded to any specific field set. The CHANGELOG's ### Migration sections are the authoritative source of truth for what needs backfilling. This means the skill works for v0.8, v0.9, and future versions without modification, as long as the CHANGELOG is maintained.

Context

Key workspace paths:

• _lumina/manifest.json — packageVersion, legacyMigrationNeeded
• _lumina/CHANGELOG.md — ### Migration sections per version (the spec)
• _lumina/scripts/lint.mjs — runs checks; L01 = missing required field (error), L11 = missing confidence (warning)
• _lumina/scripts/wiki.mjs — read-meta, set-meta, list-entities, log
• wiki/graph/edges.jsonl — citation/edge counts for confidence inference
• raw/ — source snapshots for provenance inference

Instructions

Phase 1 — Detect

Step 1.1 — Read manifest.

node -e "const m=JSON.parse(require('fs').readFileSync('_lumina/manifest.json','utf8')); console.log(JSON.stringify({packageVersion:m.packageVersion,legacyMigrationNeeded:m.legacyMigrationNeeded},null,2))"

Note packageVersion and legacyMigrationNeeded. If legacyMigrationNeeded is false or absent, migration may still be needed — the flag is advisory. Proceed to the lint check regardless.

Step 1.2 — Run lint (read-only pass).

First, get aggregate counts (tiny output, always safe):

node _lumina/scripts/lint.mjs --summary

If errors === 0 and by_check.L11 is 0 or absent, skip to the clean-exit branch below. Otherwise, you need the per-entry findings.

Important — do NOT pipe --json straight into a heredoc. On a large wiki the full findings JSON can exceed the shell tool's ~30KB stdout buffer and get truncated mid-string, breaking JSON.parse. Instead, write it to a temp file and read filtered slices:

node _lumina/scripts/lint.mjs --json > /tmp/lumi-lint.json
node -e "
const j=JSON.parse(require('fs').readFileSync('/tmp/lumi-lint.json','utf8'));
const want=new Set(['L01-frontmatter-required','L11-confidence-missing']);
const hits=j.findings.filter(f=>want.has(f.id))
  .map(f=>({id:f.id,file:f.file,message:f.message}));
console.log(JSON.stringify(hits,null,2));
"

The projected output (id + file + message only) is bounded and parseable. If even that exceeds buffer (very large wikis), read /tmp/lumi-lint.json with the Read tool instead — Read paginates, Bash stdout does not.

Collect:

• All L01-frontmatter-required findings (severity: error) — entries with missing required fields.
• All L11-confidence-missing findings (severity: warning) — entries missing the optional-but-recommended confidence field.

If summary.errors === 0 and there are no L11 warnings relevant to migration:

No migration work needed. Lint is clean.

Log and exit:

node _lumina/scripts/wiki.mjs log migrate-legacy "No migration needed — lint clean."

Step 1.3 — Read CHANGELOG migration notes.

cat _lumina/CHANGELOG.md

Locate every ### Migration section in versions between the oldest affected entry's install version and the current packageVersion. These sections describe exactly which fields were added and for which entity types. Read them carefully — they are the specification for what you will backfill.

If no ### Migration section exists but L01/L11 findings are present, the fields are listed in the findings themselves (the message field names them). Use the finding messages as the migration spec.

Step 1.4 — Build work list.

Produce a table of all affected slugs, which field is missing, and what entity type they are. Group by field for efficient processing:

Field: provenance (required, sources)
  - sources/attention-is-all-you-need
  - sources/lora-2021

Field: confidence (optional, sources + concepts)
  - sources/attention-is-all-you-need
  - concepts/softmax-temperature

Always report this plan to the user before proceeding. For work lists of 30 or fewer entries, continue without waiting for confirmation — small batches are routine and the operation is safe to re-run. For more than 30 entries, stop and ask the user to confirm before any writes. A large batch usually means a long-dormant wiki or a major schema bump, and the user should have a chance to spot-check the inference table before bulk changes land.

The safety net beneath this threshold:

• set-meta is atomic and idempotent — rerunning with a corrected value is a single command, no rollback needed.
• The inference rubric falls back to unverified when evidence is ambiguous, so wrong values err toward "honest about uncertainty," not overconfidence.
• Phase 4 re-runs lint and surfaces any remaining issues before clearing the manifest flag.

Phase 2 — Plan

For each affected slug, determine the correct inferred value before writing anything. Do not set any values yet — this phase is read-only.

For each slug, run:

node _lumina/scripts/wiki.mjs read-meta <slug>

This returns the current frontmatter as JSON. Read it to understand the entry's existing fields (url, authors, year, type, etc.).

For sources entries also check:

1. Inbound citation/edge count (how many other entries link to this one):

   grep -c '"target":"sources/<slug>"' wiki/graph/edges.jsonl 2>/dev/null || echo 0
   grep -c '"target":"sources/<slug>"' wiki/graph/citations.jsonl 2>/dev/null || echo 0
   

Inference rubrics — apply these to decide values:

provenance + raw_paths (required on sources)

Use the following inference order. Stop at the first tier that yields a result.

Tier 1 (authoritative): read the ingest checkpoint.

Ingest checkpoints are keyed by the source file's basename (_lumina/_state/ingest-<file-basename>.json), not by slug — there is no direct <slug> lookup. List every ingest checkpoint and match by content:

ls _lumina/_state/ingest-*.json 2>/dev/null

Read each match with the Read tool and check its slug field. Newer ingests (post-checkpoint-slug-merge) store "slug": "sources/<slug>" directly in the checkpoint — match the checkpoint whose slug equals the entry being migrated. Older checkpoints predate this field and won't have it; if no checkpoint's slug matches (including the case where none carry the field at all), fall through to Tier 2.

If a matching checkpoint is found and has a source_path field:

• If source_path is under raw/tmp/*: do NOT write raw_paths. Tell the user: "<slug> was ingested from a transient location (<source_path>). Move the file to raw/sources/ or raw/download/<resource>/ and re-run /lumi-migrate-legacy to backfill raw_paths properly." Set provenance to partial (if urls is non-empty) or missing (no urls).
• Otherwise: set raw_paths to [source_path] and provenance to replayable. Skip Tiers 2 and 3.

Tier 2 (heuristic): scan raw/ for matching files.

• Slug-prefix match: raw/sources/<slug>*, raw/notes/<slug>*, or raw/download/<resource>/<slug>*
• URL-derived ID match: parse the page's urls array via node _lumina/scripts/parse-ids.mjs "<url>" (one URL per call). For each validated value in the JSON output (e.g. arxiv, doi, s2), use that literal string as the ID token to scan raw/sources/, raw/notes/, and raw/download/** for filenames containing it. Do not interpolate a raw URL or path-traversal-shaped value into a glob — parse-ids.mjs already rejected non-URL inputs and wiki.mjs re-validates via safeIdToken before any path concatenation. (Legacy url string still supported for back-compat.)
• Research-pack flow: also scan raw/discovered/<topic>/<id>.json for a JSON whose id or url matches any entry in the page's urls array.

All non-raw/tmp/ matches go into raw_paths. Set provenance to replayable if any match was found.

Tier 3 (fall back to urls heuristic): no checkpoint, no file match.

• Has at least one entry in urls, no raw match → partial (leave raw_paths unset or [])
• Neither → missing

confidence (optional-but-recommended on sources and concepts)

Pick based on inbound evidence signals:

• high — Cited by 3 or more other entries (inbound edges + citations), OR the entry has multiple independent summaries or cross-references. Well-verified.
• medium — Cited by 1-2 other entries. Some corroboration exists.
• low — No inbound edges, unverified claims, or content was hand-entered with no cross-checks. Use this when you have reason to doubt accuracy.
• unverified — Default for legacy entries with no signal. Use this when you cannot determine a better value from the available evidence. This is the safe fallback — it is more honest than low when the issue is lack of evidence rather than evidence of unreliability.

Do not guess. If evidence is ambiguous, choose unverified over a higher value. Bumping confidence up is easier than correcting overconfident legacy data.

For future fields not listed above: read the ### Migration section in CHANGELOG carefully. It will specify the field, its allowed values, and how to infer the right value. Apply the same pattern: read available evidence, apply the rubric, choose conservatively when uncertain.

After the read phase, produce an inference table:

sources/attention-is-all-you-need:
  raw_paths: ["raw/sources/attention-is-all-you-need.pdf"]  (Tier 1: checkpoint source_path)
  provenance: replayable  (raw_paths non-empty, file exists)
  confidence: high        (7 inbound citations)

sources/lora-2021:
  raw_paths: []           (Tier 3: url present, no file match)
  provenance: partial     (url present, no resolvable raw_paths)
  confidence: unverified  (0 inbound edges, no cross-checks)

concepts/softmax-temperature:
  confidence: medium      (2 inbound edges)

Phase 3 — Backfill

For each entry in the inference table, set each missing field:

node _lumina/scripts/wiki.mjs set-meta <slug> <key> "<value>"

For raw_paths (an array field), pass a JSON array with --json-value:

node _lumina/scripts/wiki.mjs set-meta sources/<slug> raw_paths '["raw/sources/foo.pdf"]' --json-value

Examples:

node _lumina/scripts/wiki.mjs set-meta sources/attention-is-all-you-need raw_paths '["raw/sources/attention-is-all-you-need.pdf"]' --json-value
node _lumina/scripts/wiki.mjs set-meta sources/attention-is-all-you-need provenance replayable
node _lumina/scripts/wiki.mjs set-meta sources/attention-is-all-you-need confidence high
node _lumina/scripts/wiki.mjs set-meta sources/lora-2021 provenance partial
node _lumina/scripts/wiki.mjs set-meta sources/lora-2021 confidence unverified
node _lumina/scripts/wiki.mjs set-meta concepts/softmax-temperature confidence medium

set-meta is atomic (temp + fsync + rename) and idempotent — calling it twice with the same value is a no-op. It is safe to re-run this phase.

Schema-shape upgrade — url → urls (v0.9+):

For every source page that has a top-level url: key (singular string) in frontmatter, rewrite it as urls: (array) and remove the old key. Preserve placement — keep urls where url was.

# Detect source pages that still have legacy url: (singular)
node _lumina/scripts/wiki.mjs list-entities | node -e "
const lines=require('fs').readFileSync('/dev/stdin','utf8').trim().split('\n');
const ents=lines.map(l=>{ try{return JSON.parse(l);}catch{return null;} }).filter(Boolean);
ents.filter(e=>e.type==='sources').forEach(e=>console.log(e.slug));
" | while read slug; do
  node _lumina/scripts/wiki.mjs read-meta "$slug" | node -e "
const m=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));
if(m.url && !m.urls) console.log(process.argv[1]);
" "$slug"
done

For each slug with a legacy url: field:

# Step 1 — read current url value
URL=$(node _lumina/scripts/wiki.mjs read-meta sources/<slug> | node -e "process.stdout.write(JSON.parse(require('fs').readFileSync('/dev/stdin','utf8')).url)")

# Step 2 — write urls array
node _lumina/scripts/wiki.mjs set-meta sources/<slug> urls "[\"$URL\"]" --json-value

set-meta cannot remove a frontmatter key today — there is no --remove flag. After confirming urls: was written successfully, remove the legacy url: line with the Edit tool directly. This is the one sanctioned exception to "never hand-edit wiki frontmatter" in this skill, and it is limited strictly to deleting the obsolete url: key — do not use Edit for any other frontmatter change.

After backfilling all entries, proceed immediately to Phase 4.

Phase 3.5 — --backfill-ids (opt-in)

If the user invoked the skill with --backfill-ids, run the external_ids backfill recipe documented in references/backfill-ids.md. It is non-destructive (existing keys win) and idempotent (no diff on second run). Mismatches between URL-derived values and stored external_ids are intentionally left unchanged — lint check L16 surfaces them on the next /lumi-check. There is no --dry-run; inspect the merge with git diff wiki/sources/.

Skip this phase entirely if --backfill-ids was NOT passed.

Phase 4 — Verify

Step 4.1 — Re-run lint.

node _lumina/scripts/lint.mjs --summary

Confirm errors === 0. If you need to inspect remaining findings, re-run with --json > /tmp/lumi-lint.json and project as in Step 1.2 — never parse full --json from inline stdout on a large wiki. L11 warnings for entries you set confidence on should also be gone.

Check for L12 warnings explicitly and surface them to the user:

node -e "
const j=JSON.parse(require('fs').readFileSync('/tmp/lumi-lint.json','utf8'));
const l12=j.findings.filter(f=>f.id==='L12-raw-paths-drift')
  .map(f=>({file:f.file,message:f.message}));
if(l12.length) console.log('L12 raw_paths drift:\n'+JSON.stringify(l12,null,2));
else console.log('No L12 warnings.');
"

L12 warnings mean one or more raw_paths entries point to files that do not exist or are under raw/tmp/. Treat these as follow-up action items for the user — the migration is not blocked, but the raw_paths value is inaccurate until the referenced file is located or the entry is corrected.

If any L01 errors remain:

• Read the finding message — it names the exact field still missing.
• Return to Phase 2 and infer a value for that field.
• Apply via set-meta and re-run lint.
• Do not loop more than 3 times — if errors persist after 3 attempts, surface them to the user with the exact finding messages.

Step 4.2 — Clear the manifest flag.

node -e "
const fs = require('fs');
const path = '_lumina/manifest.json';
const m = JSON.parse(fs.readFileSync(path, 'utf8'));
m.legacyMigrationNeeded = false;
const tmp = path + '.tmp';
fs.writeFileSync(tmp, JSON.stringify(m, null, 2) + '\n', 'utf8');
fs.renameSync(tmp, path);
console.log('legacyMigrationNeeded cleared');
"

Only run this step if all L01 errors are resolved.

Step 4.3 — Log the migration.

node _lumina/scripts/wiki.mjs log migrate-legacy "Backfilled <N> entries: <field-list>. Lint: 0 errors."

Replace <N> with the count of entries updated and <field-list> with the field names backfilled (e.g., provenance, confidence).

Output Format

Report to the user:

1. Migration spec source — which CHANGELOG versions / finding messages drove the work list.
2. Entries updated — count and slugs grouped by field.
3. Inferred values — the inference table from Phase 2 (so the user can review).
4. Lint result after backfill — must show 0 errors.
5. Whether the upgrade cleanup is finished.

Examples

User: "/lumi-migrate-legacy"

Clean wiki — no migration needed:

node _lumina/scripts/lint.mjs --json
# → { "summary": { "errors": 0, "warnings": 0 } }

Report: "No migration needed — lint is clean. Nothing changed." Log entry written. Done.

User: "/lumi-migrate-legacy" (after upgrading to a version that added provenance)

Normal migration path:

node _lumina/scripts/lint.mjs --json
# → 4 L01 errors: sources/* missing provenance

# Phase 2 — for each source:
node _lumina/scripts/wiki.mjs read-meta sources/attention-is-all-you-need
# → { urls: ["https://arxiv.org/abs/1706.03762"], ... }
ls raw/sources/attention-is-all-you-need*
# → raw/sources/attention-is-all-you-need.pdf (found)
# → infer: provenance = replayable

# Phase 3:
node _lumina/scripts/wiki.mjs set-meta sources/attention-is-all-you-need provenance replayable
# ... repeat for all 4 entries ...

# Phase 4:
node _lumina/scripts/lint.mjs --json
# → { "summary": { "errors": 0, "warnings": 2 } }  -- L11 warnings remain (advisory only)
# Clear manifest flag, write log.

Report: "4 entries backfilled (provenance). Lint: 0 errors, 2 advisory warnings."

User: "/lumi-migrate-legacy" (re-run on already-migrated wiki)

Idempotency — all fields already present:

node _lumina/scripts/lint.mjs --json
# → 0 L01 errors, 0 L11 warnings

Report: "No migration needed — lint is clean. Nothing changed." Re-running this skill on a clean wiki produces zero file changes.

Guardrails

• Never write a value you cannot infer from available evidence. When in doubt, use unverified (for confidence) or read the CHANGELOG rubric for the field.
• Never modify files in raw/. Read-only.
• Never hand-edit wiki/graph/edges.jsonl or wiki/graph/citations.jsonl.
• set-meta is the only permitted write path for frontmatter changes in this skill, with one sanctioned exception: deleting the obsolete url: key during the url → urls schema-shape upgrade (Phase 3), since set-meta has no key-removal flag. Do not use Edit or Write on wiki pages for anything else.
• Do not clear legacyMigrationNeeded until lint confirms 0 errors.
• If the CHANGELOG has no ### Migration section for the detected version gap, rely entirely on the L01/L11 finding messages to identify which fields need backfilling. Do not fabricate a migration spec.

Definition of Done

Before reporting done, verify:

(a) node _lumina/scripts/lint.mjs --json shows summary.errors === 0 (b) wiki/log.md has a new ## [YYYY-MM-DD] migrate-legacy | ... entry (c) _lumina/manifest.json has legacyMigrationNeeded: false (d) Running /lumi-migrate-legacy again immediately produces zero file changes

Next step

Tell the user to run /lumi-check in a fresh session to confirm the wiki state from a blank-context perspective. Same model, blank context catches any inference bias from the migration session that just ran.