schema-unify

# Schema Unification (gbrain-base → gbrain-base-v2) v0.41.22 ships **gbrain-base-v2** — a 15-type DRY/MECE taxonomy (14 canonical + `note` catch-all) — as the install default for new brains. Existing brains on `gbrain-base` can opt in via the `pack_upgrade_available` onboard finding + the `unify-types` PROTECTED Minion handler. This skill is the playbook for that migration. ## brain_first: exempt This skill is ABOUT the brain's shape — it can't depend on the brain it's reshaping. No `gbrain search` lookup first; jump straight to onboard. ## When this skill fires - Agent runs `gbrain onboard --check` and sees `pack_upgrade_available` or `type_proliferation` warnings - User asks "what is the canonical taxonomy / how do I clean up my page types / migrate to v2" - A `dangling_aliases` finding surfaces (post-unify GC) - An agent ingesting from a custom pack wants to consult the v2 taxonomy as a reference ## Mental model (one paragraph) A production gbrain brain accreted **94 distinct `pages.type` values** over years of ingestion: tweet / tweet-thread / tweet-bundle / tweet-single / media/x-tweet/bundle / tweet-stub all coexisting; 5.5K concept-redirect pages; atom-partner-link pages that should be links; civic / framework / insight / memo / anecdote one-offs. The cure: collapse to **15 canonical types** (person, company, media, tweet, social-digest, analysis, atom, concept, source, deal, email, slack, writing, project, note) with subtypes/format/origin pushed to frontmatter, alias-rows for redirects, real link-rows for edge-shaped pages, and a catch-all that bins long-tail unknowns to `note` with `frontmatter.legacy_type = <original>` for rollback. ## Workflow ### Phase 1: Discovery Confirm the brain is actually on `gbrain-base` (not already on v2). ```bash gbrain schema active --json | jq -r '.identity' ``` Expected: `gbrain-base@1.0.0+<sha>`. If you see `gbrain-base-v2@...`, the brain is already on v2 — skip the migration. Then run onboard to see what would change: ```bash gbrain onboard --check ``` Look for the `pack_upgrade_available` finding. If it's `ok`, there's no successor declared for the active pack — done. ### Phase 2: Preview Run the per-cluster narrative: ```bash gbrain onboard --check --explain ``` This invokes the `unify-types` handler in dry-run mode and prints: - How many pages would retype per cluster (tweets, articles, companies, etc.) - How many concept-redirect pages would become alias rows - How many edge-shaped pages would convert to real links - The synthesized catch-all rules for unknown types Review the output. If the proposed changes look wrong, **don't** proceed — file an issue or write a custom pack with adjusted mapping_rules. ### Phase 3: Apply The handler is PROTECTED (manual_only) — autopilot will never auto-fire it. Submit explicitly: ```bash gbrain jobs submit unify-types \ --params '{"target_pack":"gbrain-base-v2","apply":true}' ``` On PGLite (the install default), or on any setup without a running `gbrain jobs work` worker or supervisor daemon, add `--follow` so the job executes inline: ```bash gbrain jobs submit unify-types \ --follow \ --params '{"target_pack":"gbrain-base-v2","apply":true}' ``` The persistent worker daemon is Postgres-only. Without `--follow` on PGLite, the job sits queued forever and the migration never runs. `apply` defaults to **false** (dry-run) per the handler contract, so `"apply":true` is required here or the job reports success having retyped nothing and left the active pack unflipped. Omit it to preview. Watch progress per phase (worker-daemon runs; with `--follow` the same progress streams inline): ```bash gbrain jobs get <job_id> # one job: status, progress, result gbrain jobs watch --follow # live dashboard of the whole queue ``` A job that stays `queued` here means no worker is running; resubmit with `--follow` to execute it inline. On a 186K-page brain expect ~10 minutes. The handler runs: 1. Preflight (validate target pack has `mapping_rules:`) 2. Stats snapshot (pre-state for celebration summary) 3. Acquire `gbrain-unify` db-lock (60min TTL) 4. Apply phases: - Explicit retype rules (tweets, articles, companies, etc.) - Catch-all retype (unknown types → note with legacy_type) - Page-to-link rules (atom-partner-link, symlink) - Page-to-alias rules (concept-redirect) 5. Final sync (untyped rows by path-prefix) 6. **Flip active pack** to gbrain-base-v2 7. Verify + celebration summary ### Phase 4: Verify ```bash gbrain onboard --check gbrain schema stats ``` Expected: - `pack_upgrade_available` → `ok` (active pack is now v2) - `type_proliferation` → `ok` (≤16 distinct typed values) - `dangling_aliases` → `ok` (slug_aliases all point at active canonicals) - `gbrain schema stats` shows ≤16 distinct types ### Phase 5: Post-migration Search and query `--type article` keep returning those pages post-unify: `media` declares `article` as an alias, so the type filter expands through the active pack's alias closure (the results also include other `media` pages). Direct SQL against `pages.type` needs updating to the canonical types. Search queries get a small ranking signal: pages reached via `slug_aliases` (canonicals of one or more aliases) get a 1.05x boost. Visible via `gbrain search --explain`. ## Rollback Every retyped page preserves `frontmatter.legacy_type = <original>`. Restore types in bulk (Postgres/Supabase deployments only; requires direct DB access): ```sql UPDATE pages SET type = frontmatter->>'legacy_type' WHERE source_id = 'default' AND frontmatter->>'legacy_type' IS NOT NULL; ``` On PGLite there is no SQL shell, so use the CLI surface instead: `frontmatter.legacy_type` persists per page, so individual retypes can be reverted through the normal `put_page`/CLI surface, and the soft-delete restore and pack-flip revert below work on every engine. Page-to-alias and page-to-link source pages soft-delete with 72h TTL. Restore within that window: ```bash gbrain restore <slug> ``` Revert the active pack flip: ```bash gbrain schema use gbrain-base ``` ## Anti-patterns - **Don't run unify-types under autopilot.** It's manual_only by design. Autopilot remediation should never silently change your taxonomy. - **Don't expect mapping_rules to cover every legacy type explicitly.** Use the catch-all (`*unknown*`) for the long tail. Pages get retyped to `note` with `legacy_type` preserved. - **Don't rewrite body-text wikilinks.** The slug_aliases table IS the resolver. `[[old-redirect-slug]]` keeps working via `engine.resolveSlugWithAlias` short-circuit. - **Don't bypass the dry-run.** Always run `--explain` before applying. The trust delta is real. - **Don't run two unify jobs concurrently.** The `gbrain-unify` db-lock serializes them; the second submission rejects with "already in progress." ## Decision tree ``` Active pack already gbrain-base-v2? → Skip migration. Custom pack with own mapping_rules? → Run --check --explain to see if your pack declares migration_from for the active pack. If yes, target_pack = your pack name. Brain has many custom types not covered by gbrain-base-v2 mapping_rules? → The catch-all retype binds them to `note` with legacy_type preserved. Review by inspecting frontmatter.legacy_type after the migration. Federated brain (multiple sources)? → Add --params source_id to scope the migration per-source. Each source can be migrated independently. Worried about a specific cluster's mapping? → Fork gbrain-base-v2 (`gbrain schema fork gbrain-base-v2 my-pack`), edit mapping_rules in your fork, then target the fork. ``` ## Contract Inputs: - A brain on `gbrain-base` (or any pack with `migration_from: gbrain-base-v2`). - Trusted local CLI access on the brain host: `gbrain jobs submit` grants the PROTECTED-handler opt-in itself for protected names; the remote MCP `submit_job` op cannot. - ~10 min wallclock on a 186K-page brain. Outputs: - Pages retyped to canonical types with `frontmatter.legacy_type` preserved (per-page rollback signal). - `slug_aliases` rows for concept-redirect pages (alias table IS the resolver — no link rewrite). - Real `links` rows for edge-shaped pages (`atom-partner-link`, `symlink`, etc.). - Active pack flipped to `gbrain-base-v2` atomically at end of successful run. Side effects: - Source pages soft-deleted with 72h restore TTL (`gbrain restore <slug>`). - One-time cache invalidation on KNOBS_HASH_VERSION bump (5→6); self-healing in `cache.ttl_seconds`. - Search/query `--type X` expands through the active pack's alias closure (back-compat). Failure modes: - Concurrent submission rejected by the `gbrain-unify` db-lock; second call exits gracefully. - Catch-all retype excludes `page_to_link` + `page_to_alias` source types (caught in E2E pre-merge). - Phase failures abort the run before `active_pack_flipped`; partial state restorable via op_checkpoint resume. ## Anti-Patterns DON'T: - Submit `unify-types` via the remote MCP `submit_job` op. PROTECTED handlers require trusted local callers (`gbrain jobs submit` on the brain host); remote MCP rejection is the intentional trust boundary. - Edit `mapping_rules` in `gbrain-base-v2.yaml` to skip clusters you don't trust. Fork the pack instead (`gbrain schema fork`) so the source-of-truth migration stays consistent across brains. - Run `unify-types` from inside an autopilot tick. The check is `manual_only` — autopilot deliberately never auto-fires it because pack upgrades are one-time consenting taxonomy decisions. - Hard-delete soft-deleted source pages before the 72h restore window. Use `gbrain restore <slug>` first if rollback is needed. - Assume `frontmatter.legacy_type` survives every roundtrip. The marker is canonical for the immediate post-migration window; downstream re-imports may overwrite it. ## Output Format Per phase, the handler emits to stderr: ``` [unify-types] phase=retype-explicit applied=N skipped=M cost=USD ttl=Ns [unify-types] phase=retype-catch-all applied=N [unify-types] phase=page-to-link converted=N pages soft-deleted [unify-types] phase=page-to-alias aliased=N pages soft-deleted [unify-types] phase=sync residual=N [unify-types] active_pack flipped from gbrain-base to gbrain-base-v2 ``` Final celebration summary to stderr: ``` ═══════════════════════════════════════════════════════════ gbrain-base-v2 migration complete ═══════════════════════════════════════════════════════════ Before: 94 distinct page types After: 15 canonical types Retyped: 25,632 pages Aliased: 5,521 redirects → slug_aliases table Linkified: 65 ghost pages → real link rows Soft-deleted: 5,586 pages (restorable for 72h) ═══════════════════════════════════════════════════════════ ``` For structured JSON, `gbrain call get_job '{"id": <id>}'` returns the job row; its `result` field carries the `UnifyTypesResult` shape with `per_phase`, `pack_identity_after`, `active_pack_flipped` (`gbrain jobs get <id>` prints the same result inline). ## Reference - Architecture: https://github.com/garrytan/gbrain/blob/master/docs/architecture/type-taxonomy.md - Pack-upgrade mechanism: https://github.com/garrytan/gbrain/blob/master/docs/architecture/pack-upgrade-mechanism.md - Issue: https://github.com/garrytan/gbrain/issues/1479