roam-to-logseq-db

# Roam Research → Logseq DB (v2) migration ## What this is and why it's not obvious Logseq split into two products: the old **file/Markdown version (OG)**, which has a built-in "Import Roam JSON" — and the new **DB/SQLite version (v2)**, which is now the default and has **no Roam importer at all** (only SQLite / File→DB / EDN imports). There is also **no community tool** targeting DB v2. So a naive attempt flails. This skill encodes a path that works. **The method — do not deviate without reason:** ``` Roam JSON → [scripts/roam_to_logseq.py] → Logseq FILE graph (.md + assets/ + logseq/config.edn) → [official File→DB importer] → DB graph ``` We produce a **file graph** (the universal intermediate every Roam→Logseq tool emits) and feed it to **Logseq's own File→DB importer**. We do NOT hand-write the DB (the tempting `import-edn` / `upsertNodes` path) — see the first gotcha for why that silently loses every image. ## Prerequisites - The Roam export must be **JSON** (not Markdown). Roam Markdown export **drops block references**; only JSON carries them. If the user has Markdown, have them re-export as JSON. - The converter assumes Roam's **default English daily-note titles** ("January 25th, 2026"). A graph using a different daily-note display format will silently have its daily notes become ordinary pages (journal detection fails) — adjust `DATE_RE` in the converter and confirm via the journal-count QC check. Watch for this when migrating your *other* graphs, which may differ. - The **File→DB importer** must be runnable. Two ways (pick per environment — see `references/toolchain-setup.md`): - **Headless / fully scripted** (best when a Logseq *source checkout* is available): the nbb script `deps/graph-parser/script/db_import.cljs`. One-time toolchain setup required. - **GUI fallback** (works for anyone with Logseq desktop, no source): after producing the file graph, the user clicks **Import → File to DB graph** in the app. Same engine, one click. ## Workflow Read `references/conversion-rules.md` (what the converter does to each Roam construct, and why) and `references/qc-and-troubleshooting.md` (verification queries + failure modes) before running — they carry the load-bearing detail this hub omits. 1. **Inventory the export first.** Skim the Roam JSON for scale and which constructs are present (block refs `((uid))`, `{{[[table]]}}`, images, `{{[[TODO]]}}`, code fences, daily notes, `/`-in-title pages). This tells you what to spot-check later. The converter handles all of them, but knowing the counts lets you verify nothing silently vanished. 2. **Convert:** `python scripts/roam_to_logseq.py --input <roam.json> --out <file-graph-dir>`. It downloads+localizes images (this can take a few minutes; re-runnable — it skips already- fetched files), converts every construct, and writes a Logseq file graph. Read its `--help`. 3. **Sanity-sweep the file graph** (cheap, catches converter regressions before a slow import): no leftover `{{[[table]]}}` / `{{` macros, no raw `firebasestorage` URLs, block-ref `id::` present, file counts ≈ (pages − dropped) + journals. See `references/qc-and-troubleshooting.md`. 4. **Import to a STAGING graph name first** (never straight to the user's target). This proves scale + fidelity with zero risk. Headless: `db_import.cljs <file-graph> <staging-name> -c -V`. The `-V` runs Logseq's own validator; `-c` continues past a bad page so one glitch doesn't abort a 15-min import (then check its "ignored files" report). 5. **QC the staging graph** against the source (per `references/qc-and-troubleshooting.md`): counts reconcile, **dangling-ref scan = 0**, images copied to `<graph>/assets/`, journals/tasks/assets present, no `created-at::` leaked into block content. Spot-check a table page and an image page. 6. **Final placement.** A DB graph **cannot be renamed** and the importer creates the graph *by name*, so to land it under the user's exact target name you must import directly to that name — which requires the app to release it. See the quit-Logseq gotcha. Confirm with the user before this step (it's the one action that touches their target). Scale it to the task: a 100k+ block graph deserves the full staging→QC→final dance; a tiny graph can go straight in. But the gotchas below are non-negotiable regardless of size. ## Gotchas (read these — they are why this took real effort to get right) - **build-EDN / direct-DB-write CANNOT copy image files.** `import-edn` and the HTTP API are pure DataScript transactions that never touch the filesystem, so a hand-built `#Asset` node renders as a broken image (bug #31023). Only the **File→DB importer physically copies** `assets/` and wires up `#Asset` nodes. This is *the* reason the whole method routes through a file graph. If you're ever tempted to "just write the DB directly," you will lose every image. Don't. - **Block references resolve as `[[uuid]]`, NOT Roam's `((uuid))`.** In DB v2's file importer, `((uuid))` stays literal text; the target block needs `id:: <uuid>` and the reference site must be `[[<uuid>]]`. (The converter does this.) Source-code reading suggested `((uuid))` works — it does not in v2.0.1; only empirical testing caught it. Trust the dangling-scan, not the docs. - **Quit Logseq before importing to a graph the app holds open.** The desktop app locks the open graph's SQLite via its db-worker; a second writer corrupts it. To place content under the user's *exact* target name: have them fully **quit Logseq (Cmd+Q, not just switch graphs)**, then move the empty placeholder graph dir aside (don't delete — `mv` to `.bak`), import to that name, and tell them to reopen. Verify the app is actually gone (`pgrep`) before writing. - **A stale db-worker serves OLD data and will lie to your QC.** The `logseq` CLI keeps a per-graph worker alive; after a re-import into the same name, that worker still holds the previous graph in memory, so your verification queries read stale numbers. **Kill the worker** (`pkill -9 -f "logseq_db_<name>"`) before every QC pass and after every re-import. When a fix "didn't take," suspect this first. - **A DB graph cannot be renamed** — copying/`mv`-ing the graph dir yields a graph whose worker won't start (identity is baked in). So the target name is chosen at import time; there is no cheap rename. Plan the double-import (staging → final) accordingly. - **The import is non-idempotent.** Re-running appends duplicate blocks. Import once into a fresh graph; on any failure, drop the graph and re-run from scratch. Deterministic UUIDs (the converter uses uuid-from-roam-uid) make re-runs reproducible. - **Blockquotes, code fences, and multi-line blocks swallow trailing property lines.** mldoc treats the indented `created-at::` after a `> quote` / ```` ``` ```` / multi-line block as *content*, so the property leaks into the block text. The converter skips timestamps on such blocks and forces inline-opened code fences onto their own line — but if you edit the converter, re-check for `created-at::` leaking into `:block/title` after import. - **`roam-to-logseq` (the pip package cited by blogs) does not exist** — it's AI-generated SEO spam. Real references: Logseq OG's own `roam.cljs` (conversion semantics), `sebastianpech/ roamtologseq` and `romilly/logseq-migration` (asset download) — all target the *file* version. ## Files in this skill - `scripts/roam_to_logseq.py` — the converter (Roam JSON → Logseq file graph). Self-documenting; every conversion rule has a comment explaining the empirical reason. Reuse it; don't rewrite it. - `references/conversion-rules.md` — per-construct conversion table + the reasoning behind each. - `references/toolchain-setup.md` — set up the headless File→DB importer (deps, babashka, better-sqlite3, the nbb invocation) + the GUI fallback. - `references/qc-and-troubleshooting.md` — the verification queries (dangling scan, counts, construct checks) and how to diagnose the common failures.