zeus-node-lifecycle-campaign

# Zeus Node Lifecycle Campaign This is an executable, decision-gated investigation campaign for Zeus's hardest live problem (maintainer-confirmed, 2026-07-06): **races in starting, stopping, deleting, and switching embedded nodes** (embedded LND and LDK Node), including Tokio-thread crashes and `Wallet.tsx` `fetchData` re-entrancy. Everything below was verified by reading the code at master `c5fd094fb` (v13.1.3-alpha) unless explicitly labeled HYPOTHESIS. ## When to use / When NOT to use **Use this skill when** you are reproducing, diagnosing, or fixing node start/stop/delete/switch races, or reviewing any PR that touches `views/Wallet/Wallet.tsx` connection flow, `SettingsStore` lifecycle flags, `utils/LndMobileUtils.ts`, `utils/LdkNodeUtils.ts`, or the native node modules. **Do NOT use this skill for:** | Need | Go to sibling skill | |---|---| | General symptom→triage for other bugs | zeus-debugging-playbook | | How to capture logs / measure (general tooling) | zeus-diagnostics-and-tooling | | Past incident narratives beyond lifecycle | zeus-failure-archaeology | | Backend capability matrix, adding an RPC | zeus-backends-and-capabilities | | Formal race-analysis method (generic recipes) | zeus-proof-and-analysis-toolkit | | PR/commit/review rules, the non-negotiables | zeus-change-control | | Running the app / connecting nodes at all | zeus-run-and-operate | | Settings blob, migrations, keychain | zeus-storage-and-migrations | | What counts as test evidence, jest traps | zeus-validation-and-qa | ## Jargon (one line each, both domains) - **Embedded LND**: a full LND Lightning node compiled to a mobile library (gomobile) running *inside* the Zeus app process; Zeus implementation string `'embedded-lnd'`. - **LDK Node**: a Rust Lightning node (`ldk-node` crate, ZeusLN fork) embedded via uniffi FFI bindings; implementation string `'ldk-node'`. - **Tokio**: the Rust async runtime ldk-node uses. If the *last* reference to the Node object is dropped from one of Tokio's own worker threads, the runtime is destroyed from inside itself → Rust panic → native crash (SIGABRT), killing the whole app. - **uniffi**: Mozilla's Rust↔Kotlin/Swift binding generator; the checked-in `ldk_node.kt` / `LDKNode.swift` are its output. - **gomobile / Messenger IPC**: embedded LND is a Go library; on Android it runs behind `LndMobileService.java`, a bound Android Service the RN module talks to via `Messenger` message passing (Blixt-derived code, `TODO(hsjoberg)` debt). - **Foreground service**: an Android service with a persistent notification, allowed to keep running in background; used for "persistent mode". - **Focus event**: react-navigation fires `'focus'` every time a screen becomes active — including returning from another screen. `Wallet.tsx` re-runs its connect logic on focus. - **AppState**: React Native's app-level state (`active`/`background`/`inactive`); Wallet also re-runs connect logic on `active`. - **MobX observable**: mutable store field that re-renders observers; the lifecycle flags below are plain observables with **no atomicity guarantees**. - **Neutrino**: LND's light-client chain backend (connects to Bitcoin P2P peers). **Esplora/RGS/VSS**: LDK Node's chain API server / rapid gossip sync server / remote channel-state storage (`vss.zeusln.com`). - **fetchData**: the ~800-line method in `views/Wallet/Wallet.tsx` that resets stores, starts/attaches to the node for the active backend, and loads wallet data. It IS the boot sequence (not App.tsx). ## The lifecycle state machine as it actually exists (verified 2026-07-06) There is **no single owner**. Lifecycle state is smeared across four layers: | Layer | File | Owns | |---|---|---| | Orchestration | `views/Wallet/Wallet.tsx` | `handleFocus` / `handleAppStateChange` / `handleOpenURL` → `getSettingsAndNavigate()` → `fetchData()`; the `refreshRunner` instance (`utils/WalletRefreshRunner.ts`); a 60s slow-startup alert timer | | Flags | `stores/SettingsStore.ts` | `connecting` (init **true**), `fetchLock`, `embeddedLndStarted`, `walletJustCreated`, `triggerSettingsRefresh`, `ldkNodeSyncing`; `setConnectingStatus()`; `updateNodeProperties()` (swaps `implementation`, dirs, creds on every settings write) | | JS node control | `utils/LndMobileUtils.ts` (`initializeLnd`, `startLnd`, `stopLnd`, `waitForRpcReady`, `createLndWallet`, `deleteLndWallet`) and `utils/LdkNodeUtils.ts` (`startLdkNodeWallet`, `stopLdkNode`, `waitForLdkNodeReady`, `createLdkNodeWallet`, `deleteLdkNodeWallet`) | retry loops, sleeps, error classification (`utils/LndMobileErrors.ts`) | | Native | Android `android/app/src/main/java/com/zeus/LndMobileService.java` (+ `LndMobile.java`, `LndMobileScheduledSyncWorker.java`), `android/app/src/main/java/app/zeusln/zeus/LdkNodeModule.kt`; iOS `ios/LndMobile/`, `ios/LdkNodeMobile/LdkNodeModule.swift` | the actual daemon/Node object, `lndStarted` boolean, `nodeLock`/`takeNode()` | | Mutation entry points | `views/Settings/Wallets.tsx` (switch), `views/Settings/WalletConfiguration.tsx` (create/edit/delete), `utils/WalletCreationUtils.ts` (Quick Start), `views/Settings/SeedRecovery.tsx` | write `settings.nodes`/`selectedNode` via `updateSettings`, then `setConnectingStatus(true)` + `popTo('Wallet')` | ### The guards, in evaluation order 1. **`refreshRunner.running`** (`WalletRefreshRunner`, Wallet.tsx instance) — guards `getSettingsAndNavigate` re-entry from the focus event and AppState `active`. Added as the `_navigating` instance field in `c71a04ce9` (2026-04-07) because the focus event fires twice on startup and both raced into `fetchData` → duplicate `buildNode`, second build times out holding VSS resources; moved into `utils/WalletRefreshRunner.ts` for #4751, which also clears `triggerSettingsRefresh` when a guarded refresh starts and runs at most one follow-up refresh if the flag is re-armed while it runs (only while the Wallet is focused). A `getSettingsAndNavigate` promise that never settles leaves `running` stuck `true`. **Bypassed by three call sites** that invoke `getSettingsAndNavigate()` directly: pull-to-refresh (`LayerBalances onRefresh`), the error-screen Restart button (iOS branch), and `handleOpenURL` for `zeusln://share`. 2. **`fetchLock`** (SettingsStore observable) — checked/set at `fetchData` entry (`if (fetchLock) return; SettingsStore.fetchLock = true;`). Cleared in exactly two places: the happy-path end of `fetchData`, and `setConnectingStatus(true)`. **Every early-return error path leaves `fetchLock === true`** — by design: the app then shows the error screen, and recovery requires `setConnectingStatus(true)` (Restart button, wallet switch, transient-retry loop), which clears the lock. Consequence: any code path that errors out and then expects a plain focus event to reconnect will hang silently. 3. **`connecting`** — initialized `true`, so the first `fetchData` after cold start runs the full connect branch (stop→init→start + reset of ~12 stores). Set `false` at the end of a successful connect or on fatal errors; set `true` by every wallet-switch/create/restart path. When `false`, `fetchData` becomes a light data refresh (embedded-lnd still calls `waitForRpcReady()` + full data fetch on every focus). 4. **`walletJustCreated`** — set by `createLndWallet` / LDK create paths; makes `fetchData` skip the stop→init→start cycle because the node is already running from wallet creation. Consumed (set `false`) on first use. 5. **`embeddedLndStarted`** — whether `stopLnd()` is called before re-init (`if (!recovery && SettingsStore.embeddedLndStarted) await stopLnd();`). Set `true` in `startLnd` (only when a wallet password is present), `false` in `stopLnd` and several views. 6. **Native guards** — LDK: `nodeLock` (Kotlin `synchronized`) / `takeNode()` (Swift) makes stop/build swap the node reference atomically; LND Android: a single `lndStarted` boolean inside `LndMobileService`; the Go daemon itself rejects double starts with "lnd already started". ### `setConnectingStatus(true)` side effects (SettingsStore.ts) Resets `error`/`errorMsg`/`lndFolderMissing`, calls `BackendUtils.clearCachedCalls()` (flushes the request dedup cache — see zeus-backends-and-capabilities), and clears `fetchLock`. This is the universal "reconnect" lever; treat any call to it as a lifecycle transition. ### Per-backend connect branch inside `fetchData` (only when `connecting`) - **ldk-node**: `stopLdkNode()` (unless `walletJustCreated`) → `startLdkNodeWallet` (`initializeNode` FFI + `node.start()` retried up to 5×500ms on "Node not initialized" + `syncWallets`) → later branch `waitForLdkNodeReady()` (polls `status()` every 500ms up to 60s, tolerating "Node not initialized" and not-running — added by `b54e8f138`, 2026-05-17; the companion `buildNode`-race fix `8fc9698a0`, same day, added the `walletJustCreated` skip). - **embedded-lnd**: `stopLnd()` (if started) → `initializeLnd` (writes lnd.conf, binds service) → optional express graph sync → `startLnd` (retry ladder incl. force-stop on `LND_ALREADY_RUNNING`, wallet unlock on `WALLET_LOCKED`) → `waitForRpcReady()` (polls `getInfo` 500ms up to 30s). Transient RPC errors route through `retryOnTransientError` (max 5 retries, 2s base exponential backoff; it toggles `connecting` false→true, which is also what re-clears `fetchLock`). - Remote backends (lndhub login, LNC `connect()`, NWC `connectNWC()`, REST default) have no native lifecycle but share the same flags. ### Native stop/delete rules (the Tokio contract) From `LdkNodeModule.kt` (`stop`, `buildNode`) and `LdkNodeModule.swift` (same methods), established by `d416052cd` + `0e2b89164` (both 2026-03-10): 1. Take the node reference out of the module under `nodeLock`/`takeNode()` **first**, so no other call can reach a stopping node. 2. Call `node.stop()` on a **dedicated non-Tokio thread** (raw `Thread` on Android, GCD global queue on iOS) — never on a Tokio callback thread. 3. Resolve the JS promise only **after** `stop()` returns — callers may delete wallet files or build a new node the moment `stop()` resolves. 4. Keep the reference alive **2 more seconds** after resolving (`Thread.sleep(2000)` + `hashCode()` on Android, `withExtendedLifetime` on iOS) so internal Tokio tasks drop their `Arc` refs before ours — otherwise the "Runtime dropped on worker thread" panic. 5. `buildNode` applies the same stop-and-hold to any existing node before building, and clears the JS-visible node reference up front — which is why **any FFI call during a rebuild window rejects with `"Node not initialized"`** (JS must tolerate it: `LDK_NODE_NOT_INITIALIZED` handling in `utils/LdkNodeUtils.ts`). Embedded LND has no equivalent contract; `stopLnd()` in `utils/LndMobileUtils.ts` approximates one (status check → `stopDaemon` → `killLnd` → poll status up to 10×500ms). ## Known race windows (the target list) Each window is verified by code read at `c5fd094fb`. Runtime confirmation status noted. | # | Window | Status | |---|---|---| | W1 | Double focus → duplicate `getSettingsAndNavigate` | FIXED by `_navigating` (`c71a04ce9`, now `WalletRefreshRunner`); residual: the three bypass call sites rely only on `fetchLock`, which does NOT stop a second `getSettingsAndNavigate` from re-running the pre-`fetchData` navigation logic concurrently | | W2 | JS call into LDK during `buildNode` rebuild → `"Node not initialized"` rejection | Tolerated in `waitForLdkNodeReady` (`b54e8f138`) and the pre-existing `start` retries; **unguarded** for every other store that calls the backend during the window (e.g. polling stores) | | W3 | Cross-implementation switch leaves the previous node running: `fetchData`'s `ldk-node` branch calls only `stopLdkNode()`, the `embedded-lnd` branch only `stopLnd()`. Switching embedded-lnd → ldk-node (or reverse) stops NEITHER the old node; `views/Settings/Wallets.tsx onWalletPress` only calls `setPersistentMode(false)` (dismisses the Android notification, does not stop the node) and `BackendUtils.disconnect()` for LNC | Confirmed by code read; runtime impact (two chain-syncing nodes, memory/battery, port/db contention) needs Phase 3 measurement | | W4 | Delete-active-wallet flow: `WalletConfiguration.deleteNodeConfig` stops node → `updateSettings({ justDeletedWallet: active })` → deletes files → navigates to Wallets; selecting the next wallet runs `handleJustDeletedWallet` which does `setConnectingStatus(true)` then **`sleep(2000)`** — a bare time-based wait standing in for "old node fully gone" | Open; the sleep is a fenced symptom-patch pattern (below) | | W5 | JS reload with live daemon: LND runs **in the app process** (`android/app/src/main/AndroidManifest.xml` declares `LndMobileService` with no `android:process` attribute), so a JS-only restart (`RNRestart`, dev-menu reload, JS crash) leaves the Go daemon running → next start hits `LND_ALREADY_RUNNING` → force-stop ladder in `startLndWithRetry` with 2–4s cleanup sleeps | Handled but fragile; see W6 | | W6 | `LndMobileTools.killLnd()` scans running processes for `packageName + ":zeusLndMobile"`, but no such process is ever created (no `android:process` in the manifest) — so `killLnd` can never find/kill anything and resolves `false`. Recovery therefore rests entirely on the Go `stopDaemon` call. Also `killLndProcess()` uses `getCurrentActivity()` which is null when no Activity is attached (background) → NPE risk | Code-read finding; runtime confirmation = log `killLnd` result in Phase 4. Blixt heritage: Blixt runs the service in a separate process, Zeus does not | | W7 | `LndMobileScheduledSyncWorker.java` (background sync) binds the same `LndMobileService` and — per its own FIXME at ~line 310 — calls `thread.run()` instead of `thread.start()`, executing the "thread" body synchronously on the worker's calling thread with handlers on the **main looper**. A scheduled sync racing a foreground app start can send `MSG_START_LND` against an already-started daemon (the worker's own comment: "Make sure we don't attempt to start lnd twice... better fix would be MSG_CHECKSTATUS") | Open, acknowledged debt (`TODO(hsjoberg)`) | | W8 | `LndMobileService.onUnbind`: when the last client unbinds it calls `stopLnd(null, -1)` + `stopSelf()` — but the RN module (`LndMobile.java`) binds once in `initialize()` and its `unbindLndMobileService` is flagged in `lndmobile/LndMobile.d.ts` as "function looks broken" and is never called from TS (verified: zero TS call sites). So service lifetime ≈ process lifetime; unbind-triggered stop is effectively dead code in the app (only the sync worker unbinds) | Confirmed by code read | | W9 | Background/foreground storm: AppState `active` triggers `getSettingsAndNavigate` (guarded by `WalletRefreshRunner`), `inactive` resets `NostrWalletConnectStore`; iOS fires `inactive` on mere app-switcher/notification-shade interactions | Guarded but untested under storm; Phase 0 E4 | | W10 | LDK delete-then-create same dir: `stop()` resolves before the 2s native Arc-hold expires; `deleteLdkNodeWallet` then `RNFS.unlink`s the storage dir while background Tokio cleanup may still touch it | HYPOTHESIS — the 2s hold was added for the ref-drop panic, not for file-handle safety; needs Phase 5 experiment | --- ## Phase 0 — Baseline: reproduce and measure Goal: a numeric crash/hang rate per scenario BEFORE any fix, so any fix can be judged. Do not skip; the maintainer's revert-first culture means an unmeasured "fix" is unlandable. ### Prerequisites - Dev builds on BOTH platforms (see zeus-run-and-operate; `yarn android` / `yarn ios`, Metro via `yarn start`). - Two test wallets on one device: one `embedded-lnd` (Mainnet or Testnet) and one `ldk-node`. Create via IntroSplash Quick Start + Settings → Wallets → Add. - Log capture: ```bash # Android — native lifecycle tags + crashes (works on debug and release): adb logcat -v time | grep -E "ReactNativeJS|LdkNodeModule|LndMobileService|LndMobile|AndroidRuntime|FATAL|Fatal signal" # Android — crash buffer only (count crashes across trials): adb logcat -b crash -v time # Android JS console in DEBUG builds goes to the Metro terminal, not logcat. # In RELEASE builds console.log appears under the ReactNativeJS logcat tag. # LDK Node's own Rust log (debug builds; substitute the wallet's uuid dir): adb shell run-as app.zeusln.zeus sh -c 'ls files/ldk-node/' adb shell run-as app.zeusln.zeus sh -c 'tail -f files/ldk-node/<uuid>/ldk_node.log' # Embedded LND's log: in-app Settings → Embedded node → LND Logs, or (debug): adb shell run-as app.zeusln.zeus sh -c 'tail -f files/<lndDir>/logs/bitcoin/mainnet/lnd.log' # iOS — native NSLog (simulator; process name is "zeus"): xcrun simctl spawn booted log stream --predicate 'process == "zeus"' --style compact # iOS JS console: Metro terminal. ``` ### Log milestones (the vocabulary of every experiment) All verified present in source: | Milestone | Exact string (grep-able) | Source | |---|---|---| | Focus fired, connect starting | `[Wallet] handleFocus: triggering getSettingsAndNavigate` | Wallet.tsx | | Re-entry suppressed | `skipping — getSettingsAndNavigate already in flight` | Wallet.tsx (focus + AppState variants) |