migrating-nango-deletion-detection

# Migrating Nango Deletion Detection ## Do this 1. Find legacy usage: - `deleteRecordsFromPreviousExecutions(` - `trackDeletes:` / `track_deletes` 2. For each sync + model that needs automatic deletion detection: - Add `await nango.trackDeletesStart('ModelName')` at the start of `exec` (before fetching/saving). - Replace `await nango.deleteRecordsFromPreviousExecutions('ModelName')` with `await nango.trackDeletesEnd('ModelName')`. - Keep `trackDeletesEnd` after all `batchSave`/`batchUpdate`/`batchDelete` calls. 3. Safety: - Only call `trackDeletesEnd` if the full dataset was fetched + saved between `trackDeletesStart` and `trackDeletesEnd` (otherwise you can cause false deletions). - Prefer letting exceptions bubble. If you `catch`, re-throw when data is incomplete. ## Checkpointed full refresh (multi-execution) Full refresh syncs need a pagination `checkpoint` (page/cursor/offset) in addition to delete tracking. Nango syncs run inside a time-limited execution window; without a checkpoint, a run that does not finish in that window restarts from page 1 next time, wasting compute re-fetching the same early pages and never reaching the rest of the dataset. - Read the checkpoint first (`await nango.getCheckpoint()`), and resume pagination from it when present. - Call `trackDeletesStart('ModelName')` at the beginning of every execution in the refresh window. It is safe to call repeatedly — it will not overwrite the start of a delete-tracking window that a prior execution of the same logical refresh already opened. - After each successful `batchSave()`, call `saveCheckpoint()` with the next page/cursor — on every page, including the last one. Do not guard the call with "more pages remain" (`if (nextPage) { ... }`); a run whose whole dataset fits on the first page then never saves a checkpoint at all. - Call `clearCheckpoint()` only after the last page is saved. Because every processed page must call `saveCheckpoint()`, the normal page-processing path has a checkpoint row to clear. If a distinct path creates no checkpoint at all (for example, it processes no pages), do not call `clearCheckpoint()` on that path; it throws `checkpoint_conflict` at runtime. This is not a substitute for saving the last page. - Call `trackDeletesEnd('ModelName')` only after that `clearCheckpoint()` — i.e. only in the execution that finishes saving the full dataset. ## Tests - Re-record mocks after code changes: - `nango dryrun <sync-name> <connection-id> --validate -e dev --no-interactive --auto-confirm` - `nango dryrun <sync-name> <connection-id> --save -e dev --no-interactive --auto-confirm` - `nango generate:tests && npm test` - Never hand-edit `*.test.json`. ## Before/after ```ts // Before for await (const page of nango.paginate(cfg)) { await nango.batchSave(page, 'Ticket'); } await nango.deleteRecordsFromPreviousExecutions('Ticket'); ``` ```ts // After await nango.trackDeletesStart('Ticket'); for await (const page of nango.paginate(cfg)) { await nango.batchSave(page, 'Ticket'); } await nango.trackDeletesEnd('Ticket'); ``` If the sync can exceed the execution window (large dataset, slow provider), add a pagination checkpoint so `trackDeletesStart`/`trackDeletesEnd` only wrap one logical refresh across however many executions it takes: ```ts // After (checkpointed, multi-execution safe) const checkpoint = await nango.getCheckpoint<{ page?: number }>(); let page = checkpoint?.page ?? 1; // Safe on every execution: does not overwrite an already-open window. await nango.trackDeletesStart('Ticket'); for await (const results of nango.paginate({ ...cfg, paginate: { ...cfg.paginate, offset_start_value: page, on_page: async ({ nextPageParam }) => { page = typeof nextPageParam === 'number' ? nextPageParam : undefined; } } })) { await nango.batchSave(results, 'Ticket'); // Save on every page, including the last — do not guard this with // `if (page !== undefined)`. Skipping it whenever there is no next page // means a run whose entire dataset fits on the first page never saves a // checkpoint, and the clearCheckpoint() below would then fail with // checkpoint_conflict (deleting a row that was never written). await nango.saveCheckpoint({ page }); } // Every page, including the last, was checkpointed. await nango.clearCheckpoint(); await nango.trackDeletesEnd('Ticket'); ```