sync-protocols

name	sync-protocols
description	Design delta-sync protocols for mobile with cursor-based pagination and durable checkpoints. Use when building server-to-client data sync for offline-capable apps.

Sync Protocols (Delta Sync)

Instructions

Mobile apps with offline support need a protocol that lets them catch up efficiently after long silences. A good sync protocol is boring: it is a cursor, a delta, and a checkpoint, repeated.

1. Model

Each syncable entity type has:

• A monotonic server-side revision number (rev), incremented on every write.
• A deleted_at column (soft deletes survive delta sync).
• A stable id.

The client stores per-entity-type:

• Its last seen server rev (the checkpoint).
• A queue of local pending mutations to send.

2. Pull Endpoint

GET /v1/sync/articles?since=4812&limit=500

{
  "changes": [
    { "op": "upsert", "rev": 4813, "entity": { "id": "a_01", "title": "...", "updated_at": "..." } },
    { "op": "delete", "rev": 4814, "id": "a_02" }
  ],
  "next_cursor": "4814",
  "has_more": false,
  "server_time": "2026-04-15T09:00:00Z"
}

Rules:

• since is the client's checkpoint. Server returns changes with rev > since, ordered ascending.
• limit defaults 500, max 2000.
• has_more: true means the client paginates immediately (same request with new since).
• The client only updates its checkpoint to max(rev) after successfully applying the page. On crash mid-apply, the next sync replays the same page.

Postgres-backed implementation:

const limit = Math.min(Number(req.query.limit ?? 500), 2000);
const since = Number(req.query.since ?? 0);
const rows = await db.query(
  `SELECT id, rev, deleted_at, data
     FROM articles
     WHERE rev > $1
     ORDER BY rev ASC
     LIMIT $2 + 1`,
  [since, limit],
);
const hasMore = rows.length > limit;
const page = rows.slice(0, limit);
res.json({
  changes: page.map(r => r.deleted_at
    ? { op: "delete", rev: r.rev, id: r.id }
    : { op: "upsert", rev: r.rev, entity: r.data }),
  next_cursor: String(page.at(-1)?.rev ?? since),
  has_more: hasMore,
  server_time: new Date().toISOString(),
});

3. Push Endpoint

POST /v1/sync/articles

{
  "mutations": [
    { "op": "upsert", "client_id": "cli_1", "base_rev": 4812,
      "entity": { "id": "a_03", "title": "Draft" } },
    { "op": "delete", "client_id": "cli_2", "base_rev": 4812, "id": "a_04" }
  ]
}

Each mutation carries:

• client_id for idempotency (the client retries safely).
• base_rev = the revision the client saw before making this change (for conflict detection; see conflict-resolution-backend).

Response includes the assigned server rev per accepted mutation and any conflicts:

{
  "results": [
    { "client_id": "cli_1", "status": "ok", "rev": 4815, "id": "a_03" },
    { "client_id": "cli_2", "status": "conflict", "current_rev": 4810, "current_entity": { "...": "..." } }
  ]
}

4. Monotonic Revision Assignment

The server's rev must be strictly increasing across all writes of a given entity type (not per-entity). Options:

• Postgres BIGSERIAL / GENERATED ALWAYS AS IDENTITY.
• Redis INCR per table.
• Hybrid logical clock for multi-region (rare; usually eventual global rev is good enough).

Never reuse a revision. A rollback writes a new revision that reverses state.

5. Deletes

Soft delete by setting deleted_at and bumping rev. The sync stream emits a delete op. Periodic vacuum removes rows with deleted_at < now() - 90 days, but clients must have checkpointed past that rev first.

6. Full Resync / Reset

If the gap is too large or the client was offline past the vacuum horizon, return:

{ "changes": [], "reset": true, "server_time": "..." }

The client wipes local state for this entity type, sets since = 0, and pulls again.

7. Multi-Entity Sync

Prefer one endpoint per entity type. A "sync everything" endpoint couples unrelated entities and bloats payloads.

For a single-round-trip sync screen, an aggregator (BFF) can call the per-type endpoints in parallel and return a composite.

8. Client Consumption (Kotlin Multiplatform-style pseudocode)

suspend fun syncArticles() {
  var since = store.checkpoint("articles")
  while (true) {
    val page = api.pull("/v1/sync/articles?since=$since&limit=500")
    db.transaction {
      page.changes.forEach { apply(it) }
      store.setCheckpoint("articles", page.nextCursor.toLong())
    }
    since = page.nextCursor.toLong()
    if (!page.hasMore) break
  }
}

Apply inside a transaction so checkpoint advancement is atomic with change application.

9. Backpressure and Scheduling

• Sync on app foreground, on network regain, and on silent-push hints.
• Respect Retry-After if the server signals overload.
• Throttle background sync to once every 5–10 minutes unless prompted.

10. Observability

Track, per entity type: bytes/sync, changes/sync, has_more ratio (too high means limit too low), and reset rate (too high means retention too short).

Checklist

• Monotonic rev per entity type with no reuse.
• GET /sync/{type}?since=&limit= implemented with has_more pagination.
• POST /sync/{type} accepts client_id + base_rev; returns per-mutation status.
• Soft delete plus documented vacuum horizon; reset: true fallback path.
• Client advances checkpoint only after durable apply.
• Per-type endpoints preferred; aggregator added only if justified.
• Sync scheduled on foreground, network regain, and silent-push hints.
• Metrics for bytes, changes, has_more ratio, reset rate.