intake-backfill-checkpointing

name	intake-backfill-checkpointing
description	Grouped source-intake backfill workflow for compacting vadimgest/intake batches into durable notes and then running exact batch-specific checkpoint commits.

Intake Backfill Checkpointing

Use this when processing grouped vadimgest / intake / edge-source batches that arrive with a checkpoint instruction and need note writeback plus cursor advancement.

Core flow

1. Compact the group first. Treat the batch as channel/thread groups, not raw rows. Deduplicate repeated facts/messages before any writeback.
2. Choose the right durable home.
   • If the entity is clear, update the relevant State + Log note with source provenance.
   • If routing is uncertain, write one sourced Inbox note for the batch with bullets per high-signal group.
3. Preserve provenance. Inbox summaries should include source line ranges or representative vadimgest://... URIs for each grouped conversation/activity window.
4. Treat metadata-only rows correctly. Title-only, empty-body, or no-metrics rows are usually process markers, not evidence. Keep them if they signal unresolved work; skip only true noise.
5. Advance the checkpoint only after writeback. If the batch provides an exact checkpoint snippet, run it verbatim after triage/writeback for exactly that batch. Commit only the listed source lines and never substitute a generic commit command.
6. Verify before finishing. Confirm the note path written and confirm the checkpoint output reflects the requested source/line advance.

Pitfalls

• Do not create one note entry per raw row when rows belong to the same conversation.
• Do not commit sources that were not listed in the batch snippet.
• Do not treat process-only records as new facts.
• Do not skip a batch just because routing is uncertain; use a sourced Inbox note as the fallback.

Reference

• See references/intake-backfill-checkpointing.md for a compact example of batch compaction, routing, and exact checkpoint handling.