paginated-source-backfill-hygiene

name	paginated-source-backfill-hygiene
description	Backfill grouped source batches into existing paginated notes safely, without duplicating sections or skipping checkpoint commits.

Paginated Source Backfill Hygiene

Use this when draining source-grouped intake/backfill batches into an existing note that already has many dated sections or repeated batch blocks.

When to use

• A batch contains grouped source/thread records that need compaction before writeback.
• The target note is long, paginated, or already contains earlier backfill sections.
• The batch output includes an exact checkpoint snippet or commit instruction.
• Multiple raw rows repeat the same request/title and must be deduped before writing.

Workflow

1. Compact by group first.

   • Treat each channel/thread as one unit.
   • Dedupe repeated facts/messages inside the group before drafting notes.

2. Read enough of the target note before editing.

   • If the file is paginated, read the full file or a wide enough window around the insertion point.
   • Do not patch from a truncated view when the note contains repeated headings like ## 2026-07-02 batch ....

3. Append, don’t fracture.

   • Add one compact batch section per group.
   • Keep the existing batch style and source provenance format.
   • For unclear routing, prefer one sourced Inbox section instead of splintering into speculative entity notes.
   • If the target note already contains the same batch title or a near-duplicate section, append a fresh dated addendum at EOF instead of forcing a brittle replace across repeated headings.
   • For single-record follow-up batches, still write a compact batch subsection with Q1–Q4 and provenance; do not collapse it into a bare one-line cursor update.

4. Preserve provenance.

   • Include source line ranges or representative vadimgest://... URIs.
   • Keep Q1–Q4 answers compact and grouped.
   • Treat repeated refreshes of the same request as freshness signals unless new facts appear.

5. Run the exact checkpoint snippet after writeback.

   • If the batch output gives a checkpoint script, execute it exactly as provided.
   • Commit only the listed source lines.
   • Verify the checkpoint file was updated; do not assume the writeback implies the commit happened.

Pitfalls

• Partial-file patching: repeated headings can cause fuzzy patch collisions. Re-read the file before patching if a replace fails.
• Over-splitting duplicates: do not create one note/log entry per raw row when the same thread/title repeats.
• Over-routing: if the target entity is uncertain, keep it in Inbox with provenance instead of guessing a durable entity note.
• Silent checkpoint drift: never commit extra sources beyond the snippet.

Verification

• The note contains the batch heading, source references, and compact Q1–Q4 answers.
• Repeated messages in the same thread were deduped.
• The checkpoint file reflects the exact listed source line(s) only.

Support files

• references/session-2026-07-02-paginated-backfill.md — session-specific example of a paginated backfill append, patch collision, and exact checkpoint commit.