paginated-source-backfill

name	paginated-source-backfill
description	Backfill paginated vadimgest-style source batches into durable notes with deduped group compaction, source-provenance, and checkpoint verification.

Paginated Source Backfill

Use this skill when draining grouped source records from a batch into Obsidian Inbox/State/Log notes and then advancing the intake checkpoint.

When to use

• A pre-run script or intake dump provides grouped records with line ranges or representative URIs.
• The batch is paginated or partially visible, so you must work from excerpts and then verify against the full file before overwriting anything.
• The work requires a compact writeback plus a checkpoint commit.

Core workflow

1. Compact by group first.

   • Treat each channel/thread group as the unit of work.
   • Deduplicate repeated facts/messages inside the group before writing.
   • Answer the four intake questions for the grouped window:
     1. What needs to be done?
     2. How can I help now?
     3. What facts changed?
     4. What patterns/signals changed?

2. Route by specificity.

   • If the entity is clear, update the entity note with sourced facts.
   • If routing is unclear or cross-entity, write one sourced Inbox note for the batch.
   • Do not create one note per raw row when multiple rows belong to the same conversation.

3. Preserve provenance.

   • Include source line ranges or representative vadimgest://... URIs per group.
   • For durable notes, use compact provenance in the note body so later grooming can trace every claim.

4. Verify before overwriting paginated files.

   • If you only read part of a large note, re-read the full file or at least the target region before patching large structural changes.
   • Prefer a full-file rewrite only after confirming the current end-state; partial reads can leave you patching against stale context.
   • If a patch inserts malformed markers or duplicated prefix characters, stop and inspect the exact lines before retrying.

5. Commit only after writeback.

   • Run the exact checkpoint snippet provided by the batch.
   • Commit only the listed source lines.
   • If any source in the batch could not be fully processed, do not commit it.
   • Verify the checkpoint file after the commit to ensure the cursor moved exactly as expected.

Output shape for Inbox notes

• Title the note with date + source.
• Include the source block first.
• For each deduped group, write short Q1–Q4 bullets.
• Keep the note compact and sourced; avoid one-row-per-raw-record duplication.

Pitfalls

• Don’t merge unrelated groups just because they share a source channel.
• Don’t omit the source line ranges or representative URIs.
• Don’t patch a file based on a truncated view without re-reading the relevant lines.
• Don’t advance the checkpoint before the note writeback is verified.

Verification

• Re-open the edited note and inspect the affected section.
• Confirm the checkpoint JSON reflects the committed line only.
• If the batch includes a repeated request, describe it as a deduped resend rather than a new task.

See references/backfill-checklist.md for a compact checklist and the paginated-file verification recipe.