| name | local-ingest |
| description | Onboard or backfill a company's changelog locally in Claude Code by fetching pages and extracting releases with the agent itself (and parallel sub-agents), then writing through the batch-upsert endpoint — no remote fetch dispatch, no server-side extraction inference billing. Use when you want to onboard a source without triggering the remote update workflow's billed extraction, or recover from a remote fetch that burned tokens and wrote zero releases. Includes a mandatory robots.txt / Content-Signal opt-out gate. |
Local Ingest
Onboard a changelog without dispatching the remote fetch path. The remote path (releases admin source fetch → POST /v1/workflows/update → the API worker's DeterministicUpdateWorkflow, #1946) runs the body→records extraction server-side (incremental Haiku or the tool-loop) and bills that inference — even when it writes zero releases. A local Claude Code agent is already a capable model with web-fetch + file tools, so it can do the per-page extraction inline (folded into the session you're already in) and only needs a deterministic write path.
Core principle: the agent is the extractor. You fetch, you parse, you structure the records, and you POST them to the batch endpoint, which does a plain idempotent upsert with no AI on insert. The remote extraction loop never runs.
Local Claude Code only. This skill uses the Agent tool for sub-agent fan-out and assumes a persistent local filesystem and the CLI's RELEASES_API_* env. It sits alongside seeding-playbooks as a local-only operator skill — it is not deployed to the managed-agent fleet.
Cost contract
State this explicitly to the operator before a large run, and honor it:
- Spends: your agent's own tokens for fetch + extract, plus any sub-agents you fan out (your choice of model —
sonnet for quality, haiku for bulk).
- Does NOT spend: no server-side extraction (the update workflow's incremental-Haiku / tool-loop passes), no remote
web_fetch extraction. POST /v1/workflows/update is never called.
- The batch endpoint runs no AI on insert — no summarization, no marketing classifier, no feed enrichment. It does an
insert plus a fire-and-forget vector embed (waitUntil). The AI fields (title_generated / title_short / summary) are intentionally left empty; see Manual enrichment below for the deliberate follow-up paths.
The negative signal that confirms no remote run happened: in Axiom releases-cloudflare-logs there is no POST .../fetch?sessionId=det-… and no extract-deps-worker event for the source.
When to use
- Onboarding a new source (or backfilling an existing one) interactively in Claude Code and you want to skip the remote extraction cost.
- A remote fetch burned a full extraction loop and wrote 0 releases (the Conductor / large-body /
max_tokens failure mode) — extracting inline sidesteps the loop entirely.
- The page is fetchable by your own web tools (the refusal below is a policy gate, not a fetch limit).
When NOT to use:
- You're in a managed-agent session. This is local-only (Agent-tool fan-out + local filesystem + CLI env). MA sessions should keep using the normal fetch path.
- The source has a clean, content-rich feed. Just add it (
managing-sources) and let the cron pipeline fetch it. Local-ingest is for cases where the agent doing the extraction inline is the right tool, not a replacement for feed/GitHub adapters.
- The publisher opts out (preflight refuses — see Step 1).
Workflow
Step 1 — Preflight (mandatory safety gate)
Run the opt-out check before fetching or writing anything:
bun .claude/skills/local-ingest/preflight.ts <url> --json
It fetches robots.txt (following apex→www redirects), parses the Cloudflare Content Signals policy, and gates on exit code:
| Exit | Verdict | Action |
|---|
0 | proceed | Permissive or absent — continue. |
1 | refuse | ai-input=no declared. STOP. Surface to the user. Do not fetch or write. |
2 | unknown | Couldn't fetch/parse robots.txt. Retry once; if still unknown, surface to the user — don't assume proceed. |
The gate is on ai-input only. Local ingest is search/input ingestion — feeding a search index — which is exactly what ai-input governs. ai-train=no opts out of model training, a use this path doesn't perform, so ai-train=no alone (e.g. vercel.com) correctly proceeds.
This is the Conductor lesson. conductor.build serves Content-Signal: ai-train=no, search=yes, ai-input=no — its org/source already sits paused in the registry for exactly this reason (releases admin playbook conductor). A local onboard must not silently spend tokens or ingest content against a publisher opt-out. conductor.build must be refused (via its ai-input=no) — it is the regression target for this gate.
The refusal is a policy choice, not a technical limitation: web_fetch and CF Browser Rendering can still retrieve these pages (only Cloudflare's /crawl endpoint hard-enforced the signal for Conductor). Honor the opt-out anyway. Override only with explicit publisher permission documented by the operator — there is no silent bypass.
The preflight also prints any Sitemap: URLs from robots.txt — keep them for Step 3.
Step 2 — Resolve target
Ensure the org and source exist before writing — releases attach to a source_id. Defer to managing-sources for create/naming/primary rules and to finding-changelogs for whether a structured feed exists (if it does, prefer the normal pipeline). Then:
- Read the org playbook for parse hints:
releases admin playbook <org> (see managing-sources → Playbooks).
- Capture the
sourceSlug and the src_… id from the created/resolved row — you need one of them for the batch URL.
If the source is brand-new, create it (e.g. releases admin source create "<name>" --url <url> --org <org> [--primary]) but do not trigger a fetch — that would dispatch the remote update workflow you're trying to avoid.
Step 3 — Map pages
Classify the page shape (releases admin discovery evaluate <url> --json reports pageStructure: single-page | index | unknown):
- Single-page changelog → one fetch in Step 4.
- Index → detail → enumerate the per-release URLs. Prefer
/sitemap.xml (the preflight surfaced it) filtered to the changelog path; otherwise parse the index HTML for per-release links.
- Client-rendered index → the index is a JS shell whose cards load via a client-side fetch, so
curl/non-JS scrape returns zero entries. Enumerate by rendering: drive the index with Claude in Chrome (or fan out a rendering sub-agent), paginating if the list is windowed, and collect the per-release links. Then check whether the detail pages are statically rendered — they often are even when the index isn't — and if so bulk-curl those directly (HTML→markdown locally, no render, no AI bill). Harvey's help.harvey.ai/release-notes is exactly this: client-rendered index, static detail pages. Don't conclude "no releases" from an empty non-JS scrape of the index.
Cap to a recent window (default ~25–50 most-recent releases) and log what you skipped — never silently truncate. Re-runs are idempotent, so a deeper backfill later just adds rows.
Step 4 — Fetch + extract
Reuse the parsing-changelogs conventions for type/version/date/media (see Parity below).
- A few pages: fetch and extract inline in this agent.
- Many pages: fan out parallel sub-agents — one URL-slice each, each returning records in the batch schema. Use the
Agent-tool dispatch pattern from seeding-playbooks (and superpowers:dispatching-parallel-agents for the fan-out discipline). Do not fetch release URLs in the parent agent — delegate to keep the parent's context clean.
- Parent-saves caveat: sub-agents may be blocked from making the batch write (permission limits). Plan for it — have each sub-agent return its records in its result, and let the parent (you) do the writes in Step 5. Same pattern
seeding-playbooks uses for note-saving.
- Extract media — don't ship text-only. Pull the images/video out of each release body and populate
media, unwrapping _next/image / _vercel/image optimizer wrappers to the underlying CDN URL (normalizeMediaUrl from packages/rendering/src/media-url.ts). A text-only first pass is a common miss caught only on operator review (Harvey shipped text-only, then had to re-extract _next/image → cdn.sanity.io and re-seed); the enrichment re-POST is mode: "upsert-content" (Step 5).
Each record must match the batch schema (see Step 5) and the parity rules.
Optional — deterministic extraction (Approach A, monorepo only): instead of extracting by hand you can call the production extract libs in packages/adapters/src/extract/ — extractFromBody, extractWithTools, runDirectFetchExtraction. ExtractDeps.cloudflare is nullable and repo can be a noop, so they run with just an Anthropic key (proof: scripts/smoke-toolloop.ts). This trades agent tokens for direct Anthropic-API tokens but reproduces the production tool-loop exactly. These packages are private: true — reachable from the monorepo only, never the thin CLI. The default remains Approach B (the agent is the extractor).
Step 5 — Write (batch upsert)
POST records to the batch endpoint in chunks (~50/request keeps requests bounded; the handler does the D1 bind-limit chunking server-side):
curl -sS -X POST "$RELEASES_API_URL/v1/orgs/<orgSlug>/sources/<sourceSlug>/releases/batch" \
-H "Authorization: Bearer $RELEASES_API_KEY" \
-H "Content-Type: application/json" \
-d @chunk.json
Per-release fields (the real accepted shape — workers/api/src/routes/sources.ts, postReleasesBatchHandler):
{
"version": "1.4.0",
"title": "Dark mode",
"content": "Markdown body…",
"url": "https://…/changelog/dark-mode",
"contentHash": "…",
"publishedAt": "2026-05-30",
"media": "https://…/shot.png",
"type": "feature",
"prerelease": false,
}
- Idempotent. Upserts on
UNIQUE(source_id, url), fill-don't-clobber: on collision it only backfills content/media when the stored row is empty, so a re-run never overwrites a good row. Safe to re-run and to page — same url updates rather than duplicates.
- Enrichment re-POST (
mode: "upsert-content"). Because the default is fill-only, re-POSTing richer content for a URL that already has content is a no-op — the stub stays. For a deliberate second pass (seed index summaries first, then full detail-page bodies + media), add "mode": "upsert-content" at the top level of the batch body: it overwrites content/media on a same-URL collision and skips the scrape title-dedup pre-filter (#1526). Use it only for intentional enrichment; the default path stays fill-only so a routine re-fetch can't clobber. The url must be identical to the seeded row — enrich keys on URL, so a changed URL scheme inserts a duplicate instead. If the URL scheme changed, re-seed instead: releases release delete --source <src_…> --hard then a fresh /batch (default mode).
- Auth. Requires
write scope; the static RELEASES_API_KEY (root) satisfies it. .env autoloads RELEASES_API_URL + RELEASES_API_KEY in this repo.
- No AI on insert (see Cost contract).
Chunking helper (reads a records.json array, posts in batches of 50):
bun -e '
const orgSlug = process.argv[1], sourceSlug = process.argv[2], file = process.argv[3];
const all = JSON.parse(await Bun.file(file).text());
const base = process.env.RELEASES_API_URL, key = process.env.RELEASES_API_KEY;
for (let i = 0; i < all.length; i += 50) {
const releases = all.slice(i, i + 50);
const res = await fetch(`${base}/v1/orgs/${orgSlug}/sources/${sourceSlug}/releases/batch`, {
method: "POST",
headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json" },
body: JSON.stringify({ releases }),
});
console.log(`chunk ${i / 50}: ${res.status} ${(await res.text()).slice(0, 200)}`);
}
' <orgSlug> <sourceSlug> records.json
Step 6 — Validate
releases tail <slug> --json
Confirm releases have non-empty titles, dates, content, and media. If a row is empty, fix the extraction and re-run — the default upsert backfills empty rows safely. If a row already has thin-but-non-empty content (e.g. an index-summary stub you now want to replace with the full body), a plain re-run is a no-op — use the mode: "upsert-content" enrichment re-POST from Step 5.
Step 7 — Playbook note
Record how this source was ingested so future agents don't re-derive it. Via manage_playbook (action update_notes) or releases admin playbook <org> --notes-file -. Note that the source was locally ingested (not on the cron path yet), its observed cadence, and any extraction quirks. Follow the managing-sources playbook-authoring rubric (imperative voice, keep-test, ### Fetch instructions / ### Traps / ### Coverage). Don't restate metadata the header already carries.
Parity requirements
Local output must match production shape so the source can later be picked up by the normal pipeline with no cleanup:
| Axis | Rule |
|---|
type | feature vs rollup per parsing-changelogs → Classifying Rollups. Leave unset/feature by default. |
version | Real version string, or omit. Pre-clean <UNKNOWN> / n/a placeholders — the batch path does not strip them (the single-insert path does, server-side). |
publishedAt | ISO-8601. Approximate from month/quarter/year headings rather than omit (parsing-changelogs → Dates): month → first of month, etc. |
media | Unwrap _next/image / Vercel optimizer wrappers before storing — use normalizeMediaUrl from packages/rendering/src/media-url.ts. |
url | Always populate. Dedup relies on a stable per-release url; without it the upsert can't collapse re-runs. |
Manual enrichment (out of scope for /batch, but available)
The batch path intentionally omits title_generated / title_short / summary and org overviews. When an operator wants them, set them deliberately, case by case:
- Per-release override:
PATCH /v1/releases/:id accepts summary, titleGenerated, titleShort, composition, version, … and re-embeds when content/title/AI-fields change.
- At insert time (alternative to
/batch): the single-release insert POST /v1/sources/:slug/releases accepts summary? / titleGenerated? / titleShort? — use this path instead of /batch when you want enrichment set in the same write. It also strips <UNKNOWN>/n/a versions server-side.
- Bulk AI enrichment after the fact:
bun scripts/generate-release-content.ts --orgs=<slug>.
Common mistakes
- Skipping the preflight. It's the first step for a reason — silently ingesting an opt-out source is the exact failure this skill exists to prevent.
conductor.build must be refused.
- Silent truncation. Capping the window is fine; not telling the user what you skipped is not.
- Missing
url. Breaks idempotency — re-runs duplicate instead of upserting.
- Triggering a fetch on the new source.
releases admin source fetch / manage_source action fetch starts the remote update workflow (billed server-side extraction) — the cost you're avoiding. Create the source, then write via /batch.
- Fetching detail pages in the parent agent during fan-out. Delegate to sub-agents; keep the parent context clean.
- Using this for a clean feed source. Add it and let cron fetch — don't hand-extract what the feed adapter handles for free.
- Running it in a managed-agent session. Local Claude Code only.
Related skills
finding-changelogs — is there a feed/GitHub source that makes local-ingest unnecessary? Page-structure evaluation.managing-sources — create org/source, naming, primary, playbook authoring.parsing-changelogs — the extraction conventions you must match (type, dates, rollups).seeding-playbooks — the parallel Agent-tool dispatch + parent-saves pattern reused in Step 4.- superpowers:dispatching-parallel-agents — fan-out discipline for the many-pages case.
