hiivmind-corpus-refresh-headless

name	hiivmind-corpus-refresh-headless
description	Headless (non-interactive) variant of hiivmind-corpus-refresh for use by automated pipelines and scheduled tasks. Always runs in update mode, selects all stale sources automatically, and emits a machine-readable YAML result block instead of prose. Use hiivmind-corpus-refresh for interactive use.

Corpus Refresh (Headless)

Non-interactive corpus refresh for automated pipelines. Always updates all stale sources. Emits a structured YAML result block on completion — no prose summaries, no prompts.

Follows the same source-type patterns as hiivmind-corpus-refresh. Refer to pattern documentation for source-type-specific logic.

---

State

computed:
  corpus_root: null
  config: null
  sources: []
  index_format: null           # "v1" | "v2"
  status_report: []            # per-source { id, type, status, old_sha, new_sha }
                               # status: current | updated | failed | skipped-manual
  updated_sources: []
  index_changes:
    added: 0
    modified: 0
    removed: 0
    stale_entries: []
  embedding_status: null       # "updated" | "skipped" | "no-model" | "not-installed" | "deferred"
  errors: []

---

Phase 1: Validate

Outputs: computed.config, computed.sources, computed.index_format

Resolve corpus root from corpus_path input or cwd. Read and parse config.yaml. Abort (emit result with error) if any of these fail:

• config.yaml missing
• No sources in config.sources
• index.md is a placeholder (not built)

Detect index format: index.yaml exists → v2, otherwise v1.

v1 is read-only as of this release. If the corpus is v1, do not refresh it. Write the result file with errors: ["v1-index: read-only — run hiivmind-corpus-migrate"], every source status: skipped-manual, zero index_changes, embeddings: skipped, then ABORT. (Contract unchanged — this reuses existing fields; a pipeline sees a clean "nothing done, migration required" result rather than a crash.)

---

Phase 2: Check Freshness

Outputs: computed.status_report

Check each source for upstream changes. Run in parallel (Task agents) where possible. Per-source logic by type:

Type	Check	Pattern doc
git	git ls-remote SHA vs last_commit_sha	patterns/sources/git.md
local	Always "skipped-manual" (no auto-detection — needs interactive refresh)	patterns/sources/local.md
web	Cache age > 7 days → stale	patterns/sources/web.md
llms-txt	SHA-256 of fetched manifest vs manifest.last_hash	patterns/sources/llms-txt.md
generated-docs	Same as git, using source_repo	patterns/sources/generated-docs.md
obsidian	Same as git when the vault is a clone (.source/{id}/); timestamp scan for a direct local path	patterns/sources/obsidian.md
self	git log -1 --format=%H -- {docs_root} vs last_commit_sha	patterns/sources/self.md

If all sources are current: emit result with no changes and exit.

---

Phase 3: Update Sources

Outputs: computed.updated_sources, file-level changes per source

Update every source with status != "current". Wrap each source in error handling — a failure marks that source status: failed and logs to computed.errors, but the run continues for other sources.

Git source update

${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/sources/git.md:

Clone to .source/{id} if not present (depth 50; for large repos prefer the sparse blob-filtered clone — see "Sparse Checkout for Large Repositories" in patterns/sources/git.md). Never use the GitHub compare API to list changed files: it caps at 300 files and silently truncates. Then ensure the tracked SHA is reachable for diffing:

DEEPEN_IF_NEEDED(clone_dir, source):
  IF repo is shallow:
    IF cat-file cannot resolve source.last_commit_sha:
      fetch --shallow-since={source.last_indexed_at} origin {source.branch}
      IF still cannot resolve:
        fetch --unshallow origin {source.branch}

Pull latest. Diff last_commit_sha..HEAD under docs_root to get changed files with their status (A/M/D).

Other source types

• Local: scan uploads/{id}/ for files newer than last_indexed_at
• Web: re-fetch all URLs to .cache/web/{id}/ — see patterns/sources/web.md
• llms-txt: re-fetch manifest, diff pages, re-cache changes — see patterns/sources/llms-txt.md
• Generated-docs: same shallow-clone strategy as git, using source_repo
• Obsidian: same as git when the vault is a clone (.source/{id}/), or a timestamp scan for a direct local path — see patterns/sources/obsidian.md
• Self: git diff --name-status {old_sha}..{new_sha} -- {docs_root}, filtered by include_patterns

---

Phase 4: Apply Changes to Index

Outputs: computed.index_changes

Apply every change per ${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/index-updating.md: v2 → stale-marking rules (M/A/D); v1 → direct entry edits with real titles extracted from .source/ (single or tiered). Then update config metadata per the same pattern. Track counts into computed.index_changes.

Placement of added entries follows config.build (see patterns/index-updating.md) — skip_sections exclusions are logged in the result file's errors[] as "excluded-by-config: {path}" (informational, not a failure).

---

Phase 5: Embeddings

Outputs: computed.embedding_status

${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/embeddings.md:

Skip if index-embeddings.lance/ doesn't exist, or if no entries were added/modified.

If computed.index_changes.stale_entries is non-empty, set computed.embedding_status = "deferred" and skip embedding entirely — stale entries have placeholder or outdated summaries, and embedding them would poison semantic search. The enrichment stage (hiivmind-corpus-enrich-headless) re-embeds after regenerating summaries.

Otherwise run detect.py: if "ready", run uv run ${CLAUDE_PLUGIN_ROOT}/lib/corpus/scripts/embed.py index.yaml index-embeddings.lance/ (incremental upsert). If "no-model", skip — never trigger a download during automated refresh. If not installed, skip.

---

Output Contract

See: ${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/headless-contract.md

1. Resolve result_path (input, default {corpus_root}/refresh-result.yaml).
2. Ensure the corpus .gitignore contains lines refresh-result.yaml and enrich-result.yaml — append them if missing (create .gitignore if absent).
3. Write the result file with the Write tool:

contract_version: 1
kind: refresh
corpus: {name from config}
run_at: {ISO timestamp}
sources:
  - id: {source id}
    type: {source type}
    status: current | updated | failed | skipped-manual
    old_sha: {short sha or null}
    new_sha: {short sha or null}
    files_changed: {count}
index_changes:
  added: {n}
  modified: {n}
  removed: {n}
  stale_entries:
    - {entry id}
embeddings: updated | skipped | no-model | not-installed  
 {  }   

   {}

Compute embeddings_lag post-Phase-5 the same way as status (patterns/embeddings.md § Embedding Lag): null when no index-embeddings.lance/ exists. Omitting the field is valid — the scheduler PR body reports it only when present and > 0.

4. Echo the same YAML between ---headless-result and --- markers as the final output — this is a human-readable log convenience only; pipelines MUST read the file, not parse prose.

Write the file even on partial failure or early exit (all-current, validation abort): a missing result file is indistinguishable from a crashed run. A source that fails is status: failed with the error in errors[]; other sources proceed normally.

---

Next Stage: Enrichment

This skill intentionally leaves changed entries stale: true with placeholder metadata — it detects and records change; it does not interpret content. When the result file shows index_changes.stale_entries non-empty (and embeddings: deferred), the pipeline MUST follow with:

CALL_SKILL("hiivmind-corpus:hiivmind-corpus-enrich-headless", { corpus_path })

before committing, so the branch/PR contains a complete refresh: regenerated summaries, concept assignments, verification results, and current embeddings. Standalone/manual runs may skip enrichment, but the stale flags then persist until enrich-headless or a rebuild runs.

---

Error Handling

Error	Behaviour
No config.yaml / no sources / index not built	Abort: write result file with error, emit log block, exit
Clone/fetch failed	Source marked failed, logged in errors[], continue
Shallow deepen failed	Fallback: --shallow-since → full unshallow → fail source
Index update failed	Logged in errors[], partial changes saved
Config save failed	Logged in errors[]

---

Pattern Documentation

• Git sources: ${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/sources/git.md
• Local sources: ${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/sources/local.md
• Web sources: ${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/sources/web.md
• llms-txt sources: ${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/sources/llms-txt.md
• Generated docs: ${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/sources/generated-docs.md
• Self sources: ${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/sources/self.md
• Index v2 schema: ${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/index-format-v2.md
• Index updating: ${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/index-updating.md
• Freshness checks: ${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/freshness.md
• Index rendering: ${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/index-rendering.md
• Embeddings: ${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/embeddings.md
• Result contract: ${CLAUDE_PLUGIN_ROOT}/lib/corpus/patterns/headless-contract.md

Related Skills

• Migrate v1→v2 (headless): ${CLAUDE_PLUGIN_ROOT}/skills/hiivmind-corpus-migrate/SKILL.md
• Headless status (pipelines): ${CLAUDE_PLUGIN_ROOT}/skills/hiivmind-corpus-status-headless/SKILL.md
• Headless rebuild (pipelines): ${CLAUDE_PLUGIN_ROOT}/skills/hiivmind-corpus-build-headless/SKILL.md
• Enrichment (run after refresh): ${CLAUDE_PLUGIN_ROOT}/skills/hiivmind-corpus-enrich-headless/SKILL.md
• Interactive refresh: ${CLAUDE_PLUGIN_ROOT}/skills/hiivmind-corpus-refresh/SKILL.md
• Build index: ${CLAUDE_PLUGIN_ROOT}/skills/hiivmind-corpus-build/SKILL.md