ingest

name	ingest
description	Parse a directory of documents (PDF / DOCX / PPTX / HTML) into a queryable Wikify corpus. Use when the user has local files to turn into a corpus, or to finish an arxiv harvest. Wraps `wikify corpus build` and owns parser-backend choice (docling default, lite), the --out convention, partial-failure policy, and post-build health checks.
allowed-tools	Bash(wikify corpus *)

ingest

Turn a folder of documents into a queryable Wikify corpus. This is a thin, decision-light wrapper over wikify corpus build. Its one real job is choosing the parser backend and confirming the corpus is healthy before handing it to a build workflow.

Command

wikify corpus build <source> --out data/corpora/<name> \
  [--mode additive|sync] \
  [--parser default|lite|docling] \
  [--workers N] \
  [--openalex/--no-openalex] \
  [--allow-partial] \
  [--parse-timeout SECONDS]

<source> is a directory of documents (PDF / DOCX / PPTX / HTML).

When invoking through uv run for a long build, use uv run --no-sync wikify corpus build .... Plain uv run re-syncs the venv on every call; if a wikify.exe from a prior run is still alive it holds the console script and the sync aborts with os error 32 before the build starts. --no-sync skips that mutation. If a build was killed, a stale wikify.exe/python worker may also be holding the GPU or the .ingest.lock -- the lock is now reclaimed automatically when its owning pid is dead, but kill orphaned workers before relaunching.

--parse-timeout (default 1800s) caps each file on the GPU subprocess-batched backend; a wedged parse (a giant scanned book stuck in OCR) is killed and counted as a parse failure, recoverable under --allow-partial.

GPU embeddings (opt-in)

The default onnxruntime is the CPU build, so the embedding stage runs on CPU (~5/s). The GPU variant cannot be a default dependency -- the onnxruntime-gpu Linux wheel hard-links the CUDA runtime at import and breaks CPU/CI hosts. On a CUDA box, enable GPU embeddings after uv sync by reinstalling the GPU variants in place: uv pip install --reinstall onnxruntime-gpu fastembed-gpu.

They replace the onnxruntime / fastembed modules in place; embedding._onnx_providers then selects the CUDAExecutionProvider automatically (cuDNN is preloaded from torch's bundled libs). Needs a CUDA runtime matching the onnxruntime-gpu major version on the system path. Embedding throughput rises ~50-100x (jina-v2-small ~5/s CPU -> ~300-560/s CUDA). A later uv sync reverts to CPU; re-run the reinstall.

Parser-backend choice

• default (= docling): Docling for every format. The first run downloads the Granite-Docling-258M formula model (~258 MB) plus the layout / table models; after the cache is warm the median PDF parses in ~10 s on an Ampere GPU. Pick this when equation/table fidelity matters.
• lite: CI / low-resource backend (pymupdf4llm + python-docx + python-pptx + trafilatura, no models). Use it where the model downloads are unaffordable.

(default and docling select the same backend.)

Mode

• --mode additive (default): keep documents already in the corpus and add the new ones. Use for incremental growth.
• --mode sync: drop corpus documents no longer present under <source>. Use when <source> is the authoritative set.

Citation enrichment / network

--openalex is ON by default: the refresh stage issues network requests to api.openalex.org to canonicalise bibliography metadata and surface in-corpus citation matches. Set OPENALEX_EMAIL for the polite-pool rate limit (10 req/s). Pass --no-openalex to skip that stage and run fully offline (air-gapped or no-network builds).

Output convention

Always write to --out data/corpora/<name>, never build/<name>.

Partial-failure policy

Default is OFF: any per-file parse failure aborts the whole build with exit 5. Pass --allow-partial to continue past failures and recover the successful papers; a later re-run picks up the files that failed.

Post-build health check

wikify corpus check data/corpora/<name>

The corpus dir is a positional argument (not --out); when omitted it falls back to WIKIFY_CORPUS or the cwd (see docs/filesystem-state-design.md, "Corpus build/read commands"). Confirm the doc / chunk / embedding / edge counts look right before declaring the corpus ready.

Rebuild surface

For re-deriving artifacts on an existing corpus without a full rebuild, corpus rechunk and corpus refresh are available. Mention them; do not expand on them here.

References

• ../wikify/subskills/reference/references/cli/grammar.md
• ../wikify/subskills/reference/references/cli/output-contract.md
• ../wikify/subskills/reference/references/cli/exit-codes.md