xberg-picking-a-format

<!-- AI-RULEZ :: GENERATED FILE — DO NOT EDIT Content-Hash: blake3:a2f204c34c2eb303812f0b7b0a9acf29c879edd1985b8cc3312555b347378e1e Source-Hash: blake3:4a90223a333b0f963ae8cdec4f38db88b769132c5dd1b7f39a7fe8e84f57f83e Schema-Version: v1 --> # Picking a format Xberg has two orthogonal format knobs. Get them right up front and the downstream code stays simple. | Knob | What it controls | Values | Default | | ------------------- | ------------------------------------------------- | -------------------------------------- | ---------------- | | `--format` | How the CLI prints the result | `text`, `json`, `toon` | `text` (`extract`), `json` (`batch`) | | `--content-format` | How extracted content is rendered inside `result` | `plain`, `markdown`, `djot`, `html`, `json`, `doctags` | `plain` | | `--token-reduction` | Strip whitespace / boilerplate for LLM contexts | `off`, `light`, `moderate`, `aggressive`, `maximum` | `off` | `--format json` returns an envelope wrapping the `ExtractedDocument` — the document lives under `.result` for `extract` and under `.results[]` for `batch`, with `content`, `metadata`, `tables`, and `images` as fields of that nested document. `--format text` prints just `content`. `--content-format` is what shows up inside that `content` field. ## Decision tree ```text Who consumes the output? ├── LLM (Claude, GPT, Gemini, local) — embed/prompt context │ --format text --content-format markdown ├── Vector store / RAG indexer │ --format json --content-format markdown │ (markdown preserves structure for chunking) ├── Downstream parser that expects machine-readable JSON │ --format json --content-format plain │ (cleanest text + structured metadata) ├── Human review / archival │ --format text --content-format markdown ├── HTML re-rendering / web display │ --format json --content-format html ├── Lossless intermediate for pandoc / academic tooling │ --format json --content-format djot └── Token-budget-constrained pipeline --format text --content-format plain (drops markup; add --token-reduction moderate for further savings) ``` ## Examples Feed a PDF directly into an LLM: ```bash xberg extract paper.pdf --content-format markdown ``` Index a corpus into a RAG store with tables and headings preserved: ```bash xberg batch docs/*.pdf --format json --content-format markdown \ | jq -c '.result | {content: .content, tables: .tables}' ``` Strip a file to bare text for a token-tight summarizer: ```bash xberg extract long.pdf \ --content-format plain \ --token-reduction moderate ``` Pull metadata only, ignore content: ```bash xberg extract file.pdf --format json | jq '.result.metadata' ``` ## When in doubt - **Default to `markdown`** as the content format. It is the best compromise across LLMs, RAG, and human review, and Xberg has the most faithful renderer for it. - Reach for `plain` only when downstream cannot tolerate any markup. - Reach for `djot` only if you're already in a djot/pandoc pipeline. - Reach for `html` only when re-rendering for the web. - Reach for `json` for a heading-driven content tree, or `doctags` for Docling-compatible output. ## Token-reduction (orthogonal) `--token-reduction` collapses whitespace, strips repeated headers/footers, and trims boilerplate. It composes with any `--content-format`: - `off` (default), `light`, `moderate`, `aggressive`, `maximum`. Use `moderate` as a safe starting point for LLM context windows. `maximum` is lossy — verify before relying on it. See `references/cli-reference.md` for the full flag set and `references/configuration.md` for the equivalent `output_format` and `token_reduction` keys in `xberg.toml`.