vaultli

name

vaultli

description

Manage a file-based knowledge vault with YAML frontmatter, sidecar markdown, and JSONL indexing. Use when initializing or maintaining a knowledge base, attaching metadata to markdown or non-markdown assets, rebuilding or validating INDEX.jsonl, or searching vault records by metadata. Trigger on: "knowledge base", "vault", "frontmatter", "sidecar", "INDEX.jsonl", "document this query/template/skill", "search the vault", or "set up KB docs".

vaultli

Use vaultli when the job is to make a repository's knowledge assets agent-discoverable. It is strongest at standardizing metadata, creating sidecars, building INDEX.jsonl, and validating vault integrity.

What vaultli does

vaultli turns a directory of markdown files, queries, templates, runbooks, and other assets into a structured file-based knowledge base.

• Native markdown files keep metadata inline in YAML frontmatter.
• Non-markdown assets get same-directory sidecars such as report.sql.md.
• Indexed records are written to INDEX.jsonl for fast lookup and filtering.
• Validation checks help catch stale, broken, or internally inconsistent vault state.

The source of truth is the files on disk. INDEX.jsonl is a derived cache.

What This Skill Is Not

vaultli is not a full-text search engine or vector database.

• search and show operate on INDEX.jsonl.
• Search quality depends on frontmatter, especially description, tags, and category.
• After a hit is found, use resolve, cat, or context to hydrate real file content.
• For sidecars, you often need both the .md sidecar and the source asset.

Default Invocation

Prefer the Rust binary — it is at behavioral parity with the Python reference (verified by cross-language parity tests) and starts an order of magnitude faster, which matters in tight agent loops:

# one-time build from the vaultli directory
cd <vaultli>/rs && cargo build --release

# then invoke the binary directly (or put it on your PATH)
<vaultli>/rs/target/release/vaultli --json <command> ...

<vaultli> is wherever this package lives; the binary has no other install-time dependencies and the directory can be relocated freely.

Fall back to the Python CLI only when the Rust binary is unavailable (no Rust toolchain, or debugging a suspected Rust-specific bug):

uv run python -m tools.vaultli ...

Both implementations accept the same subcommands and flags. Prefer --json in agent flows so results are machine-readable. Use --root to target the vault explicitly instead of depending on the current working directory.

Area	Rust	Python
Default agent choice	Use this first	Use as fallback
Best for	Fast repeated CLI calls	Debugging and reference reads
Parity role	Checked against Python	Oracle for Rust parity tests
Expected behavior	Same user-visible commands and JSON envelopes	Same user-visible commands and JSON envelopes

Stability Boundaries

Treat the flat-file lifecycle as stable: init, add, scaffold, ingest, index, validate, search, show, resolve, cat, set, unset, and refresh.

Treat search --semantic, context, federated-search, and git-info as implemented but still experimental helper surfaces. They are safe to use, but a new agent should prefer explicit JSON inspection and validation around them instead of assuming they are a full vector store, orchestration engine, or git history system.

Before claiming compatibility or release readiness, run:

UV_CACHE_DIR=/tmp/uv-cache uv run pytest tests/test_vaultli.py
cd tools/vaultli/rs && cargo test

Core model

vaultli is built around three simple ideas:

1. Markdown is the universal knowledge wrapper.
2. YAML frontmatter is the universal metadata format.
3. JSONL is the universal index format.

A typical vault looks like this:

kb/
  .kbroot
  INDEX.jsonl
  docs/
    guide.md
  queries/
    retention.sql
    retention.sql.md
  templates/
    campaign_report.j2
    campaign_report.j2.md

Core Workflow

1. Find or confirm the vault root.

vaultli --json root .

2. Create the vault only if .kbroot is missing.

vaultli --json init ./kb

3. Choose the right write path.

Need	Use
Add frontmatter to an existing markdown file and index it right away	add <file>
Create a sidecar for a non-markdown asset without indexing yet	scaffold <file>
Bulk scaffold missing metadata for a file or directory	ingest <path>
Preview a bulk ingest without writing files	ingest <path> --dry-run
Limit bulk ingest to safe slices	ingest <path> --include 'queries/*.sql' --exclude 'queries/tmp*'
Preview the generated metadata before writing anything	infer <file>
Edit one frontmatter field by path or id	set <target> <field> <value>
Remove one frontmatter field by path or id	unset <target> <field>
Refresh inferred metadata while preserving body content	refresh <target>
Rebuild cache state after edits	index
Audit the vault for broken links, duplicate IDs, and stale index state	validate
Look up candidate records by metadata	search, then resolve or cat; narrow with filters, --sort, --explain, or experimental --semantic
Assemble content for an answer	context [query] or context --id <id>
Search multiple vaults	federated-search --vault <root> --vault <other>
Inspect git state for a vault item	git-info [target]

4. Immediately refine generated metadata.

The inferred metadata is a draft. A new agent should treat the generated values as placeholders:

• rewrite description to be retrieval-friendly and specific
• tighten tags
• correct category if the inference is too generic
• add depends_on and related only when the IDs are real
• flesh out the markdown body so humans and later agents have context

5. Rebuild and validate.

vaultli --json index --root ./kb
vaultli --json validate --root ./kb

6. Retrieve in two stages.

vaultli --json search retention --root ./kb
vaultli --json search --root ./kb --category query --tag retention --sort priority --limit 5
vaultli --json search "retention query" --root ./kb --semantic --explain
vaultli --json resolve queries/retention --root ./kb --body --source
vaultli cat queries/retention --root ./kb --source
vaultli --json context --root ./kb --id queries/retention --token-budget 2000

Use show when you only need metadata. Use resolve when you need file paths and optional body/source content. Use cat when the next step needs raw text. Use context when answering a question and you want dependencies packed into a single JSON bundle.

Sidecar rules

For non-markdown files, vaultli uses same-directory sidecars:

Source file	Sidecar
report.sql	report.sql.md
template.j2	template.j2.md
config.yaml	config.yaml.md

The sidecar must include a valid source field, usually:

source: ./report.sql

Rules That Prevent Bad KBs

• Never edit INDEX.jsonl directly.
• Native markdown gets inline frontmatter.
• Non-markdown files need same-directory sidecars named <file>.<ext>.md.
• Sidecars must have a valid relative source, typically ./filename.ext.
• Non-markdown assets are not searchable until a sidecar exists.
• validate reports problems; it does not repair them.
• Prefer search filters over search --jq for common metadata fields.
• Treat search --semantic as experimental token-overlap matching, not vector retrieval.
• search --jq requires the jq binary to be installed.
• Sidecar hash changes come from the source asset bytes, not from sidecar prose edits.
• Use ingest --dry-run before bulk writes, and add --include/--exclude globs on large trees.
• Use set, unset, and refresh instead of hand-editing frontmatter when making simple metadata changes.

A Safe Default Loop

vaultli --json root .
vaultli --json add ./kb/docs/guide.md --root ./kb
vaultli --json scaffold ./kb/queries/report.sql --root ./kb
vaultli --json ingest ./kb --root ./kb --dry-run
vaultli --json ingest ./kb --root ./kb --dry-run --include 'queries/*.sql'
# edit the generated markdown files
vaultli --json index --root ./kb
vaultli --json validate --root ./kb
vaultli --json search report --root ./kb
vaultli --json resolve queries/report --root ./kb --body --source

When A New Agent Usually Gets Confused

• Expecting body text to be searchable without opening files.
• Assuming a source file is indexed just because it exists in the vault.
• Leaving the generic inferred description untouched.
• Forgetting to re-run index after metadata edits.
• Treating validate output as if it were self-healing.
• Reaching for Python by habit even when the Rust binary is already built.
• Opening source paths manually when resolve --source or cat --source would avoid mistakes.
• Running broad ingest on a large tree without --dry-run and include/exclude globs.

Related docs

• README.md explains what vaultli is and how the core model works.
• vaultli-spec-v1.0.md defines the storage layout and metadata schema.
• rs/ is the primary implementation (Rust); build with cargo build --release and invoke rs/target/release/vaultli.
• py/core.py is the Python reference implementation, still kept in sync and used as the parity oracle by rs/tests/parity.rs.