qi-cli

qi is a local-first knowledge search CLI. It indexes documents into SQLite and supports BM25 full-text search and vector/hybrid search (with a local or remote embedding provider). ## Quick Start ```bash qi init # Create config + database (config lists a sample ~/notes collection) qi index ~/notes # Index a directory; it is saved to config as a collection qi doctor # Verify setup qi search "your query" # BM25 keyword search (no provider needed) qi query "your semantic question" # Hybrid search; keyword-only, with a warning, if no provider is configured ``` `index`, `search`, `query`, `get`, `list`, `stats`, `delete` and `hook` need a config file, so run `qi init` first. `qi update`, `--version` and `--help` work without one, and `qi doctor` reports a missing file as a failed check. --- ## Commands ### `qi init` Writes `~/.config/qi/config.yaml` (if absent) and initializes the SQLite database. ```bash qi init ``` ### `qi index [path|collection]...` Indexes documents. A file whose size and mtime match the last run is skipped without being read; otherwise SHA-256 content hashing decides whether it changed. Files over 10 MiB and binary files (a NUL byte in the first 8000 bytes) are skipped and counted in `skipped`, as are hidden directories, common tool directories (`node_modules`, `vendor`, `dist`, `build`, …) and symlinks. Documents whose files are gone are deleted from the database in the same run. A second `qi index` against the same database waits for the first to finish rather than failing. Use `--force` to reprocess unchanged files, e.g. after a qi upgrade changes parsing. Without an `extensions` list, a collection indexes only `.md`, `.markdown`, `.txt` and `.text`. List other formats explicitly: `.mdx` (Markdown; MDX `import`/`export` and JSX tag lines are ignored), `.rst` (reStructuredText) and `.adoc`/`.asciidoc` (AsciiDoc) are split at section titles into heading paths; any other listed extension is plain text. Embeddings are generated at index time. If you add an embedding provider after an initial index run, or change its `model`, `dimension`, `base_url` or `max_input_chars`, re-run `qi index` to embed what is missing or stale. An argument starting with `/`, `./`, `../` or `~` (or `.`/`..`) is a path; anything else is a collection name. A new path is saved to config as a collection named after its own directory (`~/Projects/tools/qi` → `qi`); when two collections would share a name, each takes leading path segments until unique (`work-notes`, `personal-notes`). There is no `--name` flag. `qi list` shows current names. ```bash qi index # indexes current working directory qi index ~/notes # an absolute or ~ path qi index ./docs # a relative path needs ./ (plain "docs" is a collection name) qi index notes wiki # collections by name; several arguments are indexed in turn qi index --changed-since ORIG_HEAD . # no-op unless an indexed file changed since that git revision ``` ### `qi hook install [path|collection]` Installs `post-merge`, `post-rewrite` and `post-commit` git hooks in the repository holding the collection. After `git pull` (merge or rebase) or the commit that concludes a conflicted merge, every configured collection inside that repository is reindexed in the background with `qi index --changed-since <sha>`; pulls that touch no indexed file return immediately, and ordinary commits cost one git call. A squash pull leaves no base commit to diff against, so it runs a plain `qi index`, which still skips unchanged files by stat. Installing also registers the collection in config if it is new. Output goes to `qi-index.log` in the git directory. Respects `core.hooksPath`; reinstalling (for any collection in the same repo) rewrites the same hooks, and a hook qi did not write is left alone with the line to add printed instead. ```bash qi hook install # the current directory's repository qi hook install ~/src/monorepo/docs # a collection below the repository root ``` ### `qi search <query>` BM25 full-text search. Fast, offline, no provider needed. ```bash qi search "authentication" qi search "deploy" -c wiki -n 5 ``` ### `qi query <query>` Semantic/hybrid search. Falls back to BM25 and prints a warning on stderr when no embedding provider is configured, or when embedding the query or the vector search fails. ```bash qi query "how does auth work" qi query "deploy pipeline" --mode lexical # BM25 only qi query "deployment steps" --mode hybrid # BM25 + vector (default) qi query "critical path" --mode deep # alias for hybrid qi query "question" --explain # show BM25/vector/RRF score breakdown ``` **Modes:** - `lexical` — BM25 only - `hybrid` — BM25 + vector fused with RRF; skips vector if BM25 has a clear winner. Vector scoring covers only BM25's candidate documents (topped up with any-term matches), so a document sharing no word with the query surfaces only when BM25 finds nothing at all - `deep` — alias for `hybrid` With no `--mode`, the mode comes from `search.default_mode` in the config (`hybrid` by default). ### `qi get <id>` Retrieve the indexed source snapshot by the full `hash` returned in search JSON (or an unambiguous hex prefix). Use the hit's `start_line:end_line` for its source passage. An ambiguous prefix is an error; identical content at multiple paths is returned once, with other locations in `also_at`. ```bash qi get abc123 qi get abc123 --lines 40:80 # 1-indexed, inclusive; "40:" and ":80" also work qi get abc123 --max-bytes 4096 # truncate and say so (0 = unlimited) qi get abc123 --format json # indexed source body and citation metadata ``` Prefer `--lines`/`--max-bytes` over reading whole documents: they are what keeps a retrieved document from swallowing an agent's context. ### `qi list` List all collections defined in config (name and path). ```bash qi list ``` ### `qi delete <collection>` Delete a collection: removes all indexed data from the database and removes the collection entry from config. Irreversible. ```bash qi delete notes ``` ### `qi stats` Show document counts, chunk counts, embedding health, and database size per collection. If `Embeddings` reports missing or stale vectors, vector search lacks data for those chunks — re-run `qi index`. ### `qi doctor` Health-check config, database, collection paths, and embedding coverage (missing, stale, orphaned or corrupt vectors, with the command that repairs each). It does **not** contact the provider: a `SKIP` line means no embedding provider is configured, not that one is unreachable. When none is configured, `qi query` falls back to BM25 with a warning. ### `qi update` Update the binary in place from the latest GitHub release. --- ## Global Flags | Flag | Description | |---|---| | `--verbose` | Log debug detail to stderr (no `-v` shorthand; `-v` is `--version` on `qi` itself) | | `-f, --format text\|json\|markdown` | Output format (default: text) | | `--config <path>` | Override config path | ## `search` and `query` Flags These are per-command, not global. | Flag | Description | |---|---| | `-c, --collection <name>` | Limit to a specific collection | | `-n, --limit <N>` | Number of results (default: 10) | | `--passages <N>` | Additional supporting passages per document, 0–5 (default: 0) | | `--since YYYY-MM-DD` | Only documents dated on or after this day | | `--until YYYY-MM-DD` | Only documents dated on or before this day | | `--sort date` | Newest first instead of by relevance | | `--path <glob>` | Only documents whose collection-relative path matches the glob | Every document has a date. It comes from the frontmatter `timestamp:`, `date:` or `created:` when one of them parses, otherwise from the file's modification time — so no document is excluded from `--since`/`--until` for lack of a date. `--path` takes one glob in the same dialect as a collection's `ignore` list: it matches the path, its basename or any parent directory, so `meetings` and `meetings/*` both select `meetings/q3/retro.md` and `*.log.md` matches at any depth. Paths are relative to their collection, so without `-c` the glob is applied inside every collection. Scope is applied while candidates are still being ranked, so an in-scope hit is not lost behind out-of-scope ones. --- ## Configuration (`~/.config/qi/config.yaml`) ```yaml database_path: ~/.local/share/qi/qi.db collections: - name: notes # regenerated from the path on load path: ~/notes extensions: [.md, .txt] ignore: [drafts, "*.draft.md"] # path.Match globs on the relative path, basename or any parent dir - name: wiki path: ~/wiki extensions: [.md, .mdx, .rst, .adoc] providers: embedding: # optional — enables vector/hybrid search base_url: http://localhost:11434 # Ollama, LM Studio, llama.cpp, OpenAI-compatible model: nomic-embed-text dimension: 768 max_input_chars: 0 # optional — truncate long texts before embedding (0 = no limit) search: default_mode: hybrid # lexical | hybrid (deep is an alias for hybrid) bm25_top_k: 50 vector_top_k: 50 rrf_k: 60 chunk_size: 512 prefer_extensions: [.md, .txt] # optional — boost results with these extensions extension_boost: 2.0 # optional — score multiplier for preferred extensions (default 2.0) ``` `base_url` and `api_key` support `${VAR}` environment-variable expansion, and a trailing `/v1` in `base_url` is normalized automatically (no doubled `/v1/v1`). Relative paths resolve against the config file's directory. `.git`, `node_modules` and other tool directories are always skipped, so they need no `ignore` entry. Changing `model`, `dimension`, `base_url`, `max_input_chars` or the provider `name` makes existing vectors stale until the next `qi index` re-embeds them; `api_key` and `batch_size` (default 32) do not. ### Common provider setups **Ollama (fully local):** ```yaml providers: embedding: base_url: http://localhost:11434 model: nomic-embed-text dimension: 768 ``` **OpenAI:** ```yaml providers: embedding: name: openai # fills in base_url and api_key (from OPENAI_API_KEY) when left empty model: text-embedding-3-small dimension: 1536 ``` --- ## How search works - **BM25** — SQLite FTS5. Always available, very fast, good for keyword queries. - **Vector** — Cosine similarity, computed in Go, over embedding BLOBs stored in SQLite. Requires an embedding provider. Captures semantic intent. Used only inside hybrid mode, where it scores the chunks of BM25's candidate documents, and scans every embedded chunk only when BM25 finds nothing. - **Hybrid (RRF)** — Runs both, fuses rankings with Reciprocal Rank Fusion (`score = Σ 1/(k + rank)`), counting each document once at its best rank. Skips vector if BM25 has a dominant winner (top score > 3× second place). - Results are one per document: a document is represented by its best chunk. - **Deep** — Accepted as an alias of Hybrid. --- ## Document references Search JSON includes `hash` (full SHA-256 retrieval handle/source version), `source_uri` (collection/path identity), and `start_line`/`end_line` (1-indexed, inclusive raw-source range). Text/Markdown output includes a ready-to-run `qi get <hash> --lines A:B` command. Do not pass integer `doc_id` or `chunk_id` to `get`. Reuse the same `--config` override when retrieving a hit. Cite the source URI, hash, and line range together. `--passages N` optionally attaches bounded supporting passages that inherit the parent hash and URI; it does not add ranking votes or result slots for the document. `get` returns the indexed snapshot even if the file has since changed on disk. The hash is not a claim of live-file freshness. Reindexing can remove old versions; qi is not a revision archive. For identical-content paths, retain the search hit's URI rather than substituting get's first location. Search skips chunks with missing or invalid source ranges. Repair older indexes with `qi index <collection>`, or `qi index /path/to/directory` if the collection is no longer in config. --- ## Typical workflows **Index and search (no provider needed):** ```bash qi init qi index ~/notes # saves and indexes ~/notes as "notes" qi search "my keyword" -c notes ``` **Manage collections:** ```bash qi list # see all configured collections qi index ~/projects # add a new collection qi delete old-notes # remove collection data + config entry ``` **Semantic search with Ollama:** ```bash # pull a model: ollama pull nomic-embed-text # add embedding provider to config qi index notes qi query "how does X work" --explain ``` **Debug / inspect:** ```bash qi doctor # check config, database, and embedding coverage qi stats # see document/chunk/embedding counts qi query "question" --explain # see score breakdown qi get abc123 # read the full source document ``` --- ## Troubleshooting ### `qi query` returns keyword-only results despite embedding provider being configured Work through these steps in order: **1. Check if the provider is actually wired up** ```bash qi doctor ``` Look for an `embeddings:` line with `OK`. `SKIP embedding provider (not configured)` means qi found no provider block — common causes: wrong YAML key name (`embedding` not `embeddings`) or bad indentation (it must be under `providers:`). A missing or zero `dimension` is different: the config fails to load and every command reports the error. **2. Check if embeddings were actually generated** ```bash qi stats ``` If `Embeddings` reports missing or stale vectors, they were never written or no longer match the provider. This happens when the embedding provider was added to config *after* the initial `qi index` run, or when its model, dimension or URL changed — re-index to fix it: ```bash qi index <collection-name> ``` Only chunks missing embeddings are re-processed; unchanged files aren't re-parsed. **3. Verify the provider is reachable at query time** `qi query` embeds the query string at runtime. If the provider is down, qi falls back to BM25 and prints `embedding query failed, falling back to BM25` on stderr, but exits successfully. Test connectivity: ```bash # Ollama: curl http://localhost:11434/v1/embeddings -H "Content-Type: application/json" \ -d '{"model":"nomic-embed-text","input":["test"]}' ``` Also confirm the model is pulled: `ollama list`. **4. Use `--explain` to see what's actually happening** ```bash qi query "your question" --explain ``` This shows each result's BM25 rank, vector rank, and fused RRF score. If every result is missing a vector rank, the vector path isn't contributing — confirming one of the issues above. **Note:** hybrid mode also skips vector search if the top BM25 score is more than 3× the second result (intentional optimization for dominant keyword matches). If your queries are exact keyword matches, try a more paraphrased, natural-language query to avoid this shortcut. When BM25 finds at least one candidate, vector scoring covers only those candidates, so a document with no word in common with the query cannot be pulled in by meaning alone. When BM25 finds nothing, qi scans every embedded chunk, and such a document can surface.