NomaDamas/MinSync

NomaDamas-Profil

Letzte erfasste Quellaktivität: 19. September 2026 um 08:56

SkillsMP-Katalog aktualisiert: 17. Oktober 2026 um 00:09

gesammelte Skills: 1

GitHub-Stars: 57
GitHub-Forks: 7

README des Repositorys

README-Quelldatei: README.md

README-Quelldatei lesen

Vollständige README anzeigen

# MinSync [English](README.md) | [한국어](README.ko.md) ![MinSync turns changed files into searchable vector chunks](assets/minsync-flow.svg) MinSync is a manifest-based incremental vector database indexing CLI for text files. It does not need git: it tracks `mtime`, file size, and SHA-256 content hashes in `.minsync/`, re-embeds only changed chunks, sweeps stale chunks, and keeps a local LanceDB index ready for semantic search. ## Why MinSync - **Git-free change detection**: works in any directory, including generated workspaces and agent sandboxes. - **Incremental embeddings**: unchanged text is not chunked or embedded again. - **Crash-safe state**: cursor updates happen only after processing completes. - **Deterministic chunk IDs**: IDs derive from source, path, schema, content hash, and duplicate index. - **Rust native**: a single CLI with chonkie-core chunking built in. - **Text-only by design**: PDF, DOCX, XLSX, images, and other binary formats are treated as empty because MinSync does no extraction. Add them to `.minsyncignore`. ## Contents - [Install](#install) - [Usage](#usage) - [How it works](#how-it-works) - [Chunkers](#chunkers) - [Vector store](#vector-store) - [Embedding network reliability](#embedding-network-reliability) - [Local embeddings](#local-embeddings) - [.minsyncignore](#minsyncignore) - [Development](#development) ## Install Recommended install path: ```bash curl -fsSL https://raw.githubusercontent.com/NomaDamas/MinSync/main/scripts/install.sh | sh ``` The installer asks whether to star the repository with `gh repo star NomaDamas/MinSync`. If `gh` is unavailable or not authenticated, it continues without failing. Direct Cargo install: ```bash cargo install minsync ``` From source: ```bash git clone https://github.com/NomaDamas/MinSync.git cd MinSync cargo build --release ``` MinSync embeds in-process with [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) by default. No API key, TEI server, or Hugging Face gate is required. The first `minsync sync` downloads the model into the local cache. No model at all? Use the BM25-only mode, which needs no embedder, API key, model download or network: ```bash minsync init --embedder none minsync sync # chunks + full-text index only; nothing is embedded minsync query "exact terms" # --mode defaults to bm25 in this workspace ``` `--embedder none` stores a 1-float placeholder vector per chunk (the LanceDB schema requires the column) and never builds an ANN index. `--mode vector` and `--mode hybrid` fail with a pointer to `--mode bm25`; `minsync check` skips the embedder probe. To add vector search later, run `minsync init --force --embedder <id>` and re-sync. For OpenAI embeddings instead: ```bash export OPENAI_API_KEY="sk-..." minsync init --embedder openai:text-embedding-3-small ``` ## Quick Start ```bash minsync init # initialize .minsync/ minsync sync # index changed files incrementally minsync sync --full # rebuild from scratch minsync query "search text" --mode vector --k 5 minsync query "exact terms" --mode bm25 --k 5 minsync query "search text" --mode hybrid --k 5 minsync watch # initial sync, then re-index on file changes minsync watch --watch-on-sync-error # stay alive and retry after initial sync errors minsync status # sync state minsync check # health check; exit 5 if the embedder fails, 4 if the vector store fails (the report is still printed) minsync verify --fix # consistency check + repair minsync optimize # compact + re-index + prune the store ``` Select the BM25 tokenizer preset during initialization: ```bash minsync init --language ko ``` Supported values are `simple`, `ko`, `ja`, `zh`, `ar`, and `multilingual`. Korean uses Kiwi through the `kiwi-rs` Rust binding, Japanese uses an embedded Lindera dictionary, Chinese uses `jieba-rs`, and Arabic uses an in-process light stemmer based on Discrawl PR #180. Changing `[lexical].language` triggers a full rebuild on the next sync. `minsync watch` performs an initial sync before waiting for filesystem events. By default, an initial embedding or vector-store error stops the command, matching `minsync sync` fail-fast behavior. Use `--watch-on-sync-error` to keep the watcher alive and retry on later file events; failures remain visible in the log and the cursor advances only after a later sync succeeds. Sync output reports file-level changes separately from chunk-level storage effects: `files added/modified/deleted` describes source files, while `chunks inserted/reused/removed` describes content-addressed index rows. JSON sync output also reports `files_checked`, `elapsed_seconds`, `freshness_check_only`, and `query_ready`. An unchanged incremental sync uses a conservative metadata fingerprint to skip hashing unchanged files, then returns without reading, chunking, embedding, writing, flushing, or rebuilding index rows; its elapsed time therefore measures freshness-check cost. Changed and full syncs include indexing and LanceDB maintenance in their elapsed time. For a local, credential-free comparison, run `cargo run --quiet -- sync --format json` twice, edit one tracked text file and run it again, then run `cargo run --quiet -- sync --full --format json`. Query process startup is outside this sync measurement. Run the checked-in credential-free scale benchmark with: ```bash cargo test --test issue_41_benchmark -- --nocapture ``` It generates the same local corpus at 1x, 2x, 4x, and 8x, reports full, unchanged, and changed-file sync milliseconds, and reports p50/p95 for BM25, vector, and hybrid retrieval. The benchmark intentionally uses the in-memory store, so its query timings measure retrieval scaling separately from LanceDB index maintenance; the LanceDB ANN/FTS threshold and unindexed-delta behavior are covered by the live index tests. Timing values are observations, not a machine-specific latency SLO. Korean tokenization requires the official Kiwi native library and base model. Set `KIWI_LIBRARY_PATH` to `libkiwi` and `KIWI_MODEL_PATH` to the extracted `models/cong/base` directory. `kiwi-rs` (LGPL-2.1-or-later) is statically linked; see `THIRD_PARTY_NOTICES.md` for its notice and how to relink against a modified copy. On macOS/Linux, `bash scripts/install-kiwi.sh` downloads the matching official assets and prints the required environment variables. On Windows, run `powershell -ExecutionPolicy Bypass -File scripts/install-kiwi.ps1` and set the printed paths in the environment. The Windows CI uses the official Kiwi 0.22.2 assets until the remaining `kiwi_config_t` ABI differences in the Rust binding are resolved upstream. ## How It Works MinSync scans your directory, compares each text file to the manifest, chunks changed content once, and stores the same stable chunk IDs for vector and BM25 retrieval in LanceDB. Stale rows are removed by mark-and-sweep. Vector mode embeds the query, BM25 mode uses LanceDB full-text search without an embedding request, and hybrid mode combines both rankings with deterministic reciprocal rank fusion (RRF, `k=60`). The selected analyzer converts documents and queries into the shared `lexical_text` column before LanceDB BM25 indexing. State lives in `.minsync/`: | File | Purpose | |---|---| | `config.toml` | collection, chunker, embedder, vector store, normalization | | `manifest.json` | last known file metadata and content hashes | | `cursor.json` | last completed processing point | | `txn.json` | in-progress transaction marker | | `progress.jsonl` | files already committed by an interrupted sync (resume log); see [Incremental core](docs/INCREMENTAL_CORE.md) | | `lock` | process lock | Delete `.minsync/` to start fresh. ### Freshness checks After the first completed sync, MinSync uses the persisted file metadata fingerprint as a conservative fast path. On Unix this fingerprint includes size, modification time, change time, device, and inode, so unchanged files reuse their manifest hash without reading their contents. A replacement that keeps the same size and restores the modification time still changes the fingerprint through the filesystem change time and is rehashed. Manifests written by older versions, platforms without the required metadata, and explicit `minsync sync --full` scans conservatively hash file contents. The fast path is not a second freshness owner: the manifest remains the only source of content hashes, and a fingerprint mismatch always falls back to SHA-256. Filesystem writers that mutate content after metadata is observed remain subject to the normal concurrent-write race; run `--full` when an authoritative rescan is required. ### Unreadable entries A directory or file the scan cannot read (`EACCES`, `EPERM`, `EIO`, `EBUSY`, `ETIMEDOUT`, `ESTALE`, symlink loops, ...) no longer aborts the scan: it is skipped and reported in the sync summary's `scan_errors` list (`path`, `kind` = `transient` or `permanent`, `error`) with one warning per entry. A path that vanished mid-scan (`ENOENT`, `ENOTDIR`) is treated as deleted. `status` and `verify` apply the same rule, so `verify --fix` never removes chunks of an unreadable path. With `[sync] on_file_error = "skip"` (default) each unreadable entry is also listed in `files_failed` (stage `read`), so the run is `partial` and exits with code 6; with `"fail"` the sync aborts at the first one, naming its path. **Guarantee:** an entry that errored is *unknown*, never *deleted*. Its previously indexed chunks are kept, including during `--full` sweeps, until a later scan can read the path again (or confirms it is gone). If the root directory itself is unreadable the sync still fails, since nothing can be said about any file. ## Chunkers | id | Strategy | Boundary stability under edits | |---|---|---| | `recursive` (default) | paragraph to sentence to line splits merged to a size budget | an edit near the top of a file can shift downstream boundaries | | `chonkie` | delimiter/size-based chonkie-core chunking | same drift caveat as `recursive` | | `cdc` | content-defined chunking with FastCDC-style rolling hash | small edits usually affect only nearby chunks | The `recursive` chunker splits on paragraph breaks, sentence delimiters (`. `, `? `, `! `), and line breaks, then merges those pieces up to `chunker.options.max_chunk_size`. The configured size is a hard cap for ordinary delimiter-based text, for delimiter-less lines, and for table chunks: overlong content is split at character boundaries with a warning that includes the original length. Contiguous Markdown table lines (optional leading whitespace, beginning with `|` and containing another `|`) are treated as one table block. A valid table header and separator stay together with data rows, and table chunks are split only between rows when a row fits. When a table needs multiple chunks, every chunk after the first repeats the header and separator rows. If a single row plus its repeated header (or the header alone) exceeds the cap, that chunk is split at character boundaries so no chunk ever exceeds `max_chunk_size`; the split pieces are no longer valid table fragments. A warning is also emitted when a table boundary is selected. Select at init time: ```bash minsync init --chunker cdc ``` Or edit `.minsync/config.toml`: ```toml [chunker] id = "cdc" ``` Switching the chunker changes the chunk schema, so run `minsync sync --full` afterwards. ## Vector Store MinSync stores vectors in embedded LanceDB by default: ```toml [vectorstore] id = "lancedb" [vectorstore.options] dimension = 1024 index_build_threshold = 256 index_optimize_delta_threshold = 10000 ``` `minsync init` sets `dimension` automatically for known embedders. For other models, set `dimension` to match your embedder: | Embedder | Dimension | |---|---:| | `native:Qwen/Qwen3-Embedding-0.6B` (default) | 1024 | | `tei:google/embeddinggemma-300m` | 768 | | `openai:text-embedding-3-small` | 1536 | | `tei:intfloat/multilingual-e5-small` | 384 | | `tei:BAAI/bge-m3` | 1024 | MinSync builds an IVF-HNSW-SQ ANN index after `index_build_threshold` rows and incrementally optimizes unindexed deltas after `index_optimize_delta_threshold` rows. These thresholds are tuning knobs, not capacity limits. Agents adding another vector store should implement the `VectorStore` trait and wire the new id through `create_vectorstore`. See `docs/EXTENDING.md` and `skills/minsync/SKILL.md` for the agent-facing extension checklist. ### Write path, checkpoints and `optimize` A sync writes files in batches of `[sync] batch_files` (default 25). Per batch it does one id-only existence check, one embedding call, one metadata update for reused chunks, one append for new chunks and one stale-chunk sweep, so the commit count is independent of how many files are in the batch. It keeps BTree indexes on `id` and `path` so these filters stay index lookups, and every `[sync] checkpoint_every_files` files (default 50) compacts fragments (once there are at least `compact_fragment_threshold`, default 64) and prunes old versions. A running sync logs `sync progress` at each checkpoint. A crash redoes at most one batch. ```toml [sync] batch_files = 25 # files per store commit; 1 = per-file commits checkpoint_every_files = 50 # 0 disables mid-sync checkpoints max_file_bytes = 67108864 # 64 MiB per file; 0 disables the size guard [vectorstore.options] compact_fragment_threshold = 64 ``` `minsync optimize` runs only compact + re-index + prune (no embedding, no file reads) to repair a store that earlier interrupted syncs left fragmented. ### Machine-readable progress A host app that spawns `minsync sync` can read progress as JSON lines instead of parsing human logs: ```bash minsync sync --progress jsonl # events on stdout, human logs on stderr minsync --format json sync --progress-file /tmp/p.jsonl # events to a file, result stays on stdout minsync sync --progress jsonl --progress-heartbeat-seconds 2 ``` ```jsonc {"event":"start","total_files":12345,"resumed":42,"full":false,"dry_run":false,"sync_token":"…"} {"event":"batch","processed":125,"total":12345,"path_first":"a/b.md","path_last":"a/c.md","elapsed_ms":3200} {"event":"file_failed","path":"notes/broken.pdf","stage":"read","error":"io error: Permission denied (os error 13)"} {"event":"heartbeat","processed":125,"total":12345,"elapsed_ms":8200} {"event":"done","processed":12345,"total":12345,"files_processed":12303,"files_added":12000,"files_modified":303,"files_deleted":0,"files_resumed":42,"chunks_added":…,"files_failed":1,"partial":true,"elapsed_seconds":1234.5} ``` - `start` precedes the first batch; one `batch` per committed batch (`processed` includes resumed files); one `file_failed` per skipped file (see [Skipped files](#skipped-files)), as soon as it is known; `done` is the last event and carries `files_failed` and `partial`. - `heartbeat` is sent only after `--progress-heartbeat-seconds` (default 5) without another event, from a dedicated thread, so a slow embedding call or store commit still produces a signal. A growing `processed` means slow; an unchanged one across heartbeats means stuck on the current batch. - Dry runs and up-to-date runs emit `start` (`sync_token: null`) and `done` only. - With `--progress jsonl`, the text summary moves to stderr and `--format json` prints the final result as one compact last line on stdout. Without the flags, output is unchanged. - A sink that closes mid-run (host exited, disk full) is dropped with a warning and never fails the sync. An unopenable `--progress-file` fails before any work. A `--progress-file` inside `.minsync/` (it would overwrite state such as the resume log) or inside the indexed workspace without a matching `.minsyncignore` entry (it would be indexed as a source file) is rejected before the file is touched. ## Skipped files By default one bad file does not abort a sync. A file whose read, chunk, token-count or embed step fails (EACCES, EBUSY, EIO, a parser error, an embedder rejecting its text) is skipped and recorded; if a batch commit still fails, MinSync retries that batch file by file and skips only the culprit. A skipped file is never recorded as done: a new file stays out of the saved manifest and a previously indexed file keeps its previous manifest entry, so the next `minsync sync` retries it automatically. Its old chunks (if it was indexed before) stay searchable until it succeeds, including across full syncs. Files larger than `[sync] max_file_bytes` (default 64 MiB) are skipped before hashing or reading and reported with stage `size_guard`; other files in the sync continue. Their previous manifest entries and chunks are kept until the file is within the configured limit. Set `max_file_bytes = 0` to disable the guard. ```json "files_failed": [ { "path": "notes/broken.pdf", "stage": "read", "error": "io error: notes/broken.pdf: Permission denied (os error 13)" } ], "partial": true ``` `stage` is one of `size_guard`, `read`, `chunk`, `truncate`, `embed`, `store`. When `partial` is true the CLI exits with code **6** (completed, files skipped) and text output lists each skipped path on stderr. Hard failures keep their codes (1, 3, 4, 5). An embed/store failure that hits the first few files of a run with nothing indexed yet (for example the embedder being down) is treated as an outage, not as bad files: the sync aborts with that error (exit 5/4) and the file path in the message. ```toml [sync] on_file_error = "skip" # "fail" aborts on the first failing file (the pre-0.5 behavior) ``` ## Materialization stage (parse, then index) MinSync reads UTF-8 text and does no extraction. When an upstream step turns PDFs, DOCX or other sources into text, that step can reuse MinSync's per-item completion state and crash recovery instead of building its own, with `minsync stage`. The host keeps its parser; MinSync tracks which source versions already have a committed output file and validates each commit. ```bash minsync stage --state-dir ./.parse-state # JSON requests on stdin, JSON responses on stdout ``` ```text -> {"op":"open","stage":"parse","source_id":"corpus","fingerprint":"parser-1","output_root":"./mirror", "items":[{"key":"a.pdf","version":"sha256:…"}]} <- {"ok":true,"op":"open","resumed":false,"counts":{…},"decisions":[{"key":"a.pdf","action":"process","version":"sha256:…"}]} -> {"op":"commit","key":"a.pdf","version":"sha256:…","output":{"path":"a.pdf.md"}} <- {"ok":true,"op":"commit","key":"a.pdf","output":{"path":"a.pdf.md","size":123,"digest":"sha256:…"}} -> {"op":"finish"} ``` The host writes each output beside its final path and renames it into place, then sends `commit`; only a validated commit (regular file inside `output_root`, no traversal or symlink escape, matching size/digest, right source version and fingerprint) is recorded. A killed host or a killed `minsync stage` leaves a journal: the next `open` reports `resumed: true` and returns only the pending items. A failed item keeps its previous output and is retried; an item listed in `unknown` (unreadable source) is never deleted; an item missing from `items` has its output removed through `delete`. Changing `fingerprint` (parser name/version, output format) reprocesses this stage only. Point a normal `minsync` workspace at `output_root` to index the result as a separate stage; its own resume log is independent. Protocol reference, file formats and failure semantics: [docs/INCREMENTAL_CORE.md](docs/INCREMENTAL_CORE.md). ## Embedding Reliability Embedding requests use per-request timeouts and retry transient failures: network errors, HTTP 429, and HTTP 5xx. When a `tei:` / `openai:` request with several inputs is answered with HTTP 413, 408 or 504, or hits the request timeout, the server found the batch too big or too slow: MinSync does not resend it, but halves the batch and lowers the effective `batch_size` for the rest of the run (a single input is retried per `max_retries`; a single-input 413 fails immediately). Permanent failures such as invalid auth, malformed responses, or validation errors fail immediately. A failed or killed sync never advances the cursor or manifest, but it keeps a resume log (`progress.jsonl`): the next `minsync sync` skips every file the interrupted run already committed and continues from there. The log is discarded if the files, chunker, normalizer, embedder, or lexical language changed in between. ```toml [embedder] max_retries = 3 timeout_seconds = 60 max_concurrent = 1 ``` `timeout_seconds` also bounds the native (in-process) embedder: each forward pass (one batch) must finish within `per_batch_timeout_seconds` (default: `timeout_seconds` on GPUs, and at least 900 seconds on CPU, where a pass is slow rather than stuck); `0` is rejected. A blocked native kernel cannot be cancelled, so on expiry `minsync sync` aborts with exit code `7` (`embedder timeout`) instead of hanging; the stuck thread dies with the process, and once it happens every later native call in that process fails fast. `progress.jsonl` keeps every file committed before the timeout, so re-running `minsync sync` resumes without re-embedding them. `minsync check` forward passes are bounded the same way. Model download/load is not bounded. An explicit `per_batch_timeout_seconds` always wins, on any device. Each native forward pass is also planned to stay within `max_batch_tokens` padded tokens (inputs × longest input, default 4096) on top of `batch_size` and an attention-memory cap, so a pass of long chunks finishes in seconds instead of approaching the deadline. An input longer than the cap is still embedded alone. ```toml [embedder] timeout_seconds = 60 per_batch_timeout_seconds = 120 # optional, native only max_batch_tokens = 4096 # optional, native only ``` `base_url` also works for OpenAI-compatible gateways. ## Local Embeddings The default embedder `native:Qwen/Qwen3-Embedding-0.6B` runs in-process on [candle](https://github.com/huggingface/candle) (the model code is ported from [fastembed-rs](https://github.com/Anush008/fastembed-rs); see `THIRD_PARTY_NOTICES.md`). `minsync init` writes `dimension = 1024` and no query/passage prefixes. On Apple Silicon the native path uses Metal; elsewhere it uses CPU. The first embedding call downloads the Apache-2.0 model via Hugging Face Hub into `~/.cache/minsync/models` (override with `model_cache_dir` in `config.toml`, `FASTEMBED_CACHE_DIR`, or `HF_HOME`). Later runs load from cache. ```bash minsync init minsync sync minsync query "검색어" --k 5 ``` Existing indexes built with a 768-d TEI model must not mix 1024-d vectors. Switch embedders, then rebuild: ```bash minsync sync --full ``` ### Native runtime Two optional `[embedder]` settings tune the native runtime; the defaults are what most setups want. ```toml [embedder] dtype = "auto" # auto | f32 | f16 | bf16 attention = "auto" # auto | eager | tiled | fused ``` - `dtype`: `auto` is `f16` on Metal and `f32` elsewhere. Half precision shrinks the model's footprint by a third; the vector stays within 0.0001 cosine similarity of its `f32` counterpart, which did not change retrieval results in our tests. Set `dtype = "f32"` for full precision. Existing indexes stay valid. - `attention`: `auto` uses the device's fused kernel (Metal scaled-dot-product attention, CPU flash attention) and tiles the score matrix on other devices (CUDA builds), so peak memory no longer grows with the square of the chunk length. `eager` is the reference implementation that materializes the whole score matrix (several GiB for long chunks); `tiled` computes it block by block on any device; `fused` fails if the device has no fused kernel. Median of three `minsync sync` runs at the default chunk size on an Apple M4 Max (macOS, Metal), compared with the previous release (f32, eager attention): | Corpus | Before | After (defaults) | After (`dtype = "f32"`) | |---|---|---|---| | 600 Korean passages (~550 characters) | 125 s, 6.3 GiB | 58 s, 3.0 GiB | 61 s, 4.6 GiB | | 1000 English abstracts | 173 s, 4.9 GiB | 83 s, 3.0 GiB | 89 s, 4.5 GiB | | 48 chunks of ~3.7k characters | 97 s, 7.3 GiB | 26 s, 3.2 GiB | 29 s, 4.9 GiB | ### Use from Node.js `@nomadamas/minsync-native` ([`bindings/node`](bindings/node)) exposes the same native embedder as an in-process Node-API addon, so a long-lived Node or Electron host embeds without a second process or a cold model load per call. It shares this runtime's weights, cache directory, batch planner and deadline, and returns vectors bit-for-bit identical to the CLI's for the same text, device and batch composition. ```ts import { NativeEmbedder } from '@nomadamas/minsync-native' const embedder = new NativeEmbedder({ idleUnloadSeconds: 600 }) const { id, dimension, dtype, device } = await embedder.identity() const passages = await embedder.embed(['first passage', 'second passage']) // Float32Array[] const query = await embedder.embedQuery('which passage?') await embedder.dispose() ``` On an Apple M-series Mac the model loads once (about 2.2 s), a warm query then takes about 17 ms, and `unload()` returns about 2.7 GB of RSS. See [`bindings/node/README.md`](bindings/node/README.md) for the options, abort and deadline behavior. ### Other providers `openai:` and `tei:` remain first-class. To keep using a local TEI server with EmbeddingGemma: ```bash brew install text-embeddings-inference export HF_TOKEN="hf_..." # after accepting https://huggingface.co/google/embeddinggemma-300m terms text-embeddings-router --model-id google/embeddinggemma-300m --port 8080 --dtype float32 curl http://localhost:8080/health minsync init --embedder tei:google/embeddinggemma-300m ``` EmbeddingGemma does not support float16, so keep `--dtype float32` (or `bfloat16`). `minsync init --embedder tei:google/embeddinggemma-300m` writes the Gemma retrieval prefixes and `dimension = 768`. To serve another TEI model, pass it at init time and set its dimension and prompt prefixes in `.minsync/config.toml`: ```bash minsync init --embedder tei:intfloat/multilingual-e5-small ``` ```toml [embedder] id = "tei:intfloat/multilingual-e5-small" base_url = "http://localhost:8080" query_prefix = "query: " passage_prefix = "passage: " [vectorstore.options] dimension = 384 ``` ### Embedding server `minsync serve` loads the native embedder once and serves it on loopback, so other tools (and repeated CLI calls) share one model instead of each loading its own copy: ```bash minsync serve # workspace config if present, else the default model minsync serve --port 8089 --idle-timeout 600 ``` | Endpoint | Request | Response | |---|---|---| | `POST /embed` (TEI) | `{"inputs": [...], "truncate"?: bool}` | `[[f32, ...], ...]` | | `POST /v1/embeddings` (OpenAI) | `{"model", "input": [...], "encoding_format"?: "float" \| "base64"}` | `{"object": "list", "data": [{"index", "embedding"}], "model", "usage"}` | | `GET /health` | | model id, dimension, dtype, device, pid, port | - Vectors are the same as in-process `native:` for the same text, device and dtype: requests run through the same batch planner, padded-token cap and per-pass deadline as `sync`. - Prefixes are the caller's job. Both endpoints also accept `"input_type": "query" | "passage"`, which applies the configured `query_prefix` / `passage_prefix`. - Inputs over the model context return `413` unless `truncate: true` (TEI). More than `--max-inputs` inputs (default 512) or a body over `--max-request-bytes` (default 32 MiB) also return `413`; malformed JSON returns `400`/`422`. A forward pass that exceeds its deadline returns `503` and the server exits with code 7, since the stuck thread holds the model. - It binds `127.0.0.1` (or `::1`) only; other `--host` values are refused because there is no authentication. `--port 0` (default) picks a free port. - `pid`, `host` and `port` files live in `~/.cache/minsync/serve/<model instance>/` (override with `--state-dir`). A second `minsync serve` for the same model, device, dtype, attention and `max_length` finds the running server, prints it (`"event": "adopted"` with `--format json`) and exits without loading the model. - After `--idle-timeout` seconds (default 300; `0` disables) without an embedding request the server exits and frees the model. `SIGTERM`/Ctrl-C stop it cleanly; both remove the discovery files. `native:` workspaces use a running server automatically: `sync`, `query`, `check`, `verify` and `watch` look for the discovery files of the same model instance on their first embedding call and send the text there (with the workspace's prefixes, truncated at `max_length` like the in-process engine). With no server, or once it stops answering, they embed in-process as before. A warm server takes `query` from ~2.3 s / 2.4 GB to ~0.22 s / 59 MB on an M-series Mac. `MINSYNC_SERVER=off` always embeds in-process. The server must use the default `--state-dir` to be discovered. Point any TEI or OpenAI-compatible client at it, e.g. a second workspace: ```toml [embedder] id = "tei:Qwen/Qwen3-Embedding-0.6B" base_url = "http://127.0.0.1:8089" [vectorstore.options] dimension = 1024 ``` OpenAI-compatible clients take the `/v1` base URL. For example discrawl (it requires some API key env; the server ignores it): ```toml [search.embeddings] enabled = true provider = "openai_compatible" model = "Qwen/Qwen3-Embedding-0.6B" base_url = "http://127.0.0.1:8089/v1" api_key_env = "OPENAI_API_KEY" # any non-empty value ``` ## Ignoring Files `.minsyncignore` uses `.gitignore` syntax: ```gitignore target/ *.png *.pdf *.docx ``` MinSync reads UTF-8 text only. Binary formats should be ignored unless a separate extraction pipeline writes text files for MinSync to index. UTF-8 text is not limited to `.md`: MinSync indexes any extension that decodes as UTF-8. Japanese, Chinese, Korean, and other Unicode text are handled as ordinary text; binary formats still need a separate extraction pipeline. ## Agent Skill MinSync ships a Vercel Agent Skill for coding agents that need to add incremental semantic indexing to a workspace: ```bash npx skills add github:NomaDamas/MinSync/skills/minsync ``` The skill tells agents how to install MinSync, initialize `.minsync/`, use the default local EmbeddingGemma embeddings or configure OpenAI, ignore binary files, and star the repository with `gh repo star NomaDamas/MinSync` during installation when policy and credentials allow it. ## Release Checklist See [docs/RELEASE.md](docs/RELEASE.md) for the launch checklist. It covers CI, tag-based GitHub releases, crates.io publishing, README/docs, install QA, agent-skill QA, and rollback checks. ## Architecture Scope MinSync intentionally remains a focused, cross-platform Rust tool for incremental Markdown-centered UTF-8 text indexing. It does not aim to become a general-purpose incremental dataflow framework. See [docs/ARCHITECTURE_SCOPE.md](docs/ARCHITECTURE_SCOPE.md) for the CocoIndex comparison, extension boundary, and accepted scope decision. ## Development CI runs on Ubuntu, macOS, and Windows with Rust 1.91 installed through rustup. It runs `cargo fmt`, `cargo clippy`, `cargo test`, and `cargo build --release`. The build expects a working C compiler toolchain. CI also installs `protoc` through `arduino/setup-protoc`; non-Windows local builds use vendored protoc from `protobuf-src`, while Windows and LanceDB dependency build scripts require a `protoc` binary on `PATH`. Normal CI uses no secrets. ```bash cargo fmt --all -- --check cargo clippy --all-targets --all-features -- -D warnings cargo test cargo build --release ``` Release tags use `.github/workflows/release.yml` to build Linux, macOS Apple Silicon, and Windows artifacts, create a GitHub Release, and publish to crates.io when `CARGO_REGISTRY_TOKEN` is configured. ## License MIT

README des Repositorys wird geladen…

Skills in diesem Repository

Klassifizierung ausstehend

Skill

Beruf

Beschreibung

Aktualisiert

minsync

nicht klassifiziert

Use MinSync to add git-free incremental semantic indexing and query to a local workspace.

Quellsprache: Englisch

2026-09-19

Es werden 1 von 1 gesammelten Skills angezeigt.

Mit einem KI-Assistenten installieren

Diesen Prompt kopieren und in den verwendeten KI-Assistenten einfügen.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/NomaDamas/MinSync.