# Nowledge
ContextFS-style State/History RAG service built from the v0.6 user-event-index
specs.
## What Is Implemented
- Rust + axum API service.
- `/v1/state/*` update-style current state APIs.
- `/v1/history/*` append-only history APIs.
- HMAC-derived per-user event indexes:
`rag_events__t_{tenant_hash}__u_{user_hash}`.
- HMAC-derived per-user personal context indexes:
`rag_context__t_{tenant_hash}__u_{user_hash}`.
- Shared company context index semantics via `rag_company_context`.
- ContextFS URI nodes with L0 `.abstract`, L1 `.overview`, and L2 detail/chunk
layers.
- Staged retrieval and trace records.
- Company document preflight, revisions, and activation.
- Structured dataset snapshots, idempotent row ingestion, and deterministic
numeric summary calculation.
- Hybrid document matching backed by [turbovec](https://github.com/RyanCodrai/turbovec)
(Google's TurboQuant quantized vector index): saved documents and fragments
are embedded into a local vector index, and retrieval blends lexical scores
with fragment-level and document-level vector similarity.
- Obsidian-style bidirectional knowledge links with backlink/outbound search.
- Independent analysis API that can use a separate LLM provider/model to create
links and insight records from ingested context.
- Insight, session, RAG answer/debug, LLM status, and debug route surfaces.
- Meilisearch bootstrap hooks with in-memory persistence for local development
and tests.
## Run
```sh
cargo run
```
The default URL is `http://127.0.0.1:14242`.
Nowledge requires Rust 1.88 or newer. For a safe local starting point, copy the
checked-in example and explicitly load it into your shell; the binary does not
load `.env` files itself:
```sh
cp .env.example .env
set -a
. ./.env
set +a
cargo run
```
Useful environment variables:
```sh
RAG_HOST=127.0.0.1
RAG_PORT=14242
RAG_TENANT_ID=default
RAG_INDEX_HASH_SECRET=
RAG_ALLOW_LEGACY_WEAK_INDEX_HASH_SECRET=false
RAG_STORE_BACKEND=memory
RAG_ALLOW_EPHEMERAL_PRODUCTION=false
RAG_BEARER_TOKEN=optional-user-token
RAG_BEARER_TOKEN_SCOPE=owner
RAG_BEARER_TOKEN_OWNER_USER_ID=owner-user-id
RAG_ALLOW_LEGACY_TENANT_SERVICE_BEARER=false
RAG_ALLOW_LEGACY_SHARED_WRITER=false
RAG_ADMIN_TOKEN=optional-admin-token
RAG_AUTH_USERS=owner-user-id:user-token:user,*:service-token:user,writer-owner:writer-token:user|company_writer
RAG_RUN_MODE=development
RAG_ALLOW_UNSAFE_UNAUTHENTICATED=true
RAG_MEILI_URL=http://127.0.0.1:7700
RAG_MEILI_API_KEY=optional-runtime-document-search-key
RAG_MEILI_ADMIN_API_KEY=optional-distinct-index-admin-key
RAG_MEILI_ALLOW_INITIAL_PROVISION=false
RAG_MEILI_OPERATIONS_INDEX_CREATED_AT=required-production-rfc3339-createdAt
RAG_MEILI_AUDIT_INDEX_CREATED_AT=required-production-rfc3339-createdAt
RAG_WRITE_CONSISTENCY=read_your_writes
RAG_MEILI_SCAN_PAGE_SIZE=500
RAG_MEILI_SCAN_MAX_DOCUMENTS=100000
RAG_MEILI_CONNECT_TIMEOUT_MS=5000
RAG_MEILI_REQUEST_TIMEOUT_MS=30000
RAG_PARSER_PROVIDER=builtin
RAG_MINERU_API_URL=http://127.0.0.1:8000
RAG_MINERU_BACKEND=hybrid-auto-engine
RAG_MINERU_RETURN_MD=true
RAG_MINERU_RETURN_CONTENT_LIST=true
RAG_MINERU_RETURN_MIDDLE_JSON=true
RAG_MINERU_RETURN_IMAGES=true
RAG_MAX_JSON_BYTES=2097152
RAG_MAX_UPLOAD_BYTES=52428800
RAG_MAX_MULTIPART_FIELDS=32
RAG_UPLOAD_ALLOWED_MIME_TYPES=text/plain,text/markdown,application/octet-stream,application/pdf,application/vnd.openxmlformats-officedocument.wordprocessingml.document,application/vnd.openxmlformats-officedocument.presentationml.presentation,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,image/png,image/jpeg,image/webp,image/gif,image/tiff
RAG_MAX_BULK_EVENTS=500
RAG_MAX_BULK_ROWS=5000
RAG_MAX_SEARCH_LIMIT=100
RAG_MAX_QUERY_CHARS=8192
RAG_MAX_TAGS_PER_ITEM=64
RAG_MAX_TAG_BYTES=128
RAG_REQUEST_TIMEOUT_MS=90000
RAG_SYNC_INGEST_TIMEOUT_MS=120000
RAG_MAX_IN_FLIGHT_REQUESTS=256
RAG_RATE_LIMIT_REQUESTS_PER_MINUTE=600
RAG_CORS_ALLOWED_ORIGINS=*
RAG_ALLOW_WILDCARD_CORS=false
RAG_VECTOR_MATCH_ENABLED=true
RAG_VECTOR_MATCH_WEIGHT=4.0
RAG_VECTOR_MATCH_DOC_WEIGHT=2.0
RAG_VECTOR_MATCH_MIN_SCORE=0.25
RAG_INGEST_MAX_CONCURRENT_TASKS=2
RAG_INGEST_QUEUE_CAPACITY=16
RAG_INGEST_TASK_RETENTION_SECONDS=86400
RAG_INGEST_CLEANUP_INTERVAL_SECONDS=300
RAG_TRACE_RETENTION_DAYS=30
RAG_INGEST_WORKER_ENABLED=true
RAG_SHUTDOWN_TIMEOUT_MS=30000
RAG_PROVIDER_CONNECT_TIMEOUT_MS=5000
RAG_LLM_TIMEOUT_MS=60000
RAG_PARSER_TIMEOUT_MS=120000
RAG_PROVIDER_MAX_RETRIES=2
RAG_PROVIDER_PROXY_MODE=system
RAG_LLM_MAX_INPUT_CHARS=65536
RAG_LLM_MAX_OUTPUT_TOKENS=2048
RAG_LLM_MAX_RESPONSE_BYTES=2097152
RAG_PARSER_MAX_RESPONSE_BYTES=67108864
RAG_LLM_RATE_LIMIT_REQUESTS_PER_MINUTE=30
RAG_LLM_RATE_LIMIT_TOKENS_PER_MINUTE=100000
RAG_LLM_PROVIDER=none
RAG_LLM_MODEL=none
RAG_LLM_REASONING_EFFORT=optional-low-medium-high-xhigh
RAG_ANALYSIS_LLM_PROVIDER=none
RAG_ANALYSIS_LLM_MODEL=gpt-5.3-codex-spark
RAG_ANALYSIS_LLM_REASONING_EFFORT=optional-low-medium-high-xhigh
RAG_OPENAI_API_KEY=optional-openai-key
RAG_CODEX_AUTH_PATH=optional-explicit-codex-auth-json
RAG_REDACTION_PREVIOUS_SECRETS=optional-revoked-token,optional-prior-token
RAG_HEALTH_LLM_ENABLED=true
RAG_HEALTH_LLM_PROBE_INTERVAL_SECONDS=0
RAG_HEALTH_LLM_PROBE_TTL_SECONDS=60
RAG_HEALTH_LLM_MAX_STALE_SECONDS=120
RAG_HEALTH_LLM_TIMEOUT_MS=10000
RAG_HEALTH_REQUIRE_LLM=true
```
Production requires an independently generated `RAG_INDEX_HASH_SECRET` of at
least 32 bytes with at least 12 distinct byte values. The public development
default, the previously documented `change-me`, and literal documentation
placeholders are rejected; do not reuse an authentication credential. Generate
a new value once with `openssl rand -base64 48`, store it in the service's
secret manager, and keep it stable. This key protects owner/index derivation
plus audit and error fingerprints.
Changing this key changes every per-user event and personal-context index UID.
Before upgrading an existing Meilisearch deployment, classify its current key
without printing it. If the deployment already uses a weak key, preserve that
exact value and temporarily set
`RAG_ALLOW_LEGACY_WEAK_INDEX_HASH_SECRET=true`; rotating it in place would make
the existing indexes unreachable. New deployments must never enable this flag.
Remove it only after the `index_hash_secret_v1` migration/reindex has completed
and been verified. The compatibility path expires on 2026-10-01 / v0.13.0.
`RAG_RUN_MODE` accepts only `development`, `test`, or `production`; unknown
values are startup errors and never enable unauthenticated access by default.
`RAG_WRITE_CONSISTENCY` accepts only `eventual`, `read_your_writes`, or
`wait_for_index`. When it is unset, development and production default to
`read_your_writes` and test mode defaults to `wait_for_index`. Production
rejects `eventual` because it cannot guarantee visibility of a completed
write. The recommended production setting, and the one the reference
deployment runs, is an explicit `RAG_WRITE_CONSISTENCY=wait_for_index`: every
write waits until Meilisearch confirms its tasks before the response returns,
so searches served directly from Meilisearch (by this or a restarted process)
see it immediately. `read_your_writes` responds once Meilisearch accepts most
writes (kinds that require index confirmation still wait) and merges the process's own in-memory copy into searches until
indexing catches up, which favors write latency but relies on that single
process's mirror. The deprecated
`RAG_MEILI_WAIT_FOR_TASKS` is consulted only when `RAG_WRITE_CONSISTENCY` is
unset: `true` maps to `wait_for_index`, while `false` maps to
`read_your_writes`. Setting both variables is a startup error. This legacy
compatibility path expires on 2026-10-01 / v0.13.0.
History mutation and analysis materialization responses include additive `persistence` metadata with the
durable operation ID, operation/indexing states, ordered primary task UIDs,
and all task UIDs. If an event batch is committed but a derived context write
fails, the response is explicit (`status=partially_failed`) and retrying the
same owner-scoped idempotency key reconciles unfinished work while replaying
the original stable event IDs. Other legacy response shapes continue to fail
safely when a derived write is partial; operators can inspect and retry the
journal through the admin operation endpoints.
Every supported operation-level `idempotency_key` is bound to the canonical
tenant, target, and request payload. An exact retry replays the stable result;
reusing the key for a different request returns 409. History bulk writes accept
only the top-level batch key (nested `events[].idempotency_key` is rejected),
and ingest requests reject `idempotency_key` because ingest idempotency is not
yet supported. Keyed structured-row batches generate a distinct stable ID for
each ID-less row by batch position.
Before starting this revision against an existing Meilisearch deployment,
back up Meilisearch and add the tenant-safe operation journal. The managed
fixed-index bootstrap fails closed when the old index set exists without
`rag_operations`; it will not recreate the missing index automatically.
Startup also refuses any existing fixed index, or any registered per-user
event or personal-context index, whose primary key is not `id`.
Drain writers during this cutover and audit existing documents for legacy
idempotency metadata. `operations_v1` starts the durable journal; it cannot
reconstruct request fingerprints or response snapshots for keys accepted by an
older binary. Treat pre-journal keys as retired and do not replay them after
the upgrade.
```sh
cargo run --bin operations_v1 -- plan --out /secure/operations-v1-plan.json
cargo run --bin operations_v1 -- apply \
--plan /secure/operations-v1-plan.json --dry-run
cargo run --bin operations_v1 -- apply \
--plan /secure/operations-v1-plan.json
cargo run --bin operations_v1 -- plan \
--out /secure/operations-v1-verify-plan.json
cargo run --bin operations_v1 -- verify \
--plan /secure/operations-v1-verify-plan.json
```
Use the same `RAG_MEILI_URL` and `RAG_MEILI_ADMIN_API_KEY` as the service.
`operations_v1` manages only `rag_operations`, waits for every task,
requires the exact primary key `id`, is non-destructive, and
rejects a tampered plan. `verify` exits nonzero whenever the index, primary
key, managed settings, or exact Meilisearch `createdAt` generation are not
ready, so deployment automation must treat that command as a hard gate. A
missing-index plan creates the index but cannot know its future generation;
always capture the fresh post-apply verification plan shown above. The previous
application revision ignores this additional index, so rollback does not
require deleting it. See
[ADR 0006](doc/adr/0006-mutation-consistency.md) for consistency and recovery
semantics.
Unfinished journal records keep their complete replay plan so reconciliation
can re-apply pending steps. Once an operation is completed and confirmed
searchable, its final checkpoint is compacted: content bodies (document text,
fragment bodies, event text/payloads, row values, parsed blocks) are cleared
while ids, routing, ownership, checksums, deletion targets, the response
snapshot used for idempotent replay, and per-step payload sizes plus keyed
fingerprints (`plan.redacted_metadata.compacted_payloads`) are kept. The
process keeps at most 10,000 settled records in memory; older ones stay
readable through the durable journal (the memory backend has no journal, so
its replay window is that bound).
The operation journal currently requires one writer process per tenant.
Do not run active-active Nowledge writer replicas against the same tenant until
a durable lease or compare-and-set protocol is introduced; the in-process
mutation gate is not distributed consensus. Read replicas are outside this
revision's supported deployment topology.
### Durable shared-mutation audit rollout
This revision also adds `rag_audit_records` for the five shared-knowledge
mutation targets: company-document preflight, revision creation, revision
activation, company-document deletion, and structured-dataset schema upsert.
Authorization denials for those targets are recorded in the same index. There
is deliberately no public audit-query route in this revision.
Run the following non-destructive migration before starting the upgraded
binary against an existing managed Meilisearch deployment. Use the exact
application revision that will be deployed, back up Meilisearch first, stop or
drain writers for the cutover, and provide the service's `RAG_MEILI_URL` plus
`RAG_MEILI_ADMIN_API_KEY` when required:
```sh
cargo run --bin audit_records_v1 -- plan \
--out /secure/audit-records-v1-plan.json
cargo run --bin audit_records_v1 -- apply \
--plan /secure/audit-records-v1-plan.json --dry-run
cargo run --bin audit_records_v1 -- apply \
--plan /secure/audit-records-v1-plan.json
cargo run --bin audit_records_v1 -- plan \
--out /secure/audit-records-v1-verify-plan.json
cargo run --bin audit_records_v1 -- verify \
--plan /secure/audit-records-v1-verify-plan.json
```
Do not start the upgraded service unless `verify` exits successfully. Adding
the index to `FIXED_INDEXES` intentionally makes an old managed index set
incomplete, and startup refuses to recreate only the missing audit index.
`audit_records_v1` manages only `rag_audit_records`, requires primary key `id`,
waits for all creation/settings tasks, rejects incompatible or tampered plans,
and refuses to recreate an index that disappeared after planning. The
post-apply plan binds the exact `createdAt` generation; a plan
that originally observed a missing index is intentionally not valid evidence
for verifying the newly created generation. Plan, dry-run, and verify are
mutation-free.
For either fixed-index migration, a missing-index plan is not a generally
idempotent artifact. If creation was accepted but the CLI then crashed or lost
its response, do not retry that plan or silently adopt the UID. Stop, preserve
and inspect Meilisearch task history for the exact UID, time, and result, then
generate and review a fresh plan. An `already_present` plan may be verified;
`settings_drift` must be dry-run/applied and then verified with that bound plan;
`missing` starts a new full sequence only after a confirmed failed create task.
Treat `primary_key_mismatch` or an unexplained generation as an incident/restore
condition.
Before each authorized shared mutation, the service persists an `attempted`
record and fails closed if that record cannot be accepted. It then updates the
same record to `success` or `failure` in a supervised background write; a
finalization failure retains the attempt and emits only bounded, fingerprinted
diagnostics. Recording an
authorization denial is admitted to a bounded, supervised best-effort
background write. Capacity, shutdown, or persistence failure never delays or
replaces the original 401/403 response. Records contain trusted tenant/request
correlation plus bounded enums and HMAC-derived owner/resource/reason
identities; raw request content, identifiers, reasons, paths, queries, prompts,
tokens, and provider bodies are not part of the schema.
Audit evidence is retained indefinitely in this revision. There is no
automatic age deletion. Operators must monitor Meilisearch capacity and
`nowledge_audit_background_drops_total`, preserve legal holds, archive a
verified off-host dump before any approved cleanup, and scope any cleanup by
both tenant and occurrence time. The exact policy is recorded in ADR 0008.
The previous application ignores the additional index, so rollback does not
require deleting it. Preserve the index and migration artifact for a later
retry or forward deployment. See
[ADR 0008](doc/adr/0008-durable-shared-mutation-audit.md) for the persistence
and failure semantics. Existing installations that also require the
tenant-scope migration must follow the ordered
[production upgrade runbook](doc/runbooks/production-upgrade.md): create and
verify the operation and audit indexes before generating the final
`tenant_scope_v1` plan.
HTTP and ingest boundaries are typed startup configuration. Numeric limits
must be positive, malformed values fail startup, and the synchronous ingest
timeout must be at least the ordinary request timeout. The ordinary request
timeout must exceed `RAG_LLM_TIMEOUT_MS`, `RAG_HEALTH_LLM_TIMEOUT_MS`, and the
60 s Meilisearch task wait by at least 5 s (65 s with the shipped LLM
defaults), so a slow dependency returns its own error instead of being cut off
by a generic route 504; shorter values fail startup. Request, sync-ingest,
and shutdown deadlines are capped at seven days so deadline arithmetic cannot
overflow or panic. JSON requests are
limited by `RAG_MAX_JSON_BYTES`; multipart file data is streamed to temporary
storage and limited independently by `RAG_MAX_UPLOAD_BYTES` and
`RAG_MAX_MULTIPART_FIELDS`. `RAG_UPLOAD_ALLOWED_MIME_TYPES` is a comma-separated
list normalized to exact lowercase MIME types; wildcards, duplicates, and an
empty list fail startup. The default allows plain text, Markdown, generic binary
(`application/octet-stream`), PDF, OOXML Word/PowerPoint/Excel, and PNG, JPEG,
WebP, GIF, and TIFF. Bulk event/row counts, tags, tag byte lengths, and search
limits are rejected before mutation when they exceed their configured
maximums. Limit failures use the normal error envelope and return 413 for body
or upload size, 429 for rate/queue pressure, 503 for global capacity or a
disabled/closing worker, and 504 for route timeouts. Pressure responses include
`Retry-After`, and every response includes a server-generated `X-Request-Id`.
LLM and MinerU calls share bounded HTTP clients. Connect, operation, decoded
response-body, prompt/output, and per-principal request/token limits are
configured by the `RAG_PROVIDER_*`, `RAG_LLM_*`, and
`RAG_PARSER_MAX_RESPONSE_BYTES` variables above. `RAG_PROVIDER_MAX_RETRIES`
accepts 0 through 2; retries are limited to connection failures before a
response, HTTP 408, non-quota 429, and HTTP 500/502/503/504, and preserve one
client request ID. Response/body timeouts are not retried because the upstream
may already have consumed the request. `Retry-After` is honored within the
overall deadline. `RAG_PROVIDER_PROXY_MODE=system` uses the process proxy
environment, while `direct` disables proxies explicitly. Analysis uses a
strict JSON schema plus an independent server-side authorization/validation
pass; provider output cannot select tenant or owner scope.
`RAG_CORS_ALLOWED_ORIGINS` is a comma-separated list of exact `http://` or
`https://` origins. Development and test default to `*`; production defaults
to no allowed browser origin. A production wildcard is rejected unless it is
the sole origin and `RAG_ALLOW_WILDCARD_CORS=true` is set explicitly.
`RAG_MAX_IN_FLIGHT_REQUESTS` load-sheds excess work without consuming the
`/livez` capacity or timeout path. A request keeps its in-flight slot until its
response body finishes, but at most `RAG_REQUEST_TIMEOUT_MS` after the
response headers are ready: at that deadline the slot is released even if a
stalled client has stopped reading, and the unfinished body is cut off the
next time it is polled. `/readyz` remains public but has its own
readiness-probe rate bucket. Rate limiting is keyed by the authenticated
logical tenant/principal, so rotating or adding a second token does not create
a new owner budget. CORS exposes `X-Request-Id` and `Retry-After` to allowed
browser origins. See
[ADR 0003](doc/adr/0003-http-ingest-runtime-boundaries.md) for middleware
ordering, overload semantics, upload staging, and rollout.
Use `RAG_STORE_BACKEND=meili` with `RAG_MEILI_URL` to mirror core writes to
Meilisearch and search per-user event indexes through Meilisearch. Ordinary
production mode requires this durable backend; an intentionally ephemeral
production process must acknowledge restart data loss with
`RAG_ALLOW_EPHEMERAL_PRODUCTION=true`. Production mode requires configured auth
unless `RAG_ALLOW_UNSAFE_UNAUTHENTICATED=true` is set explicitly.
`RAG_LLM_PROVIDER` and `RAG_ANALYSIS_LLM_PROVIDER` accept only `none`, `mock`,
`openai_api_key`, or `codex_auth`; unknown values fail startup instead of
silently selecting a disabled client. Required primary and analysis profiles
both participate in readiness.
Authentication data scope is explicit. A named owner in `RAG_AUTH_USERS` creates
an owner-bound principal; the literal owner `*` creates a tenant-service
principal. Feature roles do not widen data scope: an owner-bound
`company_writer` can mutate shared company knowledge but still cannot read a
different owner's private data. `*:token:admin` creates admin scope (the
reserved `admin` value is a scope marker, not a feature role), while new admin
credentials should prefer `RAG_ADMIN_TOKEN`. Legacy named-owner
`owner:token:admin` entries temporarily retain admin scope and emit a startup
warning; migrate them to `*:token:admin` or `RAG_ADMIN_TOKEN` before the
compatibility window ends on 2026-10-01 in v0.13.0.
Every bearer, admin, and `RAG_AUTH_USERS` credential must contain at least eight
characters. Other configured secrets used by the response redactor—including
Meilisearch/OpenAI keys, a readable index-HMAC key, Codex auth-file tokens, and
explicit previous secrets—must contain at least four characters. Empty,
whitespace-padded, duplicate authentication credentials and shorter values are
startup errors; error text never echoes the rejected value.
The legacy `RAG_BEARER_TOKEN` must set `RAG_BEARER_TOKEN_SCOPE=owner` together
with `RAG_BEARER_TOKEN_OWNER_USER_ID`, or set
`RAG_BEARER_TOKEN_SCOPE=tenant_service`. Existing intentional tenant-service
clients may temporarily set `RAG_ALLOW_LEGACY_TENANT_SERVICE_BEARER=true`; that
compatibility switch is removed on 2026-10-01 in v0.13.0. Company-document
preflight, revision creation/activation, and dataset-schema upserts require the
`company_writer` role or admin scope by default. Operators may temporarily set
`RAG_ALLOW_LEGACY_SHARED_WRITER=true` to preserve ordinary authenticated access
to those shared writes while clients migrate; it emits a startup warning and
has the same removal deadline. Company-document deletion remains admin-only.
See [ADR 0002](doc/adr/0002-principal-scope-and-diagnostics.md) for rollout and
rollback details.
Tenant-service principals must select `owner_user_id` for private state-fact,
state-search, and insight-search reads; omission returns 403 instead of
implicitly searching whichever private owner happens to exist.
Health endpoints split process liveness from operational readiness:
- `GET /livez` returns process status plus build version/revision and does not
query Meilisearch or LLM providers.
- `GET /readyz` is public for load balancers. It preserves the operational
readiness decision and 200/503 status semantics while returning only coarse
dependency state. A Meilisearch-backed process remains unready until every
mandatory tenant domain has hydrated atomically; it does not expose raw
provider payloads, usage/private counts, plan data, credits, or credential
sources. It never calls an LLM provider: the `llm` and `analysis_llm`
verdicts are the last ones in the health-probe cache, kept whatever their
age, and `llm_checked_at` says when each was observed. Until a profile's
first probe completes it reports `not_probed`, which keeps the service
unready when `RAG_HEALTH_REQUIRE_LLM=true` and otherwise reports it ready
but `degraded`.
- `GET /healthz` requires an admin bearer and checks Meilisearch plus the
tenant hydration report and configured parser and LLM provider/model,
including auth validity, quota/rate-limit state, probe age, and a compact
usage summary. It re-probes an LLM profile whose cached verdict is older
than `RAG_HEALTH_LLM_PROBE_TTL_SECONDS` (concurrent callers share one
probe). If `RAG_HEALTH_REQUIRE_LLM=true`, an unconfigured or exhausted LLM
makes the service unhealthy.
- LLM health probing is sparse to save provider quota. The server probes each
LLM profile once in the background at startup; after that only `/healthz`
(and the harness eval false-ready check) re-probes, at most once per
`RAG_HEALTH_LLM_PROBE_TTL_SECONDS`. Periodic background probing is opt-in
through `RAG_HEALTH_LLM_PROBE_INTERVAL_SECONDS`: `0` (the default) disables
it, and any other value must be at least 300 seconds. The admin usage view
still reports a cached verdict older than `RAG_HEALTH_LLM_MAX_STALE_SECONDS`
as an unhealthy `stale_probe`; readiness never expires one.
- LLM health probes always run at `low` reasoning effort, whatever
`RAG_LLM_REASONING_EFFORT` or `RAG_ANALYSIS_LLM_REASONING_EFFORT` configures;
the reported `reasoning_effort` stays the profile's configured value.
- The protected health `llm.rate_limits` block carries the freshest live
provider budget snapshot. For `codex_auth` it is parsed from the ChatGPT Codex `x-codex-*`
response headers on every health probe and real completion: `primary` and
`secondary` windows with `used_percent` / `remaining_percent` / reset times
and the length Codex reports (`window_minutes`, plus a derived
`window_label` such as `7d`), `plan_type`, credits, and model-scoped
`additional_limits`. Window lengths come from Codex, not from fixed
assumptions, and either window may be absent.
`llm.rate_limit_state` becomes `near_limit` at ≥90% window usage and
`limited` at 100%, so dashboards can warn before hard 429s.
- LLM-backed responses (`/v1/rag/answer`, `/v1/analysis/insights`,
`/v1/llm/title`, `/v1/llm/test`) include real provider token counts in
their `usage` blocks (`input_tokens`, `cached_input_tokens`,
`output_tokens`, `reasoning_output_tokens`, `total_tokens`) whenever the
upstream reports them, so consumers no longer need char-based estimates.
Configured secrets are redacted before provider submission, model-output
persistence/materialization, and response serialization. Analysis parses the
provider JSON before redaction, validates proposed link locators against the
prompt's authorized context/seed locator set, deduplicates exact persisted
link/insight identities, and then sanitizes free text. Active exact-identity
matches are reused unchanged, while inactive matches conflict instead of
being overwritten or reactivated. New accepted analysis candidates are
written as one owner-bound durable operation and returned only after every
Meilisearch task is confirmed; all-reuse batches require no write.
- `GET /v1/usage` returns only the selected owner's usage counters to owner and
tenant-service principals. Global counters and service-wide Meilisearch,
parser, and LLM provider diagnostics are admin-only.
- `GET /v1/llm/status` requires authentication and reports `auth_source` as a
category such as `codex_file` or `environment`, never as a filesystem path.
- Every JSON response passes through a final configured-secret sanitizer,
including typed state/history/link responses, sync ingest results, parsed
blocks, and explicit ContextFS source-document reads. Object keys and values
are covered. JSON bodies above 16 MiB or malformed JSON bodies fail closed
instead of bypassing the sanitizer. A background task reads the Codex auth
file at most once per second on the blocking pool and atomically publishes one
credential snapshot to both LLM clients and redaction; request and liveness
paths never read the file. Observed tokens remain in the in-process inventory
across rotations and transient auth-file errors. Structural locators remove
complete configured secrets but
do not apply heuristic fragment/token-shape rewriting, preserving stable
`ctx://` navigation when an ordinary slug overlaps part of a credential.
- Provider previews are redacted before truncation. New ingest masks configured
secrets with equal-length placeholders before fragmenting so provenance
offsets remain stable; retrieval also masks configured-secret pieces at
fragment boundaries. That second boundary prevents legacy data or a
post-ingest credential rotation from sending split token halves to a provider
or returning them in content-bearing JSON fields.
Meilisearch startup is fail-closed. Production refuses both an all-missing and
a partial managed index set so a lost data directory cannot be accepted as an
empty installation. A first installation requires the one-time
`RAG_MEILI_ALLOW_INITIAL_PROVISION=true` gate; use it only while observing a
known-empty target, then capture the durable-index identity pins, remove the
gate, and restart. Development and test retain automatic first provision.
Existing registered per-user event and personal-context indexes are reconciled
in place and are never recreated during hydration. Settings are updated only
when the managed attributes differ, so ordinary restarts do not enqueue
redundant Meilisearch mutations. Use the explicit administrative reset workflow
only when destructive recreation is intended.
Production also requires distinct Meilisearch credentials. The runtime
`RAG_MEILI_API_KEY` performs document/search/task reads and must not have index
create/delete permission. `RAG_MEILI_ADMIN_API_KEY` is used only by managed
index/settings paths and maintenance binaries. Durable operation and audit
writes compare the captured index generation and exact managed settings before
the write and after its task succeeds; drift makes readiness unhealthy and the
write fails closed. Normal production startup also pins the exact `createdAt`
identities of `rag_operations` and `rag_audit_records` through
`RAG_MEILI_OPERATIONS_INDEX_CREATED_AT` and
`RAG_MEILI_AUDIT_INDEX_CREATED_AT`, so a restart cannot silently rebaseline to
a different fully provisioned cluster. Capture these values only after the
verified migrations or first-install provisioning, preserve them with the
deployment configuration, and change them only during an operator-reviewed
restore or replacement.
Before restarting after a credential rotation, place any revoked value that
can still occur in persisted documents in the comma-separated
`RAG_REDACTION_PREVIOUS_SECRETS` secret-manager value. Retain it until those
records are reingested or scrubbed. This bridges process restarts without
logging or returning the prior credential; remove entries only after verifying
that persisted data no longer contains them.
Every response carries a server-generated `X-Request-Id`. Generic 500 and 502
error bodies include that safe correlation ID without returning the underlying
cause. Request failures and best-effort background failures record only an
allowlisted cause category plus a keyed fingerprint; dynamic task/source IDs
are omitted or fingerprinted. Raw causes and identifiers are never emitted.
Document parser ingestion is an additive layer in front of the existing RAG
backend. Use `RAG_PARSER_PROVIDER=builtin` for plain text fallback or
`RAG_PARSER_PROVIDER=mineru` to call a remote `mineru-api` service. Ingestion
APIs are `POST /v1/ingest/tasks`, `GET /v1/ingest/tasks/{task_id}`,
`GET /v1/ingest/tasks/{task_id}/result`, `POST /v1/ingest/uploads`,
`POST /v1/ingest/uploads:sync`, and `POST /v1/ingest/files:sync`.
`POST /v1/ingest/tasks` and `/v1/ingest/uploads` return queued task metadata
immediately; background workers perform parsing, fragmenting, and indexing.
Ingest requests do not currently support `idempotency_key`; supplying one
returns 400 before a task record is created.
Queue capacity defaults to eight times `RAG_INGEST_MAX_CONCURRENT_TASKS` and
can be set explicitly with `RAG_INGEST_QUEUE_CAPACITY`. A full queue is rejected
immediately without creating an orphan task. Disabling the worker rejects new
asynchronous ingest while synchronous ingest remains available. Synchronous
ingest uses a separate immediate load-shed lane capped by
`RAG_INGEST_MAX_CONCURRENT_TASKS`; saturation returns 503 plus `Retry-After`
before buffering a JSON or multipart body. SIGINT/SIGTERM stop new queue
acceptance, drain in-flight HTTP requests while background audit and trace
writes are still accepted, then close the background runtime and allow
supervised work to finish within `RAG_SHUTDOWN_TIMEOUT_MS`, and mark unfinished
tasks as failed with `ingest_interrupted`; startup recovery applies the same terminal state to
persisted nonterminal tasks from a prior process.
Finished (`completed`/`failed`) task records and their stored results are
pruned after `RAG_INGEST_TASK_RETENTION_SECONDS` (default 86400; set 0 to
keep them forever), swept every `RAG_INGEST_CLEANUP_INTERVAL_SECONDS` —
covering both the in-memory maps and the mirrored Meilisearch documents.
Ingested fragments and source documents are unaffected; only the task
bookkeeping expires.
Retrieval traces (`trace_id` from context search, RAG answers, and eval
runs) are diagnostic records, not mutations. A search never takes the global
write lock or the operation journal: its trace is cached in process
immediately and a background writer persists queued traces to `rag_traces` in
batches (one document request per batch). If the queue is full or the write
fails, the search still succeeds and the trace stays readable from the
process cache. Traces are not loaded at startup; older traces are read
through by tenant-scoped id. `RAG_TRACE_RETENTION_DAYS` (default 30, at most
36500; 0 keeps traces forever) deletes expired traces from the cache and from
Meilisearch by a `created_at_ms` range filter, first about one minute after
startup and then hourly. The `rag_traces` managed settings make `created_at`
and `created_at_ms` filterable; startup reconciliation applies that through
the admin key, and older traces without `created_at_ms` are expired by their
RFC 3339 `created_at` in bounded batches. The usage `rag.trace_count` reports
traces retained by the current process, not the persisted total.
Meilisearch task confirmation polls with exponential backoff (5 ms doubling
to 200 ms, same 60 s total timeout). Several task UIDs from one step are
confirmed together through `GET /tasks?uids=...`; a server or key scope that
cannot list them falls back to per-task `GET /tasks/{uid}` probes.
Every Meilisearch HTTP request is bounded by `RAG_MEILI_CONNECT_TIMEOUT_MS`
(default 5000) for connection setup and `RAG_MEILI_REQUEST_TIMEOUT_MS`
(default 30000) for the whole exchange including the response body; the
connect bound must not exceed the request bound. Task confirmation issues one
short request per poll, so the request bound limits each poll rather than the
60 s task wait that `wait_for_index` relies on.
A failed Meilisearch response or task keeps Meilisearch's error object in the
private cause: `code`, `type`, `link`, and a secret-redacted message truncated
to 512 characters (at most 16 KiB of an error body is read; a non-JSON body
keeps a 256-character snippet). The cause category comes from the Meilisearch
`code` when present. `nowledge::meili` warnings log only the HTTP status, the
Meilisearch code and type, and the cause category and keyed fingerprint, never
the message, body, headers, or API keys.
A search that Meilisearch answers with `index_not_found` is empty only for the
per-user event and personal-context indexes, which are created on a user's
first write. A missing fixed index, or a 404 without `index_not_found`, fails
as an upstream error instead of returning no hits.
Multipart uploads are staged with a generated private filename, incrementally
hashed, and deleted after parsing or on cancellation/error. Duplicate file
parts, excess metadata fields, an empty file, invalid or disallowed MIME values,
and a supplied checksum that is not exactly 64 hexadecimal SHA-256 characters
or does not match the staged file are rejected. Each file part must carry a
`Content-Type` in `RAG_UPLOAD_ALLOWED_MIME_TYPES`; an optional `content_type`
metadata field must match it. A staged file cannot be combined with `content`,
`content_list`, `content_list_v2`, `middle_json`, or `model_json`.
MinerU receives a streaming file part when `parser_provider=mineru`; the builtin
parser performs one bounded UTF-8 read.
Parsed blocks become retrieval fragments; source documents and parse artifacts
are stored for traceback/read flows but are not searched by default.
`POST /v1/context/search` supports `compact`, `standard`, and `full` return
profiles plus optional include payloads for traceback, links, neighboring
fragments, source summaries, artifact refs, score breakdowns, and raw stage
debug. Default retrieval still returns only active fragments; standard/full
responses add source groups and location/block provenance for highlighting.
`POST /v1/rag/answer` citations preserve the same provenance, including source
document URI, page index, bounding box, block type, section path, artifact refs,
fragment offsets, and checksums when available.
Document matching is hybrid: full documents and their fragments are embedded
into a [turbovec](https://github.com/RyanCodrai/turbovec) quantized vector
index when they are saved, and context search blends the lexical substring
score with fragment-level vector similarity plus document-level vector
evidence from the fragment's source document. Inflected or reordered queries
("deployment pipelines" vs "deploy pipeline") match without an exact
substring, and document-level evidence boosts fragments that already match
on their own. Raw source document bodies stay out of default retrieval:
document evidence never admits a fragment by itself. Vector search is always restricted to the caller's
isolation-filtered candidates, embeddings are deterministic hashed lexical
features (no external embedding service), and `include: ["score_breakdown"]`
exposes the `lexical` / `vector` / `document_vector` / `combined` components
per hit. Disable with `RAG_VECTOR_MATCH_ENABLED=false`; tune the blend with
`RAG_VECTOR_MATCH_WEIGHT`, `RAG_VECTOR_MATCH_DOC_WEIGHT`, and
`RAG_VECTOR_MATCH_MIN_SCORE`. Building on Linux requires OpenBLAS
(`libopenblas-dev` on Debian/Ubuntu); macOS uses the built-in Accelerate
framework.
Link and analysis surfaces:
- `POST /v1/links` creates or updates a directed knowledge link between two
ContextFS URIs. Link search treats the stored edge as bidirectional for
navigation.
- `POST /v1/links/search` searches links by text, relation, URI, outbound edge,
or backlink.
- `POST /v1/analysis/insights` searches ingested context, includes existing
links as evidence, and can materialize new links plus insight records. It uses
`RAG_ANALYSIS_LLM_PROVIDER` / `RAG_ANALYSIS_LLM_MODEL`, so analysis can run on
a different model from `/v1/rag/answer`. `RAG_LLM_REASONING_EFFORT` is passed
to configured Responses API calls, and
`RAG_ANALYSIS_LLM_REASONING_EFFORT` can override it for analysis jobs. When
`history_event_id` is supplied, the caller must also provide or be bound to
`owner_user_id`; analysis evidence is then constrained to that owner's same
history event index.
## Deploy
The reference deployment builds from a Git checkout and runs under systemd
next to a separate Meilisearch unit. Routine deploys, verification, and
binary rollback follow [doc/runbooks/gfit-deploy.md](doc/runbooks/gfit-deploy.md);
releases with data migrations follow the
[production upgrade runbook](doc/runbooks/production-upgrade.md).
## Verify
```sh
cargo fmt --check
cargo clippy --locked --all-targets -- -D warnings
cargo check --locked --all-targets
cargo test --locked
cargo package --locked
```
CI also runs the exact route-manifest contract test, runs the Meilisearch
integration suites against a Meilisearch service container pinned to the
production version, checks the same lockfile with Rust 1.88, audits RustSec
advisories, and enforces the dependency license, source, ban, and advisory
policy in `deny.toml`.
Optional Meilisearch integration tests run when `RAG_TEST_MEILI_URL` is set.
If the server requires a key, set `RAG_TEST_MEILI_API_KEY` too:
```sh
RAG_TEST_MEILI_URL=http://127.0.0.1:7700 \
RAG_TEST_MEILI_API_KEY=$MEILI_MASTER_KEY \
cargo test --test meili_integration --test insight_history
```
Without `RAG_TEST_MEILI_URL`, or when the server is unreachable or needs a key
that was not provided, those tests skip and report as passed. Set
`RAG_TEST_REQUIRE_MEILI=1` to make each of those conditions, and a missing
`RAG_TEST_MEILI_API_KEY`, fail the test instead; CI sets it.
Optional live MinerU integration tests run when `RAG_TEST_MINERU_API_URL` is set:
```sh
RAG_TEST_MINERU_API_URL=http://127.0.0.1:8000 cargo test --test mineru_integration
```
The regression tests cover the v0.6 hard constraints around per-user event
isolation, owner mismatch rejection, state upsert shape, company-doc preflight,
structured row idempotency, and token redaction.