rask-lance-catalog

name

rask-lance-catalog

description

The Lance catalog and the governance layer above it — the Lance Namespace spec (54 ops, the 24-code problem+json error model, the `POST /v1/<object>/{id}/<action>` route grammar) and rask's own `project > warehouse > namespace > table` hierarchy: the guards, the registries, protection/trash, and `services/maintenance`. Use when adding or changing a catalog endpoint; raising an error from the catalog; touching the hierarchy guards, the project/warehouse registries or `seed_ownership`; designing create/drop/undrop/cascade or protection semantics; working on compaction, the sweep, the orphan scan or the reconciler; or answering "is this spec-conformant".

rask lance catalog — the spec and the estate's layer above it

Two contracts stack here, and confusing them is how bugs happen:

1. The Lance Namespace spec (lance.org/format/namespace/) — operations, error codes, REST grammar. Defines ONLY namespaces (recursive) + tables, and explicitly prescribes no hierarchy enforcement and nothing above a namespace.
2. rask's hierarchy — project > warehouse > namespace > table. Everything above a namespace is OURS: our objects, our guards, our lifecycle. The spec neither requires nor forbids it.

The spec surface (verified 2026-08-04 against lance.org)

• Operations: 54/54 ROUTED, 48 backend-backed. tests/integration/test_spec_conformance.py asserts both halves — every spec op has a served route, and the vendored lance_docs/ns_catalog/spec.yaml still carries 54 ops (a shrunken spec would silently weaken the check). The other 6 answer a spec-correct 501 because the native dir backend stubs them: backfill_column, alter_transaction, batch_create_table_versions, batch_commit_tables, and BOTH materialized-view ops (docs/COVERAGE.md). Spelling matters when grepping: the spec op and the served route are SINGULAR — POST /v1/table/{id}/backfill_column (spec.yaml:1570, endpoints/columns.py:146) — while alter_table_backfill_columns is the native method it wraps. rename_table is NOT one of them, and this bullet said it was for months. #5b backs it in-process (endpoints/tables.py::rename_table copies the dataset root, repoints the namespace and deregisters the source, never reaching native.call), so it answers 200; docs/COVERAGE.md:15-17 carries the correction, dated 2026-08-05, and this file kept naming the native stub the endpoint had deliberately stopped using. The spec's minimum is 8 metadata ops; we carry the whole list including versioning, tags, branches, indices and transactions.
• Route grammar: POST /v1/<object>/{id}/<action> — everything a reverse proxy needs (authN/Z, routing) is in the PATH, never only in the body. Path/body id conflict → 400. List ops are GET with query-param pagination; data ops (create/insert/query) are Arrow IPC, not JSON; count/explain return plain text.
• Identifier: segments joined by the configured delimiter ($ default) — ["a","b","t"] ↔ a$b$t. The root namespace is the delimiter itself.
• Identity headers: api_key → x-api-key, auth_token → Authorization: Bearer; arbitrary context maps BOTH WAYS through a header.<name> context KEY — {"header.x-trace-id": "abc"} is sent as x-trace-id: abc (prefix stripped) and every response header returns as header.<name> (spec.yaml:2471). The old x-lance-ctx-<key> form is superseded and survives only in the generated per-model docs. Headers beat body fields.

The error contract — NEVER invent a status

The spec defines 24 numeric error codes (0–23) — 22/23 are TableBranchNotFound / TableBranchAlreadyExists, added with the branch ops — identical across Python/Java/Rust/REST; clients dispatch on the code, not the HTTP status. On the wire they are RFC 9457 problem bodies (application/problem+json).

The one rule: an endpoint raises lance_namespace typed errors (InvalidInputError, NamespaceNotFoundError, …) and lets service_kit/lakehouse/ns_errors.py::install_problem_handlers translate — it maps all 24 codes (not-founds → 404, already-exists/not-empty/concurrent → 409, InvalidInput → 400, PermissionDenied → 403, Unauthenticated → 401, Unsupported → 501, Throttling → 429). The branch codes were MISSING until 2026-08-04 — a missing branch answered 500 on endpoints rask ships. tests/unit/test_ns_errors_contract.py now pins the map against the ENUM, so the next spec-added code fails a test instead of a client. Never HTTPException with a hand-picked status for domain errors — a 422 was shipped once and no generated client understood it; fixed to InvalidInputError (95ae4cb). NamespaceNotEmpty → 409 is the spec's own error for "container refuses while full" — use it, don't mint one.

rask's hierarchy layer

project (tenant)  >  warehouse (ONE bucket; a project holds MANY)  >  namespace (self-nesting)  >  table

Three layers, each owned in ONE place:

Layer	Owner
shape (what can exist)	the guards in catalog/api/fga_deps.py — require_parent (a table must have a namespace; rename destinations too), require_warehouse_scoped (a top-level namespace only via POST /v1/warehouses/{id}/namespaces; no-op when warehouses_enabled is off — single-bucket deployments have no warehouse to demand), require_project_exists (a warehouse's project must have a registry record — 404 naming POST /v1/projects), require_not_protected (deletion protection — the force rule is under Lifecycle)
who	the FGA model's can_* relations (service_kit/governed/auth/model.fga) — the app never invents policy
what is possible NOW	the registries, checked BEFORE the native write: project records (catalog/services/projects.py, _projects/<id>.json), warehouse records + top_ns → warehouse_id → root_uri bindings (catalog/services/warehouses.py)

A tenant EXISTS when its registry record does — not when a warehouse implies it and not when FGA holds tuples for it. POST /v1/projects writes the record AND the creator's project#admin tuple in one operation (estate-admin gated on can_observe_events at the root), which is what stops existence and permission drifting apart. There is no bootstrap exception in warehouse-create any more: one door, and a warehouse-owner cannot ride a create into project admin.

Check order at every create door: identity (401) → shape (InvalidInput 400) → parent exists (404) → authz (403) → conflict (409) → native write → tuples (seed_ownership) → events. Guards run BEFORE native.call — rejecting after leaves a real Lance object with no authz parent.

project and warehouse are control-plane objects, not spec objects — their endpoints (/v1/projects, /v1/warehouses) are ours. Keep their errors in the same problem-body format so one client error path serves the whole API.

Storage — what state lives where (and why there is NO app database)

State	Store
table data + versions	Lance datasets on object storage (rustfs)
project registry (_projects/<id>.json), warehouse registry + namespace bindings	JSON records on the control root. Id-MINTING creates are conditional (If-None-Match: * via service_kit.lakehouse.records.create_json): project mint, warehouse mint and the write-once bindings are store-arbitrated, a lost race surfaces as 409, never last-writer-wins. Mutable-field read-modify-writes are CONDITIONAL too as of 2026-08-15 — warehouses.upsert_warehouse / set_warehouse_status / projects.upsert_project go through records.mutate_json (read + ETag-guarded replace, bounded retry), so an idempotent re-POST can no longer silently revert a concurrent quarantine or protection change. put_warehouse / put_project remain as SEEDING primitives with no production caller, pinned by tests/unit/test_registry_writes_are_conditional.py. Protection and trash records are still plain overwrites and correctly so: they carry no read-modify-write, so there is no carried-forward field to lose.
authz	OpenFGA on its chart-managed Postgres
lineage	AGE (Postgres), chart-managed

A relational app-DB was removed at P7a and must not creep back for the catalog: registry writes are admin-frequency, the conditional creates arbitrate the id-mint races, and deletes are bottom-up single-object operations by design — there is no multi-object transaction to need one. The moment that changes (atomic cross-object invariants, high-frequency filtered listings), it is a design decision, not a default. The conditional-put primitive is proven live by TWO cas-marker e2e suites: test_object_store_cas_e2e.py (Lance's own manifest commits) and test_registry_cas_e2e.py (the registry seam, contended 8-way).

Lifecycle

Reclamation, scheduling and the GC design live in services/maintenance itself — read services/maintenance/services/{sweep,optimize,purge,reconcile,index_health,tiers}.py — plus packages/service-kit/src/service_kit/lakehouse/base_refs.py, the shallow-clone-source pre-pass, which lives in service-kit rather than the service because the catalog's on-demand maintenance doors (catalog/services/maintenance.py) must apply the same refusal and cannot import the sweep before changing anything the sweep, the reconciler or the orphan scan touches. This used to point at a root open_*.md, which is ephemeral by design: the plan was deleted when its work landed and the pointer dangled. The per-table/namespace/project POLICY that governs a sweep is catalog.schemas.PolicyRequest (retention, retain_versions, compact_enabled, interval, target rows), resolved winner-takes-all — an exact table match shadows the namespace record which shadows the project record, and the winner supplies EVERY field. Any surface showing an effective policy must say which record won; an inherited value rendered identically to a set one is how nobody can tell what is governing their data. The project-scoped surface is home's /projects/<p> § Maintenance.

• Creates are top-down: parent must EXIST (registry), gated on the parent's can_*.
• Deletes are bottom-up: a container refuses 409, naming its contents; cascade is explicit; warehouse delete gates on the PROJECT's can_administer; bucket purge is a separate opt-in; project delete has no cascade at all.
• force=true overrides the protected flag and NOTHING else — the FGA gate runs first and identically with or without it. Test both delete doors for force-without-authz.
• Recoverable drops are OPT-IN (#75). With LANCE_TRASH_GRACE_DAYS > 0 (default 0/OFF, because a grace period changes what drop_table means for every caller) a drop DEREGISTERS and files a _trash/ record; POST /v1/table/{id}/undrop re-registers from it; GET /v1/table/{id}/tasks shows the pending deadline (§2.4 per-object task visibility). purge=true is the explicit opt-out — a caller who means "destroy the bytes now" says so. A recoverable drop does not revoke the table's FGA tuples: the owner is the one person who needs to undrop it, and revoking made undrop unreachable for exactly that caller (found by driving the deployed catalog — the unit tests run FGA off); the grants die with the bytes, at purge or expiry. undrop re-registers with a RELATIVE location (the final path segment): register_table refuses the absolute URI describe_table had just reported, and the dir backend lays tables out flat. The sweep REPORTS expired trash and deletes nothing. COVERAGE.md's old "soft-delete is N/A, time-travel replaces it" entry was WRONG and is corrected: time-travel does not survive drop_table. The CASCADE is recoverable too (#96): with a grace period, drop_namespace(cascade) never issues the destructive native call — a trash record pointing at bytes that call deleted would be a lie — and instead DETACHES the subtree (every table deregistered, namespaces emptied deepest-first then dropped, one kind-tagged record each, shared drop-time expires_at; tuples KEPT, the same #75 rule). POST /v1/namespace/{id}/undrop is the PLURAL undrop: rebuilds every trashed namespace under the id shallowest-first, re-registers every trashed table (relative-location form), resumable (exist_ok creates; already-registered = recovered); shows the subtree's deadline; is the same explicit opt-out. A RESTRICT drop stays unrecorded on purpose — it only ever removes an empty manifest row. Declared-only tables (no recorded location) are skipped by undrop with a warning: no bytes were lost.

Gotchas

• update_table_schema_metadata MERGES — the spec text ("Replace schema metadata") is wrong about every backend. Probed against a real dir backend: posting {owner} over {owner, tier} leaves tier standing. So omitting a key cannot remove it, and the spec's request model types metadata as a strict {str: str} that cannot carry a null — which left table properties with no delete at all, and str(None) writing the literal string "None" onto the table. Since #78 a null value DELETES the key: no-null bodies stay on the native spec op, a body with any null routes to dataplane.update_schema_metadata (pylance's update_schema_metadata, the same dialect update_field_metadata already speaks). Never replace=True — the map a caller holds came from read_schema_metadata, which excludes lineage.*, so a replace silently destroys the #21 self-describing coordinates. description is the one RESERVED key (the lakehouse renders it under the table name); everything else in that map is opaque user data.

• deregister keeps bytes ON PURPOSE (external data); drop removes them. Neither leaves Lance orphans — but partially-failed writes and unpurged buckets do, and nothing reclaims those yet. services/maintenance's orphan pass REPORTS them (MAINTENANCE_ORPHAN_SCAN_ENABLED — it opens every dataset, unlike the rest of the drift report which compares three stores). It is ON in the deployed estate, which is the opposite of what this line said: maintenance/core/config.py:196 defaults it False, but chart/values.yaml:1356 ships orphanScan: true, and the chart is the single deploy artefact — so "off by default" describes a configuration nothing runs. Read the chart for what an estate does; read config.py only for what an unconfigured process does.

• Three Lance file classes look like orphans and are not. A scan that names any of them would drive a reclaimer into live data, and all three were found by running against a real estate, not by reading the layout doc: _refs/tags/*.json are TAGS, which PIN versions (cleanup_old_versions exempts tagged versions for that reason); is a structural marker; and a large binary column's bytes live in , a SIDECAR that does not name — the first live run called 29 MB of real page images reclaimable. Conversely genuinely accumulate forever (the spec keeps one per commit attempt) and nothing prunes them.