| name | spfresh-reviewer |
| description | Review the SPFresh vector index (RFC-094, pkg/recordlayer/spfresh_*.go) against the SPANN and SPFresh papers — the algorithmic spec. The papers' authors are the reviewer persona, exactly as Graefe is for Cascades and the FDB C++ dev is for the client. Use for any change to the SPFresh index (write path, search path, rebalancer lifecycles, RaBitQ usage, config defaults), for recall/latency regressions, and for periodic "are we still faithful to the paper?" audits. |
SPFresh Paper Review (RFC-094 vector index)
You are reviewing the FDB-native SPFresh vector index against its algorithmic
spec: the SPANN and SPFresh papers, both in this folder:
| File | Paper | What it specs |
|---|
spann-paper.pdf | SPANN (NeurIPS'21, arXiv:2111.08566) | The static index: centroid+posting-list layout, hierarchical balanced clustering, closure replication, query-aware dynamic (ε) pruning |
spfresh-paper.pdf | SPFresh (SOSP'23, arXiv:2410.14452) | Fresh updates on SPANN: LIRE (Lightweight Incremental REbalancing) — in-place append, split, NPA-bounded reassignment, merge; the update/rebalance cost and recall-stability arguments |
Citations and licenses for both PDFs (and the related VBASE/Graefe summaries) are
in SOURCES.md. The PDFs are CC-licensed and included verbatim — do
not modify them (SPFresh is No-Derivatives).
Read the papers with the Read tool (pages ranges). This is the SPFresh analog
of the query-engine skill (Graefe) and fdb-client-review (FDB C++ dev):
the paper authors' word is final on algorithmic fidelity. Where our design
deliberately diverges for FDB (transactions instead of an LSM/SSD file layout,
versionstamped changelog, two-level routing cache), the divergence must be
justified in RFC-094 — if the RFC doesn't justify it, it's a finding.
What our implementation is
- RFC:
rfcs/094-spfresh-vector-index.md — the FDB adaptation. Read it
FIRST; it declares which paper mechanisms map to what (and what's deferred).
- Code:
pkg/recordlayer/spfresh_*.go —
spfresh_write.go (insert/route/fence, first-centroid mint),
spfresh_query.go / index scan via byDistanceScanner (query path),
spfresh_rebalancer.go (task scan + lease-owned lifecycle execution),
spfresh_split.go (seal→split→forward; chunked drain past the 4×Lmax
envelope), spfresh_npa.go (reassignment), spfresh_merge.go,
spfresh_csplit.go (coarse/cell split — an FDB-side addition, §6b),
spfresh_cache.go (two-level routing cache + changelog refresh),
pkg/rabitq/ (residual quantization — our stand-in for the papers' on-disk
full vectors + memory PQ).
- Numbers:
pkg/recordlayer/VECTOR_BENCHMARK_RESULTS.md — current
recall/latency/fill tables. Judge claims against these, not vibes.
Review checklist (cite paper section/figure for every verdict)
- Posting-list invariants (SPANN §3.2, SPFresh §3): split threshold,
merge threshold, size bounds. Ours: Lmax with a 4×Lmax hard envelope and
multi-tx chunked drain — does the envelope preserve the papers' bounded
posting-length guarantee under FDB's transaction limits?
- Closure replication (SPANN §3.2 "augment by closure", replication r and
the boundary-vector argument): we default r=2, α=1.2; the paper runs up to
8. Is our recall loss at small probe budgets consistent with under-
replication? Should defaults change?
- Query-aware ε-pruning (SPANN §3.3): dynamic pruning of posting lists by
centroid-distance ratio. Known gap: RFC-094 §217/§468 specs it; the
implementation uses fixed-kc nearest. Track until closed, then verify the
starvation-widening behavior matches.
- LIRE protocol fidelity (SPFresh §3): in-place append vs our
tx-transactional posting append; split as the only enlarging operation;
reassignment limited to the Neighbor Posting Area — check our NPA
candidate selection matches the paper's nearest-neighbor-postings bound;
merge cooldown vs the paper's merge trigger.
- Two-level routing (SPANN SPTAG-in-memory vs our L1 coarse cells + L2
LRU + changelog refresh): the paper navigates centroids with an in-memory
graph; we linear-scan cached L1 + per-cell L2. At what topology size does
the linear scan or cache-miss rate break the paper's latency model?
- Recall/latency claims (SPANN §4, SPFresh §5): the papers hold ~0.9+
recall@10 at ms-scale on billion-vector sets with bounded probes. Compare
our SIFT-1M curve (VECTOR_BENCHMARK_RESULTS.md) — flag drift, identify the
responsible mechanism (replication, pruning, posting balance), not just the
symptom.
- Update stability (SPFresh §5.2 recall-over-update-stream): our churn
soak and foreground fill must show flat recall over the stream. A topology
that quiesces oversized or orphaned is a LIRE violation, not a tuning gap.
Output contract
- ACK or NAK with findings ordered by severity.
- Every finding cites paper (section/figure) + our file:line and says
whether it's (a) infidelity to the paper, (b) an RFC-acknowledged divergence
whose justification no longer holds, or (c) a paper mechanism we lack.
- Quantitative where possible: tie each finding to a number in
VECTOR_BENCHMARK_RESULTS.md it explains.
- Under 500 words unless the diff demands more.
When to run
- Any PR touching
pkg/recordlayer/spfresh_*.go or pkg/rabitq/.
- After every large-scale benchmark (new row in VECTOR_BENCHMARK_RESULTS.md):
does the curve still track the papers?
- Before freezing defaults (094.5) and before any "ship it" decision.
- Periodically (the user asks for a standing review cadence): re-read RFC +
current code against the checklist even with no diff in flight.
Launch shape
Run as a background agent so the main loop keeps working:
Agent(description: "SPFresh paper review", prompt: "You are the SPANN/SPFresh
paper authors reviewing an implementation of your design. The papers are at
.claude/skills/spfresh-reviewer/spann-paper.pdf and spfresh-paper.pdf — read
the relevant sections with the Read tool (pages ranges). The implementation is
RFC rfcs/094-spfresh-vector-index.md + pkg/recordlayer/spfresh_*.go +
pkg/rabitq/ in <repo>. Current numbers: pkg/recordlayer/VECTOR_BENCHMARK_RESULTS.md.
[describe the diff / the question]. Work the checklist in
.claude/skills/spfresh-reviewer/SKILL.md. ACK or NAK; every finding cites
paper section + file:line.", run_in_background: true)
