| name | vector-search |
| description | Semantic / ANN vector search over a sparq RDF graph: build a memory-mapped per-term-id embedding store (.spqv), then run cosine top-k with an in-RAM HNSW, a persistent on-disk DiskANN/Vamana graph (.spqg), or an exact brute-force baseline; verbalize entities (label+type+description) for embedding, scalar/product quantize (SQ/PQ) for large stores, fuse with another ranked signal (RRF / score blend) for hybrid retrieval, run predicate-constrained (filtered) ANN over a BGP-selected dict-id mask behind the opt-in `filtered-ann` feature, run recall-gated concept dedup + k-NN over a RAW (id, vector) matrix behind the opt-in `approx-ann` feature (build_ann/knn/dedup: merges apply only after measured ANN recall vs an exact ground truth clears a pre-registered gate), and — behind the opt-in `vec-predicate` feature — run k-NN INSIDE plain SPARQL via the `vec:nearest` / `vec:search` magic predicates. Use when adding embedding/semantic-search/nearest-neighbour/near-duplicate-merging over a sparq Graph or a raw concept-vector matrix in the sparq-vectors crate. |
sparq-vector-search
sparq-vectors is an opt-in crate that adds embedding storage + nearest-neighbour
(ANN) search to the sparq RDF engine. You store one f32 embedding per dictionary term
id in a flat memory-mapped .spqv file, then query top-k by cosine with an exact
scan, an in-RAM HNSW index, or a persistent on-disk DiskANN/Vamana graph — all
cosine-identical so their scores are directly comparable. Embeddings are produced
out-of-process (you supply the Embedder); the crate never runs a model and the
default engine build does not even compile it.
Quickstart
crates/sparq-vectors/Cargo.toml (it consumes sparq-core; no features needed for the
core flow):
[dependencies]
sparq-core = { path = "../sparq-core" }
sparq-vectors = { path = "../sparq-vectors" }
oxrdf = "*"
Embed entity labels, finalize the store, query the nearest neighbours of a term:
use sparq_core::Graph;
use sparq_vectors::{embed_labels, nearest_term_exact, HashEmbedder, VectorStore};
use oxrdf::{NamedNode, Term};
# fn main() -> Result<(), String> {
let ttl = r#"
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix ex: <http://example.org/> .
ex:bolt rdfs:label "Usain Bolt" ; a ex:Athlete .
ex:bolt2 rdfs:label "Usain Bolt Junior" ; a ex:Athlete .
"#;
let graph = Graph::load_str(ttl, "turtle")?;
let embedder = HashEmbedder::new(64);
let mut store = VectorStore::create("graph.spqv", 64)?;
embed_labels(&graph, &mut store, &embedder)?;
store.finalize()?;
let bolt = Term::NamedNode(NamedNode::new("http://example.org/bolt").map_err(|e| e.to_string())?);
let neighbours: Vec<(Term, f32)> = nearest_term_exact(&store, &graph, &bolt, 10);
# Ok(()) }
Key APIs
VectorStore::create<P: Into<PathBuf>>(path, dim: usize) -> Result<VectorStore, String>
VectorStore::open<P: AsRef<Path>>(path) -> Result<VectorStore, String>
VectorStore::open_from_bytes(bytes: Vec<u8>) -> Result<VectorStore, String>
impl VectorStore {
fn put(&mut self, id: Id, vector: &[f32]) -> Result<(), String>
fn finalize(&mut self) -> Result<(), String>
fn get(&self, id: Id) -> Option<&[f32]>; fn iter(&self) -> impl Iterator<Item=(Id, &[f32])>
fn dim(&self) -> usize; fn len(&self) -> usize; fn is_empty(&self) -> bool
}
# feature = "metadata-sidecar": opaque per-vector tags persisted in `.spqv` v4; no new dependency
store.put_with_meta(id, vector, meta: &str) -> Result<(), String>
store.meta(id) -> Option<&str>
StreamingWriter::create(path, dim); fn put(&mut self, id, &[f32]); fn finalize(self) -> Result<VectorStore, String>
ImportSpec { spqv_path, dim, ids: &[Id], binding: ImportBinding }
ImportBinding::Unbound | ImportBinding::Graph(&Graph)
VectorStore::import_npy(ImportSpec, npy_path) -> Result<VectorStore, String>
VectorStore::import_numeric_dump(ImportSpec, dump_path) -> Result<VectorStore, String>
trait Embedder { fn dim(&self) -> usize; fn embed(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>, String>; }
HashEmbedder::new(dim) / ::with_seed(dim, seed)
embed_labels(&Graph, &mut VectorStore, &impl Embedder) -> Result<usize, String>
embed_labels_with(&Graph, &mut VectorStore, &impl Embedder, &LabelConfig) -> Result<usize, String>
verbalize(&Graph, &Term, &EntityTextConfig) -> Option<String>
embed_entities(&Graph, &mut VectorStore, &impl Embedder, &EntityTextConfig) -> Result<usize, String>
VectorStore::create(p, dim)?.with_fingerprint(&Graph)
StreamingWriter::create_with_fingerprint(p, dim, &Graph)
store.fingerprint() -> Option<Fingerprint>; store.check_graph(&Graph) -> Result<(), String>
EmbeddingProvenance { model_id, model_version, content_version, metric: EmbeddingMetric, normalization: Normalization,
verbalization, reserved: Vec<u8> }
EmbeddingProvenance::new(model_id, EmbeddingMetric, Normalization)
EmbeddingMetric::{Cosine, Dot, Euclidean}; Normalization::{None, L2}
prov.compatible_with(&query_prov) -> Result<(), String>
prov.to_rdf(store: NamedNodeRef, dim) -> Vec<Triple>
VectorStore::create(p, dim)?.with_provenance(EmbeddingProvenance)
StreamingWriter::create_with_provenance(p, dim, &Graph, EmbeddingProvenance)
store.provenance() -> Option<&EmbeddingProvenance>
store.check_provenance(&query_prov, LegacyMode) -> Result<(), String>
LegacyMode::{Reject, Allow}
store.add(id, &[f32]) -> Result<(), String>
store.remove(id) -> bool
store.update(id, &[f32]) -> Result<(), String>
store.compact(out_path, &Graph) -> Result<VectorStore, String>
store.has_delta() -> bool; store.delta() -> Option<&VectorDelta>; store.take_delta() -> Option<VectorDelta>
store.apply_delta(VectorDelta) -> Result<(), String>
store.save_delta() -> Result<PathBuf, String>
store.save_delta_to(path) -> Result<(), String>
VectorStore::open_with_delta(base) -> Result<VectorStore, String>
VectorStore::open_with_delta_at(base, delta_path) -> Result<..>
VectorStore::sibling_delta_path(&Path) -> PathBuf; VectorStore::has_persisted_delta(&Path) -> bool
nearest_exact(&VectorStore, query: &[f32], k) -> Vec<(Id, f32)>
nearest_exact_tiebreak(&VectorStore, &Graph, query: &[f32], k, exclude: Option<Id>) -> Result<Vec<(Id, f32)>, String>
# feature = "metadata-sidecar": same ranking/scores, decorated after ranking
nearest_exact_with_meta(&VectorStore, query: &[f32], k) -> Vec<(Id, f32, Option<String>)>
nearest_term_exact(&VectorStore, &Graph, &Term, k) -> Vec<(Term, f32)>
nearest_term_exact_checked(&VectorStore, &Graph, &Term, k) -> Result<Vec<(Term, f32)>, String>
cosine(a: &[f32], b: &[f32]) -> f32
VectorIndex::build(&store) / ::build_with(&store, HnswConfig{ef_search, ef_construction, seed})
HnswConfig::default() / ::fast_build() / ::high_recall()
impl VectorIndex { fn nearest(&self, query: &[f32], k) -> Vec<(Id, f32)>;
fn nearest_with_ef(&self, query: &[f32], k, ef_search: usize) -> Vec<(Id, f32)>;
fn nearest_term(&self, &Term, &Graph, &VectorStore, k) -> Vec<(Term, f32)>;
fn nearest_term_checked(..) -> Result<Vec<(Term, f32)>, String> }
build_ann(vectors: &[(Id, Vec<f32>)], policy: HnswConfig) -> Result<ConceptAnnIndex, String>
knn(&ConceptAnnIndex, query: &[f32], k) -> Vec<(Id, f32)>
impl ConceptAnnIndex { fn knn(&self, query: &[f32], k) -> Vec<(Id, f32)>; fn knn_of(&self, Id, k) -> Vec<(Id, f32)>;
fn len()/is_empty()/dim(); fn ids() -> &[Id] }
exact_ground_truth(vectors, k) -> Result<GroundTruth, String>
GroundTruth::new(k, Vec<(Id, Vec<Id>)>) -> Result<GroundTruth, String>
DedupPolicy { recall_gate: f64 , merge_threshold: f32 , k }
dedup(&ConceptAnnIndex, &DedupPolicy, &GroundTruth) -> Result<DedupReport, String>
DedupReport { recall: f64, merges: Vec<(Id , Id )>, groups: Vec<Vec<Id>> }
DiskAnnIndex::build(&VectorStore, path) / ::build_with(&store, path, VamanaConfig{degree, build_beam, search_beam, alpha, seed})
DiskAnnIndex::build_for(&store, path, &Graph) / ::build_with_for(&store, path, cfg, &Graph)
DiskAnnIndex::build_with_pq(&store, path, cfg, PqConfig) -> Result<..>
DiskAnnIndex::open(path) -> Result<DiskAnnIndex, String>
DiskAnnIndex::open_from_bytes(bytes: Vec<u8>) -> Result<DiskAnnIndex, String>
impl DiskAnnIndex { fn nearest(&self, &[f32], k) -> Vec<(Id, f32)>; fn nearest_term(..) -> Vec<(Term, f32)>; fn len()/dim();
fn has_pq_cache() -> bool;
fn fingerprint() -> Option<Fingerprint>; fn check_graph(&store, &Graph) -> Result<(), String>;
fn nearest_term_checked(&Term, &Graph, &store, k) -> Result<Vec<(Term, f32)>, String> }
sibling_graph_path(&Path) -> PathBuf
IdMask::new() / ::from_ids(impl IntoIterator<Item=Id>) / FromIterator<Id>
impl IdMask { fn insert(&mut self, Id) -> &mut Self; fn contains(Id)->bool; fn len()/is_empty(); fn iter() -> impl Iterator<Item=Id> }
nearest_exact_filtered(&VectorStore, query: &[f32], &IdMask, k) -> Vec<(Id, f32)>
FilterConfig { prefilter_fraction: f32 , prefilter_floor: usize , traversal_beam_factor: usize }
impl FilterConfig { fn prefer_prefilter(mask_len, store_len) -> bool }
impl DiskAnnIndex { fn nearest_filtered(&self, &[f32], &IdMask, &VectorStore, k) -> Vec<(Id, f32)>
fn nearest_filtered_with(&self, &[f32], &IdMask, &VectorStore, k, FilterConfig) -> Vec<(Id, f32)> }
impl VectorIndex { fn nearest_filtered(&self, &[f32], &IdMask, &VectorStore, k) -> Vec<(Id, f32)> }
CostModel { scatter_penalty: f32 }
impl CostModel { fn decide(mask_len, store_len, k) -> CostEstimate }
Strategy::{PreFilter, PostFilter}
CostEstimate { mask_len, store_len, k, prefilter_cost, postfilter_cost, strategy }
postfilter_exact(&VectorStore, query: &[f32], &IdMask, k) -> Vec<(Id, f32)>
nearest_filtered_costed(&VectorStore, &[f32], &IdMask, k, &CostModel) -> (Vec<(Id, f32)>, CostEstimate)
nearest_filtered_costed_tiebreak(&VectorStore, &Graph, &[f32], &IdMask, k, exclude: Option<Id>, &CostModel) -> Result<(Vec<(Id, f32)>, CostEstimate), String>
overfetch_target(k, mask_len, store_len) -> usize
trait AnnBackend { fn candidates(&self, query: &[f32], fetch) -> Vec<(Id, f32)>; fn len()/is_empty(); }
ExactBackend::new(&VectorStore)
ApproxBackend::new(&DiskAnnIndex)
nearest_filtered_overfetch(&impl AnnBackend, query, &IdMask, k, max_rounds) -> Vec<(Id, f32)>
nearest_filtered_overfetch_default(&backend, query, &IdMask, k) -> Vec<(Id, f32)>
ScalarQuantizer::fit(dim, vectors: impl IntoIterator<Item=&[f32]>) -> Result<ScalarQuantizer, String>
ProductQuantizer::fit(dim, vectors, PqConfig{m, k, iters, seed}) -> Result<ProductQuantizer, String>
ProductQuantizer::to_bytes() -> Vec<u8> / ::from_bytes(&[u8]) -> Result<ProductQuantizer, String>
impl {Scalar,Product}Quantizer { fn encode(&self, &[f32]) -> Vec<u8>; fn reconstruct(&self, &[u8]) -> Vec<f32>;
fn encode_store(&self, &VectorStore) -> Result<EncodedStore, String> }
DistanceTable::new(&ProductQuantizer, query: &[f32]); fn distance(&self, code)->f32; fn cosine(&self, code)->f32
EncodedStore::rank_pq(&self, &DistanceTable, k) -> Vec<(Id, f32)>; cosine_from_sq_dist(sq: f32) -> f32
EncodedStore::from_parts(ids, codes, stride) -> Result<EncodedStore, String> / ::codes() -> &[u8]
fuse_rrf(lists: &[&[(T, f64)]], k: f64 , top_k) -> Vec<(T, f64)>
fuse_rrf_weighted(lists: &[(&[(T, f64)], f64)], k, top_k) -> Vec<(T, f64)>
fuse_scores(a: &[(T,f64)], b: &[(T,f64)], alpha , top_k) -> Vec<(T, f64)>
hybrid_search(query: &Q, top_k, k , &mut [Retriever<'_, Q, T>]) -> Vec<(T, f64)>
query_vec(&Graph, sparql: &str, &VectorStore) -> Result<QueryResult, String>
query_vec_with_budget(&Graph, &str, &VectorStore, &QueryBudget) -> Result<QueryResult, String>
prepare_vec(&Graph, &str, &VectorStore) -> Result<PreparedQuery, String>
rewrite_query(Query, &Graph, &VectorStore) -> Result<Query, String>
query_vec_approx(&Graph, &str, &VectorStore, &DiskAnnIndex) -> Result<QueryResult, String>
query_vec_approx_with_budget(&Graph, &str, &VectorStore, &DiskAnnIndex, &QueryBudget) -> Result<QueryResult, String>
prepare_vec_approx(&Graph, &str, &VectorStore, &DiskAnnIndex) -> Result<PreparedQuery, String>
Common recipes
1. Verbalize entities (label + type + description), then embed
Better than label-only when the graph has descriptions/types — the default
EntityTextConfig renders "<label>. a <type>. <description>":
use sparq_core::Graph;
use sparq_vectors::{embed_entities, verbalize, EntityTextConfig, HashEmbedder, VectorStore};
use oxrdf::{NamedNode, Term};
let graph = Graph::load_str(ttl, "turtle")?;
let cfg = EntityTextConfig::default();
let bolt = Term::NamedNode(NamedNode::new("http://example.org/bolt").map_err(|e| e.to_string())?);
println!("{:?}", verbalize(&graph, &bolt, &cfg));
let mut store = VectorStore::create("graph.spqv", 64)?;
let n = embed_entities(&graph, &mut store, &HashEmbedder::new(64), &cfg)?;
store.finalize()?;
Tailor the template — add a categorical literal group, set the language chain:
use sparq_vectors::{EntityTextConfig, PropertyGroup};
use oxrdf::NamedNode;
let mut cfg = EntityTextConfig::default();
cfg.languages = vec!["en".into(), "".into()];
cfg.groups.push(
PropertyGroup::literal(vec![NamedNode::new("http://example.org/occupation").unwrap()])
.with_prefix("occupation: ")
.with_max_values(3),
);
cfg.max_chars = 1024;
2. Exact vs HNSW (pick by scale)
HNSW (VectorIndex) is the APPROXIMATE backend behind the opt-in approx-ann feature — the
only thing pulling instant-distance, so the default build has NO third-party ANN dep (lean core).
It is approximate: recall < 1.0 (NOT answer-exact). Build with --features approx-ann.
[OPUS-4.8] (sq-lfo84) The HNSW squared-Euclidean distance is computed by an explicit-SIMD kernel
(src/simd.rs, approx-ann-only, no new dependency): runtime-detected NEON on aarch64 and
AVX2+FMA on x86_64, with a scalar fallback numerically bit-identical to the previous
auto-vectorised loop. It measurably cuts the graph-build time and lifts query QPS. Recall is
floor-preserved, not bit-identical: the SIMD kernels use FMA (one rounding) while the scalar path
does d*d then += (two roundings), so a SIMD squared distance differs from the scalar one by ≤1
ULP — rankings are stable up to exact near-ties, and that residual is what the HNSW floor gate
(recall@10 ≥ 0.95, tests/recall.rs) absorbs (the gate is a floor, not a bit-identity assertion).
The deterministic exact / DiskANN / PQ paths keep the scalar reduction, so their EXACT-gated
bench/vector/expected.tsv deficits are byte-stable. Full recall-QPS + build-time evaluation matrix (SIMD vs instant-distance-scalar vs
hnsw_rs, NON-CANONICAL): research/gap-vector-ann-simd-2026-07.md.
# #[cfg(feature = "approx-ann")] {
use sparq_vectors::{nearest_exact, VectorIndex, HnswConfig};
let store = sparq_vectors::VectorStore::open("graph.spqv")?;
let hits = nearest_exact(&store, query, 10);
let index = VectorIndex::build_with(&store, HnswConfig { ef_search: 100, ef_construction: 100, seed: 0 });
let approx = index.nearest(query, 10);
let fast = VectorIndex::build_with(&store, HnswConfig::fast_build());
let dense = VectorIndex::build_with(&store, HnswConfig::high_recall());
# let _ = (fast, dense);
# }
3. Persistent on-disk index that survives process restart
use sparq_vectors::{DiskAnnIndex, VectorStore};
let store = VectorStore::open("entities.spqv")?;
let _ = DiskAnnIndex::build(&store, "entities.spqg")?;
let index = DiskAnnIndex::open("entities.spqg")?;
let hits = index.nearest(query, 10);
4. Quantize a large store, then PQ-filter + full-precision re-rank (the DiskANN loop)
PQ codes are a coarse RAM-resident filter, not a final ranking — re-rank the candidates
against the full-precision store. [OPUS-4.8] sq-qamd: DiskAnnIndex::build_with_pq now drives
this loop INSIDE the index (rank each visited node's neighbours on the in-RAM codes, re-rank the
final beam off the mmap — nearest reports the exact full-precision cosine), and persists the
codebook + codes as a trailing .spqg section so open reloads the cache with no rebuild:
use sparq_vectors::{DiskAnnIndex, PqConfig, VamanaConfig, VectorStore};
let store = VectorStore::open("entities.spqv")?;
let idx = DiskAnnIndex::build_with_pq(&store, "entities.spqg", VamanaConfig::default(), PqConfig::default())?;
assert!(idx.has_pq_cache());
let top10 = idx.nearest(query, 10);
To drive the same coarse-filter-then-re-rank loop by hand (e.g. over the exact store with no graph):
use sparq_vectors::{cosine, DistanceTable, ProductQuantizer, PqConfig, VectorStore};
let store = VectorStore::open("entities.spqv")?;
let pq = ProductQuantizer::fit(store.dim(), store.iter().map(|(_, v)| v), PqConfig::default())?;
let cache = pq.encode_store(&store)?;
let table = DistanceTable::new(&pq, query);
let candidates = cache.rank_pq(&table, 50);
let mut rescored: Vec<(u32, f32)> = candidates
.into_iter()
.map(|(id, _)| (id, cosine(query, store.get(id).unwrap())))
.collect();
rescored.sort_unstable_by(|a, b| b.1.total_cmp(&a.1).then(a.0.cmp(&b.0)));
let top10: Vec<_> = rescored.into_iter().take(10).collect();
ScalarQuantizer (4×, f32→u8) is the cheaper alternative; reconstruct(code) gives the
lossy preview for re-ranking. Both quantizers and all searchers share the L2-normalized
cosine convention (cos = 1 − d²/2).
5. Hybrid retrieval — fuse text vectors with another ranked signal
Neither crate depends on the other; the fusion helpers take plain (item, score) lists.
Over-fetch each signal (e.g. 50) for a top-10 fusion so RRF has overlap to reward.
use sparq_sim::Sim;
use sparq_vectors::{fuse_rrf, fuse_scores, RRF_K};
let text: Vec<(oxrdf::Term, f64)> =
index.nearest_term(&query, &graph, &store, 50).into_iter().map(|(t, s)| (t, s as f64)).collect();
let structural: Vec<(oxrdf::Term, f64)> = Sim::new(&graph).most_similar(&query, 50);
let hybrid = fuse_rrf(&[&text, &structural], RRF_K, 10);
let blended = fuse_scores(&text, &structural, 0.7, 10);
hybrid_search packages the common RRF case — drive N retriever closures off one query
Term and fuse by Term in a single call (dedups; a term in only one list still surfaces;
RRF ignores the score scales). Widen nearest_term's f32 to f64 so both lists share the
(Term, f64) shape:
use sparq_vectors::{hybrid_search, RRF_K};
let fused = hybrid_search(&query, 10, RRF_K, &mut [
&mut |t: &oxrdf::Term| index.nearest_term(t, &graph, &store, 50)
.into_iter().map(|(t, s)| (t, s as f64)).collect(),
&mut |t: &oxrdf::Term| sparq_sim::Sim::new(&graph).most_similar(t, 50),
]);
6. Out-of-RAM build / filesystem-less open
use sparq_vectors::{StreamingWriter, VectorStore};
let mut w = StreamingWriter::create("/tmp/big.spqv", 384)?;
w.put(7, &[0.1; 384])?;
let store = w.finalize()?;
let bytes = std::fs::read("/tmp/big.spqv").unwrap();
let store2 = VectorStore::open_from_bytes(bytes)?;
let idx = sparq_vectors::DiskAnnIndex::open_from_bytes(std::fs::read("/tmp/big.spqg").unwrap())?;
wasm32 (sq-98c): the crate compiles to wasm with the default features — memmap2 is
target-gated out of every wasm32 build (a [target.'cfg(not(target_arch = "wasm32"))']
dependency, NOT a cargo feature: features are additive, so a feature could never remove the
dependency, and a target cfg can't leak into the native build via feature unification). On wasm
VectorStore::open / DiskAnnIndex::open fall back to a buffered std::fs::read into the same
f32-aligned owned backing (works on wasm targets WITH a filesystem, e.g. WASI; on browser
wasm32-unknown-unknown the read fails with a clean I/O error) — open_from_bytes is the
supported browser path for both file kinds. The CI wasm lane build+clippy-gates this and
asserts the wasm graph stays memmap2-free. import_* still need std::fs and are compiled off
the wasm target.
7. Bulk-import embeddings computed ELSEWHERE (NumPy .npy / flat dump)
When vectors come from an external pipeline (Python/sentence-transformers, a vendor batch job)
rather than an in-process Embedder, load the matrix straight into a .spqv. The matrix carries
no term identity, so you supply a parallel slice of dict ids — row i is stored for ids[i]
(the row → dict-id contract). Emit (id, text) pairs from the same scan, embed the texts
out-of-process, then import the matrix with the ids in the same order.
use sparq_vectors::{ImportBinding, ImportSpec, VectorStore};
let ids: Vec<u32> = vec![ 10, 20, 30];
let store = VectorStore::import_npy(
ImportSpec { spqv_path: "graph.spqv", dim: 384, ids: &ids, binding: ImportBinding::Graph(&graph) },
"vecs.npy",
)?;
let store = VectorStore::import_numeric_dump(
ImportSpec { spqv_path: "graph.spqv", dim: 384, ids: &ids, binding: ImportBinding::Unbound },
"vecs.f32",
)?;
Fail-closed: a dtype/byte-order/fortran-order/dimensionality mismatch, a shape[1] != dim, a
shape[0] != ids.len(), a wrong dump length, a duplicate/zero/non-finite row, or a
malformed/oversized .npy header is an Err — never a silent reinterpretation. The .npy header
length and declared shape are bounded against the actual file size before any body allocation
(sq-tzwa hardening). import_* need std::fs and are compiled off the wasm target.
8. k-NN INSIDE SPARQL — the vec: magic predicate (opt-in, feature = vec-predicate)
Express nearest-neighbour search in plain SPARQL, mirroring sparq-text's text: predicate:
the rewrite runs the k-NN, inlines the hit nodes as a VALUES table at the spargebra-algebra
level, and evaluates through the engine's prepared-query seam — so the engine, planner,
executor and wasm bundle are unchanged. This is the only SPARQL-level hook, and it is
OFF by default: the feature is the only thing that pulls sparq-engine into this crate, so
without it sparq-vectors is a pure storage+ANN crate and the base query path carries zero
vec: code (cargo tree -p sparq-vectors lists no sparq-engine/spargebra).
sparq-vectors = { path = "../sparq-vectors", features = ["vec-predicate"] }
use sparq_vectors::{query_vec, VectorStore};
let r = query_vec(&graph,
"PREFIX vec: <http://sparq.dev/vec#>
SELECT ?label WHERE {
?node vec:nearest ( <http://example.org/bolt> 5 ) . # 5 neighbours of bolt
?node rdfs:label ?label . # join to ordinary triples
}", &store)?;
let r = query_vec(&graph,
"PREFIX vec: <http://sparq.dev/vec#>
SELECT ?node ?score WHERE {
( ?node ?score ) vec:search ( \"0.1,0.9,…\" 10 )
} ORDER BY DESC(?score)", &store)?;
The argument lists ( … ) are ordinary SPARQL RDF collections (spargebra lowers them to
rdf:first/rdf:rest, which the rewrite walks). Hard query errors (not silent mismatches):
the neighbour position(s) must be variables; query/k must be constants; the object list must
be exactly ( query k ) and the vec:search subject exactly ( ?node ?score ); a query-vector
literal's dimension must match the store; any other vec: IRI is unknown. An absent/unembedded
seed IRI yields no rows. By default the unfiltered search is the exact full scan
(deterministic, answer-exact — a fine default below ~10⁵ vectors), with top-k membership at a
boundary score tie decided by ascending N-Triples codepoint order (VG-TIE-1, sq-tb9p0) so two
answer-exact implementations return the same top-k set on the same store. The rule is fail-closed
on the embeddable domain: a candidate containing a blank node (whose N-Triples label is
document-local, so it has no stable key) is a hard query error wherever its term — not its score
alone — would decide membership; IRIs, literals, and ground triple terms rank normally.
Approximate vec: for large stores (opt-in, ALSO approx-ann, sq-z589). With BOTH
vec-predicate and approx-ann on, query_vec_approx / prepare_vec_approx take an extra
on-disk DiskAnnIndex argument and run the unfiltered k-NN through that Vamana index instead
of the full scan — for large .spqv stores where brute force is the bottleneck. Everything else
(parse, rewrite, VALUES inlining, joins, error checks, seed-self-exclusion) is identical to
query_vec. APPROXIMATE: recall < 1.0 — the index can miss a true neighbour; use query_vec
(exact) when answer-exactness matters. Pass an index built with DiskAnnIndex::build_for(&store, path, &graph) (and a store bound via .with_fingerprint(&graph)) so the staleness guard catches a
stale index. The filtered path (below) is unaffected — it still uses the cost-model'd filtered
search; the approximate seam is only the unfiltered scan.
# #[cfg(all(feature = "vec-predicate", feature = "approx-ann"))] {
use sparq_vectors::{query_vec_approx, DiskAnnIndex};
let index = DiskAnnIndex::build_for(&store, "entities.spqg", &graph)?;
let r = query_vec_approx(&graph,
"PREFIX vec: <http://sparq.dev/vec#>
SELECT ?node WHERE { ?node vec:nearest ( \"0.1,0.9,…\" 10 ) }",
&store, &index)?;
# }
Automatic filtered ANN (compose with filtered-ann, sq-bvmd + sq-3tjd). When both
vec-predicate and filtered-ann are on, the rewrite derives the candidate id-set
([IdMask], recipe 9) automatically: if the vec: neighbour variable is constrained by
ordinary patterns in the same BGP, those patterns carve out the eligible nodes and the k-NN is
run as a filtered search (nearest_exact_filtered) over just that set — no separate API call.
The constraint is the join-connected sub-BGP of the neighbour variable (sq-3tjd): not only
patterns that directly mention it, but every pattern reachable from it through shared variables
— so the mask honours transitive / multi-variable constraints. A direct-mention-only BGP is a
special case of this (the connected component is just those patterns), so the single-variable
behaviour from sq-bvmd is unchanged.
let r = query_vec(&graph,
"PREFIX vec: <http://sparq.dev/vec#>
SELECT ?node WHERE {
?node vec:nearest ( \"1,0\" 5 ) .
?node <http://ex/kind> <http://ex/Car> . # ← carves out the candidate id-set
}", &store)?;
let r = query_vec(&graph,
"PREFIX vec: <http://sparq.dev/vec#> PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
SELECT ?node WHERE {
?node vec:nearest ( \"1,0\" 5 ) .
?node <http://ex/owns> ?x .
?x rdf:type <http://ex/Vehicle> . # ← reached transitively through ?x
}", &store)?;
The mask is exactly the set the engine binds to the neighbour variable when that connected
sub-BGP is evaluated and the neighbour variable projected, so the filtered top-k is identical to
post-filtering the unfiltered top-k by that same (now transitive) constraint — and therefore a
subset of the unfiltered result. Boundary-score-tie membership in the admitted pool follows the
same VG-TIE-1 N-Triples rule as the unfiltered path (nearest_filtered_costed_tiebreak,
sq-tb9p0), with a node-seed excluded from the pool before the boundary is determined — so the
post-filter equivalence holds exactly, ties included (VG-FILT-2 in answer-exact mode). A pattern disconnected from the neighbour variable (no
shared-variable path) is excluded, so it never narrows the mask. Each vec: request in a BGP gets
its own connected-component mask, derived independently. If the neighbour variable is
unconstrained (no pattern mentions it) the search falls back to the plain unfiltered
nearest_exact (recipe 8's exact behaviour, unchanged). With filtered-ann off, the vec:
predicate is always unfiltered — this composition adds nothing to the vec-predicate-only build.
Cyclic join sub-BGPs (a back-edge to the neighbour variable like ?node :owns ?x . ?x :ownedBy ?node, or a longer cycle among intermediates) are handled correctly (sq-p5oy): the
connected-component computation is a per-pattern fixpoint so it always terminates on a cycle,
and the cyclic sub-BGP is evaluated through the standard engine, so the mask is exactly the engine's
(cycle-correct) binding — filtered == post-filter holds for cyclic constraints too.
Pre-filter vs post-filter — the cost model (sq-7hx6, subsumes sq-ic0n). Once the mask is derived,
the rewrite still has a choice: pre-filter (scan only the masked ids, nearest_exact_filtered)
or post-filter (scan the whole store unfiltered, then drop the disallowed hits, postfilter_exact).
Both return the byte-identical top-k (same ids, same order — see recipe 10), so the choice is purely
about throughput. The rewrite picks per query via CostModel (recipe 10): pre-filter when the mask is
selective (mask_len · scatter_penalty ≤ store_len, default crossover ≈ half the store), post-filter
when it is broad. This is a HEURISTIC over a cost ESTIMATE, not an optimum — it never affects the answer,
only the work. (Deriving the mask itself is unconditional: we must, to stay answer-safe; the cost model
only decides how to use it.)
9. Predicate-constrained (filtered) ANN — only neighbours a BGP admits (opt-in, feature = filtered-ann)
The RDF-native vector differentiator: a SPARQL BGP carves out the eligible graph nodes (e.g.
?node a :Car ; :seats ?s), and the nearest-neighbour search returns only neighbours inside that
candidate set. You compute the exact id-set from the BGP (via graph.id_of / the permutation
indexes / a query's solution ?node column) and hand it to the search as an IdMask — the
"visit mask". OPT-IN and OFF by default; the feature is lean — it adds no new dependency (the
mask reuses the in-tree rustc-hash set) and pulls in neither the engine nor any heavy crate.
sparq-vectors = { path = "../sparq-vectors", features = ["filtered-ann"] }
use sparq_vectors::{nearest_exact_filtered, DiskAnnIndex, IdMask, VectorIndex, VectorStore};
let store = VectorStore::open("entities.spqv")?;
let car_ids = ;
let mask: IdMask = car_ids.into_iter().collect();
let hits = nearest_exact_filtered(&store, query, &mask, 10);
let idx = DiskAnnIndex::open("entities.spqg")?;
let approx = idx.nearest_filtered(query, &mask, &store, 10);
let vidx = VectorIndex::build(&store);
let hnsw = vidx.nearest_filtered(query, &mask, &store, 10);
Tune the crossover / beam with nearest_filtered_with(query, &mask, &store, k, FilterConfig { .. }).
You build the mask yourself here, which keeps this filtered API a pure sparq-vectors surface with
zero engine coupling. The engine-side automatic BGP → mask wiring — deriving the IdMask from
the surrounding BGP and running a filtered vec: search — is wired in recipe 8 (sq-bvmd, when both
vec-predicate and filtered-ann are on), and is built on exactly this nearest_exact_filtered
seam.
10. Pre-filter vs post-filter — the cost model (opt-in, feature = filtered-ann)
Deriving a mask is one cost; using it is another. With a mask in hand there are two ways to the
same top-k: pre-filter (scan only the masked ids) or post-filter (scan the whole store
unfiltered, then drop the disallowed hits). CostModel estimates both and picks the cheaper per
query. The vec: rewrite (recipe 8) calls this automatically; you can also drive it directly.
use sparq_vectors::{nearest_filtered_costed, postfilter_exact, CostModel, Strategy};
let est = CostModel::default().decide( 50, 1000, 10);
assert_eq!(est.strategy, Strategy::PreFilter);
let (hits, est) = nearest_filtered_costed(&store, query, &mask, 10, &CostModel::default());
let post = postfilter_exact(&store, query, &mask, 10);
assert_eq!(post, sparq_vectors::nearest_exact_filtered(&store, query, &mask, 10));
The model. m = |mask|, n = store_len. Pre-filter touches m scattered rows (random-access
gather into the mmap), post-filter streams n rows sequentially; a scattered row is modelled as
scatter_penalty × (default 2.0) a sequential one. Pre-filter iff m · scatter_penalty ≤ n —
default crossover ≈ half the store. k/dim cancel out of the exact-scan comparison (both branches
do dim-width work per touched row, then take k), so they do not enter the crossover; they are
surfaced in CostEstimate for a future approximate backend.
Why it is answer-safe. nearest_exact returns a complete ranking of the store, so post-filtering
it and pre-filtering the mask reduce to ranking the same admitted ids by the same comparator — the
top-k is byte-identical (asserted in tests/cost_model.rs). There is no over-fetch recall
boundary for this exact backend: a full ranking can only under-fill k when the mask admits fewer
than k stored vectors, and then the pre-filter scan is equally short. (overfetch_target(k, m, n) = ceil(k/selectivity) sizes the first fetch for the approximate backend, whose bounded candidate
list CAN under-fill — the iterative over-fetch path that handles it is recipe 11.)
Honest scope. This is a HEURISTIC over a cost estimate, not an optimum: scatter_penalty is a
single modelled constant (non-canonical, not a measured fit), real cache/SIMD behaviour is
hardware-dependent, and a planner that has not yet evaluated the sub-BGP would feed an estimated
cardinality (here the mask is materialised, so m is exact). A mis-estimate costs only throughput —
never the answer.
11. Approximate filtered ANN with iterative over-fetch (opt-in, feature = filtered-ann + approx-ann)
The cost model (recipe 10) is for the exact backend, which post-filters a complete ranking and
so never under-fills. An approximate backend returns a bounded candidate list: post-filtering it
can under-fill k when the admitted ids cluster below the fetched prefix (the recall boundary).
nearest_filtered_overfetch removes that boundary by growing the fetch and retrying.
# #[cfg(all(feature = "filtered-ann", feature = "approx-ann"))] {
use sparq_vectors::{
nearest_filtered_overfetch_default, ApproxBackend, ExactBackend, DiskAnnIndex, VectorStore,
};
let store = VectorStore::open("entities.spqv")?;
let idx = DiskAnnIndex::open("entities.spqg")?;
let exact = ExactBackend::new(&store);
let exact_hits = nearest_filtered_overfetch_default(&exact, query, &mask, 10);
let approx = ApproxBackend::new(&idx);
let approx_hits = nearest_filtered_overfetch_default(&approx, query, &mask, 10);
# let _ = (exact_hits, approx_hits); }
Honesty (load-bearing). Iterative over-fetch fixes under-fill (returning < k when k exist); it
does not make the approximate backend exact. Even a full-store fetch from ApproxBackend is the
approximate ranking, so a true admitted neighbour the index never visits is still missed — recall
stays < 1.0 (measured floor 0.90@10 in tests/overfetch.rs, NOT claimed as exactness). Only
ExactBackend is answer-exact. The A1-paper recall evidence is for the exact/transitive answer-safe
path (recipe 10), and does not transfer to this approximate path. Implement your own backend by
impl-ing AnnBackend (one candidates(query, fetch) method + len).
12. Incremental add / remove / update without a full rebuild (opt-in, feature = delta)
A .spqv is build-once-immutable, so one new/removed entity would otherwise force a full re-embed +
rebuild. The delta feature adds a delta sidecar over the immutable base — add / remove
/ update write an append map + tombstone set, and get / iter / len (hence search) transparently
union base+delta and honour tombstones. compact folds it back into a fresh base. Lean (no new dep).
# // cargo build -p sparq-vectors --features delta
use sparq_vectors::VectorStore;
let mut store = VectorStore::open("graph.spqv")?; // a finalized base
store.add(new_id, &vec_for_new_id)?; // NEW id (errs if already present)
store.update(existing_id, &new_vec)?; // replace an existing id (errs if absent)
store.remove(stale_id); // tombstone (returns bool; re-add allowed)
// search transparently sees the delta — no rebuild, no extra API:
let hits = sparq_vectors::nearest_exact(&store, &query, 10);
// PERSIST the delta so the mutations survive a restart WITHOUT a compact (sq-7e50):
store.save_delta()?; // crash-durable .spqd sibling (tmp+fsync+rename)
// …later, in a fresh process: reopen the base AND replay the persisted delta:
let reopened = VectorStore::open_with_delta("graph.spqv")?; // == the (base + delta) view it had before
// or fold the delta into a fresh base == a from-scratch rebuild over the final vector set:
let compacted = store.compact("graph.spqv", &graph)?; // bound to graph's fingerprint generation
# Ok::<(), String>(())
Generation tie. A delta carries the graph generation (the sq-32i5 fingerprint) it was built
against; apply_delta rejects a delta whose generation differs from the receiving base, so its dict-ids
can never mis-key onto a different graph. The persisted .spqd header carries that same generation, so
open_with_delta routes the replay through apply_delta and a sidecar written against a different
graph generation is rejected, never silently mis-keyed. Durability (sq-7e50). Two durability paths:
compact materializes the delta into a fresh validated .spqv; save_delta persists the delta itself
to a crash-durable .spqd (write-tmp + fsync + atomic rename) that open_with_delta replays on reopen
— so incremental mutations survive a process restart with no compact. A truncated/torn sidecar is
detected (an exact-length check) and rejected, never read out of bounds. put is unchanged (still errors
after finalize) — add is the additive path.
13. Structure-aware preprocessing — closure + type-constrained negatives (opt-in, feature = structure)
Research-grade P0 of the structure-aware-vectorisation epic. Two additive, buildable primitives an
out-of-process KGE trainer consumes — this crate trains nothing (embeddings are produced
outside it) and serves no exact answer.
- Closure-before-vectorise — run the
sparq-reason RDFS/OWL-RL closure before the
vectoriser sees the graph, so entailed rdf:type/subClassOf/domain/range triples are real
facts the encoder, type extractor, and sampler read (design §5.A; the reasoner is sound+complete
for its profile, incremental closure property-tested == from-scratch).
- Type-constrained negative sampling (Krompass et al. 2015) — a
NegativeSampler corrupts a
positive triple's head only to domain-consistent entities and its tail only to range-consistent
ones, reading declared+observed domain/range from sparq-introspect. The SamplingMode
(Unconstrained / TypeConstrained) is the on/off ablation switch (design §6.B) — a harness
measures Hits@k/MRR both ways on the same seed. No benchmark numbers exist; none are claimed.
# // cargo build -p sparq-vectors --features structure
use sparq_reason::Profile;
use sparq_vectors::{
close_for_vectorise, Corrupt, NegativeSampler, SamplingMode, TermScope, TypeConstraints,
};
let closed = close_for_vectorise(turtle_src, "turtle", Profile::Rdfs)?; // materialise entailed facts
let constraints = TypeConstraints::mine(&closed.graph); // declared+observed domain/range
let sampler = NegativeSampler::new(&closed.graph, &constraints, SamplingMode::TypeConstrained);
let negatives = sampler.sample([h, r, t], Corrupt::Tail, 16, /*seed*/ 42); // type-valid tail corruptions
// [GPT-5.6] RDF 1.2 triple terms remain excluded by default. The explicit ablation arm admits
// them, with atomic slots drawing only atomic candidates and triple-term slots only triple terms.
let scoped = NegativeSampler::new_scoped(
&closed.graph,
&constraints,
SamplingMode::TypeConstrained,
TermScope::Embeddable,
);
let triple_term_pool_size = scoped.triple_term_count();
# let _ = (negatives, triple_term_pool_size);
# Ok::<(), String>(())
Fail-open by design. A predicate with no known domain (or range) class degrades to uniform
corruption for that side — a missing/wrong schema never deadlocks the sampler. The sampler is
deterministic for a fixed (seed, mode, triple) and applies the standard "filtered" guard (never
emits a corruption that is itself a true triple). Honesty: the empirical benefit of either prior is
unproven — both ship behind the ablation precisely because the literal/type-aware KGE literature
reports inconsistent gains; this slice is the buildable inputs. The trainer that consumes them now
exists behind the kge feature (recipe 14).
TermScope::IriBlank is the Default and preserves the former named-node/blank-node entity space.
TermScope::Embeddable is an explicit RDF 1.2 triple-term measurement arm. It does not make triple
terms compositional: it exposes each term as a graph node through rdf:reifies, while the embedded
term's internal (s, p, o) remains opaque. NegativeSampler::new retains the default scope;
new_scoped is the opt-in entry point. The separate corruption pools prevent sort-trivial negatives.
14. KGE measurement foundation — DistMult/ComplEx trainer + filtered link-prediction ablation (opt-in, feature = kge)
The kge feature (implies structure) is the measurement foundation for the epic: a thin,
CPU-only, deterministically-seeded shallow KGE that consumes the P0 closure + type-constrained
negatives to produce embeddings, plus the standard filtered link-prediction harness that measures
them. It is the instrument — it makes no accuracy claim. Two scoring models behind the
ModelKind axis: DistMult (Yang 2015, symmetric) and ComplEx (Trouillon 2016, asymmetric).
No new dependency (hand-rolled SGD, no ML crate).
# // cargo build -p sparq-vectors --features kge
use sparq_vectors::{close_for_vectorise, train, ModelKind, TrainConfig, SamplingMode, TypeConstraints};
use sparq_reason::Profile;
let closed = close_for_vectorise(turtle_src, "turtle", Profile::Rdfs)?; // closure-before-vectorise
let tc = TypeConstraints::mine(&closed.graph);
// small() defaults to the ASYMMETRIC ComplEx; small_with_model(..) selects DistMult explicitly.
let cfg = TrainConfig::small(SamplingMode::TypeConstrained, /*seed*/ 7); // tiny, reproducible
let (model, report) = train(&closed.graph, &tc, cfg);
assert!(report.loss_decreased()); // it learns
let score = model.score(h, r, t); // model.model is ModelKind::{DistMult,ComplEx}
# Ok::<(), String>(())
The model axis is load-bearing. DistMult scores (h,r,t) and (t,r,h) identically, so on a
relation whose true edges are directional it is structurally near-random — it cannot place the
head above the tail. Both synthetic eval slices here are ~100 % directional, so a DistMult ablation
runs in a near-random regime where the inter-cell deltas are noise. EvalConfig::small defaults to
the asymmetric ComplEx, and any ablation delta must be read off the ComplEx run, not DistMult.
The eval harness runs the 2×2 P0 ablation matrix and reports filtered MRR / Hits@1/3/10 with a
long-tail breakdown. Because a single seed's inter-cell delta can be a handful of rank hits (noise),
report each cell as a mean ± std over several seeds with run_ablation_multiseed:
# // cargo build -p sparq-vectors --features kge
use sparq_vectors::{run_ablation_multiseed, synthetic_relational_ttl, EvalConfig};
let ttl = synthetic_relational_ttl(400, /*seed*/ 42); // or your own KG text
let template = EvalConfig::small(13); // ComplEx by default
let cells = run_ablation_multiseed(&ttl, "turtle", template, &[1,2,3,4,5])?; // [(F,F),(F,T),(T,F),(T,T)]
for c in &cells {
println!("closure={} type-neg={} MRR={:.4}+/-{:.4}",
c.closure, c.type_constrained, c.metrics.mrr.mean, c.metrics.mrr.std);
}
# Ok::<(), String>(())
// A delta smaller than the combined spread of the two cells is NOT yet evidence.
Read the effect off the PAIRED delta, not the unpaired means (sq-4891y). Comparing two cells'
mean ± std and eyeballing the gap is the unpaired view — needlessly noisy, because the four cells
of one seed share the split / init / negatives, so most per-seed variance is shared and cancels in
their difference. run_ablation_multiseed_paired returns the paired per-seed closure delta (and
type-negative delta) as a PairedDelta { mean, std, se, n } whose paired std is far smaller than
the unpaired sum-of-stds, with a significant_at(k) gate (mean − k·se > 0, requires n ≥ 2):
# // cargo build -p sparq-vectors --features kge
use sparq_vectors::{run_ablation_multiseed_paired, synthetic_gufo_ttl_sized, EvalConfig};
let ttl = synthetic_gufo_ttl_sized(400, /*density*/ 3, 9); // denser → more held-out triples/seed
let template = EvalConfig::small(13); // ComplEx; bump epochs for a tighter run
let r = run_ablation_multiseed_paired(&ttl, "turtle", template, &(0..12).collect::<Vec<_>>())?;
let d = r.closure_mrr; // headline closure-prior lift, variance-reduced
println!("closure delta={:.4} se={:.4} sig@2se={}", d.mean, d.se, d.significant_at(2.0));
# Ok::<(), String>(())
// Firm-up bar: a positive PAIRED delta clearing 2·se on a SCHEMA-BEARING slice under ComplEx.
synthetic_gufo_ttl_sized(n, density, seed) is the schema-bearing firm-up slice: the rigid
Person kind is asserted on nobody, so the RDFS closure must materialise it (the closure axis
genuinely bites — unlike schema-free WN18RR/FB237). A higher density adds more learnable held-out
triples per seed, shrinking the per-seed MRR variance, while keeping the decoy set (non-triviality)
intact. examples/kge_ablation.rs prints the paired delta + an LR sweep.
Filtered protocol (load-bearing). Each ranking removes every known-true triple (train + valid +
test) before scoring the held-out one — the established Bordes 2013 protocol; getting it wrong
invalidates every number. Train/valid/test are a leakage-free partition (a triple is in exactly one
split); the trainer sees train only. Only non-schema relations are prediction targets
(SCHEMA_PREDICATES — rdf:type/subClassOf/domain/range are structural context, never targets,
so the closure axis is not distorted by trivially-derivable entailed types). The runnable ablation is
examples/kge_ablation.rs (runs BOTH models, multi-seed; a real dataset goes behind
SPARQ_KGE_DATASET=/path/to/kg.nt). All numbers are INDICATIVE only (the work-box is
non-canonical) and are never baked into docs. Adoption gate: no prior is adopted from these
synthetic, work-box, single-seed figures — adoption requires a real dataset (WN18RR/FB15k-237
subset), on a canonical machine, under the asymmetric model, with multi-seed reporting.
14a. UFO/gUFO priors — answer-safe serve-time disjointness mask (opt-in, feature = kge)
The gUFO-prior cell is wired as an explicitly selected serve-time ablation. EvalConfig::small
sets gufo_prior = false, so the default run does not construct the mask and retains the baseline
ranking path. Set it to true to mine only UFO-provable disjointness from the input graph and remove
provably incompatible candidates using the predicate's declared rdfs:domain or rdfs:range.
Training is unchanged. The reader fails closed when identity or nature evidence is ambiguous, and an
untyped candidate or predicate without a declared signature is never excluded.
# // cargo build -p sparq-vectors --features kge
use sparq_vectors::{run_ablation, EvalConfig, GUFO_NS};
let mut cfg = EvalConfig::small(13);
assert!(!cfg.gufo_prior); // the behavioural prior is default OFF
cfg.gufo_prior = true;
cfg.gufo_ns = GUFO_NS; // or an explicitly declared non-canonical namespace
let cells = run_ablation(turtle_src, "turtle", cfg)?;
assert!(cells.iter().all(|cell| cell.gufo_prior));
# Ok::<(), String>(())
The structure feature exposes the read-only UfoPriors, UfoVocabulary, Rigidity,
OntologicalNature, and GUFO_NS API without the trainer. Use UfoPriors::mine(graph) for the
canonical namespace or mine_with_namespace(graph, ns) when the dataset explicitly uses another
namespace. proven_disjoint_pairs() and proven_subsumptions() return dictionary-id facts only;
augment_oracle() feeds the proven pairs into DisjointnessOracle::absorb_proven_pairs. These APIs
do not mint terms or write inferred triples back into the graph.
The load-bearing guards are exact: the feature is absent from default = [], kge merely implies
the already opt-in structure feature, EvalConfig::small keeps the behavioural switch false, and
tests compare OFF runs deterministically plus ON/OFF output exactly on a gUFO-free graph. The mask
may improve or preserve a filtered rank, never remove the held-out answer.
RDF 1.2 triple-term visibility is also default-off. [GPT-5.6] Every TrainConfig preset sets
term_scope to TermScope::IriBlank, preserving the existing pipeline. To measure statement-level
structure, use synthetic_rdf12_ttl (or your own N-Triples data with rdf:reifies) and the paired
run_quoted_ablation runner. Its QuotedAblation reports common-random-number ON−OFF deltas; split
membership and the ranking pool remain atomic in both arms, so the comparison changes only training
visibility. synthetic_rdf12_parts exposes the generated fixture partitions when a caller needs to
audit them. This is a measurement surface, not an accuracy claim.
Statement-level quoted-triple encoding is compositional and derived (sq-1e5kk). [SONNET-4.6]
TrainedModel::encode_quoted_term(&graph, id) (and the unpacked forms encode_statement(h, r, t) /
encode_statement_rows) returns a triple term's per-component interaction vector composed from
its (s, p, o) constituents' trained rows — DistMult h∘r∘t (sums to the score), ComplEx
h∘r∘conj(t) in real/imaginary halves (the real half sums to the score) — so the quoted content
is reachable even under the default IriBlank scope, where the term itself has no node row. It is
deterministic and derived (no new parameters, one level deep, never recursive), it lives in the
model's interaction space (never compare it against entity rows or store it beside them), and it
leaves the node-level opaque-row path above intact. No accuracy claim — adopting either
representation stays measurement-gated.
14b. Provenance-weighting w(t) — weight training by PROV-O/DQV quality (opt-in, feature = structure; measurement under kge)
Research-grade Phase 4 of the provenance-driven GenAI-KB epic: derive a per-triple
provenance-quality weight w(t) ∈ (0,1] from a graph's PROV-O / DQV annotations so a
high-assurance fact contributes a full-strength training gradient and a low-assurance / low-source
fact a down-weighted one (the CKRL confidence-weighted-loss move). ProvenanceWeights mirrors
ShaclPriors: a read-only id-level scan of pkg:confidence / pkg:assurance /
prov:wasDerivedFrom (the same terms the Phase-1 sparq-nlq join reads and the Phase-3 pkg.ttl
DQV terms declare) that pulls no engine — it consumes only sparq-core's public id-scan API, so
the structure feature stays lean.
# // cargo build -p sparq-vectors --features structure
use sparq_vectors::{close_for_vectorise, ProvenanceWeights, WeightMode};
use sparq_reason::Profile;
let closed = close_for_vectorise(ttl, "turtle", Profile::Rdfs)?;
let weights = ProvenanceWeights::mine(&closed.graph);
// w(t) = clamp(assurance_mult · head_confidence · min_source_reliability, floor, 1.0).
// Uniform mode is the ablation-OFF baseline (always 1.0); Provenance mode applies the quality weight.
let w = weights.weight_of([h, r, t], WeightMode::Provenance); // a head with NO provenance → 1.0 (fail-open)
# Ok::<(), String>(())
w(t) combines the head's pkg:confidence (epistemic weight), an assurance multiplier
(secx:Proven → high, Claimed → mid, Conjectured → low; configurable via WeightConfig), and
the min pkg:confidence over its prov:wasDerivedFrom sources (a fact is only as reliable as
its least-reliable source). It is clamped to [floor, 1.0] so a positive is down-weighted, never
dropped (a zero weight would silently delete it from the loss). The kge trainer reads it via the
new TrainConfig::weight_mode (default Uniform); under Provenance the positive step's
effective LR is scaled by w(t) (negatives are unweighted — a corruption has no provenance).
Ablation + the adopt/abandon bar. run_weight_ablation trains the same config twice per seed
(weighting ON vs OFF, holding closure + type-negatives fixed) and returns a paired per-seed MRR /
Hits@10 delta — WeightAblation::mrr_significant_at(k) is the firm-up gate:
# // cargo build -p sparq-vectors --features kge
use sparq_vectors::{run_weight_ablation, synthetic_provenance_ttl, EvalConfig};
let ttl = synthetic_provenance_ttl(160, 5); // low-assurance edges are deliberately the NOISIER ones
let ab = run_weight_ablation(&ttl, "turtle", EvalConfig::small(21), &[1,2,3,4])?;
println!("Δmrr={:.4} se={:.4} adopt@2se={}", ab.mrr.mean, ab.mrr.se, ab.mrr_significant_at(2.0));
# Ok::<(), String>(())
No accuracy claim is made. Over a provenance-free graph the two arms are byte-identical and the
delta is exactly zero (the no-op invariant — a plain graph is unchanged). The
confidence/literal-aware KGE literature reports inconsistent gains, so any lift is unproven and
dataset-dependent; the bead's discipline is adopt only if the measured lift clears a pre-registered
bar, and ABANDON (and say so) otherwise.
Integration points 2–3 (sq-oy9ya, design §USE-1). w(t) is now also threaded into the two
other points sparq-vectors has, both behind the same WeightMode ablation and both no-ops
under Uniform / a provenance-free graph:
# // cargo build -p sparq-vectors --features structure
use sparq_vectors::{ProvenanceWeights, WeightMode, Block, Encoder, Metric};
# fn demo(weights: &ProvenanceWeights, subj_a: u32, subj_b: u32) -> Result<(), String> {
// (2) confidence-weighted STRUCTURAL-SKETCH / characteristic-set pooling: pool a node's
// multi-valued contributions weighted by each value's provenance, NOT a uniform mean.
// Under WeightMode::Uniform this is EXACTLY the arithmetic mean (the ablation-off baseline).
let contribs = vec![(subj_a, vec![1.0, 0.0]), (subj_b, vec![0.0, 1.0])];
let pooled = weights.pool_weighted(&contribs, WeightMode::Provenance)?; // higher-quality value dominates
// (3) per-Block query-time FUSION weight: aggregate the incident-edge provenance into a per-block
// multiplier and attach it to a Block; the existing fuse_rrf_weighted / fuse_scores path
// consumes Block::fusion_weight() so a low-quality modality contributes less to the ranking.
let bw = weights.block_weight([subj_a, subj_b], WeightMode::Provenance); // mean of the per-subject w(t)
let block = Block::new(Encoder::Numeric, Metric::Euclidean, 0, 16).with_weight(bw);
assert!(block.fusion_weight() > 0.0); // a valid (0,1] fuse weight; None ≡ 1.0 (fail-open)
# let _ = pooled; let _ = block; Ok(()) }
The .spqv SchemaHeader round-trips the per-block weight (format v2; a v1 sidecar still
parses, every block read back as the fail-open 1.0). The weight is layout metadata, not part
of a Block's identity — PartialEq/Eq ignore it, so the header round-trip contract is unchanged.
No accuracy claim; like point 1, adoption is measurement-gated.
15. Typed-literal encoders — order-preserving numeric / boolean / date + schema header (opt-in, feature = structure)
Research-grade P1: the typed-literal encoders that turn a node vector into a structured
partitioned object (design §3). All are pure functions keyed by datatype in the encode
module — no training, no graph state, no I/O.
- Datatype
router — the datatype IRI alone selects the encoder family
(Encoder::{Numeric,Boolean,Date,Other}). Exact, free.
NumericEncoder — the order-preserving magnitude encoder (the one the design gates as
formally provable): per-predicate quantile-normalise + a strictly-monotone thermometer code
whose L2 distance is monotone in value distance over the whole observed range (NOT a periodic
Fourier code). metamorphic_monotone is the provable gate; a sine code FAILS it by design.BooleanEncoder — one ±1 sign dimension, exact, 2-valued, round-trips (true→+1,
false→−1; decode(encode(b)) == b).DateEncoder — maps xsd:date/dateTime/dateTimeStamp/gYear to an order-preserving
epoch lane (via sparq-core's Timeline) and reuses the numeric encoder, so chronological order
is preserved.SchemaHeader — the self-describing .spqv partition descriptor: contiguous Blocks
(encoder + Metric + dim span) with a metric-correctness guard (check_euclidean) so a whole-row
L2/cosine search never silently runs over a non-Euclidean (e.g. future taxonomy) block. Round-trips
to bytes (SPQS magic) for a sidecar.
# // cargo build -p sparq-vectors --features structure
use sparq_vectors::{route, Encoder, NumericEncoder, BooleanEncoder, DateEncoder,
SchemaHeader, Block, Metric};
assert_eq!(route("http://www.w3.org/2001/XMLSchema#integer"), Encoder::Numeric);
let enc = NumericEncoder::fit([1.0, 30.0, 31.0, 70.0, 5000.0], /*dim*/ 16); // per-predicate fit
let v30 = enc.encode(30.0); // order-preserving block
assert_eq!(BooleanEncoder::decode(BooleanEncoder::encode(true)[0]), true); // exact round-trip
let header = SchemaHeader::new(vec![
Block::new(Encoder::Numeric, Metric::Euclidean, 0, 16), // .with_weight(w) for a fusion weight
])?;
header.check_euclidean()?; // every block is L2-searchable
# Ok::<(), String>(())
Honesty. Order-preservation, boolean exactness, datatype-dispatch correctness, and the metric
guard are proven (encode.rs tests). Whether the typed encoders raise downstream
