event-store-semantics

name

event-store-semantics

description

The authoritative behavioral contract of Quartz's event stores — `IEventStore` and its reference SQLite implementation (`nip01Core/store/sqlite/`). Use when implementing or asserting parity with a Quartz event store (external engines like Vespa, the filesystem store, geode), answering filter-semantics questions (since/until inclusivity, tag OR/AND, multi-filter limits, ordering tiebreaks), or working on the write-path rules for replaceable/addressable supersession, NIP-09 deletions, NIP-40 expiration, NIP-62 vanish, NIP-45 counts, or NIP-50 search inside the store. Every behavior has a named rule id (STORE-Fxx/Wxx/Dxx/Sxx/Cxx) so downstream implementations can annotate divergences precisely.

Event Store Semantics — the IEventStore / SQLite-store contract

The SQLite EventStore (quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/) is the de-facto reference implementation of what a Quartz event store must do. Other implementations — the in-repo filesystem store (nip01Core/store/fs/, held to parity by quartz/src/jvmTest/.../store/fs/FsParityTest.kt) and external engines (e.g. a Vespa-backed store) — reimplement its observable behavior and assert parity in CI. This skill states that behavior as named, numbered decisions so a parity divergence becomes a lookup, not an archaeology session through QueryBuilder/MergeQueryExecutor.

Every rule below was verified against the code as of this skill's last update. When you change store behavior, update the rule here in the same PR and add a line to the Semantics changelog — downstream implementations pin Quartz by commit and review pin bumps against this file.

Key files

Concern	File
Public contract (KDoc is normative)	nip01Core/store/IEventStore.kt
High-level store (owns pool + planner)	sqlite/EventStore.kt, sqlite/SQLiteEventStore.kt
Filter → SQL, ordering, limits, counts	sqlite/QueryBuilder.kt
k-way merge fast path (feed shapes)	sqlite/MergeQueryExecutor.kt
Schema, tag hashing, immutability	sqlite/EventIndexesModule.kt, sqlite/TagNameValueHasher.kt, sqlite/SeedModule.kt
Replaceable / addressable supersession	sqlite/ReplaceableModule.kt, sqlite/AddressableModule.kt
NIP-09 / NIP-40 / NIP-62 / ephemeral	sqlite/DeletionRequestModule.kt, sqlite/ExpirationModule.kt, sqlite/RightToVanishModule.kt, sqlite/EphemeralModule.kt
NIP-50 FTS	sqlite/FullTextSearchModule.kt (see also the searchable-events skill)
Index/feature toggles	sqlite/IndexingStrategy.kt (client default) and geode's RelayIndexingStrategy.kt (relay preset)
Operational README	sqlite/README.md (concurrency, pragmas, maintenance)

Executable spec: the test suites in quartz/src/commonTest/.../store/sqlite/ (BasicTest, ReplaceableTest, AddressableTest, DeletionTest, ExpirationTest, RightToVanishTest, SearchTest, SearchRelevanceOrderTest, MergeQueryCorrectnessTest, TagMergeCorrectnessTest, QueryAssemblerTest, SnapshotIdsForNegentropyTest, FilterMatcherTest, …). If a rule here ever contradicts a test, the test wins — and this file has a bug to fix.

Kind classes (used throughout)

• Replaceable: kind 0, kind 3, and 10000 ≤ kind < 20000.
• Ephemeral: 20000 ≤ kind < 30000.
• Addressable: 30000 ≤ kind < 40000.
• Everything else is a regular event.

---

Filter matching (STORE-F)

STORE-F01 — since/until are both inclusive. since compiles to created_at >= ?, until to created_at <= ? (QueryBuilder uses greaterThanOrEquals/lessThanOrEquals everywhere). An event with created_at == since == until matches.

STORE-F02 — ids and authors are exact-match only. They compile to =/IN against the full 64-char hex columns. NIP-01 prefix matching is NOT supported anywhere in the store. (Filter's constructor logs an error for non-64-char ids/authors but still sends them; they simply never match.)

STORE-F03 — tag filter combination. Within one tag name, values are OR (tag_hash IN (…)). Across different tag names in the same filter, conditions are AND (each extra name becomes another event_tags self-join). tagsAll (NIP-91 &x syntax) demands every listed value be present on the event — one join + equality per value — and composes by AND with any plain tags in the same filter.

STORE-F04 — only single-letter tag names are indexed (by default). DefaultIndexingStrategy.shouldIndex indexes a tag iff tag.size >= 2 && tag[0].length == 1. A filter on a multi-letter tag name (#title, #alt) matches nothing in the SQLite store. Deployments can widen shouldIndex, but the stock contract is single-letter-only.

STORE-F05 — d is special-cased out of the tag index. #d values are matched against the event_headers.d_tag column, not event_tags (Filter.toFilterWithDTags()). Consequences: #d works on addressable events (which populate d_tag); when all kinds are addressable the query adds kind >= 30000 AND kind < 40000 to pin the addressable index. Only use #d via plain tags. A #d under tagsAll is handled inconsistently: on the simple (no other tags/search) path it degrades to OR semantics (toFilterWithDTags folds it into dTags), and when tags["d"] is also present it is dropped entirely; on the tag-join path it is ignored. (An event has one d-tag, so AND-across-values could never match anyway.)

STORE-F06 — tag and author matching in the tag path is hash-based. event_tags stores a 64-bit MurmurHash3 of (tag name, value) keyed by a per-database random seed (SeedModule, TagNameValueHasher); the p/e/a-owner columns are hashes too. There is no post-verification of hash matches, so a hash collision would return a false positive. Probability is negligible in practice but nonzero — a parity harness comparing against an exact-match engine should know this is the one place the reference can (theoretically) over-match.

STORE-F07 — multiple filters are a union with dedup; limit is per-filter. Each filter becomes its own row-id subquery with its own ORDER BY … LIMIT; branches are combined with SQL UNION (dedup by row). There is no global limit — a 3-filter query with limits 10/20/30 can return up to 60 events, presented in one merged created_at DESC ordering. NIP-45 counts and negentropy snapshots dedup the same way (SELECT DISTINCT / UNION).

STORE-F08 — result ordering. Non-search queries order created_at DESC. The id ASC tiebreak on equal created_at is applied only when IndexingStrategy.useAndIndexIdOnOrderBy = true — which is false in the client default and in geode's relay preset. So by default, same-second ordering is unspecified (SQLite returns them in storage order). Any newest-N is valid; a parity suite must not assert same-created_at order unless it configures the flag. One extra caveat with the flag ON: the MergeQueryExecutor tag-stream path still yields same-second ties in rowid order (its cursors run off event_tags, which has no id column) — a valid newest-N that may differ byte-for-byte from the single-SQL ordering.

STORE-F09 — the merge fast path returns the same set. Single-filter queries of the shape "authors (+kinds) + limit" or "one #x IN-list (+kinds) + limit" (≤2048 streams) route through MergeQueryExecutor, a k-way newest-first merge over per-(kind,author) / per-(tag-value,kind) index cursors with dedup by id on the tag shape. This is an optimization, not a semantics change — MergeQueryCorrectnessTest/TagMergeCorrectnessTest assert set-equality with the single-SQL plan (ordering caveat per STORE-F08).

STORE-F10 — empty filter. query(Filter()) / count(Filter()) match everything (Filter.isEmpty() → the "everything" query). delete(Filter()) is deliberately asymmetric: it deletes nothing and returns 0, so a stray empty filter can't wipe the store (documented on QueryBuilder.delete).

STORE-F11 — empty lists (kinds = emptyList() etc.) are a client error with inconsistent handling; don't rely on either outcome. On the single-filter simple path an empty list renders as 1 = 0 → matches nothing. But Filter.isEmpty() treats empty lists the same as null, so on the multi-filter union path such a filter contributes no subquery — and a list of only empty-list filters degrades to the match-everything query. Known quirk; treat empty-list filters as invalid input rather than replicating this shape.

STORE-F12 — limit edge cases. limit = 0 compiles to LIMIT 0 → zero rows. limit = null means unbounded. Negative limits are not defended against (don't send them).

STORE-F13 — the in-memory matcher is a separate (simpler) implementation. Filter.match(event) (FilterMatcher) is used for live-stream matching, not storage queries; it checks ids/authors/kinds/tags/tagsAll/since/until but not search or limit. Parity work targets the SQL semantics above, not FilterMatcher.

---

Write path (STORE-W)

Inserts run every module in one transaction: header+tags → NIP-09 side effects → expiration row → FTS row → vanish side effects. A trigger RAISE(ABORT, …) rejects the whole row with the messages quoted below (they surface as the NIP-01 OK false reason).

STORE-W01 — replaceable supersession. Unique index on (kind, pubkey) for replaceable kinds. A BEFORE INSERT trigger deletes any stored version that is older — meaning created_at smaller, or equal created_at with lexicographically larger id (NIP-01 lowest-id-wins). Inserting a version that is not newer under that ordering leaves the stored row in place and fails the unique index → rejected (UNIQUE constraint failed). Net contract: exactly one version stored; newest wins; ties broken by lowest id; older re-inserts blocked.

STORE-W02 — addressable supersession. Same as W01 with unique index (kind, pubkey, d_tag) over 30000 ≤ kind < 40000. Nuance: d_tag is populated from the parsed event class (AddressableEvent.dTag()); an addressable-range kind whose class doesn't parse as AddressableEvent stores d_tag NULL, and SQLite treats NULLs as distinct in unique indexes — such events don't supersede each other. An event with no d tag parses as dTag() = "" (empty string), which does dedupe normally.

STORE-W03 — ephemeral events are never stored but are acked as accepted. insert() returns silently and batchInsert reports Accepted for 20000 ≤ kind < 30000 without writing (the live relay stream still broadcasts them). A DB-level backstop trigger (blocked: cannot store ephemeral events) rejects any that sneak past the app-level check.

STORE-W04 — expired events are rejected at insert. App-level check (event.isExpired()) plus a trigger on the expiration-row insert (blocked: this event is expired when expiration <= unixepoch()). Single-event insert throws; batchInsert returns Rejected.

STORE-W05 — expiry is enforced at insert and by sweep, NOT at query time. Events with a future expiration store a row in event_expirations. Nothing filters them out of queries after the timestamp passes: a query between expiry and the next deleteExpiredEvents() sweep returns the expired event. Operators run the sweep periodically (README recommends ~15 min). Re-inserting an already-expired event after the sweep is rejected per W04.

STORE-W06 — GiftWrap ownership is the recipient. For kind 1059 the store computes pubkey_owner_hash from the p-tag recipient (falling back to the random signer key if absent). All owner-scoped machinery — NIP-09 re-insert blocking, NIP-62 vanish deletion and blocking — operates on that owner hash, so a user's deletions/vanish remove giftwraps addressed to them, even though the wrap's pubkey is a one-time key. (Consequently GiftWraps are also excluded from authorsMissingOutbox().)

STORE-W07 — immutability. event_headers/event_tags rows are never updated (BEFORE UPDATE triggers abort). All supersession is delete + insert; event_tags, event_expirations, event_vanish, and the FTS row follow the header by ON DELETE CASCADE / trigger.

STORE-W08 — batch insert. One outer transaction, one SAVEPOINT per row: a bad row rolls back alone and reports Rejected(reason); the rest commit. If the outer commit fails, every entry is treated as Rejected (the IEventStore.batchInsert contract). Outcomes are returned in input order; OK frames pair by event id, not order.

---

Deletion lifecycle — NIP-09 / NIP-62 (STORE-D)

STORE-D01 — delete by id. A kind-5's e tags delete stored events with those ids whose owner is the kind-5's author (pubkey_owner_hash match — recipient for giftwraps per W06). The id path has no timestamp condition: it deletes the target regardless of the relative created_at values.

STORE-D02 — delete by address. A kind-5's a tags delete events at that (kind, pubkey, d_tag) coordinate with created_at <= deletion.created_at — inclusive; a version newer than the deletion survives. Only coordinates whose pubkey equals the kind-5's author are honored. Replaceable coordinates (kind:pubkey: with no d-tag) get the same created_at <= treatment against (kind, pubkey).

STORE-D03 — cross-author kind-5s are stored but inert. A deletion naming someone else's events is inserted like any regular event (it may be useful to other relays/clients) but its delete pass removes zero rows and creates no blocking.

STORE-D04 — re-insert blocking. A BEFORE INSERT trigger rejects (blocked: a deletion event exists) any event whose id (e-hash) or address (a-hash) is named by a stored kind-5 from the same owner with deletion.created_at >= event.created_at. Note the asymmetry with D01: a backdated id-deletion (older created_at than its target) still deletes on arrival, but would not block a later re-insert.

STORE-D05 — a kind-5 CAN delete another kind-5, and doing so un-blocks its targets. Nothing excludes kind 5 from the id path (D01). Deleting a deletion removes its tombstone rows from event_tags, so events it had deleted become re-insertable. Status: known quirk, not a considered decision. NIP-09 leaves it open; at least one external implementation (vespa-eventstore) deliberately diverges by treating deletion-of-a-deletion as a no-op, which is the safer reading (tombstones shouldn't be revocable). If you change this, update this rule and the changelog — parity suites key off it.

STORE-D06 — NIP-62 vanish is relay-scoped. A kind-62 only cascades when shouldVanishFrom(relay) — its relay tags name this store's relay URL or ALL_RELAYS. (A store constructed with relay = null matches only ALL_RELAYS requests.) Out-of-scope vanish events are stored as regular events with no side effects.

STORE-D07 — vanish scope and horizon. An in-scope vanish deletes every event whose owner (W06) is the vanishing pubkey with created_at < vanish.created_at (strict — the vanish event itself survives), and blocks inserts of owned events with created_at <= vanish.created_at (blocked: a request to vanish event exists; note blocking is inclusive where deletion is strict). Newer vanish requests supersede older ones per pubkey (unique on pubkey_hash).

STORE-D08 — manual deletes. delete(id) removes one row unconditionally (no blocking created). delete(filter) deletes matching rows honoring per-filter limits, with the F10 empty-filter no-op guard. Neither creates re-insert blocking — only stored kind-5/kind-62 events do that.

---

NIP-45 count (STORE-C)

STORE-C01 — count = size of the deduped match set, honoring per-filter limits. Single filter: COUNT(*) over that filter's row-id subquery (including its LIMIT, so count(Filter(kinds=…, limit=10)) is at most 10). Multiple filters: branches are UNIONed (dedup) before counting — an event matching several filters counts once. FTS-off + search term → 0 (F-series search rules apply).

---

NIP-50 search inside the store (STORE-S)

The indexing surface (which kinds are searchable, what text they contribute) is the searchable-events skill; these rules are the store's query-side contract.

STORE-S01 — extension stripping at the store boundary. Every filter-accepting method runs strippingSearchExtensions(): NIP-50 key:value tokens (include:spam, domain:…, …) are removed before FTS. Unsupported extensions are ignored, never matched as literal text and never match-nothing — an extensions-only search collapses to an unconstrained query. Stores that do implement extensions receive the raw string through the relay layer and parse it with nip50Search.SearchQuery.parse (see the IEventStore KDoc).

STORE-S02 — relevance ordering. Search results order by FTS5 bm25 rank (best match first), with created_at DESC only as tiebreak; the LIMIT keeps the most relevant N, not the newest N. A multi-filter REQ is relevance-ordered only when every filter carries a search term (best/min rank per event across branches); mixing search and non-search filters falls back to created_at DESC.

STORE-S03 — search combines by AND with the structural parts (ids/authors/kinds/tags/ since/until) of the same filter — an FTS MATCH join on top of the normal conditions.

STORE-S04 — search grammar is SQLite FTS5 MATCH. The raw (post-strip) string is passed to FTS5, so implicit-AND terms, "phrase queries", OR, and prefix* follow FTS5 semantics. Tokenization details live in FullTextSearchModule (see searchable-events).

STORE-S05 — FTS off. With IndexingStrategy.indexFullTextSearch = false: a filter with a non-empty search term matches nothing (query/count/delete alike); an empty-string search imposes no constraint. Everything else is unchanged.

STORE-S06 — deferred FTS. Relays may set deferFullTextSearchIndexing = true (geode does): tokenization moves off the insert path to a watermark-driven catch-up (needsFtsCatchUp/ftsCatchUp), and search queries drain the backlog first — so NIP-50 results are exactly as fresh as the synchronous path.

---

Negentropy / NIP-77 (STORE-N)

STORE-N01 — snapshotIdsForNegentropy(filters) returns (created_at, id) pairs under the same filter semantics as query (per-filter limits included, multi-filter dedup), order unspecified (negentropy re-sorts). maxEntries returns up to maxEntries + 1 as an overflow sentinel. liveNegentropySnapshot serves full-corpus NEG-OPENs from an in-memory index when maintainLiveNegentropyIndex is on; the delta plumbing in SQLiteEventStore keeps it exact across replaceable displacement, kind-5s, and vanish (invalidate-and-rebuild for the non-itemizable cases).

---

Configuration presets

• Client default (DefaultIndexingStrategy()): FTS on (synchronous), optional indexes off, useAndIndexIdOnOrderBy off, no live negentropy index.
• Relay preset (geode's relayIndexingStrategy()): adds created_at-alone, pubkey-alone and tag+kind+pubkey indexes, defers FTS, maintains the live negentropy index — still leaves useAndIndexIdOnOrderBy off.
• Flag-gated indexes are runtime config, not schema: flipping one on an existing DB builds the index on next open (ensureOptionalIndexes), no migration.

For parity implementers

• Treat the rule ids above as the vocabulary for divergence notes (e.g. "diverges from STORE-D05: we no-op deletion-of-a-deletion").
• The commonTest suites are the executable spec; FsParityTest shows the in-repo pattern for holding a second engine to it.
• Remember F06 (hash-based tag matching) and F08 (unordered same-second ties by default) when diffing results byte-for-byte — both are places where a "divergence" may be the reference's own slack, not your bug.

Semantics changelog

Add one line per behavior change, newest first: YYYY-MM-DD <short sha> <rule id> — what changed.

• 2026-08-04 (baseline) — rules F01–F13, W01–W08, D01–D08, C01, S01–S06, N01 written from the code at the time this skill was introduced. Changes before this date are not itemized; archaeology starts at git log on nip01Core/store/.