| name | storage-format-port |
| description | How to keep Ahtola's on-disk SQLite file format and WAL framing byte-compatible with SQLite/Turso. Use this when touching anything in `Ahtola.Core/Storage` — pager, b-tree, WAL, page allocator, overflow, varints, headers. |
Storage format porting
Ahtola.Core/Storage implements the on-disk SQLite file format and the WAL.
This is where silent corruption bugs live: a wrong offset, a flipped flag, or
an off-by-one in varint decoding produces files that read fine in our engine
but are unreadable by SQLite/Turso (or vice versa). Byte-exactness with
SQLite/Turso is the contract. See docs/wal-interoperability-contract.md
for the WAL interop target.
Where things live
| Concern | Ahtola (C#) | Turso (Rust) |
|---|
| Database header | SqliteDatabaseHeader.cs | turso-src/core/storage/sqlite3_ondisk/header.rs (and mod.rs) |
| Pager / page store | SqlitePager.cs, SqlitePageStore.cs, SqlitePagerReadCache.cs | turso-src/core/storage/pager.rs |
| Page allocator / freelist | SqlitePageAllocator.cs, SqliteFreelist.cs | turso-src/core/storage/ allocator/freelist |
| B-tree pages (table/index, interior/leaf) | SqliteTableInteriorPage.cs, SqliteTableLeafPage*.cs, SqliteIndexInteriorPage.cs, SqliteIndexLeaf*.cs, SqliteBtree*.cs | turso-src/core/storage/sqlite3_ondisk/btree.rs |
| Cell pointer array / payload layout / overflow | SqliteCellPointerArray.cs, SqlitePayloadLayout.cs, SqliteOverflowChainReader.cs, SqliteOverflowPageView.cs | turso-src/core/storage/sqlite3_ondisk/ |
| Varint | SqliteVarint.cs | turso-src/core/storage/sqlite3_ondisk/varint.rs |
| Record codec | SqliteRecordCodec.cs | turso-src/core/storage/sqlite3ondisk record serial |
| WAL + WAL index + locks | SqliteWal.cs, SqliteWalIndex.cs, SqliteWalByteRangeLock.cs, SqliteWalSharedMemoryLocks.cs, SqliteWalReadSnapshotCoordinator.cs, SqliteWalWriterCheckpointCoordinator.cs | turso-src/core/storage/wal.rs, turso-src/core/wal/ |
| Page size / encryption | SqlitePageSize.cs, AhtolaPageEncryption.cs, AhtolaEncryptionFileSystem.cs | turso-src/core/storage/ |
Porting rules
- Match offsets, sizes, and flag bits exactly. The SQLite file format
header has fixed offsets (page size at offset 16, text encoding at 56, …).
Cross-check every field against
turso-src/core/storage/sqlite3_ondisk/ and
the SQLite file-format docs referenced in the WAL contract.
- Varint encoding is Huffman-ish big-endian, high-bit continuation. Do not
"simplify"
SqliteVarint. Mirror the upstream decode/encode edge cases
(1..9 byte forms, the 9-byte form's full 64-bit payload).
- WAL framing is the interop contract. Frame headers (page number, commit
marker, checksums, salt-1/salt-2) must match byte-for-byte; the checksum
algorithm and salt handling are specified in
docs/wal-interoperability-contract.md. A WAL file Ahtola writes must be
readable by the Turso Rust engine and vice versa.
- Locks/shmem are the one allowed P/Invoke. Byte-range locks and the
shared-memory mapping in
SqliteWalByteRangeLock.cs /
PhysicalSqliteWalSharedMemoryMapping.cs are the intentional OS interop in
Storage (see the pure-managed-closure skill). Do not spread P/Invoke
beyond this.
- In-memory vs physical:
InMemoryFileSystem.cs / PhysicalFileSystem.cs
implement IFileSystem. New storage code should go through IFileSystem,
not call OS APIs directly.
Verification
- Add/extend a conformance case that round-trips the affected structure
through disk and re-reads it.
- For WAL changes, cross-check against the Turso Rust engine per the WAL
contract — do not just "our engine reads what we wrote."
