Menu
SQLite file format, B-trees, pages, cells, overflow, freelist that is used in tursodb
How to benchmark and analyze memory usage in Turso using the memory-benchmark crate and dhat heap profiler. Use this skill whenever the user mentions memory usage, memory profiling, allocation tracking, heap analysis, memory regression, memory benchmarking, dhat, or wants to understand where memory is being allocated during SQL workloads. Also use when investigating memory growth in WAL or MVCC mode. IMPORTANT - If you modify the perf/memory crate (add profiles, change CLI flags, change output format, etc.), update this skill document to reflect those changes so it stays accurate for future agents.
Use when working with Turso yield injection or synthetic failure injection machinery for safe resumable state-machine boundaries, including YieldInjector, FailureInjector, YieldPointMarker, inject_transition_yield!, inject_io_yield!, inject_transition_failure!, CommitYieldPoint, CheckpointYieldPoint, CursorYieldPoint, deterministic interleaving tests, abandoned statement/commit tests, and concurrent-simulator yield plans.
Use when migrating Turso core code to crate::alloc, allocator-api2, cfg(nightly) allocator aliases, fallible collection APIs, try_vec!, try_collect, or LimboError::OutOfMemory handling. Covers import style, allocator-aware aliases, stable/nightly boundaries, shuttle Arc compatibility, and how to keep allocator foundation commits separate from module migrations. Read this when changing code that does allocations.
How to write tests, when to use each type of test, and how to run them. Contains information about conversion of `.test` to `.sqltest`, and how to write `.sqltest` and rust tests
General Correctness rules, Rust patterns, comments, avoiding over-engineering. When writing code always take these into account
Overview of Experimental MVCC feature - snapshot isolation, versioning, limitations
| name | storage-format |
| description | SQLite file format, B-trees, pages, cells, overflow, freelist that is used in tursodb |
┌─────────────────────────────┐
│ Page 1: Header + Schema │ ← First 100 bytes = DB header
├─────────────────────────────┤
│ Page 2..N: B-tree pages │ ← Tables and indexes
│ Overflow pages │
│ Freelist pages │
└─────────────────────────────┘
Page size: power of 2, 512-65536 bytes. Default 4096.
| Offset | Size | Field |
|---|---|---|
| 0 | 16 | Magic: "SQLite format 3\0" |
| 16 | 2 | Page size (big-endian) |
| 18 | 1 | Write format version (1=rollback, 2=WAL) |
| 19 | 1 | Read format version |
| 24 | 4 | Change counter |
| 28 | 4 | Database size in pages |
| 32 | 4 | First freelist trunk page |
| 36 | 4 | Total freelist pages |
| 40 | 4 | Schema cookie |
| 56 | 4 | Text encoding (1=UTF8, 2=UTF16LE, 3=UTF16BE) |
All multi-byte integers: big-endian.
| Flag | Type | Purpose |
|---|---|---|
| 0x02 | Interior index | Index B-tree internal node |
| 0x05 | Interior table | Table B-tree internal node |
| 0x0a | Leaf index | Index B-tree leaf |
| 0x0d | Leaf table | Table B-tree leaf |
| - | Overflow | Payload exceeding cell capacity |
| - | Freelist | Unused pages (trunk or leaf) |
Two B-tree types:
Interior page: [ptr0] key1 [ptr1] key2 [ptr2] ...
│ │ │
▼ ▼ ▼
child child child
pages pages pages
Leaf page: key1:data key2:data key3:data ...
Page 1 always root of sqlite_schema table.
[payload_size: varint] [rowid: varint] [payload] [overflow_ptr: u32?]
[left_child_page: u32] [rowid: varint]
Similar but key is arbitrary (columns + rowid), not just rowid.
[header_size: varint] [type1: varint] [type2: varint] ... [data1] [data2] ...
Serial types:
| Type | Meaning |
|---|---|
| 0 | NULL |
| 1-4 | 1/2/3/4 byte signed int |
| 5 | 6 byte signed int |
| 6 | 8 byte signed int |
| 7 | IEEE 754 float |
| 8 | Integer 0 |
| 9 | Integer 1 |
| ≥12 even | BLOB, length=(N-12)/2 |
| ≥13 odd | Text, length=(N-13)/2 |
When payload exceeds threshold, excess stored in overflow chain:
[next_page: u32] [data...]
Last page has next_page=0.
Linked list of trunk pages, each containing leaf page numbers:
Trunk: [next_trunk: u32] [leaf_count: u32] [leaf_pages: u32...]
Key files:
core/storage/sqlite3_ondisk.rs - On-disk format, PageType enumcore/storage/btree.rs - B-tree operations (large file)core/storage/pager.rs - Page managementcore/storage/buffer_pool.rs - Page caching# Integrity check
cargo run --bin tursodb test.db "PRAGMA integrity_check;"
# Page count
cargo run --bin tursodb test.db "PRAGMA page_count;"
# Freelist info
cargo run --bin tursodb test.db "PRAGMA freelist_count;"