Expert knowledge for contributing to Litestream, a standalone disaster recovery tool for SQLite. Provides architectural understanding, code patterns, critical rules, and debugging procedures for WAL monitoring, LTX replication format, storage backend implementation, multi-level compaction, and SQLite page management. Use when working with Litestream source code, writing storage backends, debugging replication issues, implementing compaction logic, or handling SQLite WAL operations.
Expert knowledge for contributing to Litestream, a standalone disaster recovery tool for SQLite. Provides architectural understanding, code patterns, critical rules, and debugging procedures for WAL monitoring, LTX replication format, storage backend implementation, multi-level compaction, and SQLite page management. Use when working with Litestream source code, writing storage backends, debugging replication issues, implementing compaction logic, or handling SQLite WAL operations.
Litestream is a standalone disaster recovery tool for SQLite. It runs as a
background process, monitors the SQLite WAL (Write-Ahead Log), converts changes
to immutable LTX files, and replicates them to cloud storage. It uses
modernc.org/sqlite (pure Go, no CGO required).
Quick Start
# Build
go build -o bin/litestream ./cmd/litestream
# Test (always use race detector)
go test -race -v ./...
# Code quality
pre-commit run --all-files
Critical Rules
These invariants must never be violated:
1. Lock Page at 1GB
SQLite reserves a page at byte offset 0x40000000 (1 GB). Always skip it during
replication and compaction. The page number varies by page size:
Page Size
Lock Page Number
4 KB
262145
8 KB
131073
16 KB
65537
32 KB
32769
lockPgno := ltx.LockPgno(pageSize)
if pgno == lockPgno {
continue
}
2. LTX Files Are Immutable
Once an LTX file is written, it must never be modified. New changes create new
files. This guarantees point-in-time recovery integrity.
3. Single Replica per Database
Each database replicates to exactly one destination. The Replica component
manages replication mechanics; database state belongs in the DB layer.
4. Read Local Before Remote During Compaction
Cloud storage is eventually consistent. Always read from local disk first:
f, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))
if err == nil {
return f, nil// Use local copy
}
return replica.Client.OpenLTXFile(...) // Fall back to remote
5. Preserve Timestamps During Compaction
Set the compacted file's CreatedAt to the earliest source file timestamp to
maintain temporal granularity for point-in-time restoration.
Small code improvements and performance optimizations
Security vulnerability reports (report privately)
Discuss First
Feature requests: open an issue before implementing
Large changes: discuss approach in an issue first
Pre-Submit Checklist
Read relevant docs from the reference table above
Follow patterns in references/PATTERNS.md
Run go test -race -v ./...
Run pre-commit run --all-files
For page iteration: test with >1 GB databases
Show investigation evidence in PR (see CONTRIBUTING.md)
Testing
# Full test suite with race detection
go test -race -v ./...
# Specific areas
go test -race -v -run TestReplica_Sync ./...
go test -race -v -run TestDB_Sync ./...
go test -race -v -run TestStore_CompactDB ./...
# Coverage
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out
Key testing areas:
Lock page handling with >1 GB databases and multiple page sizes
Race conditions in position updates, WAL monitoring, and checkpointing
Eventual consistency in storage backend operations
Atomic file operations and cleanup on error paths
Environment Validation
Run scripts/validate-setup.sh to verify your development environment is
correctly configured for Litestream development.
