| name | storage-formats |
| description | Reference for data lake storage formats and table formats — Parquet, Delta Lake, Apache Iceberg, Apache Hudi, DuckLake, Lance, F3 (Faster Frozen Format), ORC, and their interaction with DuckDB. Auto-loaded when discussing storage formats, data lakes, table formats, Parquet, Iceberg, Delta Lake, DuckLake, or change data capture. |
Storage Formats and Table Formats Reference
Comprehensive reference for columnar file formats, open table formats, and their
relevance to DuckDB extension development.
1. Columnar File Formats
1.1 Apache Parquet
The dominant open columnar format. Self-describing, column-oriented, designed for
analytics workloads. Created by Twitter + Cloudera (2013), now Apache top-level project.
Physical layout:
File:
Row Group 0 (typically 128MB - 1GB):
Column Chunk (col_a): Page 0 | Page 1 | ... | Page N
Column Chunk (col_b): Page 0 | Page 1 | ...
...
Row Group 1:
...
Footer (schema, row group metadata, column statistics)
Key features:
- Row groups: Horizontal partitions (~128MB-1GB each). Unit of parallel read.
- Column chunks: One per column per row group. Unit of I/O.
- Pages: Within a column chunk (~1MB). Unit of compression/encoding.
- Encodings: PLAIN, DICTIONARY, RLE, DELTA_BINARY_PACKED, DELTA_LENGTH_BYTE_ARRAY,
DELTA_BYTE_ARRAY, BYTE_STREAM_SPLIT. Encoding chosen per page.
- Compression: Snappy (default), ZSTD (best ratio), LZ4, Gzip, Brotli. Per-column.
- Statistics: Min/max per column chunk and page. Enables predicate pushdown.
- Nested types: Via Dremel encoding (repetition + definition levels).
- Bloom filters: Optional per-column. Probabilistic membership test for point lookups.
- Page index: Column index + offset index for fine-grained page skipping.
DuckDB integration: Native Parquet reader/writer. read_parquet(), COPY TO,
parquet_metadata(), parquet_schema(). Supports filter pushdown, projection pushdown,
row group pruning via statistics. Parquet is DuckDB's primary external format.
Limitations:
- Immutable — no in-place updates. Modifications require rewriting entire files.
- No transactions, no schema evolution, no time travel (these are table format concerns).
- Row group boundaries are fixed at write time — suboptimal for evolving data.
1.2 Apache ORC (Optimized Row Columnar)
Created by Hortonworks for Hive (2013). Similar to Parquet but with differences:
- Stripes (= Parquet's row groups), typically 256MB
- Lightweight indexes: Min/max/sum/count per stripe and per 10K-row "row index entry"
- ACID support: Built-in support for delta files (base + side files for updates)
- Better compression for many workloads (more aggressive encoding selection)
- Worse ecosystem reach than Parquet (Hive-centric)
DuckDB supports ORC reading via community extensions but Parquet is preferred.
1.3 Lance (Lance Format v2)
Modern columnar format designed for ML/AI workloads. Created by LanceDB (2023+).
- Random access: O(1) row lookup by ID (unlike Parquet which requires scanning)
- Vector search: Native support for ANN (approximate nearest neighbor) indexes
- Fast updates: Append-only log with deletion vectors (no file rewrite for deletes)
- Versioning: Built-in dataset versioning via manifest chain
- Encodings optimized for embeddings: Flat, IVF-PQ, product quantization
DuckDB integration: lance community extension.
1.4 F3 (Faster Frozen Format)
Research format from CWI (DuckDB's birthplace). Designed for fast analytical queries
on frozen (immutable) data.
Key ideas:
- Adaptive encoding: Per-column, per-block encoding selection based on data distribution
- Lightweight compression: Prioritize decompression speed over compression ratio
- Vectorized scanning: Format designed to feed directly into vectorized execution
engines (like DuckDB's) without transcoding
- Zone maps: Per-block min/max for predicate pushdown
- Cache-line alignment: Data layout respects CPU cache line boundaries
F3 achieves faster scan performance than Parquet by reducing decompression overhead,
at the cost of slightly larger file sizes.
Relevance to DuckDB: DuckDB's native storage format is influenced by F3's design
philosophy — adaptive encoding, lightweight compression, vectorized-friendly layout.
1.5 FastLanes Compression Layout and File Format
Afroozeh, Boncz. "The FastLanes Compression Layout: Decoding >100 Billion Integers
per Second with Scalar Code." PVLDB 16(9), 2023.
Afroozeh, Boncz. "The FastLanes File Format." PVLDB 18(11), 2025.
FastLanes is both a compression layout and a next-gen file format from CWI.
The core problem: Lightweight compression (LWC) schemes — bit-packing, FOR, DELTA,
RLE, dictionary — are not inherently SIMD-friendly. DELTA decoding is a prefix-sum
(sequential). Traditional implementations suffer from data dependencies.
Innovation 1: Virtual 1024-bit SIMD Register (MM1024).
FastLanes processes blocks of 1024 values as if the hardware had a single 1024-bit
register. Data organized into T rows of (1024/T) lanes (T = element bit-width).
No data dependencies between lanes. The layout auto-vectorizes to ANY SIMD width —
128-bit (SSE), 256-bit (AVX2), 512-bit (AVX-512), or NEON — using identical scalar code.
Innovation 2: Unified Transposed Layout (UTL).
For inherently sequential encodings (DELTA), FastLanes physically reorders tuples
within each 1024-value block using a specific permutation ("04261537" order). After
transposition, adjacent elements in a SIMD word come from independent DELTA chains,
eliminating cross-lane dependencies regardless of SIMD width.
Supported encodings:
- Bit-Packing: Fully parallel unpacking across all lanes
- Frame-of-Reference (FOR): Base value + bit-packed residuals
- DELTA: Independent prefix-sum streams via UTL (no sequential dependency)
- RLE: Mapped to DELTA+DICT to leverage efficient DELTA kernel
- Dictionary: Bit-packed dictionary codes via interleaved layout
Performance: Decoding speed exceeds 40-60 values per CPU cycle single-threaded.
Over 100 billion integers/second. Scalar auto-vectorized code often matches or
exceeds hand-written SIMD intrinsics.
FastLanes File Format (2025): A complete file format (Parquet successor) built on
the compression layout. Adds multi-column compression (MCC) via expression-based
cascaded encodings — columns that correlate can be jointly compressed.
GitHub: cwida/FastLanes
1.6 ALP (Adaptive Lossless floating-Point Compression)
Afroozeh, Kuffo, Boncz. "ALP: Adaptive Lossless floating-Point Compression."
SIGMOD 2024. Best Artifact Award.
The problem: IEEE 754 doubles have pseudo-random bit patterns because most decimal
values (10.12, 3.14) can't be exactly represented in binary. Traditional integer-oriented
compression (DELTA, FOR) fails. XOR-based schemes (Gorilla, Chimp, Patas) have complex
branching that defeats SIMD.
ALP algorithm (for decimal-origin floats — the common case):
- Sample values from row-group to find best exponent
e and factor f
- Encode: Multiply each double by 10^e, round to nearest integer
- Verify losslessness: Decode and check exact equality with original
- Values that don't round-trip exactly → exceptions (stored separately with positions)
- Successful integers compressed via FastLanes FOR + Bit-Packing
Example: 10.12 with e=2 → 10.12 * 100 = 1012 → FOR + bit-packed as integer.
ALP-RD (Real Doubles — for non-decimal floats):
For scientific/computed data where the decimal trick fails:
- Bit-split each 64-bit IEEE 754 representation at position
p
- Left part (high bits): dictionary-encoded (few distinct patterns)
- Right part (low bits): bit-packed to exactly
64-p bits
- Split position
p chosen at row-group level to minimize left-part variance
Performance vs alternatives (41 real-world datasets):
- Compression ratio: ALP ~4.3x avg. Patas ~2.1x. Chimp ~2.4x. Comparable to Zstd.
- Decompression: ALP ~2.6 doubles/cycle. ~8x faster than Patas. ~55x faster than Gorilla.
- SUM query on ALP data: only 1.4x slower than uncompressed (vs 3.7x Patas, 7.7x Chimp)
DuckDB integration: Integrated in DuckDB v0.10.0 (Feb 2024). Replaces both Chimp
and Patas as the floating-point compression scheme. Uses DuckDB's two-phase
analyze/compress architecture. Vectors of 1024 values.
GitHub: cwida/ALP
2. Open Table Formats
Table formats add ACID transactions, schema evolution, time travel, and change tracking
ON TOP of file formats (typically Parquet). They manage metadata about which files
constitute the current table state.
2.1 Delta Lake
Created by Databricks (2019). The most widely deployed open table format.
Architecture:
table_path/
_delta_log/ # Transaction log (JSON + Parquet checkpoints)
00000000000000000000.json # Version 0: initial table creation
00000000000000000001.json # Version 1: add files
00000000000000000010.checkpoint.parquet # Checkpoint every 10 versions
part-00000-....parquet # Data files
part-00001-....parquet
Transaction log: Sequence of JSON actions (AddFile, RemoveFile, Metadata, Protocol).
Each commit = one JSON file. Optimistic concurrency via file-system atomic rename.
Checkpoints (Parquet) consolidate log periodically for fast reads.
Key features:
- ACID transactions: Serializable isolation via optimistic concurrency control
- Time travel:
SELECT * FROM table VERSION AS OF 5 or TIMESTAMP AS OF '2024-01-01'
- Schema evolution: Add/rename/drop columns, type widening
- Change Data Feed (CDF): Tracks row-level changes (insert/update/delete) between versions.
Stored as separate
_change_data/ Parquet files with _change_type column.
- Deletion vectors: Bitmap marking deleted rows within a Parquet file (avoids rewrite)
- Liquid clustering: Replaces static partitioning with adaptive data layout
- UniForm: Auto-generates Iceberg metadata alongside Delta metadata for interop
DuckDB integration: delta extension. SELECT * FROM delta_scan('path').
Supports time travel, predicate pushdown, CDF reading.
Relevance to IVM: Delta Lake's Change Data Feed is essentially a delta table —
it tracks exactly which rows were inserted/updated/deleted between versions. This maps
directly to OpenIVM's delta_<table> concept. An IVM system on top of Delta Lake
could read CDF instead of maintaining separate delta tables.
2.2 Apache Iceberg
Created by Netflix (2017), now Apache top-level project. Emphasis on correctness,
large-scale tables (petabyte+), and multi-engine compatibility.
Architecture:
metadata/
v1.metadata.json # Metadata file: schema, partition spec, snapshot list
v2.metadata.json
snap-1234.avro # Manifest list: points to manifest files
manifest-abc.avro # Manifest: list of data files with column stats
data/
part-00000.parquet # Data files (Parquet, ORC, or Avro)
Three-level metadata:
- Metadata file: Schema, partition spec, sort order, current snapshot pointer
- Manifest list: Points to manifest files, with partition-level summary stats
- Manifest files: List individual data files with per-file, per-column statistics
This hierarchy enables O(1) snapshot reads and efficient partition pruning even for
tables with millions of files.
Key features:
- Hidden partitioning: Partition transforms (year, month, day, hour, bucket, truncate)
applied transparently. Users query without knowing the partition scheme.
- Schema evolution: Full support (add, drop, rename, reorder, type promotion).
Each column has a unique ID; renames don't break readers.
- Time travel: Via snapshot IDs or timestamps
- Incremental reads:
IncrementalAppendScan API reads only new files added since
a given snapshot. Basis for streaming/IVM use cases.
- Row-level deletes: Two modes:
- Copy-on-write: Rewrite affected data files
- Merge-on-read: Write delete files (positional or equality) alongside data files
- Branching/tagging: Git-like branching of table state (Iceberg v2)
DuckDB integration: iceberg extension. SELECT * FROM iceberg_scan('path').
Supports snapshot selection, metadata inspection.
Relevance to IVM: Iceberg's incremental scan API (read only new files since snapshot X)
is a natural fit for delta detection. Combined with merge-on-read delete files, it
provides the "before and after" information needed for IVM delta computation.
2.3 Apache Hudi (Hadoop Upserts Deletes Incrementals)
Created by Uber (2016). Designed specifically for incremental data processing.
Two table types:
- Copy-on-Write (CoW): Updates rewrite entire Parquet files. Read-optimized.
- Merge-on-Read (MoR): Updates written to row-based log files, merged on read. Write-optimized.
Key features:
- Upsert primitive: First-class support for UPSERT (insert or update by key)
- Incremental queries:
SELECT * FROM table WHERE _hoodie_commit_time > X
Returns only rows that changed since commit X.
- Record-level indexing: Bloom filter, HBase, or bucket index for fast record lookup
- Compaction: Background merging of log files into base files (MoR → CoW)
- Clustering: Re-organize data layout for query performance
Relevance to IVM: Hudi was designed from the ground up for incremental data pipelines.
Its incremental query API directly provides delta rows. The record-level index enables
efficient point lookups during delta application (like probing the MV during MERGE).
2.4 DuckLake
DuckDB Labs (2025). A new open table format that stores metadata in a DuckDB
database file rather than in the data lake filesystem.
Architecture:
ducklake.db # Metadata database (DuckDB file)
tables/ # Table metadata, schemas, snapshots
data_files/ # File registry with column statistics
data/
*.parquet # Data files (Parquet)
Key design decisions:
- Metadata in DuckDB: Instead of JSON/Avro metadata files on object storage (Iceberg/Delta),
DuckLake stores all metadata in a DuckDB database. This enables:
- SQL queries over metadata (
SELECT * FROM ducklake_table_info(...))
- ACID transactions on metadata via DuckDB's MVCC
- Sub-millisecond metadata operations (no S3 round-trips for metadata)
- Data in Parquet: Actual data stored as Parquet files on any storage backend
- Catalog integration: Registers as a DuckDB catalog — tables appear as regular tables
Key features:
- Instant table creation/schema changes: Metadata-only operations, no file I/O
- Change tracking: Built-in support for tracking which data files were added/removed
per transaction — the metadata DB records the full history
- Snapshot isolation: Via DuckDB's MVCC on the metadata database
- Multi-engine access: Metadata DB can be read by any system that reads DuckDB files;
data files are standard Parquet
Relevance to IVM: DuckLake's metadata-in-DuckDB design means IVM metadata (delta tables,
view definitions, refresh timestamps) could be co-located with table format metadata in
a single DuckDB file. Change tracking is a metadata query, not a filesystem scan.
3. Change Data Capture (CDC) Across Formats
CDC is the mechanism for detecting and capturing data changes — the foundation for
any IVM system that reads from external tables.
| Format | CDC Mechanism | Granularity | API |
|---|
| Delta Lake | Change Data Feed | Row-level (insert/update_pre/update_post/delete) | table_changes() |
| Iceberg | Incremental scan + delete files | File-level (new files) + row-level (deletes) | IncrementalAppendScan |
| Hudi | Incremental query | Row-level via commit timeline | _hoodie_commit_time > X |
| DuckLake | Metadata transaction log | File-level (added/removed per txn) | SQL on metadata DB |
| DuckDB native | WAL / delta tables | Row-level | OpenIVM's delta_<table> |
For IVM systems: The ideal CDC mechanism provides:
- Row-level granularity (which rows changed)
- Before+after images (old and new values for updates)
- Monotonic ordering (changes in commit order)
- Efficient access (no full scan of unchanged data)
Delta Lake's CDF is closest to this ideal. Iceberg's incremental scan provides
file-level granularity; row-level requires comparing file contents.
4. Storage Format Implications for IVM
4.1 Immutability and Delta Tables
All modern formats are append-oriented (Parquet files are immutable). This means:
- Inserts: Append new Parquet file. Delta detection: new files since last snapshot.
- Deletes: Write deletion vector or delete file. Delta detection: scan delete files.
- Updates: Delete + insert (CoW) or log entry (MoR). Delta detection: depends on mode.
For IVM, the CoW model is simpler (each new file is a self-contained delta), but MoR
is more write-efficient. The choice affects how delta tables are populated.
4.2 Column Statistics for Cost Models
All formats provide per-file and per-column statistics (min, max, count, null_count).
These statistics can feed IVM cost models:
- Delta size estimate: count of rows in new/changed files
- Base table size: total row count across all active files
- Selectivity estimate: min/max overlap between delta and base table columns
4.3 Partitioning and IVM
Table-level partitioning (by date, region, etc.) enables partition-scoped IVM:
- If delta only affects partitions [P1, P3], the MV refresh can scope to those partitions
- For partition-level aggregates, unchanged partitions can be skipped entirely
- Iceberg's hidden partitioning is particularly clean for this
4.4 Time Travel and IVM Correctness
Time travel (reading table state at a past point in time) enables:
- Consistent snapshots: IVM refresh reads all base tables at the same logical time
- Debugging: Compare MV state with full recompute at any historical point
- Recovery: Roll back to a known-good state if IVM produces incorrect results
5. DuckDB Storage Internals
DuckDB's native storage (.duckdb files) uses its own format:
Block-based storage:
- Fixed-size blocks (256KB default)
- Row groups (~122K rows each)
- Per-column segments within row groups
- Adaptive encoding per segment (constant, dictionary, RLE, bitpacking, FSST for strings)
Compression: Chimp (floating point), ALP (floating point), FSST (strings),
bitpacking (integers), dictionary, RLE, constant.
Buffer manager: LRU-based with memory limit. Spills to disk when memory is exceeded.
MVCC: Multi-version concurrency control for transactions. Undo buffer tracks
per-row version chains.
WAL: Write-ahead log for crash recovery. Flushed on commit.
Relevance to IVM: DuckDB's row group structure means IVM delta tables are stored
in the same row-group format as base tables. The MVCC undo buffer could theoretically
be tapped for change detection (instead of separate delta tables), though this would
require deep engine integration.
6. References
