quaq-data

name

quaq-data

description

Use this skill when the user wants to fetch, download, or manage market data for quaq backtesting. Triggers include: fetching OHLCV klines from Binance, creating or converting Parquet files, preparing CSV data files, configuring the [data] section of strategy TOML files, debugging data loading errors, understanding supported data formats (Parquet, CSV, QBF, Binance ZIP), checking timestamp conventions, or troubleshooting Parquet compatibility issues. Also use when the user asks how to get data into quaq, wire up live Binance fetch, or manage files in core/data/.

Quaq Data Pipeline

Fetch OHLCV from Binance

Use scripts/fetch_parquet.py to download klines. No pyarrow dependency -- self-contained Thrift/Parquet writer that produces files the Zig engine can read.

# Explicit date range
python3 scripts/fetch_parquet.py --symbol BTCUSDT --interval 1m \
  --start 2025-01-01 --end 2025-01-04 --out core/data/BTCUSDT_1m.parquet

# Recent N days (from now)
python3 scripts/fetch_parquet.py --symbol ETHUSDT --interval 1h --days 7

# Defaults: BTCUSDT 1m, last 3 days, output core/data/BTCUSDT_1m.parquet
python3 scripts/fetch_parquet.py

Arguments:

• --symbol: trading pair (default BTCUSDT)
• --interval: kline interval -- 1m, 5m, 15m, 1h, 4h, 1d, 1w
• --start / --end: date range as YYYY-MM-DD (UTC)
• --days: alternative to start/end, fetches N days back from now
• --out: output path (default core/data/{SYMBOL}_{INTERVAL}.parquet)

The fetcher paginates through api.binance.com/api/v3/klines using curl, 1000 bars per page, with 150ms sleep between requests.

There is also scripts/fetch_binance.py which writes CSV instead of Parquet (older script). It outputs timestamps as microseconds (ms * 1000).

TOML [data] Section

Local Parquet file

[data]
dataset = "local_parquet"
symbol = "BTCUSDT"
timeframe = "1m"
path = "data/BTCUSDT_1m.parquet"

Path ending in .parquet or .pq triggers the Parquet loader. All other extensions use CSV.

Local CSV file

[data]
dataset = "local_csv"
symbol = "BTCUSDT"
timeframe = "1m"
path = "data/BTCUSDT_1m.csv"

CSV format: header line timestamp,open,high,low,close,volume, one row per bar. Timestamps are millisecond epoch integers.

Binance live fetch (no local file)

[data]
dataset = "binance"
symbol = "BTCUSDT"
timeframe = "1h"
market = "spot"
start_date = "2024-01-01"
end_date = "2024-01-31"

When path is omitted and dataset = "binance", the engine calls Binance REST API at runtime via fetch_binance.zig (curl subprocess). Fields:

• symbol (required): e.g. "BTCUSDT", "ETHUSDT"
• timeframe (required): e.g. "1m", "1h", "1d"
• market (optional): "spot" (default) or "futures"
• start_date (optional): "YYYY-MM-DD" (default: 30 days ago)
• end_date (optional): "YYYY-MM-DD" (default: today)

If path is present, it always wins over live fetch -- the engine loads the local file.

Validation errors for Binance auto-fetch:

• E023: missing symbol
• E024: missing timeframe
• E025: invalid date format (must be YYYY-MM-DD)

Data Loading Priority

The engine (engine.zig loadData) resolves data in this order:

1. If path is set: load local file (Parquet if .parquet/.pq, otherwise CSV)
2. If dataset = "binance" and no path: fetch from Binance REST API
3. Otherwise: error.FileNotFound

Auxiliary Data Nodes

For futures strategies, these nodes provide additional data series:

• data.spot_close — Spot market close price (for basis/premium calculations)
• data.funding_rate — Perpetual futures funding rate
• data.open_interest — Open interest data

These nodes have no inputs and output y (f64). They are populated by the engine when available in the data source. When the data isn't available, outputs are NaN.

The engine uses maybeEnrichAuxSeries() to lazily fetch auxiliary data (spot close, funding rate) when these node types are present in the strategy. This avoids fetching extra data when it isn't needed.

Supported Formats

Format	Extension	Loader	Notes
Parquet	.parquet, .pq	data.loadParquetBars	PLAIN encoding, UNCOMPRESSED, REQUIRED fields only
CSV	.csv	data.loadCsvBars	Header: timestamp,open,high,low,close,volume
Binance ZIP	.zip	binance.zig	DEFLATE-compressed CSV from Binance Data Vision
QBF v2	.qbf	store.zig	Columnar binary, 32-byte header, optional DEFLATE

Parquet Compatibility

The Zig ParquetReader is minimal. It supports ONLY:

• PLAIN encoding (no RLE, DELTA, DICTIONARY)
• UNCOMPRESSED codec (no SNAPPY, GZIP, ZSTD)
• REQUIRED repetition type (no OPTIONAL, no definition levels)

The Python fetch_parquet.py writes in this exact format. DO NOT use pyarrow or pandas to_parquet() -- they default to OPTIONAL fields which add RLE definition levels that the Zig reader cannot parse.

If you must convert from pyarrow Parquet, re-export through the Python fetch script or write a converter that forces REQUIRED/PLAIN/UNCOMPRESSED.

Required Parquet columns (matched by name):

• timestamp (INT64, millisecond epoch)
• open (DOUBLE)
• high (DOUBLE)
• low (DOUBLE)
• close (DOUBLE)
• volume (DOUBLE)

Timestamp Convention

• Parquet files: timestamps stored as millisecond epoch (INT64)
• CSV files: timestamps stored as millisecond epoch (integer string)
• Internal representation: DatasetBars.ts is microsecond epoch
• Conversion: both loaders multiply ms by 1000 on load (ts * 1000)
• The old fetch_binance.py CSV script writes microseconds directly (ms * 1000)

When writing CSV manually, use millisecond epoch (the loader converts). When reading DatasetBars.ts values in Zig code, they are microseconds.

• Internal aux series: funding rate and open interest stored as-is (no timestamp conversion needed)

File Path Resolution

Relative paths in TOML are resolved from CWD. When running cd core && zig build run -- backtest strategy.toml, CWD is core/. Convention:

• Place data files in core/data/
• Reference as path = "data/BTCUSDT_1m.parquet" in TOML

Typical Workflow

1. Fetch data:

   python3 scripts/fetch_parquet.py --symbol BTCUSDT --interval 1h --days 30
   

2. Configure strategy TOML:

   [data]
   dataset = "local_parquet"
   symbol = "BTCUSDT"
   timeframe = "1h"
   path = "data/BTCUSDT_1h.parquet"
   

3. Run backtest:

   cd core && zig build run -- backtest path/to/strategy.toml
   

Or skip the fetch step and use live Binance auto-fetch (slower for large ranges):

[data]
dataset = "binance"
symbol = "BTCUSDT"
timeframe = "1h"
start_date = "2025-01-01"
end_date = "2025-01-31"

Key Source Files

• scripts/fetch_parquet.py -- Python Binance fetcher (Parquet output)
• scripts/fetch_binance.py -- Python Binance fetcher (CSV output, older)
• core/src/data.zig -- CSV and Parquet loaders (loadCsvBars, loadParquetBars)
• core/src/parquet.zig -- Parquet reader/writer (Thrift TCompactProtocol)
• core/src/fetch_binance.zig -- Zig Binance REST API fetcher (used by engine auto-fetch)
• core/src/binance.zig -- Binance Data Vision ZIP parser
• core/src/store.zig -- QBF v2 binary format reader/writer
• core/src/engine.zig -- loadData() dispatches format by extension and dataset type
• core/src/ir.zig -- DataSpec struct defines the [data] section fields
• core/src/validate.zig -- E023/E024/E025 Binance auto-fetch validation

See Also

• quaq-ingest skill — For converting data from non-Binance sources (Yahoo Finance, TradingView, custom APIs) into quaq-compatible CSV/Parquet format.