quaq-backtest

name

quaq-backtest

description

Use this skill when the user wants to run backtests, interpret backtest results, perform parameter sweeps, analyze quantitative metrics, or iterate on strategy performance in the quaq engine. Triggers include: running a backtest, reading backtest output, understanding Sharpe ratio or drawdown or win rate, diagnosing zero-trade or negative-return results, sweeping parameters, validating before running, explaining strategy structure, reading exported files (trades.csv, metrics.json, chart.json, report_stats.json), interpreting quant analysis output (P0/P1/P2 tiers), or iterating on a strategy to improve performance. Also use when the user asks about execution modes (bar, tick, microstructure), run configuration, fill modes, fees, slippage, or warmup bars.

Quaq Backtest Skill

Commands

All commands run from core/ directory.

# Validate (fast, no execution)
cd core && zig build run -- validate ../strategies/my_strat.toml

# Backtest (full run + export)
cd core && zig build run -- backtest ../strategies/my_strat.toml

# Explain (parse + print structure)
cd core && zig build run -- explain ../strategies/my_strat.toml

# Parameter sweep
cd core && zig build run -- sweep ../strategies/my_strat.toml sma_fast.period 10,20,30,50

# API mode (JSON in, JSON out)
cd core && zig build run -- api '{"command":"run.execute","toml":"...","request_id":"r1"}'

Always validate before backtesting to catch errors without waiting for a full run.

Backtest Output

CLI prints these metrics:

=== Backtest Results ===
Total Return:  X.XX%
Max Drawdown:  X.XX%
Sharpe Ratio:  X.XX
Win Rate:      X.XX%
Profit Factor: X.XX
Total Trades:  N
Avg Trade PnL: X.XX
Total Fees:    X.XX
========================

Interpreting Metrics

Metric	What it means	Thresholds
total_return	Net P&L as fraction of initial capital	Negative = lost money
max_drawdown	Largest peak-to-trough equity decline	>5% is significant for crypto
sharpe	Risk-adjusted return (annualized)	<0 bad, 0-1 weak, 1-2 good, >2 excellent
winrate	Fraction of profitable trades	<40% needs high profit_factor to compensate
profit_factor	Gross profit / gross loss	<1 losing, 1-1.5 marginal, 1.5-2 decent, >2 strong
total_trades	Number of completed round-trip trades	<10 = insufficient statistical significance
avg_trade_pnl	Mean P&L per trade	Compare to fees to check if strategy covers costs
max_consecutive_losses	Longest losing streak	High values indicate vulnerability to drawdown
total_fees	Cumulative trading fees paid	If close to total_return, strategy barely covers costs

Agent Iteration Pattern

Follow this loop to improve a strategy:

1. Run backtest, read all metrics
2. Identify the weakest metric
3. Modify strategy TOML to address it
4. Re-run backtest, compare before/after
5. Repeat until metrics meet targets

Diagnosing Common Problems

Zero trades:

• warmup_bars is higher than total data bars available
• Signal condition never triggers (thresholds too strict)
• Data range too short for the indicator periods used
• Fix: lower warmup_bars, relax signal logic, or use more data

Negative Sharpe:

• Entry logic is anti-correlated with price moves
• Stops are too tight (whipsawed out before moves)
• Fees eat all profit (reduce trading frequency)
• Fix: try opposite signal, widen stops, add trend filter, or increase timeframe

Low win rate (<30%) with low profit factor:

• Strategy catches neither trend nor mean reversion
• Fix: add a trend filter (longer SMA), or switch to mean-reversion logic

High win rate but negative return:

• Losses are much larger than wins (no stop loss or stop too wide)
• Fix: tighten stop_value, add take-profit, or reduce qty_value

Common TOML Adjustments

• Change indicator period: modify period param on ind.sma/ema/rsi/atr nodes
• Add RSI filter: add ind.rsi node + op.lt/op.gt comparison + op.and with existing signal
• Add volatility filter: add ind.atr or ind.stdev, compare to threshold
• Adjust stop/TP: change stop_value and tp_value on trade.basic node
• Change position size: modify qty_value (fraction of equity when qty_mode = "percent_equity")
• Add cooldown: set cooldown_bars on trade.basic to prevent rapid re-entry

Formula-Based Iteration

Use formula blocks to quickly prototype custom logic without writing Zig code:

• Composite score: Use logic.formula to combine indicator values:

  [[nodes]]
  id = "score"
  kind = "logic.formula"
  expr = "(a - 50) * 0.01 + b * 0.5"
  

  Wire RSI to port a, normalized SMA difference to port b.

• Custom condition: Use logic.logical_formula for complex boolean logic:

  [[nodes]]
  id = "entry_cond"
  kind = "logic.logical_formula"
  expr = "a AND (b OR NOT c)"
  

  Wire three separate boolean signals to ports a, b, c.

• Fixed thresholds: Use logic.constant instead of hardcoding values:

  [[nodes]]
  id = "threshold"
  kind = "logic.constant"
  value = 0.02
  

  Easier to sweep: sweep strat.toml threshold.value 0.01,0.02,0.03,0.05

Futures Strategy Data

For futures-specific analysis, use auxiliary data nodes:

• data.spot_close — Spot price for basis/premium analysis
• data.funding_rate — Funding rate for carry trade strategies
• Wire these to formula blocks or comparison operators to build funding-aware signals

Parameter Sweep

Sweep syntax: sweep <toml> <node_id.param> <val1,val2,...>

cd core && zig build run -- sweep strat.toml sma_fast.period 10,15,20,25,30,40,50

Output is a table with columns: Value, Return%, MaxDD%, Sharpe, WinRate%, Trades.

Use sweeps to:

• Find optimal indicator periods
• Compare stop loss levels (e.g., trader.stop_value 0.01,0.02,0.03,0.05)
• Test position sizes (e.g., trader.qty_value 0.05,0.10,0.15,0.20)
• Up to 64 values per sweep

Interpret sweep results by looking for:

• Stable regions (nearby values give similar results) over isolated peaks
• Monotonic improvement (indicates a real signal, not overfitting)
• Avoid selecting the single best value -- prefer the middle of a good region

Run Configuration

[run]
initial_cash = 10000.0      # Starting capital
fill_mode = "next_open"     # "next_open" (realistic) or "close" (optimistic)
fee_bps = 7.0               # Trading fee in basis points (7 bps = 0.07%)
slippage_bps = 2.0          # Slippage in basis points
warmup_bars = 50            # Skip first N bars for indicator warmup
max_bars = 0                # 0 = process all bars

Set warmup_bars >= longest indicator period to avoid NaN signals.

Execution Modes

Three modes, set via [run] section:

Mode	Key	Data Required	Use Case
bar	default	OHLCV bars	Standard backtesting
tick	execution_mode = "tick"	Tick data	Tick-level precision
microstructure	execution_mode = "microstructure"	L2 order book	Market microstructure analysis

Exported Files

After backtest, files appear in the path set by [exports] out_dir:

File	Content	Use
trades.csv	Entry/exit timestamp, price, side, qty, PnL, fees, reason	Trade-by-trade analysis
metrics.json	Same metrics as CLI output, machine-readable	Programmatic result consumption
chart.json	Timestamps, OHLCV, output series, markers	Visualization data
events.json	Detailed trade events (if events_json = true)	Event-level debugging
report_stats.json	Quant analysis results (if quant nodes present)	Statistical validation
run_manifest.json	Strategy metadata, active quant methods, run config	Audit trail

Enable/disable exports in the [exports] section:

[exports]
trades_csv = true
chart_json = true
events_json = false
metrics_json = true
out_dir = "./out"

Quantitative Analysis

Quant nodes run automatically after the backtest completes. They validate signal quality and detect statistical properties. Add quant nodes to strategy TOML and wire their inputs from bar data.

Do NOT wire quant node outputs to bar-loop nodes. This causes validation error E022_POST_RUN_ONLY.

Tiers

P0 -- Signal Quality: metrics.ic, metrics.ic_rolling, metrics.roc_auc, stats.quantile_returns, stats.mann_whitney, stats.ks_test, stats.cliffs_delta, stats.kruskal_wallis, sim.bootstrap_ci, sim.permutation_pvalue, sim.monte_carlo

P1 -- Time Series Models: models.granger, models.garch11, models.cointegration_eg, regime.break_chow, regime.cusum, regime.hurst_rs

P2 -- Advanced Analysis: regime.markov_switching, risk.evt_var, model.var, model.vecm, dependence.copula, feature.pca, feature.cluster_kmeans, feature.spectral_dft, feature.wavelet

Interpreting Quant Results

Key P0 outputs in report_stats.json:

• IC (Information Coefficient): correlation between signal and forward returns. >0.05 is meaningful, >0.1 is strong.
• Mann-Whitney p-value: <0.05 means signal distinguishes winners from losers.
• Monte Carlo: compare actual return to simulated distribution. If actual is above 95th percentile, signal has edge.
• Bootstrap CI: confidence interval for Sharpe. If lower bound > 0, edge is likely real.

Key P1 outputs:

• Granger causality: does signal predict returns? p < 0.05 = yes.
• GARCH(1,1): volatility clustering parameters. High persistence means regime-aware sizing helps.
• Hurst exponent: >0.5 = trending, <0.5 = mean-reverting, ~0.5 = random walk.

API Mode

For programmatic access, use the JSON API:

# Validate
cd core && zig build run -- api '{"command":"strategy.validate","toml":"...","request_id":"r1"}'

# Execute backtest
cd core && zig build run -- api '{"command":"run.execute","toml":"...","request_id":"r2"}'

# Sweep
cd core && zig build run -- api '{"command":"run.sweep","toml":"...","param_name":"sma_fast.period","values":[10,20,30],"request_id":"r3"}'

# Load TOML from file instead of inline
cd core && zig build run -- api '{"command":"run.execute","toml_path":"../strategies/sma_cross.toml","request_id":"r4"}'

Response format:

{"status":"ok","request_id":"r1","engine_version":"...","errors":[],"payload":{...}}

Status values: ok, error_validation, error_runtime, error_not_found, error_invalid_request.

Workflow Summary

validate  -->  backtest  -->  read metrics  -->  diagnose  -->  modify TOML  -->  repeat
                  |
                  +--> sweep (when optimizing a single parameter)
                  |
                  +--> explain (to inspect graph structure)