strategy-ab-test

name	strategy-ab-test
description	Head-to-head strategy comparison — snapshot state, sweep params on strategy A, restore, sweep strategy B, then compare metrics side-by-side. Use when the user asks "which strategy is better?" or "A/B test these two".

Strategy A/B Test

You are running two Pine strategies head-to-head on the same symbol/timeframe and reporting a data-backed winner.

Step 1: Lock the Environment

The comparison is only valid if both strategies run on the same chart state.

1. chart_get_state — confirm symbol, timeframe, and chart type
2. state_snapshot with tag abtest-baseline — so you can roll back cleanly between A and B, and at the end

Example baseline: ES1!, 15m, Candles.

Step 2: Define What "Winner" Means

Ask the user (or use defaults) for the ranking metric. Common options:

• Net Profit — simplest, ignores risk
• Sharpe Ratio — risk-adjusted, preferred
• Max Drawdown — lower is better (use absolute value)
• Profit Factor — gross profit / gross loss
• Composite — rank each metric, sum ranks, lowest wins

Default: report all four, pick winner by Sharpe, break ties by lower max drawdown.

Step 3: Run Strategy A

1. Load A's source: paste via pine_set_source (or pine_open if saved by name)
2. pine_smart_compile — auto-detect type, compile, surface errors
3. If errors: iterate via the pine-develop loop until clean
4. strategy_sweep — parameter sweep with cooldown + resume. Pass the param grid (e.g., { ema_len: [20, 50, 100], atr_mult: [1.5, 2.0, 2.5] })
5. Collect results — strategy_sweep returns per-combination {net_profit, sharpe, max_dd, profit_factor, trades}

Example — strategy A (trend-follow):

pine_set_source(<A source>)
pine_smart_compile()
strategy_sweep({ grid: { ema_len: [50, 100, 200], atr_mult: [1.5, 2.0] } })

Pick A's best row by the chosen metric. Record it.

Step 4: Restore, Then Run Strategy B

1. state_restore with tag abtest-baseline — wipe A off the chart
2. Load B's source: pine_set_source + pine_smart_compile
3. strategy_sweep over B's param grid
4. Pick B's best row

Example — strategy B (mean-reversion):

state_restore({ tag: "abtest-baseline" })
pine_set_source(<B source>)
pine_smart_compile()
strategy_sweep({ grid: { rsi_len: [7, 14, 21], rsi_ob: [70, 75, 80] } })

If strategy_sweep pauses mid-sweep due to cooldown, it will auto-resume — just wait.

Step 5: Compare Side-By-Side

Produce a metrics table:

Metric	Strategy A (best)	Strategy B (best)	Winner
Net Profit	$8,420	$6,110	A
Sharpe	1.42	1.78	B
Max DD	-$2,100	-$980	B
Profit Factor	1.65	1.89	B
Trades	84	142	—

Pull each row from each strategy's data_get_strategy_results for the best-params run.

Step 6: Declare Winner + Caveats

• Winner by default rule (Sharpe): Strategy B
• Caveats:
  • Trade count — if B has far more trades, is it over-trading? Factor commission
  • Regime — was the test window mostly trending or chop? A's edge may only show in trends
  • Robustness — check variance across the sweep, not just the single best row. Cherry-picked best params can overfit

Step 7: Report + Cleanup

1. Show the table + winner call + caveats
2. capture_screenshot of the winning strategy loaded so the user has a visual
3. state_restore back to abtest-baseline if the user wants their original chart
4. Optionally state_delete the baseline snapshot once confirmed

Error Notes

• PINE_COMPILE_ERROR — fix source, retry compile before sweeping
• STRATEGY_SWEEP_COOLDOWN — expected; the tool resumes automatically
• CHART_LOADING after restore — wait 1-2s and continue