validate-signal

name	validate-signal
description	Use when the user asks "is this signal real?", "should I trust this BUY signal?", "are my signals working?", "validate signals", "how reliable are the signals?", or any variation about signal quality or statistical significance. Also use proactively when presenting a new signal to the user, when signals seem to contradict the regime, or when the user has been whipsawed by recent signals.

Validate Signal

The engine fires BUY, SELL, and DANGER signals based on Livermore's rules. But a signal is only useful if it predicts something. This skill tests whether a group's historical signals are statistically distinguishable from random chance.

The script

scripts/validate.py analyzes a group's leaders and reports signal quality metrics.

./run .claude/skills/validate-signal/scripts/validate.py ~/.lafmm/data/semis
./run .claude/skills/validate-signal/scripts/validate.py ~/.lafmm/data/energy --signal SELL
./run .claude/skills/validate-signal/scripts/validate.py ~/.lafmm/data/us-indices --horizon 10

Options:

• --signal — which signal type to test: BUY, SELL, DANGER_UP_OVER, DANGER_DOWN_OVER. Default: BUY.
• --horizon N — how many trading days after the signal to measure returns. Default: 5.
• --permutations N — number of random shuffles for the p-value test. Default: 10,000. More = more precise but slower.
• --json — machine-readable output.

Interpreting the output

Hit rate: fraction of signals where the subsequent return moved in the expected direction within the horizon. Above 55% is encouraging. Below 50% means the signal is worse than a coin flip.

Sharpe ratio: mean return per signal divided by standard deviation of returns. Measures reward relative to risk. Below 0.3 is noise. Above 0.5 is tradeable. Above 1.0 is strong.

p-value: probability of seeing results this good by chance. The permutation test shuffles signal dates 10,000 times and counts how often random signals perform as well as the real ones. Below 0.05 is statistically significant. Above 0.10 means you cannot distinguish this signal from random.

Signal decay: mean return at 1, 5, 10, and 20 trading days after the signal. Shows how long the signal's edge persists. If the 1d return is positive but the 20d return is zero, the signal has a short half-life. If returns grow with horizon, the signal captures a real trend.

What to do with the results

• Significant signal (p < 0.05) with good Sharpe (> 0.5): the signal has statistical backing. Trust it more. Size positions with confidence.
• Insignificant signal (p > 0.10): you cannot prove this signal beats random. Does not mean the signal is wrong, but you have no statistical evidence that it's right. Treat with caution.
• Good hit rate but poor Sharpe: wins are frequent but small, losses are rare but large. The signal catches direction but not magnitude.
• Good Sharpe but poor hit rate: wins are infrequent but large when they happen. The signal catches big moves but generates many small losers. Needs tight risk management.

Requirements

Signals need history to validate. A group with 3 signals total cannot produce meaningful statistics. The script requires at least 3 signals of the requested type. More data means more reliable results. 20+ signals is ideal.

When to use this

• Before acting on a signal, especially after a period of whipsaws. If the signal has no statistical backing, treat it with extra caution.
• When evaluating a new group's signal history before sizing positions.
• When signals from a group seem to contradict the regime detected by detect-regime.
• During periodic review to check whether the engine's edge is holding or decaying over time.

Connecting to regime

A signal that tests well in a trending regime may test poorly in a mean-reverting one. If detect-regime shows the sector is currently mean-reverting, a historically significant BUY signal carries less weight. Consider running validate-signal separately on trending and non-trending periods to understand conditional signal quality.

Limitations

Statistical significance is backward-looking. A signal that tested well on 50 historical instances may stop working if the market regime shifts. The p-value tells you about the past, not the future.

Small sample sizes produce unreliable results. 5 signals can appear significant by chance. 20+ signals is the minimum for confidence. The script enforces a floor of 3, but 3 is barely enough to compute anything meaningful.