| name | lopez-de-prado-financial-ml |
| description | Apply and audit Marcos López de Prado's financial machine learning workflow. Use for event-based bars, triple-barrier/meta-labeling, uniqueness weights, fractional differentiation, purged and embargoed cross-validation, feature importance, hyperparameter tuning, bet sizing, backtest-overfitting controls, PSR/DSR, HRP, structural breaks, entropy, market microstructure, or scalable quant research. Ground answers in the supplied 2018 book synthesis and distinguish source claims from current evidence. |
Advances in Financial Machine Learning
Apply the book as an integrated research process, not a bag of isolated algorithms. Preserve causal timestamps, financial dependence, selection history, transaction costs, and reproducibility from data construction through deployment.
Start here
- Identify the user's decision, information available at that time, prediction horizon, and economic hypothesis.
- Read cheatsheet.md for pipeline guardrails.
- Read only the relevant chapter files from the index below. For a cross-pipeline request, also read patterns.md.
- Cite source-grounded statements with the printed page references already recorded in chapter files.
- Treat page-cited statements as source synthesis. Treat uncited failure-mode, checklist, and reproducibility guardrails as practitioner guidance added during skill construction unless a chapter says otherwise.
- Mark material modernization, inference, or evidence not contained in the book as such. Verify current libraries, infrastructure, market rules, and empirical claims externally when they matter.
Research contract
Before recommending code or interpreting results, establish:
- Decision: side, timing, size, allocation, risk control, or execution.
- As-of time: latest timestamp each feature could have been known.
- Observation: bar or event clock and the information interval represented by one row.
- Outcome: label definition, entry time, barrier/end time, and overlap with other labels.
- Validation: train/test chronology, information-interval purging, transformations fitted inside each fold, and final untouched evaluation. Add a post-test embargo when ordinary CV permits training observations after the test period; in a strictly trailing walk-forward design, purge overlaps but do not add an embargo by rote.
- Selection history: hypotheses, feature sets, assets, model families, and parameter trials attempted.
- Economics: turnover, spread, impact, delay, capacity, constraints, and failure conditions.
- Evidence: immutable inputs, code version, parameters, seeds, OOS predictions, and diagnostics.
If these cannot be established, frame the work as exploratory and state the evidence gap.
End-to-end workflow
1. Organize the research process
Separate data curation, feature research, strategy specification, validation, and production responsibilities. Require a falsifiable economic mechanism before repeated backtest search. Read Ch 1.
2. Build causal observations
Choose the market clock before engineering features. Prefer activity- or information-driven bars when chronological bars contain unequal information. Preserve point-in-time joins, contract rolls, corporate actions, and venue corrections. Read Ch 2.
3. Define outcomes and decisions
Use event-specific triple barriers for path-dependent outcomes. Separate side from participation/size through meta-labeling when a primary model already supplies direction. Read Ch 3.
4. Correct dependence
Measure label concurrency and uniqueness. Use uniqueness/return attribution in weights and sequential bootstrap where bagging would otherwise resample redundant observations. Read Ch 4.
5. Transform without erasing memory
Seek the minimum fractional-differencing order that supports stationarity while retaining useful correlation with levels. Fit the order and truncation threshold within training data. Read Ch 5.
6. Fit and validate models
Use bagging or random forests with dependence-aware samples when variance dominates. Split chronologically and purge overlapping information intervals. Add a post-test embargo for ordinary CV when later training observations can carry test information forward; a strictly trailing walk-forward split generally needs purging but not that post-test embargo. Put preprocessing, feature selection, calibration, and tuning inside a nested validation loop. Fit probability calibration only from predictions unavailable to the model that generated them. Read Ch 6–9.
7. Convert predictions to exposure
Calibrate probabilities before mapping them to size. Define what full exposure means in gross notional, capital, volatility, or risk-budget terms. Combine overlapping active signals, constrain position changes, and evaluate executable target positions rather than raw class labels. Read Ch 10.
8. Challenge the backtest
Do not use repeated backtesting as hypothesis generation. Record all trials and their dependence; never assume a dense grid represents independent trials when adjusting for selection. On development data, generate multiple purged paths with CPCV where appropriate, then preserve a final chronological holdout for the frozen workflow. If data scarcity makes a separate holdout impossible, predeclare CPCV as the primary validation design and do not tune from its path results. Report costs, concentration, drawdowns, time under water, PSR, DSR, and path dispersion. Read Ch 11–15.
9. Allocate and monitor
Use hierarchical allocation when covariance inversion is unstable and clustered dependence is meaningful. Monitor structural breaks, entropy, and microstructure as state features—not automatic trading signals. Read Ch 16–19.
10. Scale reproducibly
Vectorize first; then partition independent work, preserve a serial reference, and reduce outputs deterministically. Treat historical quantum/HPC claims as design examples requiring present-day verification. Read Ch 20–22.
Non-negotiable guardrails
- Never shuffle time-dependent financial samples as if they were IID.
- Never fit encoders, scalers, PCA, feature selection, or thresholds on the complete dataset before splitting.
- Never allow training labels to overlap a test label's information interval.
- Never apply an embargo mechanically; tie it to the direction of the train/test split and the possible information path.
- Never report the best Sharpe without the number and dependence of trials that produced it.
- Never select the best CPCV path or use outer-fold predictions to retune the workflow.
- Never call a signal study a trading backtest without executable costs, delays, and position rules.
- Never infer causality from feature importance.
- Never treat a high entropy, VPIN, structural-break, or impact statistic as alpha by itself.
- Never copy the source's Python snippets blindly; reimplement against current APIs and test against a serial or analytical reference.
- Prefer transparent baselines. Complexity earns promotion only through pre-specified, leakage-resistant OOS evidence.
Chapter index
| Ch | Topic | Read for |
|---|
| 1 | Financial ML as a Distinct Subject | research organization, Sisyphus/meta-strategy paradigms |
| 2 | Financial Data Structures | bars, ETF trick, event sampling |
| 3 | Labeling | triple barriers, meta-labeling, quantamental workflow |
| 4 | Sample Weights | concurrency, uniqueness, sequential bootstrap |
| 5 | Fractional Differentiation | stationarity-memory trade-off, FFD |
| 6 | Ensemble Methods | bagging, random forest, boosting |
| 7 | Cross-Validation in Finance | purging, embargo, leakage |
| 8 | Feature Importance | MDI, MDA, SFI, orthogonal features |
| 9 | Hyper-Parameter Tuning | purged search, scoring, log-uniform sampling |
| 10 | Bet Sizing | calibrated probability, active bets, dynamic size |
| 11 | Dangers of Backtesting | research discipline, selection bias |
| 12 | Backtesting through CV | walk-forward limits, CPCV |
| 13 | Synthetic Backtesting | controlled process experiments |
| 14 | Backtest Statistics | drawdowns, PSR, DSR, attribution |
| 15 | Strategy Risk | failure probability and required precision |
| 16 | ML Asset Allocation | HRP, clustering, recursive bisection |
| 17 | Structural Breaks | CUSUM and SADF families |
| 18 | Entropy Features | encoding, Shannon/Lempel–Ziv, redundancy |
| 19 | Microstructure | impact, PIN/VPIN, order-flow features |
| 20 | Parallel Research | atoms, molecules, online reduction |
| 21 | Combinatorial Optimization | discrete trajectories and path dependence |
| 22 | HPC and Streaming | MPI, HDF5, in-situ processing |
Supporting files
Scope
This skill represents one 2018 book. Use it to structure research and locate concepts, not to establish that an empirical claim still holds. Do not imply return certainty. Preserve the distinction between a hypothesis, an in-sample finding, an out-of-sample result, and a production decision.
