lopez-de-prado-financial-ml

name

lopez-de-prado-financial-ml

description

Apply and audit Marcos López de Prado's financial machine learning workflow. Use for event-based bars, triple-barrier/meta-labeling, uniqueness weights, fractional differentiation, purged and embargoed cross-validation, feature importance, hyperparameter tuning, bet sizing, backtest-overfitting controls, PSR/DSR, HRP, structural breaks, entropy, market microstructure, or scalable quant research. Ground answers in the supplied 2018 book synthesis and distinguish source claims from current evidence.

Advances in Financial Machine Learning

Apply the book as an integrated research process, not a bag of isolated algorithms. Preserve causal timestamps, financial dependence, selection history, transaction costs, and reproducibility from data construction through deployment.

Start here

1. Identify the user's decision, information available at that time, prediction horizon, and economic hypothesis.
2. Read cheatsheet.md for pipeline guardrails.
3. Read only the relevant chapter files from the index below. For a cross-pipeline request, also read patterns.md.
4. Cite source-grounded statements with the printed page references already recorded in chapter files.
5. Treat page-cited statements as source synthesis. Treat uncited failure-mode, checklist, and reproducibility guardrails as practitioner guidance added during skill construction unless a chapter says otherwise.
6. Mark material modernization, inference, or evidence not contained in the book as such. Verify current libraries, infrastructure, market rules, and empirical claims externally when they matter.

Research contract

Before recommending code or interpreting results, establish:

• Decision: side, timing, size, allocation, risk control, or execution.
• As-of time: latest timestamp each feature could have been known.
• Observation: bar or event clock and the information interval represented by one row.
• Outcome: label definition, entry time, barrier/end time, and overlap with other labels.
• Validation: train/test chronology, information-interval purging, transformations fitted inside each fold, and final untouched evaluation. Add a post-test embargo when ordinary CV permits training observations after the test period; in a strictly trailing walk-forward design, purge overlaps but do not add an embargo by rote.
• Selection history: hypotheses, feature sets, assets, model families, and parameter trials attempted.
• Economics: turnover, spread, impact, delay, capacity, constraints, and failure conditions.
• Evidence: immutable inputs, code version, parameters, seeds, OOS predictions, and diagnostics.

If these cannot be established, frame the work as exploratory and state the evidence gap.

End-to-end workflow

1. Organize the research process

Separate data curation, feature research, strategy specification, validation, and production responsibilities. Require a falsifiable economic mechanism before repeated backtest search. Read Ch 1.

2. Build causal observations

Choose the market clock before engineering features. Prefer activity- or information-driven bars when chronological bars contain unequal information. Preserve point-in-time joins, contract rolls, corporate actions, and venue corrections. Read Ch 2.

3. Define outcomes and decisions

Use event-specific triple barriers for path-dependent outcomes. Separate side from participation/size through meta-labeling when a primary model already supplies direction. Read Ch 3.

4. Correct dependence

Measure label concurrency and uniqueness. Use uniqueness/return attribution in weights and sequential bootstrap where bagging would otherwise resample redundant observations. Read Ch 4.

5. Transform without erasing memory

Seek the minimum fractional-differencing order that supports stationarity while retaining useful correlation with levels. Fit the order and truncation threshold within training data. Read Ch 5.

6. Fit and validate models

Use bagging or random forests with dependence-aware samples when variance dominates. Split chronologically and purge overlapping information intervals. Add a post-test embargo for ordinary CV when later training observations can carry test information forward; a strictly trailing walk-forward split generally needs purging but not that post-test embargo. Put preprocessing, feature selection, calibration, and tuning inside a nested validation loop. Fit probability calibration only from predictions unavailable to the model that generated them. Read Ch 6–9.

7. Convert predictions to exposure

Calibrate probabilities before mapping them to size. Define what full exposure means in gross notional, capital, volatility, or risk-budget terms. Combine overlapping active signals, constrain position changes, and evaluate executable target positions rather than raw class labels. Read Ch 10.

8. Challenge the backtest

Do not use repeated backtesting as hypothesis generation. Record all trials and their dependence; never assume a dense grid represents independent trials when adjusting for selection. On development data, generate multiple purged paths with CPCV where appropriate, then preserve a final chronological holdout for the frozen workflow. If data scarcity makes a separate holdout impossible, predeclare CPCV as the primary validation design and do not tune from its path results. Report costs, concentration, drawdowns, time under water, PSR, DSR, and path dispersion. Read Ch 11–15.

9. Allocate and monitor

Use hierarchical allocation when covariance inversion is unstable and clustered dependence is meaningful. Monitor structural breaks, entropy, and microstructure as state features—not automatic trading signals. Read Ch 16–19.

10. Scale reproducibly

Vectorize first; then partition independent work, preserve a serial reference, and reduce outputs deterministically. Treat historical quantum/HPC claims as design examples requiring present-day verification. Read Ch 20–22.

Non-negotiable guardrails

• Never shuffle time-dependent financial samples as if they were IID.
• Never fit encoders, scalers, PCA, feature selection, or thresholds on the complete dataset before splitting.
• Never allow training labels to overlap a test label's information interval.
• Never apply an embargo mechanically; tie it to the direction of the train/test split and the possible information path.
• Never report the best Sharpe without the number and dependence of trials that produced it.
• Never select the best CPCV path or use outer-fold predictions to retune the workflow.
• Never call a signal study a trading backtest without executable costs, delays, and position rules.
• Never infer causality from feature importance.
• Never treat a high entropy, VPIN, structural-break, or impact statistic as alpha by itself.
• Never copy the source's Python snippets blindly; reimplement against current APIs and test against a serial or analytical reference.
• Prefer transparent baselines. Complexity earns promotion only through pre-specified, leakage-resistant OOS evidence.

Chapter index

Ch	Topic	Read for
1	Financial ML as a Distinct Subject	research organization, Sisyphus/meta-strategy paradigms
2	Financial Data Structures	bars, ETF trick, event sampling
3	Labeling	triple barriers, meta-labeling, quantamental workflow
4	Sample Weights	concurrency, uniqueness, sequential bootstrap
5	Fractional Differentiation	stationarity-memory trade-off, FFD
6	Ensemble Methods	bagging, random forest, boosting
7	Cross-Validation in Finance	purging, embargo, leakage
8	Feature Importance	MDI, MDA, SFI, orthogonal features
9	Hyper-Parameter Tuning	purged search, scoring, log-uniform sampling
10	Bet Sizing	calibrated probability, active bets, dynamic size
11	Dangers of Backtesting	research discipline, selection bias
12	Backtesting through CV	walk-forward limits, CPCV
13	Synthetic Backtesting	controlled process experiments
14	Backtest Statistics	drawdowns, PSR, DSR, attribution
15	Strategy Risk	failure probability and required precision
16	ML Asset Allocation	HRP, clustering, recursive bisection
17	Structural Breaks	CUSUM and SADF families
18	Entropy Features	encoding, Shannon/Lempel–Ziv, redundancy
19	Microstructure	impact, PIN/VPIN, order-flow features
20	Parallel Research	atoms, molecules, online reduction
21	Combinatorial Optimization	discrete trajectories and path dependence
22	HPC and Streaming	MPI, HDF5, in-situ processing

Supporting files

• cheatsheet.md — pipeline decisions, leakage firewall, stop conditions.
• patterns.md — reusable methods with trade-offs.
• glossary.md — compact terminology.
• source-manifest.md — provenance, checksum, page map, and extraction limitations.

Scope

This skill represents one 2018 book. Use it to structure research and locate concepts, not to establish that an empirical claim still holds. Do not imply return certainty. Preserve the distinction between a hypothesis, an in-sample finding, an out-of-sample result, and a production decision.