assess-covariance-method

name	assess-covariance-method
description	Rigorously and honestly assess a NEW or proposed covariance / correlation / precision estimator, or a new covariance scoring rule, using precise. Use when someone proposes, asks to evaluate, or wants to compare a covariance methodology. Covers implementing it to the contract, conformance, benchmarking against the registry, out-of-sample validation, and statistically defensible inference.

Assess a new covariance/correlation methodology

A protocol for turning "here's a covariance idea" into a defensible verdict. Work the steps in order; stop early only if a step fails. Install: pip install precise[research] (the research/ scripts use scikit-learn and randomcov).

0. Classify the method first

• Estimator (produces a matrix) vs assessor (scores a matrix)? Different paths below.
• Online (updatable per observation) or batch? precise is an online library; a batch method can still be wrapped, but say so.
• Does it target the covariance, correlation, or precision? It must expose, or be convertible to, covariance_.
• Does evaluating it need the ground-truth covariance (simulation only) or work on real data?

1. Implement it to the contract

Subclass BaseOnlineCovariance. The cheapest correct route: copy the simplest existing estimator and modify it — read precise/empirical.py (and precise/base.py for the exact hook signatures, typically _init_state / _update_state / _state_to_cov / _state_to_mean). The base class owns partial_fit, fit, and the derived attributes; the subclass only supplies the state update and the map to a covariance. Register it in precise/registry.py so all_estimators() includes it.

A scoring rule instead? Implement an Assessor (see precise/assessment/), set needs_data / needs_truth, and follow the higher = better convention.

2. Conformance — non-negotiable

Run the conformance suite (tests/test_estimators.py parametrizes over all_estimators()), or check the invariants directly:

• covariance_ is symmetric and PSD (eigenvalues ≥ 0);
• correlation_ has unit diagonal; precision_ @ covariance_ ≈ I when well-conditioned;
• streaming rows via partial_fit equals fit(X) for non-windowed estimators;
• set_state(get_state()) round-trips;
• it runs on the numpy-only install (no hidden heavy deps).

If it fails any of these, fix the implementation before any benchmarking — numbers from a non-conformant estimator are meaningless.

3. Pick the right judge BEFORE looking at results

This is where most covariance evaluations go wrong. In high dimensions (p comparable to n), do not rank estimators by the held-out Gaussian likelihood — it is dominated by unidentifiable small eigenvalues and ranks estimators below chance. Use the assessor panel and choose by regime (see the score-covariance-estimate skill): BlockPseudoLikelihood / SchurLikelihood / VariogramScore / GMVVariance in high-d; LogLikelihood only when low-d and well-conditioned; a QLIKE/Bregman-consistent loss (not RMSE) when the target is a noisy variance proxy. To choose defensibly, measure the statistical power of candidate judges — the probability they reproduce a known quality ordering — with research/metric_power.py.

4. Benchmark against the registry

research/bakeoff.py runs every estimator over discriminating scenarios and scores them with all_assessors(). Add the new estimator and compare. Report relative error vs a naive / shrinkage baseline within each scenario, then averaged (RMSE is scale-sensitive and can be won by doing well only in high-volatility regimes). Always include a 0/historical-mean baseline for returns-like targets.

5. Sweep the data-generating process — results are ensemble-sensitive

A win on one generator need not transfer. Generate ground truth across several ensembles (LKJ, Wishart, factor/spiked, Toeplitz/AR, equicorrelation) with randomcov plus plain numpy, sample from each, and report per-ensemble, not just pooled. A single ensemble can manufacture or hide any effect.

6. Out-of-sample and real data; be honest about selectors

• Out-of-sample on synthetic ensembles: research/oos.py. On real equity data: research/oos_equity.py (bundled Ken French returns, no key needed).
• If the method is a recommender / selector (chooses among estimators), evaluate it leave-one-generative-family-out (leave_one_family_out_trained in research/oos.py), not leave-one-sample-out — selectors generalize across samples far more easily than across novel structure, and the difference is exactly where they fail.
• Report nulls. If it ties or loses, say so plainly with the number; a method that "matches the best fixed estimator and avoids catastrophe" is a real but smaller claim than "beats it."

7. Inference done right

Pairwise significance is the easiest thing to get wrong. The loss differentials are almost always dependent: overlapping forecast horizons, rolling/trailing targets, repeated expanding-window splits, cross-asset correlation, and many pairwise comparisons. Consequently:

• A naive Diebold–Mariano statistic, or standard-error bars computed as if splits were independent, overstates significance. Treat such bars as descriptive, not as confidence intervals.
• Use a block bootstrap over dates (and clustering over correlated assets), or the Model Confidence Set (Hansen, Lunde & Nason 2011), which handles dependence and multiplicity together — preferable to a wall of pairwise stars.
• State one- vs two-sided, and any multiple-comparison treatment.
• Watch the trailing-vs-forward target trap: if the "target" is a rolling window that overlaps already observed data (e.g. next value of a 7-day trailing std), the task is partly mechanical and apparent forecastability is an artifact. Define targets so the forecast origin uses only past information.

8. Honest reporting checklist

• State exactly what was estimated and scored, and which assessor (and why, given the dimension).
• Give effect sizes and relative errors, not just p-values.
• Report per-ensemble and on real data; note where the method loses.
• Don't read point-error gains as economic/trading signals.
• Ship the code and tests so the numbers are reproducible (this repo's research/ scripts are the model: each headline number has a runnable script and a guarding test).

Reference

• Estimator contract: precise/base.py, precise/empirical.py; registry: precise/registry.py.
• Assessors: precise/assessment/; panel via all_assessors().
• Experiments: research/bakeoff.py, research/metric_power.py, research/oos.py, research/oos_equity.py; Schur-likelihood theory: research/schur_*.py.
• Background on why the high-dimensional likelihood fails and what to use instead: https://precise.microprediction.org/papers/schur-likelihood/.