| name | timeseries-retroactive-outlier-rescaling |
| description | Walk backward through a time series and multiplicatively rescale segments when jumps exceed a fraction of the running mean to correct data collection anomalies |
Retroactive Outlier Rescaling
Overview
Administrative time series (business counts, registrations, economic indicators) sometimes contain abrupt level shifts caused by data collection changes rather than real trends. Walking backward from the most recent value and rescaling earlier segments when the jump exceeds a threshold (e.g., 20% of running mean) corrects these artifacts while preserving genuine trends.
Quick Start
import numpy as np
def fix_outliers(series, threshold_frac=0.20):
values = series.values.copy().astype(float)
n = len(values)
for i in range(n - 2, -1, -1):
running_mean = np.mean(values[i + 1 : min(i + 4, n)])
if running_mean == 0:
continue
jump = abs(values[i] - values[i + 1])
if jump > threshold_frac * running_mean:
scale = values[i + 1] / values[i] if values[i] != 0 else 1.0
values[: i + 1] *= scale
return values
df["value_clean"] = df.groupby("entity_id")["value"].transform(fix_outliers)
Workflow
- Group by entity, sort by time (most recent last)
- Walk backward from the second-to-last observation
- At each step, compute the running mean of the next 2-3 values
- If
|value[i] - value[i+1]| exceeds threshold_frac * running_mean, compute scale factor
- Multiply all values before index i by
value[i+1] / value[i]
- Use cleaned series for both training and inference
Key Decisions
- Threshold: 0.20 (20%) catches major level shifts without overcorrecting normal volatility
- Direction: backward walk anchors to the most recent (most reliable) data
- Running window: 2-3 points balances stability vs. sensitivity to recent changes
- Multiplicative vs. additive: multiplicative preserves proportional relationships across entities with different scales
References
