debug

name	debug
description	Use when debugging a pytrendy bug, reproducing a regression, or inspecting intermediate pipeline stage output. Covers the tests/test.py sandbox convention, phase 1 (process_signals debug=True plots), phase 2 (segments_refine stage-by-stage bisect), and handoff to the test skill for formal regression tests.

Debugging in pytrendy

Scratch sandbox: tests/test.py

tests/test.py is the committed reproduction sandbox. Key facts:

• Not pytest-collected — filename test.py doesn't match the test_*.py glob, so pytest ignores it. Its module docstring states this explicitly.
• # %% cell markers throughout → VSCode treats it as an interactive notebook. Run cells inline with Shift+Enter; plots render inline via the interactive backend.
• Magic setup at top: %load_ext autoreload / %autoreload 2 — edits to pytrendy/ reload without restarting the kernel.
• # TODONE: / # TODO: inline notes track fix status per cell — the file's distinctive convention (see line 30, 53, 89, etc. for examples).
• Existing sections are specific bug reproductions (synth data + detect_trends() call + inspect plot), not generic templates. Agent-added sections must match: one # %% cell per bug, specific scenario/data/params.

When adding a section

• # ---------- <bug summary> header (matches existing convention — see line 325 "New Reproduction: abrupt padding all-flat fallback on zero-baseline market entry").
• # %% cell loading the data source once. Do not copy or rename the DataFrame per cell.
  • CSV fixture: load via pd.read_csv(...). Keep columns as-is; pass the actual column name to value_col in each detect_trends() call (e.g. value_col='zero_baseline_market_entry_2').
  • pytrendy built-in data (pt.load_data(...)) or inline synth: load once and reuse the reference across cells. Only reload if the scenario genuinely requires different data modifications.
• # %% cell calling detect_trends(...) — with debug=True for phase-1 bugs, plain for phase-2 bugs.
• # TODONE: note on the cell once the bug is fixed (matches the file's tracking convention).
• Cell stays in place after migration to formal tests — not deleted.

After the fix

Mark the reproduction cell's note # TODONE:. Leave the cell in place; it stays useful for re-running after future changes. Then migrate the reproduction into a formal regression test — see the test skill ("Adding a regression test from a debug reproduction").

Phase 1 — process_signals (rolling window flags)

Bugs in flagging propagate everywhere downstream. Diagnose here first.

Pass debug=True to detect_trends(). process_signals.py:196 emits 7 diagnostic plots in sequence, each showing the value column on the left y-axis and the metric/flag on the right:

1. SNR + THRESHOLD_NOISE line (2.5) — noise detection input.
2. noise_flag — where SNR fell below threshold.
3. smoothed — Savitzky-Golay output (window=15).
4. smoothed_std — rolling std (window=7) for flat detection.
5. flat_flag — where smoothed_std ≤ min non-zero rolling std.
6. smoothed_deriv + ±derivative_limit lines — trend direction input.
7. trend_flag — final per-day classification (1=Up, -1=Down, -2=Flat, -3=Noise).

If a flag is wrong here (e.g. noise_flag fires on a legitimate abrupt shift, or trend_flag misses an obvious uptrend), the bug is in process_signals.py. The knobs: THRESHOLD_NOISE=2.5, THRESHOLD_SMOOTH=0.001, WINDOW_SMOOTH=15, WINDOW_FLAT=7. Reproduce with a single series + debug=True in a # %% cell of tests/test.py and iterate.

Phase 2 — segments_refine (post-processing bisect)

If process_signals flags look correct but the final segments are wrong, the bug is in the post-processing chain. pytrendy/post_processing/segments_refine/__init__.py:refine_segments() runs ~14 stage calls in a specific order, and bugs compound — a misclassification at classify_trends propagates through grouping, shaving, cleanup, and re-classification.

Stage order from __init__.py:

1. classify_trends — DTW gradual/abrupt labels.
2. group_segments (1st pass — sporadic flats/noises).
3. expand_contract_segments — gradual boundary adjust (±7d extrema).
4. shave_abrupt_trends — abrupt changepoint z-score shaving.
5. clean_artifacts — remove overlaps from expand/contract.
6. group_segments (2nd pass).
7. clean_artifacts (again).
8. classify_trends (reclassify — some graduals → abrupts).
9. If changed: shave_abrupt_trends (2nd pass, second_pass=True, init_segments=...).
10. group_segments + clean_artifacts.
11. fill_in_flats — fill gaps with flats.
12. group_segments (3rd pass, final) + clean_artifacts (inverse_only=True).

Primary bisect — comment in source

Open refine_segments() in pytrendy/post_processing/segments_refine/__init__.py. Comment out stages one at a time (working from the bottom of the chain upward, or from classify_trends onward). After each edit, re-run the tests/test.py reproduction cell (Shift+Enter in VSCode) to see the effect on the plot/segments. The first uncomment that reintroduces the wrong behavior isolates the culprit stage.

Alternative bisect — snapshot helper (exploration only, not persisted)

For faster iteration without touching source, run this loop outside tests/test.py (REPL, separate scratch notebook, inline eval). Prints segment state after each stage; the divergence point is the culprit. Throwaway — never persist this in tests/test.py.

import pytrendy as pt
from pytrendy.post_processing.segments_get import get_segments
from pytrendy.post_processing.segments_refine import (
    classify_trends, group_segments, expand_contract_segments,
    shave_abrupt_trends, clean_artifacts, fill_in_flats,
)

df = pt.load_data('series_synthetic')[['date', 'gradual']].set_index('date')
value_col = 'gradual'
method_params = dict(abrupt_padding=0)

segs = get_segments(df)
print('pre-refine', len(segs), [(s['direction'], str(s['start']), str(s['end'])) for s in segs])

for name, fn in [
    ('classify_trends', lambda s: classify_trends(df, value_col, s)),
    ('group_segments#1', lambda s: group_segments(s)),
    ('expand_contract', lambda s: expand_contract_segments(df, value_col, s)),
    ('shave_abrupt', lambda s: shave_abrupt_trends(df, value_col, s, method_params)),
    ('clean_artifacts#1', lambda s: clean_artifacts(df, value_col, s, method_params)),
    ('group_segments#2', lambda s: group_segments(s)),
    ('clean_artifacts#2', lambda s: clean_artifacts(df, value_col, s, method_params)),
    ('classify_trends#2', lambda s: classify_trends(df, value_col, s)),
    ('fill_in_flats', lambda s: fill_in_flats(df, s)),
    ('group_segments#3', lambda s: group_segments(s)),
    ('clean_artifacts#3', lambda s: clean_artifacts(df, value_col, s, method_params, inverse_only=True)),
]:
    segs = fn(segs)
    print(name, len(segs), [(s['direction'], str(s['start']), str(s['end'])) for s in segs])

Re-classification loop gotcha

Steps 8-10 only fire if classify_trends (reclassify) changes anything (if segments_refined != init_segments at __init__.py:52). If your bug only appears on series with mixed gradual/abrupt, the 2nd-pass shave_abrupt_trends with second_pass=True, init_segments=... is a likely culprit — it has different behavior than the 1st pass.

Handoff to formal tests

Once fixed, the test skill takes over — migrate the reproduction into tests/tests_crashes_edgecases/test_*.py using assert_segments_match / assert_segments_in_a_haystack from tests/conftest.py. The tests/test.py cell stays in place.