trend-analysis

name	trend-analysis
description	Identify temporal trends across a text corpus — rising/falling topics, entities, or keywords over time. Use after topic-analysis or ner-extraction when the user wants "what am I talking about more / less than before" or "when did X first show up".

Trend Analysis

Temporal shape of topics, entities, and keywords.

Prereqs

• Documents must have timestamps (creation date, captured-at, published-at).
• Feed comes from topic-analysis, ner-extraction, or word-frequency.

Procedure

1. Choose a bucket: day / week / month / quarter / year. Rule of thumb: aim for ≥30 documents per bucket. If the corpus is sparse, go wider.
2. Count per bucket: mentions, topic-shares, or document-counts per bucket.
3. Normalize: raw counts mislead when corpus volume varies over time. Use share of bucket (percent) rather than absolute count.
4. Detect movement:
   • Rising: terms whose share in the latest 25% of buckets exceeds their share in the first 75% by ≥2x and passes a minimum-count floor.
   • Falling: inverse.
   • New: first appears in the latest N buckets.
   • Dormant: not seen in the latest N buckets.
5. Smoothing: apply a 3-bucket rolling mean before plotting. Raw weekly/monthly counts are noisy.
6. Output:
   • trends.csv: term, bucket, count, share, smoothed_share
   • trends-summary.md: top 10 rising, top 10 falling, top 10 new.
   • Optional line plot.

Pitfalls

• Corpus bias: if voice-note volume tripled last quarter, everything "rose". Always normalize.
• Small numbers: a term going from 1 mention to 3 is not a trend. Set a minimum bucket-count floor (e.g. ≥5).
• Seasonality: if the data spans years, separate seasonal (yearly) effects from secular trends before claiming a movement.

No LLM needed

This is a pure classical-NLP / stats task. Do not propose a cloud LLM.