word-frequency

name	word-frequency
description	Count word/token occurrences across a corpus with stopword filtering, stemming/lemmatization options, and n-gram support. Use when the user wants a simple frequency export — "how often does X come up", "top 100 words in my notes", "bigram frequencies".

Word Frequency

Classical, mechanical. No LLM involvement.

Procedure

1. Tokenize: spaCy tokenizer (handles punctuation, contractions) or nltk.word_tokenize. For code-adjacent corpora, preserve underscores/hyphens.
2. Normalize: lowercase. Optionally lemmatize (spaCy) or stem (Porter/Snowball). Lemmatization is usually better for reporting.
3. Filter:
   • Remove stopwords (spaCy / NLTK lists). Allow custom additions (user-domain boilerplate).
   • Remove pure-numeric tokens unless the user wants dates/ids.
   • Length filter (drop tokens <3 chars).
4. Count: collections.Counter. For n-grams, nltk.ngrams or sliding window.
5. Export:
   • word-frequency.csv: term, count, docs_containing, avg_per_doc
   • Top-N summary with percentile cutoffs.
6. Optional enrichments:
   • TF-IDF instead of raw counts (surfaces distinctive words, not just common ones).
   • Per-metadata-slice frequencies (e.g. per month, per category) — chain with trend-analysis.

Scale

Trivial. Processes millions of docs on a laptop in minutes. No cost.

Pitfalls

• Voice-note corpora: run synonym-cluster first or you'll double-count variants ("GitHub", "git hub", "get hub").
• If the user wants "how often am I talking about X" semantically (not literally), escalate to topic-analysis — word frequency alone misses paraphrases.