| name | word-frequency |
| description | Count word/token occurrences across a corpus with stopword filtering, stemming/lemmatization options, and n-gram support. Use when the user wants a simple frequency export — "how often does X come up", "top 100 words in my notes", "bigram frequencies". |
Word Frequency
Classical, mechanical. No LLM involvement.
Procedure
- Tokenize: spaCy tokenizer (handles punctuation, contractions) or
nltk.word_tokenize. For code-adjacent corpora, preserve underscores/hyphens.
- Normalize: lowercase. Optionally lemmatize (spaCy) or stem (Porter/Snowball). Lemmatization is usually better for reporting.
- Filter:
- Remove stopwords (spaCy / NLTK lists). Allow custom additions (user-domain boilerplate).
- Remove pure-numeric tokens unless the user wants dates/ids.
- Length filter (drop tokens <3 chars).
- Count:
collections.Counter. For n-grams, nltk.ngrams or sliding window.
- Export:
word-frequency.csv: term, count, docs_containing, avg_per_doc- Top-N summary with percentile cutoffs.
- Optional enrichments:
- TF-IDF instead of raw counts (surfaces distinctive words, not just common ones).
- Per-metadata-slice frequencies (e.g. per month, per category) — chain with
trend-analysis.
Scale
Trivial. Processes millions of docs on a laptop in minutes. No cost.
Pitfalls
- Voice-note corpora: run
synonym-cluster first or you'll double-count variants ("GitHub", "git hub", "get hub").
- If the user wants "how often am I talking about X" semantically (not literally), escalate to
topic-analysis — word frequency alone misses paraphrases.
