| name | topic-analysis |
| description | Identify topic clusters in a text corpus and track how those topics evolve over time. Use when the user has a body of notes, voice notes, articles, or documents and wants to know "what is this corpus mostly about" or "how have my interests shifted". Supports classical topic modeling (LDA/BERTopic) and LLM-assisted labeling. |
Topic Analysis
Find the latent topics in a corpus and (if timestamps exist) chart their evolution.
When to use
- "What are my notes about?"
- "What themes show up in this voice notepad export?"
- "How have the topics in my journal shifted over the last year?"
Pre-flight
Run choose-approach first if the corpus is >1k documents.
Procedure
- Ingest: load corpus into
{id, text, timestamp?, metadata?} records. Accept JSONL, CSV, folder of text/md files, or a transcript dump.
- Clean: lowercase, strip URLs, remove stopwords, lemmatize. For voice-note corpora, run
synonym-cluster first to collapse transcription variants.
- Choose method:
- BERTopic (preferred default): sentence-transformer embeddings + UMAP + HDBSCAN + class-based TF-IDF. Handles short docs well, gives readable topic words, supports dynamic/temporal topics out of the box.
- LDA via gensim: classic, fast, lower quality on short text.
- Embedding + k-means when the user wants exactly k topics.
- LLM topic labeling: only to name the clusters produced above — not to do the clustering itself.
- Temporal view (if timestamps present): bucket by month/quarter/year, compute topic share per bucket, produce a stacked-area or line chart of topic prevalence over time.
- Label: for each cluster, either present top-10 c-TF-IDF terms, or send those terms + 3 exemplar docs to a cheap LLM ("Give a 3-5 word human-readable label for a cluster whose top terms are … and examples are …").
- Output:
topics.json: [{topic_id, label, keywords, size, exemplar_ids}]topic-evolution.csv: date_bucket, topic_id, count, share- Optional
topics.png plot.
Cost control
- BERTopic runs locally — embedding generation is the expensive step. Use
sentence-transformers/all-MiniLM-L6-v2 (fast, free, local) unless quality is failing.
- If using cloud embeddings (OpenAI
text-embedding-3-small, Voyage), batch to 100+ per request; est. $0.02 per 1M tokens.
- LLM labeling: one call per cluster (~20-50 clusters total), not per document. Cheap.
Pitfalls
- Voice-note corpora: short docs kill LDA. Use BERTopic.
- Single-topic corpora (e.g. all notes about one project): HDBSCAN will dump everything into one cluster. Lower
min_cluster_size or switch to k-means with explicit k.
- Time-evolution plots mislead on small corpora — require ≥30 docs per time bucket or don't plot it.
