alterlab-text-as-data

name	alterlab-text-as-data
description	Analyzes text as social-science data — topic modeling (BERTopic with embeddings + class-based TF-IDF, LDA/NMF via scikit-learn or gensim), document embeddings (sentence-transformers), dictionary/lexicon methods, and supervised text classification — choosing the method that matches the inferential goal (discovery vs measurement vs prediction) and validating topic reliability rather than trusting one stochastic run. It uses the verified stack (BERTopic, scikit-learn, gensim CoherenceModel, spaCy, sentence-transformers) with pinned patterns. Use when the request mentions topic modeling, text as data, computational text analysis, document embeddings, dictionary/sentiment lexicons, or classifying a corpus. For training or fine-tuning transformer models prefer alterlab-transformers; for humanities close-reading corpora prefer alterlab-digital-humanities. Part of the AlterLab Academic Skills suite.
license	MIT
allowed-tools	Read Bash(python:*)
compatibility	Requires (declare in-session, no runtime install on Anthropic API): bertopic>=0.16, scikit-learn>=1.3, gensim>=4.3, spacy>=3.7 (+ a model like en_core_web_sm), sentence-transformers>=2.2 (pip). Runs locally via `uv run python`; no API key.
metadata	{"skill-author":"AlterLab","version":"1.0.0","depends_on":"alterlab-ssci-design-gate, alterlab-transformers (model training), alterlab-digital-humanities; audited by alterlab-ssci-inference-gate"}

Text-as-Data — Match the Method to the Inferential Goal

Skill type: ANALYSIS MODULE. Turns a corpus into measurements. The discipline is choosing by goal — discovery (what themes exist?), measurement (how much of concept X?), or prediction (label new documents) — and validating that a topic solution is reliable, not a single lucky stochastic run.

Core Mission

PICK BY GOAL: DISCOVERY vs MEASUREMENT vs PREDICTION. THEN VALIDATE THE TOPICS — ONE RUN IS NOT A RESULT.

When to Use This Skill

• "Run topic modeling on my corpus (BERTopic / LDA)."
• "Measure how much each document expresses concept X (dictionary/lexicon)."
• "Embed my documents and cluster / compare them."
• "Classify these texts into categories."

Does NOT Trigger

The request is really about…	Route to	Why not this skill
Training / fine-tuning a transformer model	alterlab-transformers	Model training, not corpus measurement.
Humanities close-reading / annotation of texts	alterlab-digital-humanities	Interpretive, not quantitative text-as-data.
Whether a text method fits the question at all	alterlab-ssci-design-gate	Design routing, upstream.
Plain tabular statistics	alterlab-statistical-analysis	No text.

Method by goal (verified stack, pinned)

Goal	Method	Verified call
Discovery (emergent themes, contextual)	BERTopic (v0.17)	from bertopic import BERTopic; topics, probs = BERTopic().fit_transform(docs); get_topic_info(), get_topic(0). Plug embeddings via embedding_model=SentenceTransformer("all-MiniLM-L6-v2"), a vectorizer_model=CountVectorizer(min_df=10).
Discovery (bag-of-words, classic)	LDA / NMF	sklearn: ; or gensim .