text-mining-science

name	text-mining-science
description	Scientific text mining with topic modeling, NER for scientific entities, claim extraction, and trend detection in research literature corpora.
tags	["text-mining","topic-modeling","scientific-nlp","information-extraction","literature-analysis"]
version	1.0.0
authors	["@xjtulyc"]
license	MIT
platforms	["claude-code","codex","gemini-cli","cursor"]
dependencies	{"python":["numpy>=1.24","pandas>=2.0","scikit-learn>=1.3","scipy>=1.11","matplotlib>=3.7","nltk>=3.8","gensim>=4.3"]}
last_updated	2026-03-17
status	stable

Scientific Text Mining

When to Use This Skill

Use this skill when you need to:

• Extract topics from scientific abstracts using LDA or NMF
• Identify named entities (genes, chemicals, diseases, methods) in text
• Detect research trends and emerging topics over time
• Extract scientific claims and hypothesis statements
• Compute semantic similarity between papers or research areas
• Mine full-text papers for methodological information
• Build a search engine or recommendation system for papers

Trigger keywords: text mining, topic modeling, LDA, NMF, BERTopic, scientific NER, claim extraction, scientific literature, abstract analysis, trend detection, keyword extraction, TF-IDF, word embeddings, document similarity, information extraction, GENIA, BioNER, chemical NER, method extraction.

Background & Key Concepts

Latent Dirichlet Allocation (LDA)

LDA assumes each document $d$ is a mixture of topics, and each topic is a distribution over words:

$$P(w | d) = \sum_{k=1}^{K} P(w | z=k) \cdot P(z=k | d) = \sum_k \phi_{kw} \cdot \theta_{dk}$$

where $\theta_d \sim \text{Dir}(\alpha)$ and $\phi_k \sim \text{Dir}(\beta)$.

Non-negative Matrix Factorization (NMF)

Factorize term-document matrix $\mathbf{V} \approx \mathbf{W}\mathbf{H}$ where all entries $\geq 0$. $\mathbf{W}$ gives topic-word loadings, $\mathbf{H}$ gives document-topic activations.

TF-IDF Weighting

$$\text{tf-idf}(t, d) = \text{tf}(t, d) \times \log\frac{N}{df(t)}$$

where $\text{tf}(t,d)$ is term frequency in document $d$, $N$ is total documents, $df(t)$ is number of documents containing term $t$.

Coherence Score (Topic Quality)

$$C_V = \frac{1}{|T|} \sum_{t \in T} \text{score}(t)$$

where $\text{score}(t)$ measures pointwise mutual information of top-$n$ words in topic $t$.

RAKE (Rapid Automatic Keyword Extraction)

Score phrases by word frequency and word degree in the co-occurrence graph:

$$S(w) = \frac{\text{degree}(w)}{\text{freq}(w)}$$

Environment Setup

pip install numpy>=1.24 pandas>=2.0 scikit-learn>=1.3 scipy>=1.11 \
            matplotlib>=3.7 nltk>=3.8 gensim>=4.3
python -m nltk.downloader punkt stopwords averaged_perceptron_tagger

import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
 sklearn.decomposition  LatentDirichletAllocation, NMF
 matplotlib.pyplot  plt
()