| name | linguistic-semantic-algorithms |
| description | Mapping out an unfamiliar codebase via NLP and graph algorithms โ 40 algorithms across topic modelling, semantic embeddings, code graphs, repository mining, clone detection, IR-based bug localization, identifier linguistics, and complexity metrics. Trigger when hunting bugs across many files, scoping a new feature, identifying domain entities, or analyzing commit history โ even if the user doesn't explicitly mention algorithms โ apply when they ask "where does X live in this codebase?", "what is this codebase about?", "find duplicated logic", "what changed recently?", "who owns this code?", or "is this function risky?". |
pproenca Linguistic and Semantic Algorithms Best Practices
Reference of 40 algorithms an agent should reach for when extracting structure, meaning, history, or risk signals from source code and commit data. Categories are ordered by insight-per-effort โ how much non-obvious truth the technique exposes relative to how easy it is to apply. The first two categories target the highest-leverage questions: what business entities live in this code? and where else does this concept already exist? โ questions that grep and intuition cannot answer.
When to Apply
Reach for these algorithms when:
- Orienting in an unfamiliar codebase: PageRank the import graph to find the core, run LDA over identifier tokens to discover business themes, mine change coupling to surface hidden architectural couplings.
- Hunting a bug from a description: BM25 + history prior + embedding re-rank produces a ranked file shortlist far better than grep.
- Scoping a feature: find prior PRs that did similar work via embedding similarity; map the feature's vocabulary against the codebase's domain via TF-IDF and noun-phrase mining.
- Reviewing a refactor: AST-level GumTree diff reveals semantic impact text diff hides; PDG isomorphism finds the "same logic, different code" twin you should also update.
- Auditing risk: hotspots (churn ร complexity), bus factor, defect-magnet density, dead-code candidates โ together they direct attention to the parts of the codebase that pay back attention.
- Identifying domain entities and bounded contexts: noun-phrase mining + TF-IDF rare-term extraction + Louvain communities + Jensen-Shannon divergence on per-cluster vocabulary.
Rule Categories by Priority
| Priority | Category | Impact | Prefix | Question answered |
|---|
| 1 | Concept & Domain Extraction | CRITICAL | concept- | What business entities live in this code? |
| 2 | Semantic Similarity & Feature Mapping | CRITICAL | sim- | Where else does this concept already exist? |
| 3 | Architectural Topology | HIGH | graph- | What is the shape of this codebase? |
| 4 | Co-Change & Temporal Mining | HIGH | mine- | What hidden couplings does history reveal? |
| 5 | Clone & Duplication Detection | MEDIUM-HIGH | clone- | Where are we repeating ourselves? |
| 6 | Bug & Feature Localization | MEDIUM-HIGH | local- | Given a description, where in code? |
| 7 | Identifier Linguistics | MEDIUM | ling- | How to prepare tokens so the other algorithms work? |
| 8 | Complexity & Risk Metrics | MEDIUM | risk- | Where is the danger concentrated? |
Quick Reference
1. Concept & Domain Extraction (CRITICAL)
2. Semantic Similarity & Feature Mapping (CRITICAL)
3. Architectural Topology (HIGH)
4. Co-Change & Temporal Mining (HIGH)
5. Clone & Duplication Detection (MEDIUM-HIGH)
6. Bug & Feature Localization (MEDIUM-HIGH)
7. Identifier Linguistics (MEDIUM)
8. Complexity & Risk Metrics (MEDIUM)
How to Use
Pick the category that matches the user's question, then read one or two specific rules from that category. Most rules cite combinable partners ("Combine with mine-change-coupling...") that compound the signal โ read the partner rule when you need higher precision.
For unfamiliar repos, the highest-ROI starting sequence is:
graph-pagerank-core โ read the top-20 most central filesconcept-lda-topic-modeling + concept-tfidf-rare-terms โ identify the business themesmine-hotspots-churn-complexity โ find where the bugs concentratemine-change-coupling โ uncover hidden architectural couplings
For a single-task bug or feature, the pipeline is:
local-bm25-saturation (broad candidates) โ local-embedding-bug-text (semantic re-rank) โ local-history-prior-localization (fix-history boost)sim-cross-pr-feature-mapping for prior precedent on new featuresmine-change-coupling to surface partner files that historically move together
Always preprocess identifier tokens via ling-camel-snake-split โ ling-abbreviation-expansion โ ling-porter-stemming before any vocabulary-based algorithm. Skipping this step silently degrades every downstream signal.
Cross-language parsing. Most rule code examples use Python's built-in ast module for brevity. For real cross-language work (Go, Rust, Java, TS, C++ in the same repo), use tree-sitter โ it provides robust parsers for 40+ languages with a uniform API. Every AST-based rule in this skill (PDG clones, GumTree, Zhang-Shasha, POS-tag heads, identifier co-occurrence) maps cleanly onto tree-sitter ASTs.
Reference Files
