| name | vocabulary-overlap-analysis |
| description | Analyze vocabulary overlap and identify named entities unique to domain/document corpora. |
| required_roles | {"scribe":"roles/scribe.viewer"} |
| personas | ["information-architect","content-strategist","taxonomist","linguist"] |
Vocabulary Overlap & Domain Entity Analysis Skill
Analyze a set of target documents to identify domain-specific vocabulary and named entities that are unique to your corpus. By contrasting your content's vocabulary against general-purpose reference corpora or a secondary document set, this skill helps surface proprietary terminology, key subject matter entities, and vocabulary overlaps.
Inputs
TARGET_PATH - The directory or file path of the target document corpus to analyze (e.g., "/documentation/product-guides").REFERENCE_PATH - (Optional) A path to a secondary document corpus to compare against. If not provided, the analysis will use general web, general English, and common domain knowledge baselines.MIN_FREQUENCY - (Optional) The minimum number of occurrences for a term or entity to be included in the analysis (default: 2).EXCLUDE_COMMON - (Optional) Boolean, whether to automatically filter out standard English stopwords and common business/industry jargon (default: true).
Workflow
Step 1: Corpus Profiling & Tokenization
Extract and profile all text from files in TARGET_PATH.
- Extract raw text from markdown, HTML, PDF, or text files.
- Clean and normalize the text (punctuation stripping, markup cleaning, etc.). Do not perform blanket lowercase conversion during tokenization and n-gram generation to preserve case casing cues essential for Named Entity Recognition (NER).
- Tokenize the text into individual words (unigrams) and common phrases (bigrams and trigrams) maintaining original casing.
- Identify potential Named Entities (proper nouns, technical abbreviations, capitalized concepts).
Step 2: Contrastive Frequency Analysis
Contrast target frequencies against reference datasets to isolate characteristic terms.
- Calculate term frequencies (TF) and document frequencies (DF) in the target corpus.
- If
REFERENCE_PATH is provided, perform the same frequency extraction for the reference corpus.
- If no reference path is provided, evaluate the term list against standard baseline general English corpora (e.g., Wikipedia or common web text datasets) and standard industry terminology.
- Calculate contrastive scores (e.g., TF-IDF or Log-Likelihood ratio) to highlight terms that are significantly more frequent in the target corpus than in the baseline.
Step 3: Domain Entity Extraction & Classification
Isolate, consolidate, and categorize unique vocabulary and named entities.
- Filter terms to isolate those exclusive to
TARGET_PATH (zero or very low overlap with the reference baseline).
- Variant Consolidation & Concept Mapping: Analyze casing, punctuation, and orthographic variants (e.g., "Scattered Spider", "scattered spider", "Scattered-Spider") and merge them into a single canonical concept. Define a capitalized or formal Preferred Term, and map all alternative representations as Variant Terms.
- Classify these consolidated unique terms into distinct semantic groups:
- Proprietary Entities: Brand names, proprietary product names, internal tool names, code names.
- Domain Jargon & Technical Terms: Specialized technical, operational, or industry-specific terms.
- Key Subject Matter Concepts: Essential concepts defining the core subject matter of the documents.
- Acronyms & Abbreviations: Short-hand terms and their expansions.
Step 4: Overlap & Commonality Assessment
Assess the degree of similarity and overlap between the corpora.
- Calculate vocabulary overlap coefficients (e.g., Jaccard Similarity index or overlap coefficient) between
TARGET_PATH and REFERENCE_PATH (if provided).
- Generate a mapped list of shared terminology representing common thematic ground.
Required Outputs
A VOCABULARY_OVERLAP_REPORT in markdown format containing:
- Executive Summary: Key vocabulary metrics including total unique tokens, density of domain-specific terms, and overlap coefficients.
- Unique Domain Entities: A structured table of named entities unique to the target corpus, classified by category (e.g., Product, Brand, Concept, Acronym) showing the Preferred Term, consolidated Variant Terms, frequency counts, and contextual examples.
- Domain Jargon & Technical Glossary: A list of specialized terms and phrases unique to this corpus, complete with contextual definitions derived from the source material.
- Vocabulary Overlap Matrix: A detailed analysis of shared vs. exclusive vocabulary, highlighting areas of alignment and divergence.
- Actionable Recommendations: Strategic guidance on how to apply these findings to:
- Custom taxonomy and metadata schema development.
- Glossary and controlled vocabulary creation.
- Search engine optimization (SEO) and query expansion strategies.
- Automated content classification and tagging.
Quick Reference
- Purpose: Uncover unique named entities and domain-specific terminology to establish controlled vocabularies and map content characteristics.
- Outcome: An actionable, structured domain vocabulary map and catalog of proprietary named entities.
References
- Aghaei, E., Niu, X., Shadid, W., & Al-Shaer, E. (2022). SecureBERT: A Domain-Specific Language Model for Cybersecurity. arXiv preprint arXiv:2204.02685. https://doi.org/10.48550/arxiv.2204.02685
- Aghaei, E., Jain, S., Arun, P., & Sambamoorthy, A. (2025). SecureBERT 2.0: Advanced Language Model for Cybersecurity Intelligence. arXiv preprint arXiv:2510.00240. https://doi.org/10.48550/arxiv.2510.00240
- Ashfaq, F. (2025). Enhancing ECG Report Generation With Domain-Specific Tokenization for Improved Medical NLP Accuracy. IEEE Xplore.
- Beltagy, I., Lo, K., & Cohan, A. (2019). SciBERT: A Pretrained Language Model for Scientific Text. arXiv preprint arXiv:1903.10676. https://doi.org/10.48550/arxiv.1903.10676
- Provilkov, I., Emelianenko, D., & Voita, E. (2020). BPE-Dropout: Simple and Effective Subword Regularization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.170
- Radaideh, R., & Khreis, A. (2026). SMSI: System Model Security Inference: Automated Threat Modeling for Cyber-Physical Systems. arXiv preprint arXiv:2604.23905. https://doi.org/10.48550/arxiv.2604.23905
- Ruiz-Ródenas, Á., Pujante Sáez, J., García-Algora, D., Rodríguez Béjar, M., Blasco, J., & Hernández-Ramos, J. L. (2025). SynthCTI: LLM-Driven Synthetic CTI Generation to enhance MITRE Technique Mapping. arXiv preprint arXiv:2507.16852. https://doi.org/10.48550/arxiv.2507.16852
- Teklehaymanot, H. K., Fazlija, D., & Nejdl, W. (2025). MoVoC: Morphology-Aware Subword Construction for Geez Script Languages. arXiv preprint arXiv:2509.08812. https://doi.org/10.48550/arxiv.2509.08812
- Teklehaymanot, H., Fazlija, D., & Nejdl, W. (2026). LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation. arXiv preprint arXiv:2603.22629. https://doi.org/10.48550/arxiv.2603.22629
