Complete toolkit for Natural Language Processing with NLTK 3.9.2, covering tokenization, stemming, lemmatization, POS tagging, parsing, WordNet integration, corpus access, and text classification. Use when building Python programs that work with human language data including text preprocessing, linguistic analysis, sentiment scoring, or educational NLP workflows.
Complete toolkit for Natural Language Processing with NLTK 3.9.2, covering tokenization, stemming, lemmatization, POS tagging, parsing, WordNet integration, corpus access, and text classification. Use when building Python programs that work with human language data including text preprocessing, linguistic analysis, sentiment scoring, or educational NLP workflows.
NLTK (Natural Language Toolkit) is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a comprehensive suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, semantic reasoning, and more.
NLTK 3.9.2 (October 2025) supports Python 3.9 through 3.13. It resolved the pickled-models security vulnerability (CVE-2024-39705), added Python 3.13 support, updated download checksums to SHA256, and made WordNet interoperable with various taggers and tagged corpora.
The project is in maintenance mode — welcoming bugfixes and minor enhancements. It is freely available under the Apache 2.0 License and has been called "a wonderful tool for teaching, and working in, computational linguistics using Python."
When to Use
Tokenizing text into words, sentences, or subword units
Stemming and lemmatization for morphological normalization
Part-of-speech tagging with Penn Treebank or Russian tagsets
Language modeling with n-gram models and smoothing
Semantic reasoning with Discourse Representation Theory (DRT), first-order logic, and resolution provers
Educational NLP workflows and introductory computational linguistics
Core Concepts
Tokenization: Breaking text into words, sentences, or other units. NLTK provides multiple tokenizers for different domains — TreebankWordTokenizer for standard English, TweetTokenizer for social media text, PunktSentenceTokenizer for unsupervised sentence boundary detection, and RegexpTokenizer for custom patterns.
Stemming vs Lemmatization: Stemming strips affixes to produce word stems (Porter, Snowball, Lancaster). Lemmatization uses WordNet to map words to dictionary forms (lemmas), producing valid words rather than arbitrary stems.
Part-of-Speech Tagging: Assigning grammatical categories to tokens. NLTK provides UnigramTagger, BigramTagger, TrigramTagger, PerceptronTagger (default for pos_tag), BrillTagger, and HMM-based taggers. English uses the Penn Treebank tagset; Russian uses the Russian National Corpus tagset.
Parsing: Building syntactic structure from tagged tokens. NLTK supports context-free grammars with chart parsers (TopDownChartParser, BottomUpChartParser, EarleyChartParser), CCG parsing, dependency parsing, and interfaces to Stanford CoreNLP.
Corpora: Pre-packaged text datasets accessible through uniform APIs. Over 50 corpora are available including Brown, Gutenberg, Treebank, WordNet, SentiWordNet, movie reviews, Twitter data, and many others.
Frequency Distributions: FreqDist and ConditionalFreqDist from nltk.probability track occurrence counts and support plotting (with Matplotlib).
Installation / Setup
NLTK requires Python 3.9–3.13. Install via pip:
pip install --user -U nltk
After installing the package, download required data:
import nltk
nltk.download('popular') # most commonly used datasets and models
Or from the command line:
python -m nltk.downloader popular
For all data (including corpora, taggers, parsers):
nltk.download('all')
Data is installed to ~/.nltk/ by default. Set NLTK_DATA environment variable for custom locations. For central installation: /usr/local/share/nltk_data (Mac), /usr/share/nltk_data (Unix), or C:\nltk_data (Windows).
Usage Examples
Basic text processing pipeline
import nltk
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.stem import WordNetLemmatizer, PorterStemmer
from nltk import pos_tag
text = "The runners' shoes were amazing. They ran quickly through the park."# Sentence tokenization
sentences = sent_tokenize(text)
print(sentences)
# ['The runners\' shoes were amazing.', 'They ran quickly through the park.']# Word tokenization
tokens = word_tokenize(text)
print(tokens)
# ['The', 'runners', "'s", 'shoes', 'were', 'amazing', '.', ...]# POS tagging
tagged = pos_tag(tokens)
print(tagged[:6])
# [('The', 'DT'), ('runners', 'NNS'), ("'s", 'POS'), ('shoes', 'NNS'), ...]# Lemmatization
lemmatizer = WordNetLemmatizer()
lemmas = [lemmatizer.lemmatize(word, pos=tag[0].lower()) for word, tag in tagged]
print(lemmas)
Sentiment analysis with VADER
from nltk.sentiment import SentimentIntensityAnalyzer
sia = SentimentIntensityAnalyzer()
scores = sia.polarity_scores("This movie was absolutely fantastic!")
print(scores)
# {'neg': 0.0, 'neu': 0.325, 'pos': 0.675, 'compound': 0.8129}
WordNet lookup
from nltk.corpus import wordnet as wn
synsets = wn.synsets("run")
print(synsets[0].definition())
# "change from one state or mode to another"print(synsets[0].lemmas()[0].name())
# "run"
Translation & Language Models: IBM Models 1–5, phrase-based translation, n-gram language models with smoothing (Kneser-Ney, Laplace, Witten-Bell) → Translation & Language Models
