natural-language-processing

name	Natural Language Processing
description	Build NLP applications using transformers library, BERT, GPT, text classification, named entity recognition, and sentiment analysis

Natural Language Processing

Overview

This skill provides comprehensive tools for building NLP applications using modern transformers, BERT, GPT, and classical NLP techniques for text classification, named entity recognition, sentiment analysis, and more.

When to Use

• Building text classification systems for sentiment analysis, topic categorization, or intent detection
• Extracting named entities (people, places, organizations) from unstructured text
• Implementing machine translation, text summarization, or question answering systems
• Processing and analyzing large volumes of textual data for insights
• Creating chatbots, virtual assistants, or conversational AI applications
• Fine-tuning pre-trained transformer models for domain-specific NLP tasks

NLP Core Tasks

• Text Classification: Sentiment, topic, intent classification
• Named Entity Recognition: Identifying people, places, organizations
• Machine Translation: Text translation between languages
• Text Summarization: Extracting key information
• Question Answering: Finding answers in documents
• Text Generation: Generating coherent text

Popular Models and Libraries

• Transformers: BERT, GPT, RoBERTa, T5
• spaCy: Industrial NLP pipeline
• NLTK: Classic NLP toolkit
• Hugging Face: Pre-trained models hub
• PyTorch/TensorFlow: Deep learning frameworks

Python Implementation

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from collections import Counter
import re
import nltk
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer
import torch
from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
                         AutoModelForTokenClassification, pipeline,
                         TextClassificationPipeline)
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
import warnings
warnings.filterwarnings('ignore')

# Download required NLTK resources
try:
    nltk.data.find('tokenizers/punkt')
except LookupError:
    nltk.download('punkt')

print("=== 1. Text Preprocessing ===")

def preprocess_text(text, remove_stopwords=True, lemmatize=True):
    """Complete text preprocessing pipeline"""
    # Lowercase
    text = text.lower()

    # Remove special characters and digits
    text = re.sub(r'[^a-zA-Z\s]', , text)

    
    tokens = word_tokenize(text)

    
     remove_stopwords:
        stop_words = (stopwords.words())
        tokens = [t  t  tokens  t   stop_words]

    
     lemmatize:
        lemmatizer = WordNetLemmatizer()
        tokens = [lemmatizer.lemmatize(t)  t  tokens]

     tokens, .join(tokens)

sample_text = 
tokens, processed = preprocess_text(sample_text)
()
()
()


()


texts = [
    ,
    ,
    ,
    ,
    ,
    ,
    ,
    ,
    ,
    
]

labels = [, , , , , , , , , ]  


tfidf = TfidfVectorizer(max_features=, ngram_range=(, ))
X_tfidf = tfidf.fit_transform(texts)


clf = MultinomialNB()
clf.fit(X_tfidf, labels)


predictions = clf.predict(X_tfidf)
()
()
()
()


()

:
    
    sentiment_pipeline = pipeline(
        ,
        model=
    )

    test_sentences = [
        ,
        ,
        ,
        
    ]

    ()
     sentence  test_sentences:
        result = sentiment_pipeline(sentence)
        ()
        ()

 Exception  e:
    ()


()

:
    ner_pipeline = pipeline(
        ,
        model=,
        aggregation_strategy=
    )

    text = 
    entities = ner_pipeline(text)

    ()
    ()
     entity  entities:
        ()

 Exception  e:
    ()


()

 sklearn.metrics.pairwise  cosine_similarity


vectorizer = CountVectorizer(max_features=)
docs = [
    ,
    ,
    
]

embeddings = vectorizer.fit_transform(docs).toarray()


similarity_matrix = cosine_similarity(embeddings)
()
(pd.DataFrame(similarity_matrix, columns=[  i  ((docs))],
                  index=[  i  ((docs))]).())


()

corpus = .join(texts)
tokens, _ = preprocess_text(corpus)


vocab = Counter(tokens)
()
()
 word, count  vocab.most_common():
    ()


()

:
    
    zero_shot_pipeline = pipeline(
        ,
        model=
    )

    sequence = 
    candidate_labels = [, , , ]

    result = zero_shot_pipeline(sequence, candidate_labels)
    ()
     label, score  (result[], result[]):
        ()

 Exception  e:
    ()


()

sample_texts = [
    ,
    ,
    
]

stats_data = []
 text  sample_texts:
    words = text.split()
    chars = (text)
    avg_word_len = np.mean([(w)  w  words])

    stats_data.append({
        : text[:] +   (text) >   text,
        : (words),
        : chars,
        : avg_word_len
    })

stats_df = pd.DataFrame(stats_data)
(stats_df.to_string(index=))


()

fig, axes = plt.subplots(, , figsize=(, ))


word_freq = vocab.most_common()
words, freqs = (*word_freq)
axes[, ].barh(((words)), freqs, color=)
axes[, ].set_yticks(((words)))
axes[, ].set_yticklabels(words)
axes[, ].set_xlabel()
axes[, ].set_title()
axes[, ].invert_yaxis()


sentiments = [, , , , ]
sentiment_counts = Counter(sentiments)
axes[, ].pie(sentiment_counts.values(), labels=sentiment_counts.keys(),
              autopct=, colors=[, ])
axes[, ].set_title()


im = axes[, ].imshow(similarity_matrix, cmap=, aspect=)
axes[, ].set_xticks(((docs)))
axes[, ].set_yticks(((docs)))
axes[, ].set_xticklabels([  i  ((docs))])
axes[, ].set_yticklabels([  i  ((docs))])
axes[, ].set_title()
plt.colorbar(im, ax=axes[, ])


text_lengths = [(t.split())  t  texts]
axes[, ].hist(text_lengths, bins=, color=, edgecolor=)
axes[, ].set_xlabel()
axes[, ].set_ylabel()
axes[, ].set_title()
axes[, ].grid(, alpha=, axis=)

plt.tight_layout()
plt.savefig(, dpi=, bbox_inches=)
()


()
()
()
()
()

()

Common NLP Tasks and Models

• Classification: DistilBERT, RoBERTa, ELECTRA
• NER: BioBERT, SciBERT, spaCy models
• Translation: MarianMT, M2M-100
• Summarization: BART, Pegasus, T5
• QA: BERT, RoBERTa, DeBERTa

Text Preprocessing Pipeline

1. Lowercasing and cleaning
2. Tokenization
3. Stopword removal
4. Lemmatization/Stemming
5. Vectorization

Best Practices

• Use pre-trained models when available
• Fine-tune on task-specific data
• Handle out-of-vocabulary words
• Batch process for efficiency
• Monitor for bias in models

Deliverables

• Trained NLP model
• Text classification results
• Named entities extracted
• Performance metrics
• Visualization dashboard
• Inference API