القائمة
Process and analyze text using modern NLP techniques - preprocessing, embeddings, and transformers
يُحدَّد Prompt الذي يراجع المصدر أولًا بشكل افتراضي. يمكنك التبديل إلى أمر مباشر أو تنزيل نسخة محلية.
اقرأ SKILL.md وأي ملفات مرافقة يعرضها SkillsMP قبل أن تقرر التثبيت.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
يتجاوز الأمر المباشر Prompt المخصّص للمراجعة. افحص المصدر قبل تشغيله.
npx skills add https://github.com/pluginagentmarketplace/custom-plugin-machine-learning --skill nlp-basicsيبقى الأمر في سطر واحد. مرّر أفقيًا لمراجعته كاملًا قبل النسخ.
تفضّل نسخة محلية؟ نزّل الملفات المتاحة حاليًا لدى SkillsMP.
Discover patterns in unlabeled data using clustering, dimensionality reduction, and anomaly detection
Build computer vision solutions - image classification, object detection, and transfer learning
Build and train neural networks with PyTorch - MLPs, CNNs, and training best practices
استنادا إلى تصنيف SOC المهني
عرض SKILL.md
| name | nlp-basics |
| description | Process and analyze text using modern NLP techniques - preprocessing, embeddings, and transformers |
| version | 1.4.0 |
| sasmp_version | 1.4.0 |
| bonded_agent | 05-nlp |
| bond_type | PRIMARY_BOND |
| parameters | {"required":[{"name":"text","type":"string|list","validation":"Non-empty text or list of texts"}],"optional":[{"name":"model_name","type":"string","default":"bert-base-uncased"},{"name":"max_length","type":"integer","default":512,"validation":"1 <= max_length <= 512"}]} |
| retry_logic | {"strategy":"exponential_backoff","max_attempts":3,"base_delay_ms":1000} |
| logging | {"level":"info","metrics":["tokenization_time","embedding_dim","batch_size"]} |
Transform unstructured text into structured insights.
from transformers import AutoTokenizer, AutoModel
import torch
# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModel.from_pretrained('bert-base-uncased')
# Tokenize
text = "Machine learning is transforming industries."
inputs = tokenizer(text, return_tensors='pt', padding=True, truncation=True)
# Get embeddings
with torch.no_grad():
outputs = model(**inputs)
embeddings = outputs.last_hidden_state.mean(dim=1) # [CLS] pooling
print(f"Embedding shape: {embeddings.shape}")
import re
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
class TextPreprocessor:
def __init__(self):
self.lemmatizer = WordNetLemmatizer()
self.stop_words = set(stopwords.words('english'))
def clean(self, text):
# Lowercase
text = text.lower()
# Remove URLs
text = re.sub(, , text)
text = re.sub(, , text)
tokens = word_tokenize(text)
tokens = [.lemmatizer.lemmatize(t) t tokens
t .stop_words]
.join(tokens)
| Type | Model | Use Case |
|---|---|---|
| Static | Word2Vec, GloVe | Simple, fast |
| Contextual | BERT, RoBERTa | SOTA accuracy |
| Sentence | all-MiniLM | Similarity, search |
from sentence_transformers import SentenceTransformer
# Sentence embeddings
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(['Hello world', 'Hi there'])
# Similarity
from sklearn.metrics.pairwise import cosine_similarity
similarity = cosine_similarity([embeddings[0]], [embeddings[1]])[0][0]
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import Trainer, TrainingArguments
# Load pretrained model
model_name = 'bert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(
model_name, num_labels=2
)
# Training arguments
training_args = TrainingArguments(
output_dir='./results',
num_train_epochs=3,
per_device_train_batch_size=16,
evaluation_strategy='epoch',
save_strategy='epoch',
load_best_model_at_end=True
)
# Train
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset
)
trainer.train()
from transformers import pipeline
# NER pipeline
ner = pipeline('ner', aggregation_strategy='simple')
text = "Apple Inc. was founded by Steve Jobs in Cupertino, California."
entities = ner(text)
for entity in entities:
print(f"{entity['word']}: {entity['entity_group']} ({entity['score']:.2f})")
from sentence_transformers import SentenceTransformer
import numpy as np
class SemanticSearch:
def __init__(self, model_name='all-MiniLM-L6-v2'):
self.model = SentenceTransformer(model_name)
self.corpus_embeddings = None
self.corpus = None
def index(self, documents):
self.corpus = documents
self.corpus_embeddings = self.model.encode(documents)
def search(self, query, top_k=5):
query_embedding = self.model.encode([query])[0]
scores = np.dot(self.corpus_embeddings, query_embedding)
top_indices = np.argsort(scores)[-top_k:][::-1]
return [(self.corpus[i], scores[i]) for i in top_indices]
# TODO: Fine-tune BERT for sentiment classification
# Use the IMDB dataset
# TODO: Build a semantic search engine
# Index 1000 documents and search by query
import pytest
def test_preprocessing():
"""Test text preprocessing."""
preprocessor = TextPreprocessor()
text = "Hello World! Visit https://example.com"
cleaned = preprocessor.clean(text)
assert 'http' not in cleaned
assert cleaned.islower()
def test_embeddings_shape():
"""Test embedding dimensions."""
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(['test'])
assert embeddings.shape == (1, 384)
| Problem | Cause | Solution |
|---|---|---|
| OOV tokens | Rare words | Use subword tokenization |
| Slow inference | Large model | Use distilled model |
| Poor accuracy | Small dataset | Data augmentation |
| Memory error | Long sequences | Reduce max_length |
05-nlpdeep-learningcomputer-visionVersion: 1.4.0 | Status: Production Ready