transformers

# Hugging Face Transformers - Modern AI Models Transformers provides APIs and tools to easily download and train state-of-the-art pretrained models. It reduces compute costs and carbon footprint by allowing researchers to reuse models instead of training from scratch. ## When to Use - Natural Language Processing (Summarization, Translation, Named Entity Recognition). - Scientific Sequence Analysis (Protein folding, DNA/RNA sequence modeling). - Chemical Property Prediction (Using molecular strings like SMILES). - Computer Vision (Vision Transformers - ViT, Image Classification). - Time Series Forecasting with foundation models. - Fine-tuning Large Language Models (LLMs) on domain-specific scientific literature. - Multimodal tasks (Document AI, Visual Question Answering). ## Reference Documentation **Official docs**: https://huggingface.co/docs/transformers/ **Model Hub**: https://huggingface.co/models **Search patterns**: `pipeline`, `AutoModel`, `AutoTokenizer`, `Trainer`, `PEFT` (Parameter-Efficient Fine-Tuning) ## Core Principles ### The "Auto" Classes Hugging Face uses "Auto" classes (`AutoModel`, `AutoTokenizer`) that automatically infer the correct architecture from the model name/path. This makes code highly portable. ### Tokenization Before data enters a model, it must be converted into numerical tokens. The Tokenizer handles this, including padding, truncation, and special tokens (like `[CLS]`, `[SEP]`). ### Pipelines The simplest way to use a model. It abstracts away tokenization, model execution, and post-processing into a single `pipe(data)` call. ## Quick Reference ### Installation ```bash pip install transformers datasets tokenizers # Requires a backend (PyTorch or JAX) pip install torch ``` ### Standard Imports ```python from transformers import pipeline, AutoModel, AutoTokenizer, TrainingArguments, Trainer import torch ``` ### Basic Pattern - Using a Pretrained Pipeline ```python from transformers import pipeline # 1. Initialize a pipeline (automatically downloads model) classifier = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english") # 2. Run inference results = classifier("The molecular structure of this compound is fascinating.") print(results) ``` ## Critical Rules ### ✅ DO - **Use the Auto Classes** - Always prefer `AutoTokenizer.from_pretrained()` and `AutoModel.from_pretrained()` for flexibility. - **Set the Device** - Explicitly set `device=0` (for CUDA) or `device="mps"` (for Mac) in pipelines to ensure GPU acceleration. - **Cache Models** - Models are large. Use the `HF_HOME` environment variable to manage where models are stored on disk. - **Handle Truncation** - Most models have a maximum sequence length (usually 512). Always use `truncation=True` in tokenizers. - **Use Datasets Library** - For training, use the `datasets` library to handle data loading and streaming without filling RAM. - **Save Tokenizers with Models** - When fine-tuning, always save the tokenizer alongside the model to ensure consistency. ### ❌ DON'T - **Load Models in a Loop** - Loading a model takes seconds and GBs of RAM. Load once, reuse many times. - **Upload Private Data** - Be careful when using models that might send data to an API (though transformers is mostly local execution). - **Ignore Padding** - For batch processing, ensure `padding=True` so all sequences in the batch have the same length. - **Use Wrong Model for Task** - A "BERT" model is for understanding; "GPT" is for generation. Use the right architecture. ## Anti-Patterns (NEVER) ```python from transformers import AutoModel, AutoTokenizer # ❌ BAD: Re-initializing the model inside a function called frequently def get_prediction(text): model = AutoModel.from_pretrained("bert-base-uncased") # ❌ SLOW & RAM HEAVY return model(text) # ✅ GOOD: Load once globally or in a class model = AutoModel.from_pretrained("bert-base-uncased") def get_prediction(text): return model(text) # ❌ BAD: Manual string splitting for "tokens" # tokens = text.split(" ") # ❌ Not compatible with model vocabulary # ✅ GOOD: Use the model's specific tokenizer tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") inputs = tokenizer(text, return_tensors="pt") # ❌ BAD: Forgetting to move model to GPU # model = AutoModel.from_pretrained("...") # output = model(inputs.to("cuda")) # ❌ Error: Model is on CPU! ``` ## Tokenization Deep Dive ### Preparing Data for Models ```python tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") texts = ["Science is cool.", "Quantum physics is hard."] # Batch encoding inputs = tokenizer( texts, padding=True, truncation=True, max_length=128, return_tensors="pt" # Returns PyTorch tensors ) print(inputs['input_ids'].shape) # (batch_size, seq_len) ``` ## The Trainer API ### Simplified Training Loop ```python from transformers import TrainingArguments, Trainer training_args = TrainingArguments( output_dir="./results", num_train_epochs=3, per_device_train_batch_size=16, evaluation_strategy="epoch", logging_dir="./logs", ) trainer = Trainer( model=model, args=training_args, train_dataset=tokenized_train, eval_dataset=tokenized_val, ) # trainer.train() ``` ## Scientific Applications ### 1. Protein Sequence Analysis (ESM) ```python # ESM-2 is a powerful protein language model tokenizer = AutoTokenizer.from_pretrained("facebook/esm2_t6_8M_UR50D") model = AutoModel.from_pretrained("facebook/esm2_t6_8M_UR50D") protein_seq = "MAPLRKTYLLG" inputs = tokenizer(protein_seq, return_tensors="pt") with torch.no_grad(): outputs = model(**inputs) # The 'last_hidden_state' represents a "feature vector" for each amino acid embeddings = outputs.last_hidden_state ``` ### 2. Chemical Property Prediction (SMILES) ```python # Using a model trained on molecular strings pipe = pipeline("text-classification", model="seyonec/ChemBERTa-zinc-base-v1") smiles = "CC(=O)Oc1ccccc1C(=O)O" # Aspirin result = pipe(smiles) print(f"Prediction: {result}") ``` ### 3. Named Entity Recognition (NER) for Papers ```python # Extracting genes, proteins, or chemicals from text ner_pipe = pipeline("ner", model="dslim/bert-base-NER") text = "The expression of the BRCA1 gene was observed in the sample." entities = ner_pipe(text) ``` ## Performance and Efficiency ### 1. Quantization (bitsandbytes) Running large models on consumer GPUs by reducing precision (8-bit or 4-bit). ```python from transformers import BitsAndBytesConfig # Load model in 4-bit precision quant_config = BitsAndBytesConfig(load_in_4bit=True) model = AutoModel.from_pretrained("model_name", quantization_config=quant_config) ``` ### 2. Using pipeline with GPU ```python # 'device=0' targets the first CUDA device pipe = pipeline("translation_en_to_fr", model="t5-base", device=0) ``` ## Common Pitfalls and Solutions ### "Out of Memory" (OOM) on GPU ```python # ❌ Problem: Batch size is too large for GPU RAM # ✅ Solution: # 1. Reduce 'per_device_train_batch_size' # 2. Use 'gradient_accumulation_steps' to keep effective batch size # 3. Use 'fp16=True' in TrainingArguments ``` ### Model Output is a Dictionary, not a Tensor ```python # ❌ Problem: outputs[0] works, but is confusing # ✅ Solution: Access by name outputs = model(**inputs) hidden_states = outputs.last_hidden_state ``` ### Slow Tokenization ```python # ✅ Solution: Use "Fast" tokenizers (written in Rust, usually default) tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=True) ``` Hugging Face Transformers has democratized AI for the scientific community. By providing a unified interface to the world's most powerful models, it allows researchers to spend less time on engineering and more time on discovering insights from data.