菜单
Load large LLMs with 4-bit NF4 quantization and optional double quantization via BitsAndBytes to reduce GPU memory by 4x while preserving inference quality
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Memory-efficient Keras generator that streams sharded CSV files in chunks, renders strokes to images on-the-fly, and yields batches for training on datasets too large for memory
Stack 1D convolutions for local feature extraction before bidirectional LSTMs to classify variable-length stroke sequences into hundreds of doodle categories
Partition a large dataset into N balanced shards using integer key modulo arithmetic for reproducible, class-interleaved splits across CSV files
Normalize raw stroke coordinates to 0-255 range, resample at uniform arc-length spacing, then apply Ramer-Douglas-Peucker simplification
Render stroke sequences to grayscale images with temporal intensity encoding where earlier strokes are brighter and later strokes fade to encode drawing order
Score each face in a video as anomalous by computing L2 distance from the embedding centroid of all faces, then convert to probability via logistic function
| name | llm-4bit-nf4-double-quantization |
| description | Load large LLMs with 4-bit NF4 quantization and optional double quantization via BitsAndBytes to reduce GPU memory by 4x while preserving inference quality |
NormalFloat4 (NF4) quantization maps weights to a 4-bit data type optimized for normally-distributed neural network weights. Double quantization further compresses the quantization constants themselves. Together they reduce a 7B model from ~14GB (fp16) to ~4GB, fitting on a single consumer GPU. Quality loss is minimal for inference tasks like scoring, generation, and classification.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
"mistralai/Mistral-7B-Instruct-v0.1",
quantization_config=bnb_config,
device_map="auto",
torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(
"mistralai/Mistral-7B-Instruct-v0.1")
BitsAndBytesConfig with NF4 quant type and bfloat16 compute dtypedouble_quant=True for additional memory savings (~0.4GB on 7B)device_map="auto" for automatic GPU placement