huggingface-on-aws

name	huggingface-on-aws
description	Deploy and train Hugging Face models on AWS using SageMaker and Neuron (Trainium/Inferentia). Use when working with HF models on AWS, Neuron SDK, SageMaker endpoints, Trainium training, TorchNeuron, or when the user mentions Hugging Face with AWS infrastructure.

Hugging Face on AWS — SageMaker & Neuron

Help developers deploy and train Hugging Face models on AWS infrastructure using SageMaker and Neuron hardware (Trainium / Inferentia).

Capabilities

• Deploy HF models to SageMaker endpoints with Neuron containers
• Launch fine-tuning jobs on Trainium via SageMaker
• Configure TorchNeuron (device placement, torch.compile, distributed training)
• Optimize models for Neuron (BF16, compilation, vLLM serving, NKI)
• Recommend instance types based on model size and use case

Quick Reference

Instance Selection

• < 3B params: ml.trn1.2xlarge (train) / ml.inf2.xlarge (infer)
• 3B - 13B: ml.trn1.32xlarge (train) / ml.inf2.8xlarge (infer)
• 13B - 70B: ml.trn1.32xlarge (train) / ml.inf2.48xlarge (infer)
• 70B+: ml.trn2.48xlarge (both)

Container Selection

• General inference: huggingface-pytorch-inference-neuronx DLC
• LLM serving (7B+): huggingface-vllm-inference-neuronx DLC
• Training: huggingface-pytorch-training-neuronx DLC

Key Environment Variables

• NEURON_CC_FLAGS="--model-type=transformer" — for transformer models
• NEURON_RT_STOCHASTIC_ROUNDING_EN=1 — for training convergence
• HF_MODEL_ID — Hugging Face model identifier
• TENSOR_PARALLEL_SIZE — number of NeuronCores for sharding

Two Paths

Current (production): torch-neuronx + optimum-neuron wrappers Coming: TorchNeuron native backend — just model.to('neuron') + standard PyTorch

Reference Docs

• For TorchNeuron patterns and migration: see docs/torchneuron.md
• For SageMaker deployment: see docs/sagemaker-deployment.md
• For distributed training: see docs/distributed-training.md
• For optimization: see docs/model-optimization.md
• For examples: see docs/examples/

Important Reminders

• Always use torchrun even for single worker
• Always set --bf16 and --use_cpu True for Neuron training
• Always remind users to delete endpoints after testing
• Use SageMaker SDK v2 (pip install "sagemaker<3.0.0")
• First deployment of large models takes 15-30 min