ai-hardware-selection

name	ai-hardware-selection
description	Selecting accelerators for AI workloads: GPU vs TPU vs NPU vs FPGA vs CPU, and the metrics that actually decide it — memory capacity & bandwidth, TOPS/ FLOPS, interconnect, and cost/Watt. Architect-level hardware-fit reasoning. USE WHEN: choosing AI hardware/accelerators, "which GPU", "TPU vs GPU", "NPU", "FPGA", "HBM/memory bandwidth", "TOPS", "cost per token", VRAM sizing for a model, training vs inference hardware, accelerator interconnect. DO NOT USE FOR: serving software topology (use `inference-serving-topology`); on-device runtimes (use `edge-inference`); generic CPU perf (use systems/hardware-aware-design).
allowed-tools	Read, Grep, Glob

AI Hardware Selection

The metric that usually decides: memory, then bandwidth

For LLM inference, the binding constraint is typically VRAM/HBM capacity (weights + KV-cache must fit) and memory bandwidth (decode is memory-bound) — not raw FLOPS. Size first: weights ≈ params × bytes/param (e.g. 70B × 2B(FP16) ≈ 140GB → multi-GPU or quantize). Add KV-cache (grows with context × batch). Only then look at TOPS.

Accelerator families

Type	Strength	Use
GPU (NVIDIA H/B-series, AMD MI)	Flexible, huge ecosystem, HBM	Training + inference, the default
TPU	Matmul-dense, pod-scale interconnect	Large-scale training/inference on GCP
NPU	Perf/Watt at low power	Edge / mobile / AI-PC inference
FPGA	Custom low-latency dataflow	Niche ultra-low-latency / fixed pipelines
CPU	Available, fine for small/batch	Small models, embeddings, light load

Other levers

• Interconnect (NVLink, InfiniBand): decisive for multi-GPU training and tensor parallelism — bandwidth between accelerators bounds scaling.
• Precision support: FP8/INT4 support multiplies effective throughput/capacity.
• Cost/Watt & TCO: cloud per-hour vs owned; power/cooling; utilization. The honest metric is cost per token (or per request) at target latency.
• Training vs inference: training needs FLOPS + interconnect + memory; inference needs memory capacity/bandwidth + latency.

When to recommend what

• Default / flexibility / training → NVIDIA GPUs (size by model VRAM).
• Edge/mobile/low-power inference → NPU.
• Hyperscale training on GCP → TPU pods.
• Fixed ultra-low-latency pipeline → FPGA (only if justified).