AI Hardware
AI Chip Architecture for LLM Inference
A concise map of the compute, memory, and interconnect constraints behind modern language-model serving.
LLM inference places two distinct workloads on an accelerator. Prefill processes many prompt tokens in parallel and can use dense matrix multiplication efficiently. Decode generates tokens sequentially and repeatedly streams model weights and cache data. A chip that looks excellent on peak arithmetic may still underperform during decode.
The memory wall
Decode often has low arithmetic intensity. Each step touches a large fraction of the model while producing relatively little computation per byte moved. High-bandwidth memory, cache hierarchy, and data placement therefore matter as much as nominal FLOPS.
The relevant question is not “How fast is the accelerator?” but “How fast can this workload move the data required for its next operation?”
Architecture layers
Compute arrays
Tensor units determine which data types and matrix shapes run efficiently. Model quantization only creates value when the hardware and kernels can execute the lower-precision path without expensive conversions.
On-chip memory
Registers and SRAM feed compute arrays with much lower energy and latency than external memory. Tiling strategies decide whether operators reuse data effectively or repeatedly fetch it.
High-bandwidth memory
HBM capacity determines model and cache residency. Bandwidth shapes decode speed. Capacity and bandwidth should be evaluated separately because techniques such as quantization can help both in different ways.
Interconnect
Large models span devices. Collective communication, topology, and synchronization affect tensor parallelism and pipeline execution. At cluster scale, network behavior becomes part of the accelerator architecture.
Benchmark the serving system
A useful benchmark specifies model, precision, context distribution, batch policy, latency target, and parallelism strategy. Report prefill and decode separately. Peak tokens per second without a latency constraint can hide queueing that users would reject.
Hardware-aware model design is most effective when it begins early. Attention structure, activation size, sparsity, and precision choices establish the data movement pattern that the final system must support.