HBM is pricey and supply-constrained. The memory 'dies' themselves are not fundamentally different from standard DDR; the real divergence is in architecture and packaging, distilled into two ideas:
First, 'stacking.' There are two ways to expand capacity: increase bit density per die, or pack more dies into the same footprint. The first path has hit a wall—DRAM scaling is hard, as shrinking cells too far lets electrons leak across boundaries, corrupting data; the most advanced nodes are only 11–12nm, and even by 2030 may reach just ~10nm, far from logic at 2–3nm.
More importantly, logic follows Moore's Law, roughly doubling density every ~2 years, while memory does not enjoy that tailwind. Since density gains from process shrinks are limited, the industry chose the second route—stacking dies vertically to increase capacity.
HBM4 already stacks 12 layers, triple the first generation. Market chatter suggests it could reach 16 layers around 2026–27.
Second, 'attaching.' To deliver bandwidth, HBM must sit next to the GPU and be co-packaged on the same silicon interposer. Consumer GPUs place DRAM chips around the core on the PCB, but server GPUs must cram HBM into the same small area as the core, and that area cannot simply be enlarged.
Space is even tighter because the four edges around the core are reserved first for I/O and external interfaces. What is left for memory is mainly the top and bottom edges, with the left and right already taken, so when you run out of horizontal room, the only way is to build upwards.
In short, 'stacking' brings capacity, and 'attaching' brings bandwidth. The technical barriers and high cost of HBM are the price paid for these two choices.