Hot Chips Deep Dive: HBM Moves Toward Customization and 3D Stacking; Samsung, NVIDIA, and Micron Unveil Roadmaps

The explosion in AI computing demand is pushing HBM memory to a crossroads in architectural evolution. At this year's Hot Chips conference, Samsung, Micron, SK Hynix, and NVIDIA intensively disclosed their respective technology roadmaps, with core topics highly concentrated: HBM is shifting from a standardized commodity to a customized platform, 3D vertical stacking has become the common direction for next-generation architectures, and thermal and power constraints have become the primary bottlenecks limiting expansion.

Samsung proposed a three-stage roadmap evolving the HBM base die from a communication layer to a customized AI platform, and unveiled the ZHBM concept—vertically stacking HBM directly on top of an XPU. The goal is to reduce total DRAM power consumption by approximately 70% compared to HBM5, with an absolute power reduction of over 100W and a bandwidth increase of more than 2.3 times. Meanwhile, NVIDIA revealed that its labs possess at least three RVA23 processors and completed the first public demonstration of CUDA running on RISC-V hardware, showcasing a specific path for third-party custom CPUs to access the NVLink Fusion ecosystem.

From a market perspective, the continuous acceleration of AI capital expenditure by hyperscale cloud providers is the core narrative thread running through the conference. Storage analyst Jim Handy pointed out that the number of GBs produced per wafer for HBM is only one-third that of standard DDR. Plus, given that the industry has seen almost no large-scale new fab construction in the past decade or so, even under an aggressive timeline, new capacity will take about two years to come online, severely limiting the industry's ability to respond to current demand shocks. HBM spot prices have risen approximately sevenfold from previous levels, with Samsung, SK Hynix, Micron, and major NAND suppliers all recording exceptionally strong revenue growth.

HBM Supply and Demand Structure: Capacity Bottlenecks Unlikely to Ease in the Short Term

Handy's analysis outlines the core contradictions in the current memory market.

On the demand side, almost all mainstream AI accelerators—including NVIDIA GPUs, Google TPUs, and various custom silicon chips—rely on HBM. Even some network components have begun adopting HBM, creating a new category of demand that barely existed a few years ago. On the supply side, because historical process shrinkage was sufficient to meet bit growth, DRAM suppliers have not undergone large-scale capacity expansion for over ten years, making short-term new capacity additions nearly hopeless.

Handy also refuted the common assertion that "improved algorithmic efficiency will suppress hardware demand." He argued that when a new technology reduces memory requirements by six times, the typical response from hyperscale cloud providers is to process six times the number of tokens with the same budget, rather than cutting capital expenditure. Efficiency gains change the productivity of AI infrastructure but do not necessarily reduce the total amount of capital invested.

The expansion of AI infrastructure is simultaneously tightening the NAND and HDD markets. After META tested the deployment of low-cost QLC SSDs between TLC SSDs and HDDs and observed performance improvements, hyperscale data centers are adopting SSDs more widely. The pull from AI infrastructure on HDD demand has also caused shortages, prompting QLC SSDs to begin filling new storage tiers and even replacing scarce HDDs, thereby pushing NAND into a state of shortage, from which Kioxia and SanDisk (SNDK) are benefiting.

Micron's Perspective: Silicon Consumption, Thermal Management, and Reliability as Architectural Constraints

Ragu, HBM Design Architect at Micron, quantified the cost of HBM expansion from two dimensions: cost and physical limits. In a typical GPU package containing four 12-layer HBM stacks, the total memory silicon area is about eight times that of the GPU silicon, meaning that approximately 90% of the silicon in the system-in-package is related to memory. The failure of a single DRAM die can affect the entire package, and reliability challenges intensify significantly as stack height increases.

Ragu cited data from Meta's Llama 3 research: approximately 17% of unexpected training interruptions are attributed to HBM. This data directly reveals the substantial impact of HBM reliability on large-scale AI training clusters.

Thermal issues have risen to become the primary architectural constraint. The HBM base die is one of the most severe heat-generating areas. Heat is generated from the bottom while the cooling device is located above the DRAM stack, forcing heat to traverse every layer. Micron stated that HBM design is increasingly centered around thermal limitations from the outset, with local power density and hotspots becoming as important as total power consumption.

Regarding stacking limits, Micron believes that 16-layer HBM is feasible in terms of technical path, but the 20-layer stacking discussed by JEDEC still faces numerous unresolved challenges—TSV density, power density, thermal management, mechanical integrity, and manufacturing yield all constitute strict constraints, rather than the signal transmission distance itself (the entire stack height remains below approximately 1 millimeter).

From the perspective of bandwidth expansion, HBM achieves performance leaps through extreme parallelization: HBM3E has 128 banks per DRAM die, increasing to 256 in HBM4; external interface I/O lines double from about 1,000 to 2,000 while the interface length (shoreline) remains basically unchanged. Thus, HBM has expanded from HBM1's 128 GB/s bandwidth and 1 GB capacity to HBM4's bandwidth of over 2.8 TB/s and single-die capacity of over 24 GB, at the cost of continuously rising silicon consumption intensity.

Micron expects that as AI workloads segment, HBM will move from general-purpose products to customized architectures for specific workloads, with processors and paired memory becoming increasingly co-optimized. Packaging technology will also evolve from micro-bumps and thermocompression bonding to fusion bonding and hybrid bonding with single-digit micron spacing.

Samsung Roadmap: Base Die Evolution and ZHBM Vision

Samsung's three-stage roadmap was one of the most forward-looking technical disclosures at the conference.

The first stage focuses on migrating the memory controller from the XPU to a customized HBM base die. Samsung estimates that the controller accounts for about 5% to 10% of the XPU silicon area, and releasing this space to compute cores is expected to improve performance by 10% to 20%. Samsung stated that most customers are actively exploring this architecture.

The second stage utilizes the unused edge area (shoreline) around the customized HBM base die to directly connect a second layer of memory via dedicated controllers and PHYs. It is expected that latency and bandwidth performance will be superior to PCIe-based expansion solutions. This second layer of memory can be LPDDR or even HBF. Samsung believes that as context windows and KV caches expand, the importance of capacity is approaching that of bandwidth.

The third stage is ZHBM: eliminating the traditional 2.5D interposer and vertically integrating HBM directly on top of the XPU. By removing lateral PHY/D2D paths, I/O and TSV structures can cover the entire chip projection area. The target energy efficiency is approximately 0.5 pJ/bit, with total DRAM power consumption reduced by about 70% compared to HBM5, and an absolute power reduction of over 100W. However, current thermal limitations point to a stack of about 4 layers, rather than 12 or 16 layers, indicating that engineering implementation is still quite distant.

Samsung remains restrained in its compute offloading strategy: thermal limitations make it unreasonable to place a large number of dense compute units in the base die. The priority is to offload memory-bound Attention computations in LLM inference to HBM, while compute-intensive Prefill and FFN operations remain on the XPU. Regarding interface strategy, Samsung currently prefers proprietary HBM vendor interfaces over UCIe, citing that UCIe has a larger area and higher power consumption; its Heat Path Block can reduce peak temperature by more than 35% when covering sufficient hotspot areas.

ZHBM also requires wafer-level integration and hybrid copper bonding at pitches of less than 6 microns. The long-standing boundary between DRAM and SoC design must be eliminated, with both requiring co-design from the outset around routing, thermal management, power density, and physical interconnects.

SK Hynix: Packaging Limits and Hybrid Bonding Path

SK Hynix's presentation focused on the packaging engineering challenges of expanding from 12 layers to 16 and even 20 layers.

In its 16-layer HBM3E test chip, even though the total stack thickness increased from 720 microns to 775 microns, the thickness of individual DRAM dies still needed to be reduced by about 10% compared to the 12-layer solution, and the inter-die gap shrunk by about 50%, significantly increasing the difficulty of controlling chip warpage and gap filling. SK Hynix pointed out that the total thickness of HBM4 at 775 microns is approaching the practical limit of current packaging methods, and future expansion cannot rely indefinitely on increasing package height.

Regarding hybrid bonding, SK Hynix believes it will become highly attractive at around 20 layers, but does not expect it to be adopted in the HBM4E phase. In a hypothetical 20-layer stack, die thickness could increase by about 20% to 24%, and thermal conductivity would improve by about 35%. Even so, hybrid bonding cannot solve local hotspot issues in the base die—SK Hynix is developing IHBM, which improves local thermal management by adding silicon heat spreaders above hotspot areas, requiring co-optimization from the initial design stage rather than being added later.

SK Hynix also judges that training and inference may eventually adopt different memory architectures: training requires balancing high bandwidth and high capacity, supporting continued increases in stack height; inference may adopt smaller pools of extremely high-bandwidth memory, placing bandwidth-insensitive data in LPDDR or other low-cost tiers.

NVIDIA and RISC-V: New Paths for Opening the CUDA Ecosystem

NVIDIA's presentation marked substantial progress in its AI ecosystem strategy. A year ago, the biggest obstacle to bringing CUDA to RISC-V was the lack of suitable RVA23 hardware; today, NVIDIA labs possess at least three RVA23 processors, and SiFive completed the first public demonstration of CUDA running on RISC-V hardware during the Hot Chips conference.

NVIDIA speaker Franz clarified that their starting point is compliance with the RVA23 profile and RISC-V server platform specifications, as NVIDIA does not wish to create a separate RISC-V specification for CUDA. The specific additional requirements for CUDA are only about two pages, with cores including PCIe cache coherence (avoiding explicit CPU cache operations during GPU DMA) and PCIe peer-to-peer communication (supporting direct data exchange between GPUs).

NVLink Fusion provides the business logic for this path: customers can use custom CPUs or accelerators while retaining most of NVIDIA's rack-level architecture. Thus, RISC-V CPUs can replace NVIDIA's ARM-based host processors in specialized systems, with the key requirement being the integration of NVIDIA's C2C network—which can provide up to 5 times the bandwidth and coherence of PCIe. Franz's clear statement is that the appeal of RISC-V lies not in the shortcomings of x86 or ARM, but in the ability of numerous vendors to create customized implementations for important niche markets, which may not receive specialized support from large existing CPU suppliers.

In conjunction with NVIDIA's push, Canonical has established RVA23 as the official RISC-V baseline in Ubuntu 26.04 LTS, with about 95% of regular Linux software package archives already available; Krste Asanović of the RISC-V International Foundation and SiFive stated that multiple suppliers will bring RVA23 server-class chips to market this year, with most matrix extension work expected to be approved within the next 12 to 18 months. The collaborative maturity of the RISC-V software and hardware ecosystems is accelerating.