At Hot Chips 2026, Samsung and SK Hynix presented their HBM roadmaps. They showed how they plan to increase bandwidth across the coming generations, the challenges they expect, and how they intend to solve them.
In this article I’ll focus only on the bandwidth aspect of HBM, and give you my view on who is doing it right, who is lagging, and the headwinds and tailwinds the memory vendors face.
HBM3e to HBM4: The pins doubled
Going from HBM3e to HBM4, the width of the interface doubled from 1024-bits to 2048-bits. The speed of those data lanes also went up. So, the 8 stacks of HBM3e which yielded 8 TB/s of aggregate bandwidth in Blackwell, will now support ~22 TB/s in Rubin’s HBM4.
But this doubling of the interface width from 1024 to 2048 bits came at a cost to both the GPU/xPU as well as the HBM stack.
What it cost the GPU/xPU
In a standard HBM system, the xPU die has to also include a HBM memory controller and a full JEDEC-compatible HBM PHY for each of the HBM memory stacks in the package. So a system like Rubin or Blackwell requires eight controllers and eight PHYs.
In HBM4, doubling the interface width made each of those PHYs larger. GPUs are already at reticle limit, so the HBM4 PHYs eat further into the GPU’s precious die area.
What it cost the HBM base die (B-Die)
One of the central structures in an HBM stack is the MPGA (Micro Pillar Grid Array), which connects the DRAM stack to the base die. Every extra I/O added in HBM4 needs another driver, receiver, and path through the TSV array. Also, just like in the case of the xPU, the PHY was already the largest single block on the base die, and doubling the interface makes it larger.
More importantly, one of the primary contributors to energy per bit is the length of the channel between the xPU and the base die. For instance, when data is retrieved from the DRAM, the bits leaves the driver on the base die, runs laterally across the bump field on the base-die to reach a micro-bump, then goes down and crosses the interposer, and then runs laterally again within the xPU die into the PHY. All these segments add to power and a longer channel costs considerably more. In HBM4, the channel got longer.
So the problem going from HBM3e to HBM4 is:
Power increased
Die area on the xPU gets eaten up
The memory vendors are addressing both by:
Moving the HBM base die to an advanced logic process
Replacing the JEDEC parallel interface with a high-speed serial die-to-die link, of the kind used by NVLink or UCIe
The base die moves to advanced logic
At their Hot Chips presentation, Samsung said that MPGA power continues to rise despite improving energy efficiency. So starting from HBM4, adopting advanced logic processes in B-die design is essential to curb power growth. Moving to a smaller node also minimizes active area.
In the chart below, from Samsung’s presentation, you can see how the split between the DRAM die(C-Die) power and the base-die (B-die) power changes, by just moving the base-die to a logic node.
From a parallel bus to a D2D serial link
The JEDEC-style width doubling stops with HBM4. Samsung calls that part sHBM4, for standard HBM4.
The path forward is to replace the parallel interface with a narrower, faster serial die-to-die link. Samsung calls this custom HBM, or cHBM.
This only becomes possible because moving the HBM base die to a logic process opens up a lot more area. This allows the xPU to offload the memory controller entirely to the HBM base-die and talk to it through a pure D2D link. The interface between the base die and the DRAM stack above stays JEDEC-standard.
The result is a smaller footprint and a shorter channel, which improves energy efficiency, and reclaimed silicon on the xPU that can go to compute.
Who is building it
This recipe of replacing the JEDEC parallel interface with D2D has already been in the works for a while. Marvell announced a custom HBM4 solution for its ASIC customers, designing the base die themselves and having it manufactured in partnership with Samsung, SK Hynix and Micron. Samsung had discussed the same approach well before its detailed Hot Chips presentation.
And one day after Hot Chips, NVIDIA announced NVHBM, with AWS’s Annapurna Labs as its first partner. Seoul Economic Daily has since reported that the NVHBM proof-of-concept was done in partnership with Samsung. Though NVIDIA did say it will be offered through multiple memory suppliers. Samsung looks to be first through the door.
What NVHBM claims
Compared to standard HBM4e, NVIDIA reports that NVHBM:
Frees up to 25% of accelerator die area, because the memory controller moves to the base die and the JEDEC PHY is replaced by a smaller D2D block
Cuts HBM power by 15%, because the channel is shorter
Improves bandwidth by 30%, by running the link faster than HBM4e’s 16 Gbps, the speed Samsung demonstrated at GTC in March
Power density, thermal issues, and the heat path block
Shrinking the PHY and raising the link speed concentrates power into a smaller area, and power density on the base die climbs. Samsung’s figures show standard HBM4e at 14 Gbps has a power density of 0.5 W/mm², and standard HBM5 above 28 Gbps exceeding 2 W/mm².
This increased power density of the base-die will affect the stack of temperature sensitive DRAM. So both Samsung and SK Hynix have chosen to solve this the same way. A dedicated conductive structure above the D2D PHY that gives the hotspot a vertical path out to the cold plate. Samsung calls it the Heat Path Block. SK Hynix calls the technology iHBM, with the embedded element labelled ICE.
Samsung says this approach is proven on their cHBM4 designs, reducing peak temperature by more than 35% with >50% PHY coverage.
Beyond HBM5 — zHBM
This is the part I’m most excited about. Here’s a statement from Samsung’s roadmap:
The ultimate solution (zHBM): true 3D vertical integration of the xPU and C-die stack, eliminating the 2.5D interposer.
In zHBM, the DRAM stack is placed directly on top of the GPU, instead of next to it. Here the xPU and HBM stack are coupled through hybrid bonding, with a thin interlayer die between them taking over some of the duties of the base die. There are three obvious advantages to this approach.
The interface is no longer limited by beachfront. Placing memory on top opens the entire face of the xPU to the DRAM banks. So, instead of sipping data through a single bus, all the compute units can be more closely coupled with DRAM banks.
Data travels vertically much shorter distances, so pJ/bit goes down. For an indication of the magnitude, d-Matrix measured 0.37 pJ/bit on their 3D stacked DRAM against 2 to 3 pJ/bit for HBM4.
The shoreline gets freed for something else. Once memory is no longer consuming the die edge, that perimeter can go to scale-up links, scale-out networking, or off-package I/O.
Samsung compared one GPU with four HBM4e stacks against one GPU with four zHBM stacks. On a 1200W budget, they show DRAM bandwidth rising about 230%, from roughly 13.6 to 32 TB/s, while DRAM power falls by about 100W.
Problems that need to be solved with zHBM
The first is methodology. zHBM requires a unified DRAM and SoC design and sign-off flow, with shared timing and function models, and joint signal integrity, power integrity and physical verification closure. Before any of this reaches production, this integrated design and sign-off flow spanning the xPU and the DRAM has to be fleshed out.
Then there are the thermals. Since zHBM puts DRAM directly above the hot compute die, the obvious question is how many layers of DRAM you can stack before the stack cooks. We’ll have to wait for answers on this one.
Who Is Positioned to Win It
We are at the beginning of a new era in AI inference chip architectures, and the next wave of memory demand is heading towards custom HBM and zHBM-style memory systems.