HBM is turning memory into AI’s costliest bottleneck

Home Semiconductor News HBM is turning memory into AI’s costliest bottleneck
Samsung HBM4E

Packaging limits and inference demand are pushing HBM and DRAM prices higher

In sum – what to know:

  • Prices keep climbing – A 16-gigabit DDR5 chip went from about $8.70 to $48 in eleven months, and AMD-launch coverage puts DRAM at about 75% of server cost.
  • Supply stays tight – Samsung, SK hynix and Micron describe 2027 HBM supply as sold out or largely contracted, and packaging is now the tighter limit.
  • Inference needs capacity – Growing KV caches shift the constraint from bandwidth to capacity, pushing flash, pooling and software alternatives to HBM.

A single 16-gigabit DDR5 memory chip cost about $8.70 in October 2025. By September 2026, the same class of component cost roughly $48, according to Team Group. That’s an increase of roughly 450% in eleven months. DRAM, the fast working memory inside everything from laptops to AI servers, has spent decades getting cheaper per bit. That relationship has broken, and it has broken at the exact moment the AI industry decided it needed vastly more of it.

Logic compute has scaled much faster than the memory attached to it. Modern accelerators can perform staggering numbers of calculations, but those calculations still depend on data arriving on time and having somewhere to sit. The result is a bottleneck that runs through high-bandwidth memory, or HBM, which stacks DRAM close to a processor; advanced packaging, which physically connects those pieces; conventional DRAM and NAND flash storage pricing; and the growing amount of memory needed to hold an AI model’s working context.

Memory helps determine the cost of a rack and the tokens per watt that rack can deliver, meaning how much AI output it produces for a given amount of electricity. According to coverage of AMD’s November MEXT launch, DRAM alone now accounts for about 75% of server cost. The direction’s hard to miss.

The bill is landing well outside data centers, of course. Apple’s manufacturing purchase obligations rose by roughly $33 billion in one quarter to $89.7 billion, though Apple didn’t say how much of that was memory. It did name memory as the main driver of its expected margin decline, and CEO John Ternus called memory pricing “unprecedented.” The shortage is now a consumer technology story too.

How long it lasts is where agreement ends. Acer CEO Jason Chen expects mainstream component pricing to reverse in the second half of 2027, arguing that only premium parts will stay short and that suppliers have an incentive to talk up the shortage. Team Group’s CEO sees no balance until the second half of 2028 or later, warning that “even money can’t buy goods,” while its chairman expects budget pricing to hold out until 2029. Both are interested parties. So are the memory makers expecting demand to exceed supply for years.

HBM4 turns memory into custom logic

HBM4, the fourth generation of high-bandwidth memory, is in volume production at Samsung, SK hynix and Micron. Samsung says it shipped the first commercial version in February 2026, sustaining 11.7Gbps per pin and reaching as much as 13Gbps with a 4nm logic base die. Micron says its 36GB, 12-high HBM4 is in high-volume production at more than 2.8TB/s per stack. SK hynix began shipping HBM4 “in earnest” in the third quarter and expects it to pass HBM3E in shipment volume and revenue in the fourth.

Shipping first and owning the market are different things. Counterpoint estimates put SK hynix at about half of second-quarter 2026 HBM revenue, Samsung at 33% and Micron at 18%, and SK hynix reportedly holds roughly two-thirds of NVIDIA’s Rubin HBM4 allocation, the one that matters most, though that estimate is unconfirmed.

The technical change underneath those numbers is significant. JEDEC’s HBM4 specification doubles the interface from 1,024 to 2,048 bits and doubles the independent channels per stack from 16 to 32, each divided into two pseudo-channels. More channels mean more simultaneous paths into the same memory. The base die, which controls how the DRAM stack communicates with the processor, is moving onto foundry logic processes instead of staying entirely on memory processes. HBM is becoming partly a custom logic product, and that changes who designs it, who manufactures it and who gets paid.

NVIDIA’s NVHBM takes that further by moving the memory controller from the accelerator into the HBM base die, which the company says brings up to 30% more bandwidth and up to 15% lower HBM power than standard HBM4E. It has reportedly asked all three suppliers to build those base dies on TSMC’s 3nm process, though none of the companies has confirmed it.

Capacity per package is climbing too. NVIDIA’s Rubin GPU carries 288GB of HBM4 and, after the revision described below, 19.2TB/s of bandwidth, while AMD’s Instinct MI455X carries 432GB across 12 stacks. Rubin Ultra was announced with 1TB of HBM4E per package, though NVIDIA is reportedly testing versions with as little as 192GB. At 1TB, feeding the memory becomes a power and cooling problem as well as a procurement one.

NVIDIA also changed a specification without announcing it. Archived versions of the Vera Rubin NVL72 page show per-GPU HBM4 bandwidth falling from 22 to 19.2TB/s between September 1 and September 11, while compute figures stayed put.

Industry reporting linked the change to suppliers struggling to hit the original HBM4 targets, and NVIDIA’s own pages remain inconsistent, so the advertised numbers deserve some caution.

Buyers can’t simply respond by ordering more. Samsung says 2027 HBM supply “has been negotiated at target-level pricing and is already sold out,” SK hynix says its DRAM and HBM orders are effectively booked through 2029, and Micron says its 2026 HBM was sold out and it has agreements for the vast majority of its 2027 HBM bit supply.

HBM is also expensive in wafers. Micron says HBM3E consumes roughly three times the wafer area of DDR5 for the same capacity, a gap it expects to widen, and TrendForce expects HBM to take about 30% of total DRAM wafer input among the top three suppliers by the end of 2027 while producing only about 13% of DRAM bits. Every additional AI memory stack removes a disproportionate amount of conventional supply.

The immediate limit has also moved downstream to packaging, the process of stacking, testing and connecting memory to the rest of the chip. According to ETNews, SK hynix is targeting full-scale operation at its P&T6 packaging site in the first half of 2027, potentially adding more than 20,000 wafers per month of HBM output. SK hynix hasn’t confirmed the figures.

New wafer capacity achieves little if packaging lines don’t scale with it.

Inference shifts the constraint from bandwidth to capacity

AI inference has two stages. Prefill reads the prompt and is generally limited by compute. Decode generates the response one token at a time, and each new token requires reading the model’s weights and its key-value, or KV, cache. That cache stores intermediate information about the conversation so the model doesn’t have to recompute everything for each word.

Decode leaves arithmetic units well below their peak. Adding more floating-point operations per second does relatively little if the processor is waiting on memory.

The KV cache grows linearly with context length and the number of simultaneous users. Agentic workloads make that worse because they reuse context across turns, tools and sessions instead of throwing it away. A 128K context on a 70-billion-parameter model needs roughly 42GB of KV cache for a single request at FP16 with grouped-query attention. Open Compute Project and IBM figures for a 128K-token FP16 serving case reach about 825GB, more than an entire GPU’s HBM. Either way, capacity stops being a secondary concern.

NVIDIA divides the memory hierarchy into G1 GPU HBM, G2 host DRAM, G3 node-local SSDs and G4 shared storage. KV cache can outgrow the first three yet is too latency-sensitive for conventional shared storage, so its BlueField-4 STX and CMX platform adds a G3.5 tier of Ethernet-attached flash inside Vera Rubin pods.

NVIDIA claims up to five times the tokens per second and five times the power efficiency of general-purpose storage.

A test from Supermicro, IBM and NVIDIA produced a more concrete result. With persistent KV cache, the companies reported up to 56 times faster time-to-first-token on 130,000-token prompts using a 120-billion-parameter model. Reusing stored context clearly beats recalculating it, and it creates a large new market for storage sitting directly beside the GPU.

Other companies are questioning whether HBM needs to be involved at all. Intel’s Crescent Island accelerator uses LPDDR5X, the lower-power memory familiar from laptops, in capacities up to 480GB on partner-built cards (160GB on Intel’s own), trading peak bandwidth for cheaper inference capacity. SK hynix and SanDisk’s High Bandwidth Flash, or HBF, stacks NAND to offer up to 512GB at roughly 0.4 to 3TB/s, with only the top grade near HBM-class speeds, although NAND endurance, meaning how many times it can be written before wearing out, makes a frequently updated KV cache a difficult workload.

AMD’s MEXT is a software answer. It keeps cold memory pages on flash and pre-fetches the hot ones into DRAM, and AMD claims 20-75% lower DRAM requirements at similar performance, citing Azure and Meta examples of overprovisioned and stranded memory and estimating that 50-80% of installed DRAM is underutilized because pages go cold.

Google combines several of these ideas, doubling on-chip SRAM in its TPU 8i Max to hold active KV data and moving caches and model weights over the network for disaggregated serving. All of these routes treat memory as an architecture problem rather than a single component specification. If any of the non-HBM ones works at scale, HBM’s pricing power looks less durable than the current margins imply.

The price shock reaches consumers, finances, and the power budget

The DDR5 chip’s move from $8.70 to $48 is only the cleanest illustration. Nikkei reports that DRAM prices have roughly doubled since the end of 2025, and TrendForce’s September 30 forecast had conventional DRAM contract prices rising another 10-15% in the fourth quarter, and its November 11 update puts PC DRAM at 13-18%.

One inversion shows how unusual the market has become. TrendForce found that per-wafer revenue from 64GB DDR5 server modules overtook HBM in the first quarter of 2026. Conventional DRAM had become the better earner, reducing the incentive to divert its limited supply into HBM despite AI demand.

JPMorgan’s September forecast nevertheless has blended HBM selling prices rising 54% in 2027 and another 25% in 2028. Contracts now increasingly include price ranges that move with the market.

Buyers want guaranteed supply badly. Micron has signed roughly two dozen strategic customer agreements with take-or-pay terms, and its customer contract liabilities jumped from $568 million to $12.9 billion in one quarter, which means customers are effectively prepaying for allocation.

China’s growing output doesn’t offer an immediate escape. CXMT is the fourth-largest DRAM maker with about 9% of third-quarter DRAM revenue, according to Counterpoint, but it concentrates on conventional DRAM. It began HBM3E risk production in August and is targeting mass production by the end of 2026. Alibaba, Tencent and ByteDance are reportedly seeking long-term Korean supply agreements because domestic production can’t cover their needs.

The financial results explain why suppliers are in no hurry to restore the old market. SK hynix reported an 80% operating margin in the third quarter, and Micron’s fiscal fourth-quarter revenue rose 379% year over year to $54.23 billion, flattered by a 14-week quarter against 13 a year earlier.

These are extraordinary numbers for an industry accustomed to brutal boom-and-bust cycles. Margins this high fund the expansion, but they also invite scrutiny and eventually encourage customers to find substitutes.

Some customers are already buying less. Notebook shipments fell 15.8% quarter over quarter in the third quarter of 2026, and PC makers are specifying less memory and smaller SSDs for 2027 systems. Budget laptops are disappearing from some markets.

Relief through new fabs takes years. TrendForce puts DRAM supply 1-2% below demand in 2026, with the gap widening in 2027. Micron’s first Idaho fab produces initial wafers in mid-2027, its second in late 2028 and its Clay, New York site in 2030, by its own estimate, and SK hynix expects demand to exceed supply through 2030.

There are also reasons to question the demand behind those sold-out order books. Some operators report high hardware idle rates while buying ahead of need, and analyst estimates of $1.2-1.3 trillion in 2027 hyperscaler capital spending aren’t company guidance. Data Center Watch counted at least 75 US data-center projects, worth about $130 billion, blocked or delayed by local opposition in the first quarter of 2026 alone.

Then there’s NVIDIA. Jensen Huang resigned on November 12, and is now lead by interim CEO and CFO Colette Kress. Suppliers say signed agreements protect near-term orders, but NVIDIA is the largest HBM customer and its leadership change could affect 2027 order books and custom base-die specifications.

A US judge’s November 14 refusal to dismiss a DRAM price-fixing class action against Samsung, SK hynix and Micron isn’t a finding of liability, but discovery could still expose internal documents about capacity, allocation and pricing.

The power connection closes the loop. Nearly every new memory product is sold on tokens per watt, with NVHBM claiming up to 15% lower HBM power. These are vendor figures under different test conditions, not a common benchmark, but they all point the same way.

Micron frames the goal as delivering more useful tokens from every megawatt. That’s the right measure, and it leaves a genuinely open question. Will the industry spend its way into enough memory, or will better tiers, pooling and software make today’s scarcity look like an argument for using less? Memory is how a megawatt becomes tokens. The power wall and the memory bottleneck are two views of the same constraint.

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More