Susana SchwartzMarvell talks KV Cache offload and the ‘memory wall’ gap
In today’s RCR AI TechTalk, Marvell Technology’s Ravi Mahatme talks about bridging the gap between ultra-fast GPU HBM and slow NVMe storage networks. The “missing tier” is the industry’s first optically connected disaggregated shared memory appliance that provides tens-of-terabytes of memory across multiple server racks with sub-microsecond (~200–350 ns) near-NUMA latency.
That, according to Mahatme, allows GPUs across different servers to reuse KV cache and data without falling off the “performance cliff” into slow NVMe storage.
That’s important, as KV caches are growing rapidly, with longer prompt lengths, more involved context windows, and larger numbers of concurrent users.
When it comes to AI token generation, an all-optical fabric achieves what copper cannot: near-local memory speeds, with literally tens-of-terabytes of memory accessible at hundreds-of-nanoseconds of latency across 25 or 30 meters.
Learn more about resolving the “missing tier” of memory in today’s RCR AI TechTalk.
Susana Schwartz
Technology Editor
RCRTech
AI Infrastructure Top Stories
Solving cost, power problems: Nvidia claims the Vera Rubin NVL72 rack system delivers up to a 30× higher agentic-AI throughput per MW than the preceding GB300 NVL72 system, delivering 45× lower cost per million tokens.
Rebellion NPUs go live: So. Korea’s Rebellions’ neural processing unit infrastructure is running across 4 commercial AI services operated by SK Telecom, taking ATOM-Max NPUs into consumer-facing, real-time workloads.
AI Today: What You Need to Know
Trump-Xi Jinping summit: During a high-stakes summit at the White House, President Trump announced that AI (which he now calls “Super Intelligence”) would be a “big topic of discussion,” but he affirmed his goal is to “leave it as it is.”
Bipartisan Anti-NDA bill: Rep. Greg Landsman (D-OH), Rep. Tom Barrett (R-MI), and Rep. Jamie Raskin (D-MD) proposed the No Secrets for Data Centers Act today – a response to increasing community backlash over NDAs.
Oracle’s Project Jupiter setback: Bloomberg reported that Oracle sent a “force majeure” to the developer of its New Mexico “Project Juniper” data center to shield itself from near-term costs amid pipeline delays and local resource opposition.
California regulatory push: California Governor Gavin Newsom signed a comprehensive data center law package requiring projects to prove they will not shift infrastructure costs to regular ratepayers.
Stalled capacity: Data center pauses and blocks have jumped from 6 in 2024 to 25 in 2025, with over 20 more in early 2026 alone – amounting to more than $170 billion in projects affected by local backlash or power constraints.
W. Texas as a DC hub: FiberLight, a provider of high-capacity fiber-optic networks, announced a $350 million capital investment to build approximately 1,400 route miles of new, high-capacity network infrastructure across West Texas.
RCR Events
Industry Resources
Report: NTN in motion — evolving standards, expanding services
