In today’s RCR AI TechTalk, Marvell Technology’s Ravi Mahatme talks about bridging the gap between ultra-fast GPU HBM and slow NVMe storage networks.
In sum, what to know:
Most GPUs are stalled for more than 50% of their cycle time because they were waiting for data
HBM is too expensive and small to hold massive KV caches, and offloading them creates massive bandwidth bottlenecks
Marvell aims to create an entirely new, ultra-fast shared optical memory tier that solves the KV cache memory wall
Ravi Mahatme says there is a critical “memory wall” gap in modern AI clusters, which lack a large, low-latency shared memory tier between fast, local accelerator memory and slow persistent storage. As Marvell Technology’s senior director of product management in the Photonic Fabric business, Mahatme’s team is offering the industry’s first optically connected disaggregated shared memory appliance, which is designed to solve the “missing tier” of memory in large-scale AI computing:
Tier 1: GPU-Attached HBM, whichsits directly on or next to the accelerator silicon, which delivers a few hundred gigabytes of local HBM. That means it is physically limited in capacity (roughly 80 GB to a few hundred gigabytes per GPU), according to Mahatme, and exceptionally expensive. It leaves little room left for surging context requirements, such as large Key-Value caches in LLMs.
Tier 2: Host CPU-Attached DRAM offers higher capacity, which Mahatme says can be hundreds of gigabytes to a few terabytes – at a lower per-GB cost – but again, “very local to the server and shared with other software running on server as well.” Because it is bound locally to the host server, it must run the operating system, orchestrate host workloads, and serve AI frameworks, and, crucially, an accelerator in Node B cannot natively or efficiently read Node A’s CPU DRAM without substantial software overhead and network stack penalties. “After that, there is really no good tier of memory available, so you go to storage. Storage is great for persistent data, but it’s slow. Even if it’s in servers to NVLink flash-based storage, the access latencies can be order of hundreds of microseconds if not milliseconds.
Tier 3: Network-attached Storage (NVMe Flash SSDs / Storage Networks), which provide massive capacity of hundreds of terabytes to petabytes, as well as persistent data retention, but at the expense of speed. Here, latency is roughly three orders of magnitude too slow. Moving from nanoseconds (DRAM/HBM) to hundreds of microseconds or milliseconds (NVMe / distributed file systems) causes GPUs to idle, and wastes large amounts of accelerator compute during real-time inference.
The “missing” tier, what Mahatme would call “Tier 2 1/2”, is an optically connected, shared memory pool that provides tens-of-terabytes of memory across multiple server racks with sub-microsecond (~200–350 ns) near-NUMA latency, allowing GPUs across different servers to reuse KV cache and data without falling off the “performance cliff” into slow NVMe storage. This tier 4 is designed to more effectively use and store memory as KV caches grow – and they grow rapidly, with longer prompt lengths, more involved context windows, and larger numbers of concurrent users. As caches exceed local HBM/DRAM, systems, it becomes necessary to evict the cache to storage or recompute the prompt, which leads to huge latency penalties.
According to Mahatme, the Marvell Photonic Fabric Memory Appliance provides this “tier 4” tier: “We want memory that is actually ‘memory’ and that reduces the time it takes for a GPU to request and receive data from microseconds and milliseconds down to hundreds-of-nanoseconds.” Mahatme explains that this memory is shared across up to 16 servers, with a reach of about 50 meters,” which is far greater than the meter to a meter-and-a-half a signal can travel through copper. “For a memory application, we think about 25 or 30 meters is more than adequate.” When it comes to AI token generation, he says the all-optical fabric achieves near-local memory speeds (near-NUMA latency). “What we have here is literally tens-of-terabytes of memory accessible at hundreds-of-nanoseconds of latency that’s shared across 25 or 30 meters.”
“The shared-memory pool means every processor and server has access to the entire 16 or 32 terabytes of memory in a flat-memory space. So any GPU or XPU can read and write from the same memory space which allows them to share data and be more performant at AI processing.”
Ravi Mahatme, senior director of product management, Marvell Technology Photonic Fabric Group
Shared memory tier: un-stall GPUs
Mahatme considers the shared-optical memory a game changer in what it can enable for AI clusters. When working for a major hyperscaler, he says he often saw in AI inferencing workloads that “most GPUs were stalled for more than 50% of their cycle time because they were waiting for data.” Because GPUs are the most important component, and extremely expensive, “you want them working as much as possible, and as efficiently as possible,” says Mahatme, noting that total-cost-of-ownership should be thought of at a system level in terms of GPUs.
With the photonic memory appliance, he believes it’s possible to achieve very low latency so that data is shared and accessible across up to 16 servers. “Rather than be stalled, your GPUs can go access the required KV cache or the model weights that can be stored on this memory appliance and continue to execute on the workload.”
From simple inference to multi-turn inferencing, agentic AI
For multi-turn inferencing and agentic AI, a model has to go through repeated loops of generating reasoning thoughts and tokens, which Mahatme says is “very slow.” He explains, “People either pin the KV cache into one single server so the next iteration also has to land on that simple server, or they push it to storage and fetch it back.”
In the shared-memory tier, he says you can instead foster “scheduling flexibility” so that if GPU 1, for example, does the first iteration, and GPU 3 does the next, and GPU the next, “you get more juice from them so that overall cluster-level utilization goes up, and you get a lot more tokens and throughput at the cluster level from the same compute infrastructure.”
Break free of traditional electrical wiring
With light moving enormous amounts of data between processors and pooled memory with near-HBM speeds, minimal delay, and very low power consumption, it’s possible for AI clusters to drastically scale up. When processors aren’t sitting idle waiting for data, data centers can break free from the limitations of traditional electrical wiring.
“Photonic fabric technology is a core innovation for low-latency, low-power optical interconnect, the key feature of which is the high bandwidth we can push through our Gen-1 appliance – with each memory module pushing 7.2 terabits of data every single second. That’s almost like HBM3E,” he says, explaining it’s almost as fast as on-chip or directly attached high-end memory. “The shared-memory pool means every processor and server has access to the entire 16 or 32 terabytes of memory in a flat-memory space. So any GPU or XPU can read and write from the same memory space which allows them to share data and be more performant at AI processing.”
By optimizing the physical connection (such as placing optical I/O right in the middle of the silicon die rather than the edges, and eliminating power-hungry components like digital signal processors), the technology achieves nanosecond-level response times while consuming very little energy (about 2.4 picoseconds per bit). “So we can connect a very high bandwidth, low latency optical interconnect between the shared memory pool, processors, and servers. That is what gives us the differentiating capability to move data very, very fast and with very low power.”
In part-2, Mahatme addresses the issue of reliability in high-heat compute envelopes and open standards around optical to help photonic fabric map onto open scale-up protocols.
