Table of Contents
Ultra-dense GPU racks are pushing power, cooling, and engineering demands beyond the limits of many existing facilities, raising questions about the viability of modular designs
Within data centers, a significant shift is happening at the rack level. AI workloads are pushing rack densities to unforeseen levels and some of the core assumptions behind modular and reconfigurable data center designs are beginning to break.
Up until a few years, rack densities in data centers typically ranged between 5 and 10 kilowatts per rack. For advanced deployments, the builds went up to 20 to 30 kW. Today, it is pushing toward 600 kW, with leading AI infra providers’ roadmaps including up to 1MW density in the next 5 years. For reference, 600 kW is equivalent of the electricity required to power 500 U.S. homes.
These insane power levels are driven by three key factors: GPU-heavy deployments required for training large language models (LLMS) and multi-modal models; advanced cooling systems like liquid and immersion technologies; and high-bandwidth interconnects.
When Nvidia introduced its 600kW rack architecture, Kyber, at GTC, it sparked a debate around whether today’s data centers are at all capable of supporting such ultra-dense, vertically integrated infrastructure.
Kris MacGee, senior vice president at eXp Commercial wrote on LinkedIn. “One rack at 600 kW requires the cooling equivalent of 57 American homes running their AC simultaneously. Flat out. That’s Africa hot. And at that scale, managing the heat is going to change what a data center site needs to look like.”
Others agreed that these power densities far exceed the design limits of existing electrical infrastructure, and data centers may not be able to adapt quickly to accommodate these servers.
There are a few points behind this argument. First, 600 kW racks pack hundreds of GPUs, and as a result, are extremely power-hungry. According to Goldman Sachs, 2027 AI server rack designs will consume 50 times more power than server racks running the internet today. Most existing data centers use ~48V DC at the rack level. That works fine for most loads, except with 600 kW builds which one company estimates require currents of roughly 12,500 amps, making conventional AC architectures redundant. New substations and backup systems are required to support these new electricity distribution demands.
Secondly, the 600 kW racks are 10 to 12 times denser than today’s standard 30 to 50 kW racks, which means, they are that much more thermally demanding and mechanically intense. Deploying them requires a new class of immersion cooling systems because of their erratic thermal profile. At the same time, reinforced flooring and re-engineered cabling, plumbing, and ventilation are key to support such high densities.
Last but not the least, operators would require real estate that can support that kind of power density.
Simply put, the data center designs required to accommodate these builds are much more complex and expensive, as there is little opportunity to retrofit existing assets around it. According to estimates, Capex per MW could jump 2 to 4 times, with power consumption rising by 165% by 2030.
These requirements also run counter to the principles of modular data center designs which many operators are turning to for flexibility and scalability. Based on the idea of portability and reconfiguration, these designs introduce standardized containers or pods that can be rearranged easily and flexibly. Ultra-dense racks by contrast require purpose-built architectures optimized around specific generations of GPUs and cooling systems.
A bigger concern is technological obsolescence. If chipmakers push toward 600 kW, 1MW, and beyond, operators risk building facilities tightly optimized for one generation of accelerators that may become unusable in a short amount of time.
Against that backdrop, there is a growing separation between AI facilities and legacy data centers, with extreme technical challenges around upgrading older campuses pushing enterprises to find economically viable colocations and hosted AI infrastructure. The future of AI campuses, according to some, may still be modular, but not at the rack level, but rather at the pod level.
