Table of Contents
In a discussion about inferencing it became clear that ‘getting AI out to end users’ at SMBs and enterprises requires rapid distribution of high-end CPUs or low-end GPUs at massive scale, to edge locations, and at low latency.
In the world of AI infrastructure, there’s a trend toward pre-validated “turnkey” IaaS solutions that span bare metal servers to fully integrated hybrid cloud and edge computing platforms. The key players include the world’s largest cloud providers and hyperscalers, but Hivelocity distinguishes itself by combining the raw performance of bare metal with the flexibility and automation of the cloud, connecting more than 50 data centers with a carrier-neutral approach that fosters flexibility and competitive pricing for SMBs and enterprises of all sizes.
According to CEO Jeremy Pease, companies are in different places in the AI adoption curve, ranging from initial experimentation to full-scale production integration. “There are so many ways to deploy AI infrastructure, so the choice to either build on-prem or to use managed services is based on many factors, like the size of the company and the expertise of its IT team and development staff.”
Oftentimes, resource-poor companies need fully managed, cloud-based platforms, while larger companies with robust IT teams and development staffs prefer the control and customization of hybrid or on-prem infrastructure. “Bare metal servers can be automatically deployed and spun up through the web, all the way to cloud infrastructure in which we monitor, patch, and manage all the pieces of the infrastructure for the customer.” But hardware isn’t the only thing companies are looking for as connectivity is also essential. “We have 60 data centers across 6 continents, but without the backbone of internet providers and internet connectivity, we’d might as well be BestBuy or CompUSA,” said Pease, referring to the need for carrier-neutral data centers. With access to a wide variety of carriers and ISPs, including NTT Communications, Comcast, Telefónica, Level 3, Cogent, GTT, as well as peering and backbone providers like PacketFabric and Digital Reality, Hivelocity is trying to set itself apart.
“It’s the networking that [facilitates] the automation, with organizations connecting their servers across data centers, pulling everything together into an infrastructure that supports their applications,” said Pease.
In addition to that connectivity, power is another differentiating factor, and that’s where Hivelocity’s bare metal servers come in — offering an alternative to the shared, virtualized environments typically found in hyperscaler environments. “Our bare metal platform is where developers can just get in there, spin up what they need, put containers on it, and back it up with object-based storage. They want the [Digital Realty ServiceFabric] for their production environment, and they want all sorts of ranges of compute and storage across different components,” explained Pease, referring to the Digital Reality connectivity platform that connects on-prem systems to cloud services through Hivelocity’s global network of data centers.
“Combining high-performance physical servers with DR’s network helps our customers customize and maximize what they’re running their application on.” That’s a big change from 25 years ago, when businesses would deploy infrastructure first, and then figure out what applications were going to work on top. “Now, data drives everything. Companies know exactly what type of infrastructure they need. Is it a CPU? Is it a GPU? Do I need this much RAM? Do I need this much storage? Our customers dictate all of that.” According to Pease, bare metal is crucial because it gives companies direct hardware access, enabling them to install their preferred operating system and configure CPU, memory, storage and other resources to what they need.
By allowing enterprises to deploy bare metal infrastructure directly into their own facilities, they can eliminate the “noisy neighbor” hassles of shared virtual machines, which Pease says translates into speed and low latency — particularly important for inferencing workloads that require real-time processing and the responsiveness (as with autonomous vehicles, virtual assistants, and chatbots). With increasing demands for speed and low latency, deployments will be happening closer to the end-users, at the edge. “Inferencing AI is where our model makes the most sense. That’s where users utilize AI in a meaningful way — with real impact at the end of the day. That’s the point where the model has completed its training and its learning, and it’s become actually usable,” added Pease.
That inferencing sweet spot requires accessibility by anyone, anywhere. “We’re talking distributing high-end CPU or low-end GPUs at massive scale, to the edge locations, at low latency. That’s a big thing. That’s why our internet backbone and our geography dispersion is so critical. That’s how these companies are going to get that AI out to their end users.”
Pease sees two different parts to capitalizing on inferencing workloads:
- “What are the products and technologies we need to offer to enable the next step of AI, and where is the compute and an infrastructure on that inference side going to go?
- If it starts to scale up rapidly, how do we make sure that we’re able to accommodate and have the capacity and capabilities are customers need?”
Unlike training models that require “one really massive location,” inference requires highly distributed networks that reach end users in an affordable way, with less compute needed than with training models, but with a greater need for accessibility around the globe. “Network connectivity is so crucial, we are always looking for the best way to accomplish it, whether using someone else’s facility or growing and scaling our own,” said Pease, noting the smaller footprint of storage is a game changer for the company and its customers. “Just 5 to 10 years ago, a facility of 30 to 40 megawatts, maybe 50 megawatts to 100 megawatts, was considered a ‘massive facility’ because everything was in a 5-kilowatt rack. Now, we’ve got 100 to 200 kilowatts per rack.”
As businesses develop their AI infrastructure structure, that capacity can fuel what companies want with their applications. “Self-serve, one-click applications with open-source, web-based interfaces [like Open WebUI or LLMs like Llama 3 8B] are in demand. Companies want them optimized for dialogue and general AI tasks.” That’s why Hivelocity is now focusing more on data analytics.”
Those applications and the data analytics around how those applications are being engaged and consumed are going to be what justifies the incredible capital expense, products, and facilities that are going toward AI infrastructure. “It won’t mean much without a serious look at the data and how it is analyzed to make the most of those investments.” For that reason, Hivelocity is investing in data warehousing and business intelligence. “We want to stay ahead of what’s coming next, rather than waiting until they’ve already happened. That’s the two ways I think about things — end products to customers, and analytics on the back end to make sure we’re ready for what’s coming next,” said Pease.
