A large AI job isn’t just an application running over the network. Thousands of GPUs could be working together on the same job, repeatedly exchanging data and synchronizing before they can continue. That makes communication performance part of compute performance. In other words, a slow GPU is a problem, but so is a GPU waiting for data from another GPU.
As these systems have grown, the industry has started describing GPU connectivity in three broad domains known as scale-up, scale-out, and scale-across.
I touched on two of these topics in another recent blog post, but in this article we’ll go deeper into each of these areas.
- Scale-up is how GPUs within a tightly coupled system communicate, like within a single rack.
- Scale-out is how we connect many of those systems into a much larger GPU cluster.
- Scale-across is what happens when that cluster can no longer fit within one physical data center.
The boundaries aren’t always perfectly clean, but this model is a useful way for us to understand the different problems we’re trying to solve.
Scale-Up Makes Many GPUs Behave Like One System
Scale-up networking is the innermost and highest-performance communication domain.
GPU systems often use specialized, high-bandwidth interconnects such as NVIDIA NVLink rather than conventional Ethernet or InfiniBand networking. Think of this as something similar to the backplane of a large chassis switch, where line cards communicate over a dedicated, extremely high-performance internal fabric. However, emerging technologies like UALink are also bringing Ethernet into the scale-up domain.
Scale-up connects accelerators so tightly that they can operate more like components of one large computing system than independent hosts attached to a network. A traditional data center network would normally be designed around servers communicating with other servers, but in the scale-up domain, we’re effectively moving the network deeper into the computing system itself.
In rack-scale architectures, the scale-up fabric extends beyond individual servers through specialized switching infrastructure integrated into the rack. This allows GPUs across multiple compute trays to participate in the same high-bandwidth communication domain.
This is very important for parallelism techniques where GPUs need to exchange large amounts of data frequently. For example, tensor parallelism can divide computation for individual model layers across multiple GPUs. Those GPUs would typically need to communicate during each layer of computation, making communication latency especially important.
NVIDIA’s latest implementation is the sixth-generation NVLink (right), used in its Vera Rubin NVL72 rack-scale architecture. NVLink 6 provides 3.6 TB/s of bidirectional bandwidth per GPU and 260 TB/s of aggregate bandwidth across 72 GPUs. The architecture also incorporates NVLink switches that provide all-to-all GPU connectivity and 130 TFLOPS of in-network compute to accelerate collective operations such as AllReduce (source).
Other accelerator ecosystems have their own approaches, and technologies such as UALink and Ethernet-based scale-up are developing as alternatives. But the important point isn’t the vendor – it’s what the network is trying to accomplish.
Notice in the image on the right how the GPU compute trays connect through the rack’s integrated NVLink switching infrastructure. The obvious limitation here is that we can only make the scale-up domain so large. Eventually we have to connect multiple systems together, which is where scale-out comes in.
Scale-Out is for Building the GPU Cluster
Scale-out is probably the most familiar domain for a traditional data center network engineer because it more closely resembles a typical data center spine-leaf architecture. Instead of making the individual rack system larger, we add more systems and connect them with a high-performance network.
Scale-up and scale-out aren’t mutually exclusive. A distributed training job might use NVLink for communication between GPUs within a rack while simultaneously using Ethernet or InfiniBand to exchange data with GPUs in other racks.
This is the backend compute fabric, and Ethernet and InfiniBand are the two major technologies used here today. Clos and fat-tree architectures are common because they provide large amounts of east-west bandwidth and multiple paths between endpoints. NVIDIA reference architectures, for example, use leaf-spine fabrics and rail-optimized connectivity between GPUs.
Switch buffer architecture is also important, especially because synchronized GPU communication can generate significant traffic bursts. The goal isn’t simply to minimize buffering, but to prevent congestion and packet loss from reducing effective GPU performance.
But calling this “just a leaf-spine network” misses what makes AI networking interesting. AI workloads can create highly synchronized communication patterns. Collective operations such as AllReduce, AllGather, ReduceScatter, and All-to-All can cause many GPUs to transmit large amounts of data at roughly the same time which creates conditions very different from what we’re used to in general-purpose data center networks.
For example, during distributed training, an AllReduce operation combines values from participating GPUs and makes the result available to each participant. If 4,000 GPUs are participating, the collective operation can be delayed by the slowest participant. That means congestion affecting even a small portion of the cluster can slow down the entire job which is why tail latency, congestion, packet loss, and jitter matter so much in AI fabrics.
Ethernet-based GPU fabrics therefore use mechanisms like RDMA (RoCEv2), congestion control, adaptive routing, sophisticated load balancing, and various network telemetry to maintain high effective bandwidth. InfiniBand does the same with a networking stack developed specifically for high-performance computing environments.
The scale-out fabric also needs huge aggregate capacity. As we grow GPU clusters from thousands to tens of thousands of GPUs, each with hundreds of gigabits or potentially terabits of network connectivity, the bandwidth inside the fabric can grow to incredibly large as well.
This is one reason AI clusters use flatter, high-radix topologies and multiple network planes. Adding switching tiers increases scale, but it also adds latency, optics, power consumption, cost, and additional places where congestion hotspots can happen. Therefore scale-out becomes an optimization problem of connecting as many GPU systems as possible while preserving the communication characteristics the distributed workload expects.
For a long time, scale-out defined the outer edge of the AI compute fabric, but then AI clusters got bigger.
Scale-Across For When One Data Center Isn’t Enough
Scale-across is the newest of these three terms. Simply put, scale-across connects GPU clusters across physically separate data centers, allowing those resources to participate in larger distributed AI workloads.
This isn’t just a traditional data center interconnect with a new name. Traditional DCI would normally carry application traffic, replication, backups, storage traffic, disaster recovery, or connectivity between independent application environments. Scale-across is trying to solve a difficult problem – allowing GPU resources in physically separate facilities to participate in a larger distributed AI system. A main reason we would we want to do this is power.
At some point, building a larger GPU cluster stops being primarily a switching problem and becomes a power, cooling, real estate, and infrastructure problem. If enough capacity can’t be delivered to one building (or even one campus), the compute has to be spread across multiple locations. And of course that means the AI network has to be there to connect it.
Broadcom describes scale-across as connecting distributed scale-out clusters over metro and regional distances, while NVIDIA describes its Spectrum-XGS implementation as extending AI fabrics between facilities separated by anything from campus distances to tens or hundreds of miles.
In a scale-across architecture, physics naturally becomes a much bigger problem. Light in fiber travels approximately five microseconds per kilometer. Therefore, a ten-kilometer link introduces roughly 50 microseconds of propagation delay in one direction before accounting for switching, serialization, queueing, or anything else in the path, and there’s no amount of networking magic that can eliminate that. That becomes especially important for congestion-control mechanisms that rely on feedback from the receiving endpoint because as distance increases, the time required to detect and respond to congestion also increases.
So scale-across isn’t about making two geographically separated GPU clusters behave exactly like GPUs connected through a local scale-up fabric. Instead, the architecture and workload need to tolerate and manage a much larger latency domain.
Not every GPU communication pattern is appropriate for scale-across. Latency-sensitive operations, like those associated with tensor parallelism, are usually kept within a local scale-up or scale-out domain. Other distributed computing operations can tolerate greater distances, depending on how frequently GPUs need to exchange data and synchronize. Engineers truly need to understand the workloads and communication patterns running on the network.
NVIDIA’s Spectrum-XGS, for example, introduces distance-aware congestion control and adaptive routing for this kind of environment. Cisco and Broadcom are also developing architectures specifically around scale-across networking, which is a good indication that this is becoming a broader architectural category rather than just an NVIDIA product term.
For network engineers, scale-across sits somewhere between the traditional worlds of high-performance data center networking and large-scale transport networking. From a hardware perspective, scale-across architectures benefit from deeper switch buffers to absorb traffic bursts and accommodate congestion across longer-distance links. However, buffering needs to be balanced against queueing latency, and effective congestion control is always still extremely important in these environments.
Three Networks Solve Three Different Problems
What’s interesting to me is how AI infrastructure is changing the relationship between networking and computing. Scale-up networking effectively becomes part of the computing system itself, scale-out turns those systems into enormous GPU clusters, and scale-across allows those clusters to extend beyond a single physical location.
For network engineers, these aren’t new concepts. We’re still dealing with bandwidth, latency, congestion, routing, and physical infrastructure. But we’re now designing networks where communication performance can directly determine how efficiently thousands of GPUs perform a single computing task.
And as GPU clusters continue to grow, I think understanding how the network affects distributed computing is going to become a more important part of our jobs.
Thanks,
Phil
Leave a Reply