Computer Networking › Section F · The link layer and LANs › Topic F.6
Data center networking
Search, streaming and cloud services run on data centres: buildings holding tens of thousands of servers or more. Their internal network must give every server fast, reliable paths to every other, and ordinary campus designs cannot.
A building full of computers
A large data centre houses tens of thousands of servers or more, and a single user request, such as a search, may be answered by hundreds of them working together. So traffic between servers inside the building, called east-west traffic, is often far larger than traffic entering or leaving, called north-south. The network therefore has to provide high bandwidth between any pair of servers, low and predictable delay, and resilience: with this many switches and cables, something is always failing, and a failure must not take a service down.
Racks, tiers and load balancers
Servers sit in racks of a few dozen machines, each wired to a top-of-rack (TOR) switch. TOR switches connect upwards to higher-tier switches, and at the top, border routers link the data centre to the internet. External clients do not contact individual servers. They send requests to a public address owned by a load balancer, which picks a server, rewrites the destination to that server's internal address and relays the response back, rather like network address translation. This spreads work, hides the internal layout and lets servers be added or removed. The web traffic such a balancer handles is the subject of The web and HTTP.
Leaf-spine and Clos networks
A simple tree of switches leaves its upper links shared by everyone beneath them. A leaf-spine design flattens it: every leaf (TOR) switch connects to every spine switch. Any two servers on different leaves are three switch hops apart, via leaf, spine, leaf, and have as many equal-length paths as there are spines. Capacity grows by adding spines, and a failed spine only removes one path. The pattern comes from the Clos network, a scheme for building large non-blocking switches from small ones, first described in the 1950s for telephone exchanges.
Fat-tree sizing
A well-known multi-tier Clos arrangement is the fat tree, which can be built entirely from identical \(k\)-port switches. It has \(k\) pods; each pod holds \(k/2\) edge and \(k/2\) aggregation switches, and \((k/2)^2\) core switches join the pods. Each edge switch serves \(k/2\) servers, so the whole fabric supports \(k^3/4\) servers using \(5k^2/4\) switches. For \(k = 4\) that is 16 servers and 20 switches. For \(k = 48\) it is 27,648 servers and 2,880 switches. Two servers in different pods have \((k/2)^2\) shortest paths: 4 when \(k = 4\), and 576 when \(k = 48\).
Worked example: oversubscription
A leaf switch has 48 server ports at 25 Gbit/s and 8 uplinks at 100 Gbit/s, one to each of 8 spines. 1. Downward capacity is \(48 \times 25 = 1200\) Gbit/s. 2. Upward capacity is \(8 \times 100 = 800\) Gbit/s. 3. The oversubscription ratio is \(1200/800 = 1.5\), written 1.5:1. 4. If every server sends off the rack at once, they share 800 Gbit/s, so each gets \(800/48 \approx 16.7\) Gbit/s, two thirds of its line rate. 5. Traffic between servers on the same leaf never touches the uplinks, so placing chatty services in one rack makes the ratio matter less.
Failing gracefully
Many parallel paths mean that a failure removes a share of the capacity rather than a connection. In the leaf-spine example, if one of the 8 spines fails, each leaf keeps 7 uplinks, or 700 Gbit/s, and the oversubscription ratio rises from 1.5:1 to \(1200/700 \approx 1.7\):1. The loss is one eighth of the uplink capacity, and flows on the failed path are moved to the survivors. A tree with a single aggregation switch could be cut off entirely by one fault. This is why operators prefer many small, identical switches to a few large, special ones.
Equal-cost multipath
With many equal paths, switches must spread traffic across them. Equal-cost multipath (ECMP) lets a router keep several next hops for one destination and choose between them by hashing fields of each packet, usually the source and destination addresses, protocol and port numbers; packets sharing all of these form one flow. Hashing keeps all packets of one flow on one path, so they stay in order, which TCP needs. The weakness is that hashing is blind to size. Two huge flows can land on the same uplink while another sits idle, and a path failure or a new hash can briefly disturb running flows.
Worked example: when hashing collides
Four long-lived flows, each wanting 10 Gbit/s, cross four 10 Gbit/s uplinks, and the hash places each flow on a uniformly random uplink. 1. For all four flows to land on different uplinks, the first can go anywhere, the second must avoid 1 uplink, the third 2, the fourth 3. 2. The probability is \(\tfrac44 \times \tfrac34 \times \tfrac24 \times \tfrac14 = 24/256 \approx 9.4\%\). 3. So with probability about \(90.6\%\) at least two flows share an uplink and get half their rate or less. A simulation of 100,000 trials agrees (about 90.5%). Operators respond with finer-grained spreading, more uplinks, or by steering flows from a controller.
Trends
As of 2026, server links of 25 Gbit/s or more are typical in new builds, and links between switches usually run at 100 Gbit/s or more, faster still in clusters built for AI. Inside the building, operators often use congestion signals from switches (explicit congestion notification) to slow senders before queues overflow, and some run remote direct memory access over Ethernet to give applications very low delay. Some operators run routing protocols such as BGP between switches, and many manage the whole fabric with software controllers, as described in Generalized forwarding and SDN. These practices vary between operators and keep changing.
Common mistakes to avoid
First, assuming most data-centre traffic goes to and from the internet; much of it is east-west. Second, confusing oversubscription with failure: a ratio above 1:1 is a deliberate saving that is fine until demand is high everywhere. Third, forgetting that ECMP balances flows, not bytes. Fourth, treating a load balancer as a router; it picks servers for requests. Fifth, mixing up the fat-tree formulas: servers grow with \(k^3\), not \(k^2\).
Key takeaways
Data centres are dominated by east-west traffic between servers, so the network must offer high bandwidth, low delay and resilience. Servers connect to top-of-rack switches, and load balancers spread external requests. Leaf-spine and fat-tree Clos designs give many equal paths; a fat tree of \(k\)-port switches supports \(k^3/4\) servers. Oversubscription compares downward to upward capacity. ECMP hashes flows over equal paths, which keeps packets in order but can collide big flows.
Flashcards
Flashcard
Traffic between servers inside the data centre
Test your understanding
Question 1 of 5
A leaf switch has 32 server ports at 10 Gbit/s and 2 uplinks at 40 Gbit/s. What is its oversubscription ratio?
Question 2 of 5
How many servers can a fat tree built from 8-port switches support?
Question 3 of 5
Why does ECMP hash on flow identifiers instead of sending each packet over a random path?
Question 4 of 5
A leaf-spine fabric has 6 spine switches. How many equal-length paths join two servers on different leaf switches?
Question 5 of 5
What does a data-centre load balancer do?
