Routing the Campus

Or "How I Learned to Stop Bridging and Love the Router"

Since its inception in the early 1980s, Carnegie Mellon University’s network infrastructure has been fairly unique in that it is a class B network (with some 65,000 possible hosts) and is almost entirely bridged. When the original designers of CMU’s network infrastructure had to make a decision about how to implement the internal infrastructure, they had two choices. They could either make custom modifications to end-user TCP/IP stacks to support subnetworking in a class B (an unheard of concept in 1983!) or put additional intelligence into the local routing devices to support standard stack implementations.

In the end, they chose the latter course. All hosts on the CMU network, wherever they are, are configured in exactly the same way, as if they are on a flat class B network. The benefits of this are fairly obvious in that end-user configuration is standardized regardless of network location, making technical support that much simpler. But as we see today, 15 years later, the rationale behind the choices made in 1983 doesn’t necessarily apply anymore.

Today, CMU does not run an entirely flat Class B network in the classical sense. Certain sections of campus that need extra attention are actually routed. The best example of this is the residence hall network. Given the uncontrollable nature of the hosts on this network, it was quickly determined it was the best interest of the University to put this network behind a router both for bandwidth and security reasons. However, by adding proxy arping to the router’s configuration, users in the dorms may still configure their machines in the standard fashion and communicate with the rest of campus and the Internet without problem.

So the question remains; why change from a primarily bridged environment to one more geared to routing? One of the inherent problems in a large-scale bridged network is a broadcast storm. Some newer operating systems, such as Microsoft Windows 95 and NT, utilize chatty protocols that use the network broadcast address to determine neighboring hosts. The more machines of this type that are on the same wire, the more traffic you see when these machines decide to talk to each other. Using routing, one can compartmentalize the machines into smaller and smaller groups, minimizing the network impact of these transactions. Ordinarily, the usage of switches would help minimize the traffic, but because these are being sent to broadcast addresses, using switches won’t help as much as desired. In the end, bridged networks can’t scale as effectively as networks utilizing routed subnetworks.

Another problem is one of hardware capabilities. With the widespread addition of switches on CMU’s network to reduce unnecessary traffic replication, it has become necessary to purchase high-end switches, as less expensive ones cannot handle the sheer number of unique MAC addresses on our network. Moving towards a more routed infrastructure would allow CMU to use switches that cannot handle quite so many MAC addresses, but are much less expensive, saving up to 75% in hardware costs.

The answer for CMU’s network is to adopt a network design that takes the best of both the bridged and routed network environments. For the backbone network, maintain the bridged network we now have in place, keeping the core as something of a DMZ for local network traffic. However, once we get to the building level, at that point we start routing, for improved network management and broadcast control.