Campus Network Design

"Case Study: Designing Campus Network With Management in Mind"

Using a star topology to link subnetworks to a single backbone-in-a-room has given Carnegie Mellon University a flexible and highly manageable campus network.

by John Leong
John Leong is director of Networking and Communications at Carnegie Mellon University in Pittsburgh, Penn.

Carnegie Mellon is a relatively small,private university with approximately 5,500 students and 1,500 faculty and staff members. Computing has played a significant role in all departments, from the Computer Science Department and Robotics Institute, to the Drama School and the Graduate School of Industrial Administration. In the past, the computing needs of the university were supported by a small number of very expensive machines. With the exception of the Computer Science Department, and possibly the Electrical & Computer Engineering Department, most departments simply could not afford to purchase their own equipment. This is no longer the case. Today, there are hundreds of moderately priced minicomputers and work- stations, as well as thousands of relatively inexpensive IBM PCs and compatibles and Apple Macintoshes located in buildings all over the campus. Besides providing the engines for academic, research, and administrative computing, the computers support a well established and heavily used electronic mail and bulletin board system for the campus.

While there are significant advantages to distributed computing, it does make resource and information sharing potentially more difficult. Unless the evolution to distributed processing is properly managed, it can quickly lead to fragmentation and chaos. The foundation of a coherent distributed system is comprehensive data communications support.

This article gives an overview of the evolution and development of data communications at Carnegie Mellon University. In it I will describe the cable plant, networking technology, protocols, LAN interconnection, network management, and applications we use. Hopefully, this discussion will help you design a campus network of your own that is easy to manage.

The Cable Plan

Cable plant installation is an expensive and disruptive undertaking, so it is desirable not to have to repeat the exercise during the life of a building. Since networking technology changes every five years or so, it is important to select a good, generic cabling system rather than one that is tailored for a currently popular type of network. A good cabling system should have the ability to support all of today's important networking technologies. It should also have the potential to meet the requirements for higher performance networking in the future.

There are two aspects to our cabling system: inter-building cabling and intra-building cabling. The intra-building cabling system implementation is the more costly and complex of the two.

The Inter-Building Cabling System

A coaxial broadband system offers multi-media capability, a large band-width, and is proven in the field. However, upon close examination, there are problems with these perceived advantages. Coaxial cable can indeed be an excellent media for analog video, and it can support voice and data as well. The typical cable television (CATV) system can easily carry more than 100 video channels. Broadband systems can also handle high-quality audio, even in stereo.

A broadband system has a large amount of bandwidth available through the use of multiple channels. However poor adherence to channel allocation standards for data usage has been a significant problem. Equipment from different vendors often use the same or overlapping frequencies, and frequency agile modems remain expensive.

There are, indeed, years of CATV broadband operational experience. However, CATV is primarily a rooted-tree network with a small number of centrally located signal sources. Almost all the nodes (which are television sets) on the network are receive only stations. There is not that much operational experience with large scale, two-way broadband systems where every station has the potential to disrupt or disable the entire network.

On the whole, my colleagues and I favor a star-shaped, fiber optic network. From an operational point of view, the star topology is much easier to maintain than bus or circular ring systems. With star topology networks, most of the troubleshooting can be handled at one location --the hub. This certainly beats having to dispatch technicians to the field for every problem call. Another big advantage of fiber-optic networks is that they are not affected by lightning. This is especially important for an inter-building cabling system in the Pittsburgh area.

Today a lot of development work is being done to increase the data handling capability of fiber-optic cabling. Digitized voice is the most commonly used high speed, fiber based service. Long distance telephone companies use fiber cables to support inter-city services at rates over 1 gigabit per second. In the data communications industry, Proteon Inc. has been marketing an 80 megabit-per-second (Mbit/sec) ring over fiber for almost two years while the 100-Mbit/sec fiber distributed data interface (FDDI) ring is in an advanced stage of development. For broadband data systems, the typical higher speed services operate at 10 Mbits/sec over a 12-megahertz channel.

In the early 1980s, we interconnected most of the campus buildings with 144 fiber-optic cables. At the recommendation of our local telephone company, 50-micron cables were used. We soon discovered that the telecommunications and data communication industries were on different wavelengths. While the telecommunication industry had standardized on 50-micron cable, the data industry at the time backed the 100-micron cable.

When you connect a small 50- micron cable into a wide 100-micron tap, only 25 percent of the light gets into the cable. This significant loss at the connection is quite a problem. We are lucky that our campus is compact. Hence, even taking 75 percent loss at launch, most of the links still manage to function satisfactorily.

From an operational point of view, the star topology is much easier to maintain than bus or circular ring systems. Over the past few years, we have increased our fiber plant by installing additional cable. This time around we used 62.5-micron fiber, which is the converging standard for both the tele-communications and data communications industries. Today we have more than 400 cables connecting all buildings on campus, including student residences.

The Intra-Building Cabling System

We have two main selection criteria for our intra-building cabling system. First, it must have a star topology. For larger buildings, multiple interconnected stars is an acceptable configuration. Since we have conscientiously uncoupled the cable plant selection from a particular network technology, our second criteria is that the cabling system must be able to support a variety of network types.

A bus type of network, such as Ethernet, has the advantages that it is easy to install and has shorter aggregate cable lengths. However, when the network becomes quite large, a bus network can be extremely difficult to maintain. Many problems can only be determined by the low-tech approach of trial and error, un-coupling stations from the network until the error causing culprit is found.

In a bus network with hundreds of stations spread over several floors, it can take considerable time to get to every station, even if you know where they are all located and if none of the rooms housing the stations are locked. In the case of a star-shaped network, most of the problem isolation can be done at the hub of the star, typically in a wiring closet. The effect on problem isolation time can be dramatic. It can reduce the down time of a network from hours to minutes.

Like many universities, Carnegie Mellon has a heterogeneous computing environment and similar networking requirements. Installing a cabling system that will only support one specific networking technology is simply not acceptable. Given that we had decided on a star topology system in 1985, the alternatives available for a general purpose intra-building cabling plant were the AT&T Premises Distribution System (PDS) and the IBM Cabling System (CS). Both systems call for an integrated set of voice and data cables. The main difference between the two is the way in which data is handled. The AT&T PDS scheme calls for the installation of an additional set of four voice-grade, unshielded, twisted pairs for data. The IBM CS uses a set of two higher quality, data-grade, shielded twisted pairs instead.

While it is possible to run relatively high-speed data over voice pairs, the attenuation typically confines such operations to a very limited distance. Furthermore, satisfying FCC radiation regulations can be a problem. On the other hand, shielded twisted-pair data cable has much better capability, particularly when dealing with high-speed networks. Since labor accounts for more than half of the cost of most cable plant installations, the impact of the higher cost for the IBM cabling material on the overall budget is not that significant. Moreover, the benefits of a better quality cable can be greater in the long run. Labor cost is a big factor in an intra-building wiring project. It is important to note that labor costs vary considerably across the country, so the "cost per outlet" figure for such projects should be treated with caution.

With the IBM cabling system, we can handle all the networking technologies we need to support, including IBM's Token-Ring, Proteon's ProNet, standard serial lines, IBM 3270 terminals, Apple Computer Inc.'s AppleTalk, AT&T Starlan, and even full-motion color video. When we first evaluated the IBM cabling system back in the mid-1980s, we discovered a serious problem: It is unable to support Ethernet. Ethernet requires four shielded twisted pairs for its transceiver (drop) cable, while there are only two data pairs in the IBM cable. This problem was resolved when SynOptics Communications Inc. was founded by a group of people from Xerox. SynOptics supplies a network called LattisNet which turns the IBM cabling system into a star-shaped Ethernet. Since then, a number of vendors have introduced products that even support Ethernet over telephone wiring.

We selected the IBM approach, and managed to complete the rewiring in the early fall of 1987 after 18 months of intense construction. Over 12,000 outlets have been installed, with almost every room in every building outfitted with at least one outlet.

A quick overview of our IBM cabling design is as follows. Type 2 cables with both voice and data pairs are used to connect the wall outlets back to the wiring closets. The voice pairs are punched down into standard telephone style punch blocks, and the data pairs are terminated in a patch panel. Typically, there are a number of strategically located wiring closets in a building. One of those closets is designated as the main wiring closet for the building, or the building entrance facility. All the closets are connected to the main wiring closet by Type 1 cables, which have two data pairs. The ratio of Type 2 to Type 1 cables is 4:1. The phone pairs are trunked from the wiring closets to the main closet using the standard cables with 25, 50, or 100 pairs. The main wiring closet serves both as the cross connect between the different closets and as the building entrance facility where the inter-building fiber network terminates.

An important point to emphasize is that the IBM cabling system does not mean IBM Token-Ring. The cabling system is really a collection of wires that needs to be patched to the appropriate electronics inside the wiring closet to form a network. The electronics can be for Ethernet, IBM Token-Ring, AppleTalk, or other network technologies.

About the time we were completing the rewiring project, Digital Equipment Corp. (DEC) introduced its own cabling system called DECconnect. It is essentially the AT&T PDS scheme with the addition of a 50-ohm (RG-58) coaxial cable for Ethernet and a 75-ohm (RG-59) coax for video. In that scheme, low-to-moderate speed data is handled by the voice-grade twisted pairs, while high-speed data service is provided by the thin Ethernet cable. It is interesting to note that DEC, the main purveyor of Ethernet, has also acknowledged that a star-shaped Ethernet is a more appropriate distribution scheme than the bus topology system.

Upon examination, we think the IBM cabling system is still preferable. In the DEC cabling system, the high-speed, 50-ohm, RG-58 cable is rather limiting, and is of questionable use for networks other than Ethernet. As for the 75-ohm cable, it is questionable whether a broad- band network will work well with a layout designed for a telephone and data network. Typically, broadband network architecture has very different configuration parameters and constraints.

Building LANs on the Cabling System

We have extensive experience with Ethernet, having used this technology since the early 1980s. This technology is reasonably mature and has been adopted by virtually all the scientific and engineering computer vendors. In general, it is a very stable and efficient network. However, due to the lack of built-in network management features troubleshooting an Ethernet network can be tricky. As I mentioned earlier, we have come to the conclusion that the standard bus type of Ethernet is very difficult to maintain.

In addition, the evolution from Ethernet Version 1 to IEEE 802.3 has been painful at times. We have many Version 1 transceivers in place, while equipment coming on-stream conforms to the 802.3 specifications. The most infuriating problem with incompatible Ethernet equipment is that it functions in a "flaky" manner rather than failing outright.

We have reconfigured a few of the more important Ethernets into a star topology using SynOptics' equipment over the IBM cabling system. We are also supporting thin Ethernet operation over the IBM cabling system using modified baluns and multi-port thin Ethernet repeaters from Cabletron and DEC. The use of the star topology is our strategic direction for Ethernet deployment. Currently, we have more than 22 Ethernets on campus with more than 60 segments. There are almost 1,000 workstations and minicomputers on those networks.

As mentioned earlier, we also support AppleTalk. LocalTalk, which is Apple's cabling system for AppleTalk, uses a daisy-chain type of bus configuration. To achieve the star configuration we prefer, we used PhoneNet and the Star Controller from Farallon Computing Inc. Since the Macintosh is an inexpensive machine, coming up with an acceptable cost per connection was a big challenge for us. We did a series of experiments and concluded that each Star Controller port can be used to support up to three parallel PhoneNET runs over our IBM cabling system, providing that they are properly terminated In addition, each of the runs can support a Macintosh and an Apple LaserWriter, or three Macintoshes. We have been deploying AppleTalk over the IBM cabling system since April of 1988. By the end of the year, we had more than 800 nodes attached to over 50 subnets. We anticipate that the number of nodes will exceed 1,000 in the near future.

We have also been using the IBM 4-Mbit/sec Token-Ring since its inception, and currently have 39 IBM Token- Rings on the campus supporting almost 1,300 nodes. Approximately 60 percent of these nodes are IBM PCs and compatibles; the rest are IBM RT workstations. In general, the IBM Token-Ring is a well-engineered technology with extensive network management capabilities that make maintenance relatively simple. It is not as popular as Ethernet since it has not been around as long. Furthermore, whereas Ethernet chip sets are available from multiple vendors Texas Instruments Inc. is currently the only publicly available source of the token ring chip set, which is also relatively costly.

One frequently asked question concerns the slower data rate of the IBM 4-Mbit/sec Token-Ring versus the faster 10 Mbit/sec Ethernet. In our experience the difference is not all that noticeable. If you take into account the time required by software for protocol and application handling, the raw data transfer latency is insignificant. On the other hand, we are mildly interested in the new 16-Mbit/sec Token-Ring offering, since the difference in price between the 4-Mbit/sec and the price 16-Mbit/sec products is not that great.

Connecting LAN's With an Inverted Backbone

The fact that the backbone is in one location, and that every network appears in that location, makes operation management much easier. We can, therefore, carry out most of the troubleshooting and reconfiguration in our data communications room and reduce the occasions we have to dispatch technicians to the field. We used Ethernet as the backbone network primarily because we had Ethernet operational longer than any other network. For the time being, it is offering us marginally satisfactory service in terms of bandwidth. However, sooner or later it will be saturated. The issue is how to prolong the life of your investment, particularly in network interface cards, and what to do when the crunch comes. An important consideration in network configuration is the 80/20 rule; if possible, you should try to configure the network so that 80 percent of the traffic is confined within the subnets, and only about 20 percent makes it onto the backbone. We monitor the backbone network load very carefully. Our current average load is under 25 percent, with peaks at more than 60 percent. When will the load become unacceptable? That very much depends on the application profile. For some applications, an average network load of 30 percent is not even noticeable, while for others this load may be intolerable.

For our particular network application mix, we plan to take action when the average traffic load approaches 30 percent. There are two possible alternatives at that point: change to a higher speed backbone, or segment the backbone into multiple interconnected networks.

The first approach looks relatively straightforward. However, you should not consider the raw network bandwidth by itself. It has to be examined in the context of a networking system, including components such as subnets and routers. If the stations attached to the high-speed network, particularly the routers, do not have the power to handle the higher speed, especially in bursty situations, the chances of packet loss will increase significantly. Once a packet is lost, the typical software time-out for recovery will take seconds, which can seriously damage the effective throughput of the communicating stations.

Hence, simply increasing backbone bandwidth without looking at the traffic profile is like trying to squeeze more capacity out of a saturated highway by increasing the speed limit. If the entrance and exit systems are not upgraded accordingly, the resulting congestion at those points can actually make system performance worse than it was before. Furthermore, upgrading to a different (and often more expensive) networking technology can mean big bucks since a lot of network interfaces, software, and even routers, have to be changed.

Because of the economic implications, we are somewhat less enthusiastic about FDDI than other people. We are looking at ways to preserve most of our investments while at the same time obtaining gigabit-per-second level performance. Segmenting the backbone network into multiple subnets is a reasonably short-term solution. But it can be tricky and requires careful study of the network traffic pattern.

Choosing Protocols for Internetworking

Occasionally, these machines need to communicate with different vendors' computers. A typical application is electronic messaging and terminal emulation. While transfer of object files between dissimilar machines is of questionable use, transfer of data and source files is another matter. In a distributed processing environment, it is also highly desirable to allow machines from multiple vendors to access a common set of network services. Thus, in various scenarios, a machine has to communicate with other machines that do not implement its "native" protocol.

There are three main approaches to overcoming the protocol incompatibility problem. First, you can implement most of the protocols on most of the machines. This requires a tremendous effort and is not really a viable approach, unless the number of protocols involved is very small. A second alternative is to use translation machines. This is easier, since multiple protocols need only be implemented in these machines, but it still requires a lot of work. Furthermore, the translation machines will become bottlenecks.

The third and most effective approach is to choose a standard protocol that each machine can implement over and above its native protocol. This is similar to using English as the lingua franca. We opted for this approach.

The standard protocol should not be chosen primarily for its technical merit. While that is an important consideration, even more important is how widely available it is. The only two candidates that match our criteria at this point are IBM's Systems Network Architecture (SNA) and the Defense Advanced Research Projects Agency's (DARPA) TCP/IP protocols. Because of IBM's dominance in the data processing market, virtually every computer vendor has SNA interconnection capability. We decided not to adopt this protocol, however, since we have no local user base and little SNA expertise on campus.

DARPA's TCP/IP, on the other hand, is one of the main protocols used on campus. It is embedded in Berkeley UNIX, which runs on a wide variety of machines. Furthermore, it is required by the Department of Defense for its contractors, which provides a strong incentive for equipment manufacturers to support the protocol.

Finally, TCP/IP is also the protocol used by the popular Advanced Research Projects Agency Network (ARPANET) as well as a host of national and regional research networks such as the National Science Foundation Network (NSFNET). We found implementation of this protocol readily available on a wide variety of machines, from the small Macintoshes and IBM PCs and compatibles to workstations, minicomputers, mainframes, and even supercomputers. The International Standards Organization (ISO) set of protocols is a potential contender. However, the definition of these protocols is just being completed. Currently, there are few implementations, although this situation will likely change within the next five years.

For Carnegie Mellon and most academic and research institutions, it is clear that TCP/IP is the appropriate choice for now. Note that having a standard protocol does not mean that it is the only permitted protocol on campus. Therefore, VMS VAXes will continue to communicate among themselves using DECnet. However, when they wish to communicate with a processor that does not support DECnet, TCP/IP will be used.

Network Interconnectlon

Historically, we have had a variety of non-commercial routers in our networks. All the routers are based on off-the-shelf hardware with software originally developed by the Carnegie Mellon Computer Science Department. The exception is the Kinetics Inc. Fastpath-4 gateway for connecting LocalTalk to Ethernet. It is a commercial product with software that originated at Stanford University. All in all, we have the capability to interconnect a variety of networking technologies. We have also connected some networks using link layer bridges. Bridges are protocol independent. They are specific, however, to a particular networking technology. We have used bridges because we realized that: (1) using a standard instead of native protocol reduces the capability of the communicating stations, and (2) there are some campus departments that have a strong communications requirement that can best be met by using non-standard protocols. Generally, we prefer routers to bridges since they give us more network management capabilities as well as a lower cost per connection.

Currently, all the local area networks on campus are interconnected. This provides a comprehensive and high-speed data communications utility that extends to almost every building on the campus. Today, this campus internetwork has more than 3,000 attached stations. With the decreasing cost of workstations and personal computers, the number of attached stations is expected to increase over the next few years.

Network Management

Operation management deals with "real-time" operational issues and network policing. Real-time operational issues are problem determination, location, isolation, and recovery, which are usually handled by operators working under a fair amount of stress. Once a problem is discovered, we must be able to isolate it quickly from the rest of the network. (Half an hour is a good upper limit before users start getting restless and hostile.)

Policing the network means watching for intentional and accidental abuse of the network. It is a background monitoring function that calls for continuous verification of the administrative view of the network against the real world. Normally, nothing much happens. When things go seriously wrong, however, the network can deteriorate in a matter of seconds.

Evolution management deals with planning for the growth of and changes to the network based on traffic load and traffic patterns. While this activity has significant strategic importance, it does not have the time pressures associated with operation management. It is interesting to note that operation management is an around-the-clock activity handled by moderately paid technicians, with support from a more highly skilled engineering staff. Evolution management, on the other hand, is typically carried out by experienced network planners.

In order to provide reliable service 24 hours a day, seven days a week, to take into account the spectrum of knowledge of any given user. Hence generic tools for collecting statistics may be helpful for long-term network planning or producing research papers, but they are of little use to the operations staff.

We have been exploring the use of expert systems for network management, particularly operation management. This type of system should reduce the demand on the support engineers, as well as provide a valuable training tool for new operators. Using the model as a guideline, we have acquired or built a substantial repertoire of network management tools. Judging by the number of strange problems that mysteriously "disappear," we still have a way to go.

Furthermore, our good tools must be available to the operations staff. Tools must be designed tools need to be better integrated. A key aspect of this effort is the implementation of the Simple Network Management Protocol (SNMP) on all network components, including workstations. Using SNMP will allow all devices to report exceptional conditions, and also provide network managers with some control over their operation. Managing a data communications network is far more complex than managing traditional telecommunications systems. In the telecommunications world, the types of equipment attached to the network are usually limited, and their functions are relatively restricted and well-defined.

When managing data communications networks, you not only have to be well acquainted with the ins and outs of the networking hardware, you also need to understand communications protocols, and sometimes even implementations and quirks of the software running in the workstations. It is often very difficult to distinguish a real network problem from a problem induced by bad software.

Conclusion

Reprinted with permission of LAN Technology. Copyright 1989 by M&T Publishing, Inc. All rights reserved.