Synopsis

Both the PacketShaper and NetEnforcer products give network administrators a higher degree of control over quality-of-service policy enforcement than traditional network switches and routers. They offer a myriad of methods for traffic classification (IP address, TCP/UDP ports, QoS tags, and others). After classification, policy-based bandwidth control mechanisms are applied to the traffic. Alternately, the boxes can mark ToS/CoS[1] bits and allow other network elements to perform the actual traffic policing.

While both products are strong contenders, our analysis is that the NetEnforcer line is a stronger platform for implementing bandwidth control policies. The concept of equitable bandwidth allocation is fundamental to the design of the unit, and its strategy of unifying the inbound and outbound traffic for purposes of queuing seems to give it a fundamental advantage over the PacketShaper.

Need for Quality of Service

Before diving into a review of the products, it’s useful to understand the rationale behind the quality of service policy enforcement products. The products in the field are slowly gaining popularity as the need becomes apparent. Specifically, traditional Ethernet-and-IP network elements (routers and switches) perform well in lightly-loaded situations, but problems arise as traffic increases. For example, shared-access Ethernet rapidly degrades in throughput after about 30% utilization. In this situation, the number of collisions and retransmissions grows quickly, reducing the network efficiency.

While the issues of half-duplex shared-Ethernet are not a problem on full duplex core network links, oversubscription of links, especially outbound links, is a problem. When machines try pushing more data than administratively allowed, routers are forced to begin queuing packets or eating into burst-buffer space. As the link utilization increases dramatically beyond the limit, constant over-subscription and queuing become the norm. Queuing has limits, however, as does user patience. While a large router with nearly infinite queues might guarantee packet delivery, most users aren’t willing to wait several days. Indeed, infinite queuing is not a solution. Most routers implement a ‘tail-drop’ system, where packets are simply dropped as the queue limit is reached. In this case, high-rate connections can overpower low-rate connections (such as simple web browsing) by simply sending or receiving more data.

The solution, as the sales people keep reminding us, is to install a QoS policy shaper, such as the PacketShaper or NetEnforcer. The policy shapers provide controls to administrators that allow far greater control of traffic than routers. Different types of traffic can be categorized and grouped, and specific policies applied to the groups. Categorization of traffic happens by inspecting the packets and matching on certain elements of the packet. This can include the source or destination IP address, source or destination TCP/UDP port numbers, or pre-defined quality of service tags. Today, categorization engines are also looking ‘deeper’ into packets, inspecting application layer content such as HTTP URLs and peer-to-peer filenames.

After categorization, administrators can then apply specific policies to the different groups of traffic. Standard policies that can be implemented include giving certain traffic a higher priority (and thus access to the bandwidth) than other traffic. Alternatively, a policy might grant a specific amount of bandwidth to traffic in a certain group, or a certain amount of bandwidth to each unique connection (for example, giving each voice-over-IP connection a specific bandwidth allocation).

The expectation, in categorizing and policing the traffic, is to reduce the need for queuing on the router. The packet shapers use different methods to manipulate TCP’s built-in flow control mechanisms, slowing down lower priority traffic. This causes some connections to begin flowing at slower speeds than they would otherwise and reduce congestion at the router.

The subtext, however, is that the packet shapers will be most effective when specific policy is encoded. They can certainly use the rate control mechanisms to reduce congestion at the router, but in general this will continue to cause high-rate connections to dominate low-rate connections. At minimum, users will continue to receive sub-par bandwidth for what they consider “important” uses (typically, interactive applications such as web browsing). Consequently, creating policies that prioritize important connections, while slowing those that do not require immediacy, can reduce the consumed bandwidth while making the network appear more responsive.

Example Policy Configuration

In a deployment of the NetEnforcer unit to packet shape egress traffic, a sample configuration might be to have an LDAP server with the policy. This would enable easy delegation of administrative control, as well as provide rapid updates as needed.

The primary method of traffic balancing would be to use traffic priorities, enabling traffic from a low-priority group to consume more bandwidth if higher-priority classes are unused. Priority 10 is the highest priority and priority 1 the lowest. A sample provisioning of priority values might be:

Priority Level	Type of Traffic
10	Network critical traffic (routing protocols) Interactive traffic (telnet, SSH) below a configured rate
8	Inbound web traffic Priority outbound traffic (certain web servers)
7	Other inbound traffic (internal connections)
6	Other outbound traffic (external connections)
5	Specific non-interactive outbound traffic (large FTP, SMTP)
3	Specifically identified low-priority ports or identified applications (peer-to-peer) importing or exporting data

Products

PacketShaper

The Packeteer PacketShaper is a popular product among Universities, including peer institutions (e.g. Stanford). They boast 9 patents and 29 pending patents on the packet shaping technology in the unit. The range of Packeteer products includes those that shape from 2Mbps to 200Mbps (full-duplex). Related products to the PacketShaper include: PolicyCenter, a central policy administration server, and ReportCenter, a central server for recording monitoring data. The underlying PacketWise technology utilizes absolute priority queuing, TCP Rate Control technology, and bandwidth partitions. The TCP Rate Control mechanism uses TCP window sizing to control and pace the transmission rate.

NetEnforcer

The Allot Communications NetEnforcer comes with recommendations from Merit Networks, but is a relatively unknown product. The range of Allot products includes those that shape from 128Kbps to 155Mbps (full-duplex). Allot also sells NetPolicy, a central policy administration server. Available as add-ons for the NetEnforcer include: NetAccountant, which tracks usage information (so-called “Internet call records”), NetBalancer, which redirects client requests to available content servers, and CacheEnforcer, which redirects local content requests through a central cache. The key shaping mechanism on the NetEnforcer is per-flow queuing.

Feature Matrix

The matrix below highlights many of the basic features of the units. Important differences are displayed boldfaced. The PacketShaper has the concept of “partitions” of bandwidth with “classes” beneath the partition. Classes identify specific types of traffic. The NetEnforcer similarly uses “pipes” to indicate bandwidth partitions, and “Virtual Channel” (VC) to identify specific types of traffic.

	PacketShaper		NetEnforcer
	6500	8500	601C	701C	701F
Shaping Rate – Total (In+Out)	200M	400M	200M	310M	310M
List Price (Thousands)	$24	$32	$32	$38	$38
Maximum Partitions/Pipes	512	512	4,000	4,000	4,000
Maximum Classes/VCs	1,024	2,048	28,000	28,000	28,000
Maximum Dynamic Partitions	5,000	20,000	28,000	28,000	28,000
Maximum IP Hosts	25,000	100,000	n/a	n/a	n/a
Maximum Flows	150,000	300,000	256,000	256,000	256,000

Configuration
Web Management Interface	♦	♦	♦	♦	♦
CLI Management Interface	♦	♦	♦	♦	♦
External LCD (+ Keypad)	LCD	LCD	LCD +	LCD +	LCD +

Policy Source
Local Policy Configuration	♦	♦	♦	♦	♦
Single Policy Config Screen			♦	♦	♦
COPS			♦	♦	♦
LDAP	◊[2]	◊	♦	♦	♦

Classification
IP/MAC Address & Port	♦	♦	♦	♦	♦
ISL/802.1q Classification	♦	♦	♦	♦	♦
DiffServ/ToS	♦	♦	♦	♦	♦
HTTP URL	♦	♦	♦	♦	♦
MPLS	♦	♦	♦	♦	♦
Extensive Layer 7	♦	♦

Packet Shaping
Admission Control (ACL)	♦	♦	♦	♦	♦
TCP & UDP Rate Control	♦	♦	♦	♦	♦
DiffServ/ToS/Cos Marking	♦	♦	♦	♦	♦
Proactive TCP Flow Control	♦	♦
Packet Queuing Control	♦	♦	♦	♦	♦
Flow-Based Traffic Control			♦	♦	♦
Fairness of Equal Flows			♦	♦	♦
Per-Class Bandwidth Limits	♦	♦	♦	♦	♦
Per-Class Priority Queuing	♦	♦	♦	♦	♦
Relative Priority Weighting			♦	♦	♦
Dynamic Host Limits	♦	♦	♦	♦	♦
Connection Number Limits	◊[3]	◊	♦	♦	♦
Bandwidth Limit Tree	♦	♦
Bandwidth “Borrowing”	♦	♦

Monitoring
Top Clients/Servers	♦	♦	♦	♦	♦
Top Applications	♦	♦	♦	♦	♦
Total Bandwidth Utilization	♦	♦	♦	♦	♦
Historical Data Store	♦	♦	♦	♦	♦
Slowest Clients/Servers	♦	♦
Top Traffic Classes/Utilization	♦	♦	♦	♦	♦
Dropped Packets/Efficiency	♦	♦	♦	♦	♦
Response Times	♦	♦
TCP Connection Metrics	♦	♦	♦	♦	♦
TCP Connection Health	♦	♦

Accounting
RADIUS Accounting			♦	♦	♦
ODBC/SQL Accounting			♦	♦	♦
SNMP Export	♦	♦	♦	♦	♦
XML/CSV Data Export	♦	♦

Reliability
Dual Power Supply	♦	♦	♦	♦	♦
Hot-Swap Power Supply		♦	♦	♦	♦
Expandable – 2 PCI	♦	♦
Redundant Unit Support	♦	♦	♦	♦	♦
Automatic Layer-1 Bypass	♦	♦	♦		♦
Bypass on Non-Fatal Error	♦	♦	♦	♦	♦

Interfaces
Fast Ethernet – Copper	♦		♦
Gigabit Ethernet – Copper		♦		♦
Gigabit Ethernet – Fiber		♦			♦
Fast Ethernet Management			♦	♦	♦
Console – DB-9	♦	♦	♦	♦	♦

Physical
19” Rackmount Units	2 RU	2 RU	2 RU	2 RU	2 RU

Configuration

Configuration of both units is principally accomplished by way of the web interfaces. Both units additionally have a command line interface for configuration. The PacketShaper uses a proprietary, Cisco-like command line interface (CLI). The NetEnforcer, running embedded Linux, provides the traditional administration commands one would expect to find. Configuration of the NetEnforcer-specific parameters is accomplished using specific binaries.

The PacketShaper web interface uses fairly standard HTML and JavaScript[4]. Multiple administrative users (with “touch” access) can use this interface simultaneously. There are five main tabs for configuration and reporting capabilities, however there are myriad links among the sections. Policy configuration generally requires several round-trips loading web pages[5]. Configuration changes take effect immediately. Because of the use of standard HTML and the web modality of click-reload, the box can provide consistent behavior for multiple simultaneous administrators. We would expect this interface to be accessible to machines with substandard or no support for Java. It was tested and worked with Internet Explorer on Windows and Mozilla under Linux. Surprisingly, however, Internet Explorer and Mozilla under MacOS X were unable to use the Javascript-based menus.

The NetEnforcer web interface is a Java-based console[6]. Only a single user with read-write access may be connected to the web interface, though multiple read-only users may be connected. When attempting to connect while another user has administrative access, the new user is offered the ability to take read-write control. The unit has separate areas for box setup, policy configuration, and monitoring/reporting. Changes in policy or setup are made interactively with standard Java user interface elements and committed to the box when complete[7]. The interface worked using Internet Explorer on Windows and MacOS X, as well as Mozilla under Linux.

Analysis

Both units provide functional web interfaces for configuration. Due to the relatively complicated nature of policy definitions, the web interface is definitely the best way to configure and monitor the devices, though both have CLIs that can theoretically perform all of the functions (modulo the graphs).

While the use of Java may restrict the platform/browsers that can reliably use the interface, we greatly preferred the NetEnforcer’s interface. Policy configuration on the PacketShaper was slow and arduous, since each operation involved a round-trip to the unit. Changes take effect immediately, so the implications of intermediate steps must be considered. While this is standard for Cisco configuration, for example, the policy configuration can be quite complex and much more dependent upon the whole picture. Additionally, the PacketShaper defines separate policies for inbound and outbound traffic, potentially requiring twice the work for similar policy configuration. Configuration is done using multiple screens that configure specific elements of the policy.

The NetEnforcer’s policy configuration is done through a single main window. The interactive nature of the dialog makes the configuration process swift. The box makes extensive use of so-called “catalogs” to define elements of the policy. For example, host groups are formed using catalogs. The quality of service parameters (such as the queue priority and bandwidth limits) are defined in a separate catalog space. The catalog elements are then accessible on the main configuration screen. Once the new configuration is complete, a “save” operation is performed, committing the new parameters.

The monitoring and reporting capabilities of the NetEnforcer are also superior to those of the PacketShaper[8]. The unit comes with several standard graphs (Bandwidth, Utilization, Top Servers, Top Clients) in a well-defined subsection of the Java interface. The graphs are designed for real-time analysis and updated every 30 seconds. Additional historical data is also stored on the unit, and can be exported using SQL/ODBC, RADIUS, or SNMP. Custom graphs, however, cannot be created through the web interface.

On the other hand, the PacketShaper has a few basic graphs (utilization, network efficiency) but other graphs must be custom-defined[9]. Because of the many variables that can be graphed, non-basic graphs require manual definition. The graphs are optimized or long term reporting (the minimum graph size is one-hour, while the NetEnforcer scales the graphs automatically, starting with the first data point). Additionally, custom-defined graphs require two separate browser windows to keep active, and are difficult to use for short-term analysis. Long term historical data can be exported using a XML or CSV formatted data over an HTTP connection, or SNMP. The PacketShaper maintains 1 hour averages of data variables for 2 months and 24 hours of 1 minute data points. Long term graphing on both units is best accomplished by exporting data samples to external utilities (both companies provide additional reporting packages), though the PacketShaper’s web interface can access longer term data onboard.

System Testing

Setup

The goal of testing the packet shaping devices is to assess the behavior of them in scenarios that we believe reflect those of the larger network. Both units are installed inline with single buildings. This provides us the ability to assess the performance of the units in monitoring and shaping the true network traffic. Since we have little control of this traffic, however, the tests are conducted using controlled connections.

The testing network consists of three laptops on these existing building networks and two server machines across the core. While certainly the number of machines involved in the test is relatively small, the small number of machines involved allows close analysis of the performance of the packet shaping units. The laptops were running MacOS X, Linux, and Windows XP, while the servers were running Linux. All of the machines were connected using full-duplex Fast Ethernet, to minimize any effects of Ethernet retransmissions.

Both units are configured with a 5Mbps limit (inbound and outbound) for the three laptop testing machines. Establishing specific limits is done to prevent skew in the results due to non-test network traffic. Additionally, it prevents the testing from interfering with the normal operation of the network. Recording of the test results is accomplished using client and server reports of the net effective rates, as well as rates reported by the packet shaping units. Measurements of the transfer rates were observed during all parts of the test, but the reported values are taken from the middle of the test. This was done to minimize the effects of ramp-up or ramp-down during the tests.

Three primary testing modules are used to examine the units: the “Web In” module, the “Web Out” module, and the “IPerf-Out” module.

· Web In
The web-in module is designed to simulate the behavior of web browsing. All three laptops run the test, which simultaneously initiates 4-8 retrievals of files from a web server running on Server 1. The file sizes were two each among 500KB, 1MB, 2MB, and 4MB. While larger than typical web files, the larger file size allows more accurate computation of the transfer rates. The module automatically restarts the download of a file after the previous instance of the download completes.

· Web Out
The web-out module simulates the effects of a large internal web server pushing content to external clients. A web server on Laptop 1 provides two large files (50MB and 90MB), and Server 1 initiates a download of these two files. The individual tests are constructed such that the file downloads are completed as part of the testing duration, so no restart of the file retrieval is required.

· IPerf-Out
The iperf utility is used during the test to generate additional traffic on the wire. Most commonly it is used to simulate low-priority traffic and is queued separately from the other web traffic. The laptops initiate TCP iperf connections to Server 2, and the principal flow of data is from the client to the server (unlike web connections).

Three major classification and queuing strategies are implemented on both boxes, and the results from each test recorded. The strategies are:

· Traffic Bandwidth Allocation
A common recommended strategy for implementing a packet shaping device is to create classes for various traffic types. Categorization by application layer protocol or transport layer port is employed to assign traffic into the classes. Specific bandwidth allocations (limits) are given to the classes. The limits apply to the aggregate of all traffic matching the policy definition.

For purposes of our testing, bandwidth allocations are placed for outbound traffic. Web-Out is assigned a specific allocation and IPerf-Out is assigned a different allocation. The assumption is that Web-In does not need a specific limit.

· Traffic Priorities
Like the specific bandwidth allocation strategy, this strategy places different priorities on the various traffic classes. The PacketShaper products support an absolute priority mechanism. Traffic in a higher priority class is completely serviced prior to traffic in a lower class receiving any bandwidth. The NetEnforcer products support a relative priority system: higher priority queues receive more bandwidth than lower priority queues by a specific multiplier. For example, traffic at priority 8 receives 2.7 times the bandwidth at priority 3.

In our testing, Web-Out and IPerf-Out traffic is assigned different priorities (using the different strategies).

· Host-Based Fair Bandwidth Allocation
Unlike the other modules, an approach of having the packet shaper fairly allocate bandwidth amongst all the active users does not rely upon application identification.

During the tests, the three laptops are given equal bandwidth allocations for outbound bandwidth. Clients running both an IPerf-Out and Web-Out should expect to receive the same slice of the available bandwidth as clients running just the Web-Out module.

Test Network Map

Results

Base Case

To begin, we initiate the Web-In, Web-Out, and IPerf-Out modules. All three Web-In clients are configured to attempt simultaneous downloads of 8 files. This is conducted with no specific policies except the 5Mbps limit in both directions. The observed transfer rates are:

The results of this base case test are rather surprising. The tests ensure that the 5Mbps pipe can be completely saturated in both directions. It appears that the NetEnforcer’s strategy of treating the inbound and outbound portions of each connection as a single entity, as well as its strategy of equally allocating bandwidth among connections, provides better overall results. While the NetEnforcer is moving the full 5Mbps in each direction, the PacketShaper moves a paltry 1.6Mbps outbound. We presume this is because the inbound web traffic is preventing even TCP ACKs from moving across the inbound link.

Priority Queuing

The next test involves the creation of priority queue values, differentiating the three classes of traffic. The settings are chosen to maximize the bandwidth for Web-In traffic, assuming users will be most likely to perceive network response as a function of interactive network use, such as web browsing. Web-Out is given slightly less priority than Web-In, and IPerf-Out is given the least bandwidth. The IPerf-Out traffic is meant to simulate peer-to-peer or other low-priority outbound traffic

The settings used in this test are below. The NetEnforcer uses a scale of 1-10 and uses a relative queuing system. That is, traffic at a certain priority is serviced more often, with a specific multiplier, than traffic at lower priorities. The PacketShaper priorities are based on a scale of 1-7, and indicate absolute priorities. Given the NetEnforcer settings, Web-In traffic should receive 5 times the bandwidth of Web-Out, and Web-Out should receive 1.6 times the bandwidth of IPerf-Out.

The results of the Priority Queuing test are below. All of the tests could individually consume the entire 5Mbps if run independently.

Again, the results of the PacketShaper are surprising: the IPerf-Out traffic is actually receiving more than 5 times the bandwidth of Web-Out, even though the strict queuing should have allocated no bandwidth to the IPerf. The NetEnforcer results are in line with expectations, as the IPerf-Out result (1.8) times the expected multiplier (1.6) is 2.88, slightly more than the bandwidth that Web-Out ultimately received. The Web-In module requires only a small amount of outbound bandwidth for the TCP ACKs, which it appears to be receiving using both boxes.

Bandwidth Allocations

Continuing with the tests, we change the policies to grant specific bandwidth allocations to the traffic types. On both boxes, Web-Out is allocated 2Mbps of outbound bandwidth, while IPerf-Out is allocated 1.2Mbps.

The results of this test are in line with previous tests. Namely, the Web-In traffic continues to cause havoc on the PacketShaper, while the NetEnforcer allocates the bandwidth nearly as expected. Because the outbound policies on the PacketShaper do not affect inbound traffic at all, the PacketShaper is unable to shape the traffic on the inbound side to provide the outbound bandwidth allocations.

Fair Bandwidth Allocation

The final test strategy is to configure the units to perform fair bandwidth allocation based on IP source address. The rationale is to divide the bandwidth amongst the active network users, providing extra bandwidth to users only after all the active users receive an equal share.

This allocation strategy is supported by both units with differing degrees. The PacketShaper provides the ability to create so-named “Dynamic Sub-partitions” at multiple points along a tree-based policy definition. It is therefore more flexible than the NetEnforcer’s capability of creating “Template VCs” at just one level. The PacketShaper also supports the creation of dynamic sub-partitions by simply describing the sub-partition criteria (source address, for example). The NetEnforcer requires that the members of the sub-partition be enumerated.

While attempting to configure the NetEnforcer with several thousand addresses, the Java interface slows to a crawl and is nearly unusable. Other methods of definition include flat-file and LDAP backend, but these methods were not examined. Using these template VCs would only be possible if the actual members was not enumerated in the Java interface.

Additionally, whether deployment of fair bandwidth allocation per-source address across campus is a good idea is an open question. Both the PacketShaper and NetEnforcer would need to be tested with a high number of dynamically created subpartitions/VCs to be assured of predictable and acceptable behavior. There is some concern that the units would being to slow dramatically as the unique hosts increased.

To test the capability of the units to handle this strategy, however, we configure the policies using the web interfaces. Each is configured to fairly allocate the outbound bandwidth. Throughout the test the Web-Out module is running, consuming outbound bandwidth. The IPerf-Out test is not used. Laptop 1 is configured to retrieve all 8 files using the Web-In module. Laptop 2 is configured to only retrieve 6 files, while Laptop 3 is setup to retrieve 4 files.

First, both boxes are tested without the fair allocation. This ‘Best Effort’ mode imposed only the 5Mbps bidirectional limit. The Web-In test will run indefinitely, but testing is stopped after approximately three minutes, when it is clear that the network is in a steady state. Then, the fair allocation policies are configured and the test is restarted. After determining the new allocation rates, the test is concluded.

The results clearly indicate that both boxes can perform fair bandwidth allocation. However, the wisdom in doing so is still an open question. Both units have substantial uncertainty about the performance with many active hosts, and the NetEnforcer interface is not suited to configuring this for a large number of hosts.

Conclusion

While both the Packeteer PacketShaper and Allot NetEnforcer provide much more flexibility in the application of quality of service policies to enterprise networks, our testing and experience indicates that the NetEnforcer is more suited to our environment. The NetEnforcer configuration is simple and responsive. While it does not support an entire tree of policy settings, the NetEnforcer’s two-level classification and policy application configuration will be acceptable for all but the most complex needs. The top NetEnforcer model can be purchased with fiber interfaces, allowing simple integration into our network, and the separate Fast Ethernet management interface is a nice addition.

Apart from the configuration and management capabilities, we are impressed by the NetEnforcer’s fundamental support of equitable division of bandwidth among the connections. During testing, similar connections among the laptops results in equitable bandwidth division, even when fair bandwidth allocation policies are not configured. Similar testing on the PacketShaper results in a widely skewed bandwidth allocation – some flows are treated much better than others. We believe the NetEnforcer also performs better given its method of unifying the inbound and outbound connection for purposes of queuing. This is evident in the test results, as the NetEnforcer achieves better total throughput under saturated-network conditions.

© 2002 Carnegie Mellon University. All rights reserved. PacketShaper and PolicyCenter are trademarks or registered trademarks of Packeteer, Inc. NetEnforcer, NetPolicy, NetBalancer, and CacheEnforcer are trademarks or registered trademarks of Allot Communications Ltd.
Appendix A: PacketShaper Setup View

Appendix B: PacketShaper Manage View

Appendix C: PacketShaper Monitoring View

Appendix D: NetEnforcer Setup View

Appendix E: NetEnforcer Policy Configuration

Appendix F: NetEnforcer Monitoring View

[1] “Type of Service” is a byte in the IP header; “Class of Service” is a byte in the Ethernet header

[3] The number of subpartitions can be bounded.