Simulating slow WAN links with Dummynet and VMWare ESX

At work I’m developing a feature that needs to be tested across a variety of WAN links like DSL, cable modems, T1s, T3s, etc.  There are commercial network impairment generators that do this for you, but they’re too expensive to be cost effective.

Fortunately I recalled that FreeBSD includes a module called dummynet, which interacts with the IP stack to queue and drop packets however you want.  Using this in concert with VMWare ESX 3.5, I was able to set up an environment in which two Windows machines communicate with one another through a dummynet bridge, thus simulating any sort of network performance I want.

It was not easy to get going, particularly since much of the dummynet info on the ‘net haven’t been updated to reflect the significant changes in FreeBSD 7.  Since I might want to do this again someday, I’m recording the secret handshake required to get this working.

Setup

The initial setup was quite a PITA, primarily because FreeBSD’s bridging functionality changed drastically recently, and most of the docs on the web are still gears towards the earlier implementation.  However, I finally got it going; the following describes my journey.

ESX Virtual Switches

There’s a default virtual switch, vSwitch0, on our ESX box, into which all VM NICs are plugged by default.  I created another one, ‘Bandwidth Wilderness 1′, which is not associated with any physical NICs.

The Windows VM that will be on the ‘LAN’ is plugged into vSwitch0, and the Windows VM that will be on the ‘WAN’ is plugged into ‘Bandwidth Wilderness 1′ (or whatever you call yours).

Oh, and the promiscuous mode policy for both switches must be Accept, which is not the default.  If you don’t set this, you’ll get some pretty fucking weird ARP resolution problems which will result in some pretty confusing behavior.

Firewalls

To simulate a secure WAN configuration, I’ve enabled the Windows Firewall on both sides, and removed all exceptions but Core Networking and the Visual Studio Debugger stuff.  This will break things like file sharing and pings.  Turn the firewall off temporarily if you’re having trouble with such things.

FreeBSD VM

You’ll need a VM with three NICs.  They will appear in FreeBSD as em0, em1, and em2.  em0 is the one the box will use to accept SSH connections and generally communicate with the network.  em1 will be plugged into vSwitch0 just like em0, but it won’t have an IP address assigned.  em2 will be plugged into Bandwidth Wilderness 1.  Once the VM is configured right, it will bridge traffic between em1 and em2, thereby allowing any VM plugged in to ‘Bandwidth Wilderness 1′ to communicate with anything in vSwitch0 transparently, with all packets running through the FreeBSD machine.  This is how dummynet is able to introduce packet losses and delays.

Custom kernel

You should start with a FreeBSD install that includes full kernel and userland sources, and a full development environment.  Instructions for building a custom kernel are here.  I called my custom kernel DUMMYNET instead of MYKERNEL.  I used the following kernel options in the config file:

options IPFIREWALL
options IPFIREWALL_VERBOSE
options DUMMYNET

I left all the existing options values unchanged.  I then built and installed the kernel per the instructions, and rebooted.

/etc/rc.conf

In FreeBSD much of the service enable/disable configuration resides in /etc/rc.conf.  Here’s what I put in mine (not including default stuff that was already there):

# Enable ipfw
firewall_enable="YES"
firewall_script="/etc/rc.firewall.anelson"

# Set up a bridge beteen em1 and em2
ifconfig_em1=”up”
ifconfig_em2=”up”
cloned_interfaces=”bridge0″
ifconfig_bridge0=”addm em1 addm em2 up”

The above should be rather self-explanatory.  The bridging bit is a little dodgy, but I pulled it right from the if_bridge man page.

/etc/rc.firewall.anelson

The firewall rules I used are stored in a new file.  Make sure you chmod it +x so it’s executable; it is a shell script.  All it does is run ipfw (IP FireWall) to load the rule set which routes the bridged traffic through dummynet:

#!/bin/sh
ipfw -q /etc/ipfw.rules

/etc/ipfw.rules

The actual firewall rules are stored in this file.  It includes all possible bandwidth configurations, all but one of which must be commented out at all times:

#!/bin/sh
#
# These rules control the dummynet parameters which simulate shitty
# WAN links.
#
# em1 is the LAN interface, while em2 is the WAN interface
#
# This config file is set up so that bandwidth from LAN->WAN may be different
# than WAN->LAN, as in the case of an ADSL link
#
# The setup of the rules is pretty simple.  Traffic that comes in on
# em1 is bridged to go out on em2, and traffic coming in on em2 is bridged
# to go out on em1.  Thus, the rules for LAN->WAN traffic are attached
# to traffic coming in on em1 ('recv em1'), and the WAN->LAN rules are
# attached to traffic going out on em1 ('xmit em1').  So even though both
# sets of rules refer to em1, the LAN-side interface, they distinguish
# between LAN and WAN traffic by the traffic direction
#
# This setup is heavily influenced by the 2006-April-07 posting to
# the freebsd-ipfw list by John Nielsen titled
# "Notes on using dummynet with ip_bridge"

flush
queue flush
pipe flush

#
# traffic from lan (em1) to wan (em2) goes through pipe 1
# traffic from wan (em2) to lan (em1) goes through pipe 2
#

# No delay, max bandwidth both ways
#pipe 1 config delay 0
#pipe 2 config delay 0

# SDSL
# 40ms RTT, 0.1% round-trip packet loss, 1536Kbit/s bandwidth
# symmetrical (same both ways)
#pipe 1 config delay 20ms plr 0.0005 bw 1536Kbit/s
#pipe 2 config delay 20ms plr 0.0005 bw 1536Kbit/s

# Long-distance T1
# 40ms RTT, 0.1% round-trip packet loss, 1.5Mbit/s bandwidth
# symmetrical (same both ways)
#pipe 1 config delay 20ms plr 0.0005 bw 1536Kbit/s
#pipe 2 config delay 20ms plr 0.0005 bw 1536Kbit/s

# High-latency T1
# 200ms RTT, 0.1% round-trip packet loss, 1.5Mbit/s bandwidth
# symmetrical (same both ways)
#pipe 1 config delay 100ms plr 0.0005 bw 1536Kbit/s
#pipe 2 config delay 100ms plr 0.0005 bw 1536Kbit/s

# Long-distance T3
# 40ms RTT, 0.1% round-trip packet loss, 45Mbit/s bandwidth
# symmetrical (same both ways)
#pipe 1 config delay 20ms plr 0.0005 bw 45Mbit/s
#pipe 2 config delay 20ms plr 0.0005 bw 45Mbit/s

# 100Mbit local ethernet
# 0ms RTT, 0.0% round-trip packet loss, 100Mbit/s bandwidth
# symmetrical (same both ways)
pipe 1 config bw 100Mbit/s
pipe 2 config bw 100Mbit/s

# Add firewall exceptions for localhost
# Localhost traffic not on the lo0 interface is bogus and dropped
add allow all from any to any via lo0
add deny all from any to 127.0.0.0/8
add deny all from 127.0.0.0/8 to any

# Don't firewall anything on em0 which is used to talk to this box
add skipto 60000 all from any to any via em0

# direct LAN->WAN traffic through pipe 1
add pipe 1 all from any to any out recv em1
add skipto 60000 all from any to any out recv em1

# same as above but for WAN->LAN traffic through pipe 2
add pipe 2 all from any to any out xmit em1
add skipto 60000 all from any to any out xmit em1

#The above 'add skipto 60000' are like goto statements to this rule
#which allows everything
add 60000 allow all from any to any

/etc/sysctl.conf

This file contains overrides for sysctl values, which control various kernel behaviors.  This is what I added to the bottom of the file:

#Only forward IP traffic across the bridge interface
net.link.bridge.pfil_onlyip=1

#Use IPFW for layer-2 filtering
net.link.bridge.ipfw=1

# Got this from BSD list “Notes on using dummynet with ip_bridge”
net.inet.ip.fw.one_pass=0

That’s pretty much it.

Testing

To test this I installed Cygwin on both the LAN and WAN windows boxes, and built iperf from source on them.  I ran this command on the WAN box:

iperf --server --len 1024K

And this command on the LAN box:

iperf --client 10.23.4.86 --time 60 --len 1024K

Where 10.23.4.86 is the IP address of the WAN box.  This sort-of accurately simulates a bulk transfer and shows you how fast it is.  This way you can verify that you set up the dummynet parameters correctly.

Changing the simulated bandwidth

To select the bandwidth configuration to use, edit /etc/ipfw.rules and comment out all the ‘pipe 1′ and ‘pipe 2′ lines except the two that correspond to the bandwidth config you want.  Once you’ve done that, run /etc/rc.firewall.anelson as root and it will apply the new settings.  Always test with iperf after you do this, to make sure you didn’t fuck something up and break network connectivity.

References

• Dummynet tutorial (beware, the bridging-related info is out of date and useless)
• Dummynet home page
• FreeBSD custom kernel build guide
• “Notes on using dummynet with if_bridge” listserv post
• if_bridge man page