Simulating slow WAN links with Dummynet and VMWare ESX
At work I’m developing a feature that needs to be tested across a variety of WAN links like DSL, cable modems, T1s, T3s, etc. There are commercial network impairment generators that do this for you, but they’re too expensive to be cost effective.
Fortunately I recalled that FreeBSD includes a module called dummynet, which interacts with the IP stack to queue and drop packets however you want. Using this in concert with VMWare ESX 3.5, I was able to set up an environment in which two Windows machines communicate with one another through a dummynet bridge, thus simulating any sort of network performance I want.
It was not easy to get going, particularly since much of the dummynet info on the ‘net haven’t been updated to reflect the significant changes in FreeBSD 7. Since I might want to do this again someday, I’m recording the secret handshake required to get this working.
Setup
The initial setup was quite a PITA, primarily because FreeBSD’s bridging functionality changed drastically recently, and most of the docs on the web are still gears towards the earlier implementation. However, I finally got it going; the following describes my journey.
ESX Virtual Switches
There’s a default virtual switch, vSwitch0, on our ESX box, into which all VM NICs are plugged by default. I created another one, ‘Bandwidth Wilderness 1′, which is not associated with any physical NICs.
The Windows VM that will be on the ‘LAN’ is plugged into vSwitch0, and the Windows VM that will be on the ‘WAN’ is plugged into ‘Bandwidth Wilderness 1′ (or whatever you call yours).
Oh, and the promiscuous mode policy for both switches must be Accept, which is not the default. If you don’t set this, you’ll get some pretty fucking weird ARP resolution problems which will result in some pretty confusing behavior.
Firewalls
To simulate a secure WAN configuration, I’ve enabled the Windows Firewall on both sides, and removed all exceptions but Core Networking and the Visual Studio Debugger stuff. This will break things like file sharing and pings. Turn the firewall off temporarily if you’re having trouble with such things.
FreeBSD VM
You’ll need a VM with three NICs. They will appear in FreeBSD as em0, em1, and em2. em0 is the one the box will use to accept SSH connections and generally communicate with the network. em1 will be plugged into vSwitch0 just like em0, but it won’t have an IP address assigned. em2 will be plugged into Bandwidth Wilderness 1. Once the VM is configured right, it will bridge traffic between em1 and em2, thereby allowing any VM plugged in to ‘Bandwidth Wilderness 1′ to communicate with anything in vSwitch0 transparently, with all packets running through the FreeBSD machine. This is how dummynet is able to introduce packet losses and delays.
Custom kernel
You should start with a FreeBSD install that includes full kernel and userland sources, and a full development environment. Instructions for building a custom kernel are here. I called my custom kernel DUMMYNET instead of MYKERNEL. I used the following kernel options in the config file:
options IPFIREWALL
options IPFIREWALL_VERBOSE
options DUMMYNET
I left all the existing options values unchanged. I then built and installed the kernel per the instructions, and rebooted.
/etc/rc.conf
In FreeBSD much of the service enable/disable configuration resides in /etc/rc.conf. Here’s what I put in mine (not including default stuff that was already there):
# Enable ipfw
firewall_enable="YES"
firewall_script="/etc/rc.firewall.anelson"
# Set up a bridge beteen em1 and em2
ifconfig_em1=”up”
ifconfig_em2=”up”
cloned_interfaces=”bridge0″
ifconfig_bridge0=”addm em1 addm em2 up”
The above should be rather self-explanatory. The bridging bit is a little dodgy, but I pulled it right from the if_bridge man page.
/etc/rc.firewall.anelson
The firewall rules I used are stored in a new file. Make sure you chmod it +x so it’s executable; it is a shell script. All it does is run ipfw (IP FireWall) to load the rule set which routes the bridged traffic through dummynet:
#!/bin/sh
ipfw -q /etc/ipfw.rules
/etc/ipfw.rules
The actual firewall rules are stored in this file. It includes all possible bandwidth configurations, all but one of which must be commented out at all times:
#!/bin/sh
#
# These rules control the dummynet parameters which simulate shitty
# WAN links.
#
# em1 is the LAN interface, while em2 is the WAN interface
#
# This config file is set up so that bandwidth from LAN->WAN may be different
# than WAN->LAN, as in the case of an ADSL link
#
# The setup of the rules is pretty simple. Traffic that comes in on
# em1 is bridged to go out on em2, and traffic coming in on em2 is bridged
# to go out on em1. Thus, the rules for LAN->WAN traffic are attached
# to traffic coming in on em1 ('recv em1'), and the WAN->LAN rules are
# attached to traffic going out on em1 ('xmit em1'). So even though both
# sets of rules refer to em1, the LAN-side interface, they distinguish
# between LAN and WAN traffic by the traffic direction
#
# This setup is heavily influenced by the 2006-April-07 posting to
# the freebsd-ipfw list by John Nielsen titled
# "Notes on using dummynet with ip_bridge"
flush
queue flush
pipe flush
#
# traffic from lan (em1) to wan (em2) goes through pipe 1
# traffic from wan (em2) to lan (em1) goes through pipe 2
#
# No delay, max bandwidth both ways
#pipe 1 config delay 0
#pipe 2 config delay 0
# SDSL
# 40ms RTT, 0.1% round-trip packet loss, 1536Kbit/s bandwidth
# symmetrical (same both ways)
#pipe 1 config delay 20ms plr 0.0005 bw 1536Kbit/s
#pipe 2 config delay 20ms plr 0.0005 bw 1536Kbit/s
# Long-distance T1
# 40ms RTT, 0.1% round-trip packet loss, 1.5Mbit/s bandwidth
# symmetrical (same both ways)
#pipe 1 config delay 20ms plr 0.0005 bw 1536Kbit/s
#pipe 2 config delay 20ms plr 0.0005 bw 1536Kbit/s
# High-latency T1
# 200ms RTT, 0.1% round-trip packet loss, 1.5Mbit/s bandwidth
# symmetrical (same both ways)
#pipe 1 config delay 100ms plr 0.0005 bw 1536Kbit/s
#pipe 2 config delay 100ms plr 0.0005 bw 1536Kbit/s
# Long-distance T3
# 40ms RTT, 0.1% round-trip packet loss, 45Mbit/s bandwidth
# symmetrical (same both ways)
#pipe 1 config delay 20ms plr 0.0005 bw 45Mbit/s
#pipe 2 config delay 20ms plr 0.0005 bw 45Mbit/s
# 100Mbit local ethernet
# 0ms RTT, 0.0% round-trip packet loss, 100Mbit/s bandwidth
# symmetrical (same both ways)
pipe 1 config bw 100Mbit/s
pipe 2 config bw 100Mbit/s
# Add firewall exceptions for localhost
# Localhost traffic not on the lo0 interface is bogus and dropped
add allow all from any to any via lo0
add deny all from any to 127.0.0.0/8
add deny all from 127.0.0.0/8 to any
# Don't firewall anything on em0 which is used to talk to this box
add skipto 60000 all from any to any via em0
# direct LAN->WAN traffic through pipe 1
add pipe 1 all from any to any out recv em1
add skipto 60000 all from any to any out recv em1
# same as above but for WAN->LAN traffic through pipe 2
add pipe 2 all from any to any out xmit em1
add skipto 60000 all from any to any out xmit em1
#The above 'add skipto 60000' are like goto statements to this rule
#which allows everything
add 60000 allow all from any to any
/etc/sysctl.conf
This file contains overrides for sysctl values, which control various kernel behaviors. This is what I added to the bottom of the file:
#Only forward IP traffic across the bridge interface
net.link.bridge.pfil_onlyip=1
#Use IPFW for layer-2 filtering
net.link.bridge.ipfw=1
# Got this from BSD list “Notes on using dummynet with ip_bridge”
net.inet.ip.fw.one_pass=0
That’s pretty much it.
Testing
To test this I installed Cygwin on both the LAN and WAN windows boxes, and built iperf from source on them. I ran this command on the WAN box:
iperf --server --len 1024K
And this command on the LAN box:
iperf --client 10.23.4.86 --time 60 --len 1024K
Where 10.23.4.86 is the IP address of the WAN box. This sort-of accurately simulates a bulk transfer and shows you how fast it is. This way you can verify that you set up the dummynet parameters correctly.
Changing the simulated bandwidth
To select the bandwidth configuration to use, edit /etc/ipfw.rules and comment out all the ‘pipe 1′ and ‘pipe 2′ lines except the two that correspond to the bandwidth config you want. Once you’ve done that, run /etc/rc.firewall.anelson as root and it will apply the new settings. Always test with iperf after you do this, to make sure you didn’t fuck something up and break network connectivity.
References
- Dummynet tutorial (beware, the bridging-related info is out of date and useless)
- Dummynet home page
- FreeBSD custom kernel build guide
- “Notes on using dummynet with if_bridge” listserv post
- if_bridge man page
Gotcha cloning Ubuntu VM
I cloned an Ubuntu 7.04 VMWare Workstation 6.0 virtual machine, and found that the cloned VM booted without network access. dmesg | grep eth0 showed that the pcnet32 driver was loaded fine and detected eth0, but ifconfig doesn’t list eth0, and ifconfig -a lists eth1 which is down. WTF?
Turns out this is a feature of Ubuntu and possibly other distros, which remember which MAC address goes to which device using /etc/iftab. I never would’ve figured it out if not for this thread. The fix seems to be to manually edit /etc/iftab to put in the clone VM’s MAC address.
That’s pretty fucking lame. Both Windows and the BSDs don’t have that problem.
Upgrading achilles from VMWare GSX Server 3.x to VMWare Server RC 2
I decided to use the long weekend to break in VMWare Server RC2. As per the release notes, I uninstalled GSX Server and rebooted prior to installing VMWare Server.
Impressively, it didn’t require another reboot, and appears to be fully compatible with my old VMs.
According to the admin guide, in order to take advantage of the new features in Server, you have to upgrade the virtual machines. This changes some of the virtual hardware, so may have some implications for the behavior of the guest OSs.
I’ll start with carlotta, the W2k3 box I don’t use that often. I did so, and was exhorted to upgrade the VMWare tools.
…
It seemed to work fine. Now the FreeBSD and OpenBSD boxes…
One problem is that the VMWare Console port seems to have changed back to its old default, 900-something. I had previously changed it to tcp/8080, since the fuckwits at CI Host run a facist firewall in front of my box that only passes a few known ports. I don’t remember how I changed it last time, but I can’t find it in the VMWare Server GUI so it’s probably a registry setting.
Sure enough, page 81 of the admin guide has this to say:
To change the port number on a Windows host or client
Add the following line to config.ini inC:\Documents and Settings\All Users\Application Data\VMware\VMware VMware Server:
authd.port =where <portNumber>is the port number that all consoles connecting to virtual machines on this host must use.
So, I added authd.port = 8080 to the config.ini file and rebooted.
The upgrade went all, and all the VMs seem to be working normally. Too easy.
Maddening problem with clock slip in FreeBSD under VMWare
A few weeks ago my father pointed out that the date stamps on my blog posts were behind by a week. Upon investigation, I found that bonzo‘s clock was a week behind. I updated it and declared victory.
Then, he pointed it out again a few days ago. Sure enough, it had slipped by several days. When I logged into the VMWare Console to check for options to sync the clock or whatever, I noticed a repeated error from the FreeBSD kernel that I’ve been getting on bonzo forever and always ignored:
calcru: runtime went backwards from [some big number] usec to [another] usec for pid [pid]
I googled this message, and found a whole community of FreeBSD users suffering under slipping clocks when running FreeBSD under VMWare. There’s something on the freebsd-current list, and VMWare’s own support forums.
There are a few proposed fixes, most involving the kern.timecounter.hardware sysctl. I tried changing it from its default of APIC to TSC and i8245, but none worked.
I then ran across a post on the VMWare forums suggesting:
In FreeBSD:
'tools.timeSync = "true"' added to .vmx file sysctl -w kern.timecounter.hardware=i8254 kldload vmmemctl (from vmware-tools) and have vmware-guestd running add 'kern.hz="250"' to /boot/loader.confI don’t have APIC or ACPI disabled in my FreeBSD host either
Now, I don’t want to run the VMware tools just to keep the clock in sync, but I did put kern.hz="250" in /boot/loader.conf and kern.timecounter.hardware=i8254 in /etc/sysctl.conf, then rebooted.
It’s been several minutes now, and the clock seems to be holding. I’m afraid I don’t understand in detail why this helps, though a VMware knowledgebase article alludes to a problem of missed timer interrupts, with a fix being reducing the frequency of the timer interrupts requested by the OS. I think that’s kern.hz="250". The importance of switching the time counting method from APIC to i8254 is less clear, unless it’s just a more reliable source of ticks.
At any rate, this problem has caused me to notice that VMWare server is in RC-2. As it’s the free successor to GSX Server 3, I really need to upgrade. Perhaps over the coming long weekend…
Extending swap space on FreeBSD 5.4
When I first created bonzo, I allocated 96MB of RAM in VMWare. As I ran Gallery 2, Drupal, ByteHoard, etc, it became clear from the out-of-memory errors that I needed to boost the memory space. I since increased the allocation to 256MB, but the swap file is still only 160MB. Consequently, I’m plagued by kernel out of space space errors like:
Dec 5 18:09:18 bonzo kernel: swap_pager_getswapspace(6): failed
and
Dec 6 03:03:07 bonzo kernel: swap_pager: out of swap space
Dec 6 03:03:07 bonzo kernel: swap_pager_getswapspace(4): failed
Dec 6 03:03:07 bonzo kernel: pid 15958 (httpd), uid 80, was killed: out of swap space
I’ve expanded bonzo‘s virtual disk by an additional gigabyte, using the steps from my previous post on growing VMWare disks, and now I need to get FreeBSD to use that new space as swap.
I ran sysinstall to go through the FreeBSD installer again, which is how I created the FreeBSD disk label initially. The disk label is what I as a Windows guy would think of as the partition table; it’s the list of partitions on the disk and their file system types.
To access the disk labeler again I went to ‘Expert’ mode in the installer, then chose ‘Label’. I hit ‘C’ to create a new label, and got:
Not enough space to create an additional FreeBSD partition
Sure enough, the label editor shows:
Disk: ad0 Partition name: ad0s1 Free: 0 blocks (0MB)
Clearly it doesn’t ‘see’ the 1GB I added to the end of the disk.
When I first accessed the ‘Label’ function, I got:
│WARNING: A geometry of 113179/15/63 for ad0 is incorrect. Using │
│a more likely geometry. If this geometry is incorrect or you │
│are unsure as to whether or not it's correct, please consult │
│the Hardware Guide in the Documentation submenu or use the │
│(G)eometry command to change it now. │
│ │
│Remember: you need to enter whatever your BIOS thinks the │
│geometry is! For IDE, it's what you were told in the BIOS │
│setup. For SCSI, it's the translation mode your controller is │
│using. Do NOT use a ``physical geometry''. │
However, I get that alot for large disks or RAID arrays, not just VMware disks, so I didn’t pay it any mind. Perhaps I should.
Apparently there is a tool, growfs, which has been in the FreeBSD base install for years. From the growfs(8) man page:
The growfs utility extends the newfs(8) program. Before starting growfs
the disk must be labeled to a bigger size using bsdlabel(8). If you wish
to grow a file system beyond the boundary of the slice it resides in, you
must re-size the slice using fdisk(8) before running growfs.
Awesome. So I have to resize the slice with fdisk, label the disk to a bigger size with bsdlabel, and only then can I expand the filesystem with growfs. Hmm. I’ll try later.
UPDATE: I wimped out. I’m rather scared to death of damaging bonzo’s filesystem. Even though it’s backed up nightly to jane, that doesn’t mean I relish the prospect of rebuilding the filesystem under duress. Thus, I’ve taken the easy way out; I’ve added a new, 2GB hard drive using VMware, and will make that into a swap volume.
I ran sysinstall again, to use its GUI fdisk-er and label-er. I’ve created a single partition, ad1s1b, which is a 2GB swap partition. I’ve added it to /etc/fstab (sysinstall tried, but the kernel hadn’t picked up the newly created filesystem on /dev/ad1 yet, so it failed. Not a big deal.). A reboot, and I’ll inspect my handiwork.
Sweet:
bonzo# swapinfo
Device 1K-blocks Used Avail Capacity
/dev/ad0s1b 162632 0 162632 0%
/dev/ad1s1b 2097112 0 2097112 0%
Total 2259744 0 2259744 0%
Growing a VMWare Virtual Disk
I need to grow the download disk on hotsoup-p2p, a VMWare Workstation virtual disk that stores my P2P downloads while they’re in progress. It’s only 10GB; I need at least 20GB, probably more.
From this forum posting, I see there’s a utility installed in the VMWare Workstation directory, vmware-vdiskmanager.exe. Running it I get:
Usage: vmware-vdiskmanager.exe OPTIONS diskName | drive-letter:
Offline disk manipulation utility
Options:
-c : create disk; need to specify other create options
-d : defragment the specified virtual disk
-k : shrink the specified virtual disk
-n <source-disk> : rename the specified virtual disk; need to
specify destination disk-name
-p : prepare the mounted virtual disk specified by
the drive-letter for shrinking
-q : do not log messages
-r <source-disk> : convert the specified disk; need to specify
destination disk-type
-x <new-capacity> : expand the disk to the specified capacity
Additional options for create and convert:
-a <adapter> : adapter type (ide, buslogic or lsilogic)
-s <size> : capacity of the virtual disk
-t <disk-type> : disk type id
Disk types:
0 : single growable virtual disk
1 : growable virtual disk split in 2Gb files
2 : preallocated virtual disk
3 : preallocated virtual disk split in 2Gb files
The capacity can be specified in sectors, Kb, Mb or Gb.
The acceptable ranges:
ide adapter : [100.0Mb, 950.0Gb]
scsi adapter: [100.0Mb, 950.0Gb]
ex 1: vmware-vdiskmanager.exe -c -s 850Mb -a ide -t 0 myIdeDisk.vmdk
ex 2: vmware-vdiskmanager.exe -d myDisk.vmdk
ex 3: vmware-vdiskmanager.exe -r sourceDisk.vmdk -t 0 destinationDisk.vm
dk
ex 4: vmware-vdiskmanager.exe -x 36Gb myDisk.vmdk
ex 5: vmware-vdiskmanager.exe -n sourceName.vmdk destinationName.vmdk
ex 6: vmware-vdiskmanager.exe -k myDisk.vmdk
ex 7: vmware-vdiskmanager.exe -p m:
(A virtual disk first needs to be mounted at m:
using the VMware Diskmount Utility.)
So I turned off the virtual machine, and did:
vmware-vdiskmanager -x 25GB "C:\VMs\hotsoup - Shareaza\Downloads.vmdk"
It took forever, but then it was done.
Trouble is, the Windows partition is still only 10GB. I need to expand it.
Using Disk Management, I converted the disk to ‘Dynamic’ from ‘Basic’. Once that’s done, I right-click the disk and chose ‘Expand’. I expanded by 15GB, to use the full capacity of the disk. Too easy.
New VMWare Player!
I just got an email from VMWare announcing the new VMWare Player. As the name implies, it allows anyone who downloads the free player to run virtual machines created w/ VMWare Workstation.
The download site seems to be pretty heavily loaded, so I’ve not had a chance to review its limitations, but if they’re reasonable, this could be huge. Perhaps some enterprising group will create a stock Windows XP image, activate all the DRM players on it, and share it the world over for communal music/video access.
Already, Oracle, IBM, BEA, Red Hat, et al are offering prefab VMs for evaluation purposes, which has significant appeal for both the vendor and the client. Clients don’t get stuck w/ reams of resource-hungry eval-ware bogging down their primary machines, and vendors don’t have to support the half-upgraded Windows 98 bastard box which inevitably someone will run an eval on.
Now, if only they were smaller, and not monolithic; in other words, I’d like to be able to download pre-installed components which could just be folded into an existing VM, but I’m not sure how that would work anyway.
Update:
It’s even better than I thought:
On Windows hosts, VMware Player also opens and runs Microsoft® Virtual PC
and Virtual Server virtual machines and Symantec® LiveState Recovery system images.
Kickass!
Some limitations:
- VMware Player does not support Virtual SMP. You cannot use VMware Player to power on a virtual machine that has more than one virtual processor assigned.
- VMware Player can run only one virtual machine at a time. You must close the virtual machine currently running in VMware Player before you can open another virtual machine.
And, obviously, you can’t create new virtual machines.
Still, I’m amazed VMWare is giving away so much virtualization technology. They’ve really thrown down the gauntlet to MS with its too-little-too-late Virtual PC offering.