As a follow-up to my previous article on Ruby HTTP client performance I’ve revamed my test rig and revised my tests to cover more variables, more implementations, and more Ruby versions.
This time I compare Ruby 1.8.6, 1.8.7, and 1.9.0, exclusively on CentOS 5 Linux. I compare the stock
right_http, and a number of
Net::HTTP variations with slight performance tweaks. I evaluate clock time, CPU time, and CPU time over clock time, for five sites with varying network characteristics. As before, my test rig and results are available on my SVN repository for further experimentation.
For the conclusion and pretty pictures skip to the bottom. For the gory details read on.
I have a simple HTTP client task, downloading a 10MB zip file from each of five data centers around the world (Seattle, Dallas, Chicago, Washington DC, and London). I’ve implemented that task using each of the HTTP client implementations I’m testing. I then use each implementation to download the file from each of the data centers using my CentOS 5 VPS box located in Future Hosting’s Dallas data center. I keep track of how much wall clock and CPU time is consumed by each implementation with the help of the Ruby
benchmark library, and log that information to a CSV file.
I repeat this for ruby 1.8.6, 1.8.7, and 1.9.0. I then aggregate the results and generate pretty graphs.
I have tested the following HTTP client implementations:
Net::HTTPlibrary that ships with Ruby. In 1.8.7 and beyond this library is a bit improved by a larger 16K read buffer size, but is otherwise unchanged between revisions
net_http_notimeout– A subclass of
Net::HTTPthat overrides the timeout logic to eliminate the timeout feature, which is cause for some shitty performance. This implementation also forces a 16K buffer size even under Ruby 1.8.6
net_http_select– A subclass of
select()to implement timeouts instead of the rather inefficient stock timeout implementation. This, like all my custom HTTP impls, forces a 16K buffer size.
Net::HTTPsubclass that has a modified read loop which uses the same pre-allocated String buffer for each read. This implementation also uses
select()for timeout, and a hard-coded 16K buffer size.
net_http_zerocopy_sysread– A variation of
readpartialwith no timeout for socket reads, along with the existing preallocated buffer optimizations of its parent.
rfuzz– Uses a slightly modified version of the lightweight HTTP client in the
rfuzzbase implementation as well as this tweaked one do not implement timeouts. The
rfuzzlibrary is required or this implementation will be skipped.
right_http_connection– Uses the
right_http_connectionHTTP client implementation from the
right_http_connectionworks by monkey-patching
Net::HTTP, which is why I had to modify my test rig to run each implementation test in a new instance of the Ruby interpreter. Bad form.
eventmachine– Uses the EventMachine HTTP client unmodified
libcurlUses the Ruby bindings for the
curlnative HTTP library
Unfortunately, I could not get
revactor working on any of my Ruby versions, so I was unable to evaluate those implementations.
Try this at home
To reproduce my results, do the following:
- Check out my SVN repository at
- Run the tests with Ruby 1.8.6. On my machine that’s the version in the path:
ruby -v ruby 1.8.6 (2007-09-24 patchlevel 111) [i686-linux]
which means the following command runs all available impls:
ruby -w -rubygems test_all_impls.rb
That will run all the available impls (some, like
rfuzz, aren’t available if you haven’t installed the necessary gem), and log the results to
(date)is the YYYY-MM-DD date.
- Run the tests with Ruby 1.8.7. On my machine I had to build that from source:
~/ruby18/bin/ruby -v ruby 1.8.7 (2008-08-11 patchlevel 72) [i686-linux]
To run the tests you have to pass the Ruby command on the command line, since I couldn’t figure out how to programmatically determine the path to the Ruby interpreter. On my system that’s:
~/ruby18/bin/ruby -w -rubygems test_all_impls.rb "~/ruby18/bin/ruby -w -rubygems"
Again, gems are required for some impls.
- Run the tests with Ruby 1.9.0. Same deal as 1.8.7.
~/ruby19/bin/ruby -v ruby 1.9.0 (2008-10-06 revision 19702) [i686-linux]
The command is similar:
~/ruby19/bin/ruby -w -rubygems test_all_impls.rb "~/ruby19/bin/ruby -w -rubygems"
- After running all three, you’ll have a bunch of CSV files in the
resultssubdirectory for today’s date. Here’s what I have:
results/2008-11-09/ruby-1.8.6-i686-linux-eventmachine.csv results/2008-11-09/ruby-1.8.6-i686-linux-libcurl.csv results/2008-11-09/ruby-1.8.6-i686-linux-net_http_notimeout.csv results/2008-11-09/ruby-1.8.6-i686-linux-net_http_select.csv results/2008-11-09/ruby-1.8.6-i686-linux-net_http_zerocopy.csv results/2008-11-09/ruby-1.8.6-i686-linux-net_http_zerocopy_sysread.csv results/2008-11-09/ruby-1.8.6-i686-linux-rfuzz.csv results/2008-11-09/ruby-1.8.6-i686-linux-right_http_connection.csv results/2008-11-09/ruby-1.8.6-i686-linux-stock_net_http.csv results/2008-11-09/ruby-1.8.7-i686-linux-eventmachine.csv results/2008-11-09/ruby-1.8.7-i686-linux-libcurl.csv results/2008-11-09/ruby-1.8.7-i686-linux-net_http_notimeout.csv results/2008-11-09/ruby-1.8.7-i686-linux-net_http_select.csv results/2008-11-09/ruby-1.8.7-i686-linux-net_http_zerocopy.csv results/2008-11-09/ruby-1.8.7-i686-linux-net_http_zerocopy_sysread.csv results/2008-11-09/ruby-1.8.7-i686-linux-rfuzz.csv results/2008-11-09/ruby-1.8.7-i686-linux-right_http_connection.csv results/2008-11-09/ruby-1.8.7-i686-linux-stock_net_http.csv results/2008-11-09/ruby-1.9.0-i686-linux-eventmachine.csv results/2008-11-09/ruby-1.9.0-i686-linux-net_http_notimeout.csv results/2008-11-09/ruby-1.9.0-i686-linux-net_http_select.csv results/2008-11-09/ruby-1.9.0-i686-linux-net_http_zerocopy.csv results/2008-11-09/ruby-1.9.0-i686-linux-net_http_zerocopy_sysread.csv results/2008-11-09/ruby-1.9.0-i686-linux-right_http_connection.csv results/2008-11-09/ruby-1.9.0-i686-linux-stock_net_http.csv
- To generate aggregate data files suitable for analysis,
cdinto the results directory and run:
ruby -w -rubygems ../../combine_csv.rb
This will aggregate all the .csv files in the directory and format them into three aggregate files:
total_cpu_time.txt. These files are formatted with columns for the server locations, rows for the HTTP implementation names, and values corresponding to the wall clock time, clock time over cpu time, and cpu time for each implementation and site. These are ready-made for generating the bar charts below in Excel. Note that you’ll need the FasterCSV library in order for this to work.
Running all the tests is a pain in the ass. If you fetch my SVN repository, you’ll find the raw data files that I got from my tests under
results/2008-11-09. Or, you can just read my analysis below.
As you can see above, each implementation takes more or less the same amount of wall clock time to download from a given site, with significant variations between sites. This is expected, as downloading a file over the Internet is a mostly network-bound operation. We don’t care so much how long it takes, as how much the CPU has to work while it’s happening. Which brings us to…
Wow, stock 1.8.6
Net::HTTP really is teh suck! At least twice as much CPU usage as the nearest competitor. Going from a 1K read buffer to 16K in 1.8.7 made a big difference.
Going further down the list, you can see the Ruby 1.9.0
Net::HTTP implementations with zero copy reads and
readpartial, and the
notimeout variant, are the best performers, with
eventmachine close behind. It’s encouraging that a pure-ruby impl like
rfuzz can compete with a mostly native impl like
It’s also important to note that each of the downloads, be it the super-fast Dallas or the slow London, hit the CPU the same way. This really jumps out when you look at CPU time over wall clock time:
Percent of wall clock time spent using the CPU
Here you see the various transfer times for each site, but you can also see the widely ranging performance of the various HTTP implementations. No real surprises here;
eventmachine are doing very well, while the 1.8.6 stock
Net::HTTP continues to blow.
If you need an HTTP client in Ruby, DO NOT use the 1.8.6
Net::HTTP. The 1.8.7 version is considerably better, but
eventmachine are all better still.
Net::HTTP family, dropping the inefficient timeout implementation and optimizing the read code to reuse the same buffer are both pretty low-hanging optimizations which should be considered for a future Ruby release. For now, I’d recommend
libcurl if you’re on Linux, or 1.8.7
Net::HTTP on Windows (since
rfuzz doesn’t have timeouts, and
eventmachine is hard to get going under Ruby on Windows).
UPDATE: It turns out there is a binary gem release of eventmachine 0.12.0 for Windows, so if you’re doing Windows development and need a performant HTTP client implementation, you should definitely look into eventmachine. Thanks to Abdul-Rahman Advany for the tip.