apocryph.org Notes to my future self

16Jun/079

The Totally Bullshit Ruby Extension Experience on Windows

In my quest to wrap Wireshark to dissect packet captures into something Ruby can handle, I’ve eliminated the PDML export option, and am now trying to write a Ruby C extension to wrap the Wireshark libraries.

As a Windows user of open source software, I’m used to being a second class citizen, worth little more than some gruding attention from a UNIX programmer who can barely be troubled to scrape together a shitty nmake makefile for a twelve year old version of Visual C++. It’s understood that without installing Cygwin and accepting alot of UNIX-on-Windows kludginess, you’ve basically a snowball’s chance in hell of building any moderately complex open source tool under Windows.

I finally managed to build Wireshark from sources using the instructions on the Wireshark dev site, but that’s only because Wireshark is one of the most Windows-friendly open source projects in the history of mankind. Ruby extensions, on the other hand, exemplify perfectly the my-way-or-the-highway conceit of the UNIX developer community. Allow me to explain:

If all you know about Ruby extensions you know from reading the Dave Thomas book Programming Ruby, then you can’t possibly understand what I’m saying. You know that building a Ruby extension in C is as easy as whipping up a few lines of extconf.rb, running ruby to generate the Makefile, then make and make install. Like so much else in Ruby, it Just Works.

Uh, yeah. If you’re running an operating system that the Ruby digerati have blessed as worthy, that’s probably how it works. Certainly under FreeBSD, most Linux distros, and Mac OS X, it’s that easy. I can only assume that us Windows users are such fuckwit n00bs we deserve to suffer as second class citizens; it’s what we get for betraying Le Resistance.

You see, in most UNIX environments, you can make a few simple assumptions. You can assume the system has a C compiler installed, that there’s only one compiler that could be used to build system binaries, that make and cc are in the path, and that you can let the compiler more or less decide what runtime libraries to link with, knowing that whatever it chooses will run anywhere on the machine. You can further assume that you’re building binaries for your machine alone, since everyone knows how to do builds so why bother with binary tarballs?

Under Windows, each of these assumptions is wrong. Windows doesn’t ship with a compiler, and even if one is installed, there are several flavors of Visual C++ ranging from Visual C++ 6.0 circa 1999 through Visual C++ 2005 SP1 circa 2006. Though each version is similar, most have different runtime libraries that do not interoperate, and later versions have a bizarre manifest system whereby runtime libraries are linked at runtime. If you write an executable to use Compiler Platform A, and link with a DLL built with Compiler Platform B, any attempts to use runtime objects like FILE pointers or memory management functions between the EXE and the DLL will end in tears.

This brings me back to the Ruby extension mechanism. The idea is simple enough: use a Ruby script to describe the headers, libraries, and various other bits an extension takes as input, and let Ruby itself generate a makefile custom tailored to your environment. This sucks for a couple of reasons: compiler version and runtime library.

It seems the one-click installer of Ruby 1.8.4 for Windows was built using Visual C++ 6.0, since the config.h file in c:\ruby\lib\ruby\1.8\i386-mswin32\config.h has this little nugget:

 #if _MSC_VER != 1200
 #error MSC version unmatch
 #endif

For those of you born after 1990, 1200 is the value for _MSC_VER used to indicate code is being compiled under Visual C++ 6.0. So, right off the bat, any code that #includes config.h (that is to say, all Ruby extensions) will fail loudly when built with anything other than Visual C++ 6.0. Why, you might ask yourself, would the developer responsible for this code limit it to the oldest version of Visual C++ still in use today? I’ve nfi, especially since C/C++ purists have no better friend on Windows than Visual C++ 2005, and no worse enemy than Visual C++ 6.0. But, be that as it may, the decision was made, and we suffer for it.

You might be wondering ‘what would happen if I remove that code and build something with Visual C++ 2005?’. Great question; it turns out it can be made to work. First you must comment out those lines in config.h. After you do that, and use ruby extconf.rb to generate a Makefile for your extension, you have to run nmake (Microsoft’s version of make). You get this not-at-all-reassuring output:

 Microsoft (R) Program Maintenance Utility Version 8.00.50727.762
 Copyright (C) Microsoft Corporation.  All rights reserved.

      cl -nologo -I. -Ic:/ruby/lib/ruby/1.8/i386-mswin32 -Ic:/ruby/lib/ruby/1.8/i386-mswin32 -I. -MD -Zi -O2b2xg- -G6  -c -Tcrcapdissector.c
 cl : Command line warning D9035 : option 'Og-' has been deprecated and will be removed in a future release
 cl : Command line warning D9002 : ignoring unknown option '-G6'
 rcapdissector.c
      cl -nologo -LD -Fercapdissector.so rcapdissector.obj msvcrt-ruby18.lib oldnames.lib user32.lib advapi32.lib ws2_32.lib  -link -incremental:no -debug -opt:ref -opt:icf -dll -libpath:"c:/ruby/lib" -def:rcapdissector-i386-mswin32.def -implib:rcapdissector-i386-mswin32.lib -pdb:rcapdissector-i386-mswin32.pdb
 Creating library rcapdissector-i386-mswin32.lib and object rcapdissector-i386-mswin32.exp

I like to think most competent programmers are conditioned to feel aversion and discontent when a build generates warnings, particularly warnings like “‘-G6′? What the fuck does that mean? Screw it, I’m ignoring it” and “Uh, nobody uses ‘Og-’ anymore, what the hell is wrong with you?”. Apparently, I’m a naive little fuckwit, because you’re just expected to suck up those warnings if you want to build Ruby extensions under Visual C++ 2005.

Think you’re done yet? Fuck no. If you try to pull a nmake install now, it’ll look like it worked:

 Microsoft (R) Program Maintenance Utility Version 8.00.50727.762
 Copyright (C) Microsoft Corporation.  All rights reserved.

 install -c -p -m 0755 rcapdissector.so c:\ruby\lib\ruby\site_ruby\1.8\i386-msvcrt\rcapdissector
 install -c -p -m 0644 .\lib\field.rb c:\ruby\lib\ruby\site_ruby\1.8\rcapdissector
 install -c -p -m 0644 .\lib\packet.rb c:\ruby\lib\ruby\site_ruby\1.8\rcapdissector
 install -c -p -m 0644 .\lib\packet_element.rb c:\ruby\lib\ruby\site_ruby\1.8\rcapdissector
 install -c -p -m 0644 .\lib\protocol.rb c:\ruby\lib\ruby\site_ruby\1.8\rcapdissector

Hooray! But wait. Let’s try it out with irb first:

 irb
 irb(main):001:0> require 'rcapdissector/rcapdissector'
 LoadError: 126: The specified module could not be found.   - c:/ruby/lib/ruby/site_ruby/1.8/i386-msvcrt/rcapdissector/rcapdissector.so
         from c:/ruby/lib/ruby/site_ruby/1.8/i386-msvcrt/rcapdissector/rcapdissector.so
         from c:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
         from (irb):1

Between the require and the LoadError is a stern message box saying:

 ---------------------------
 ruby.exe - Unable To Locate Component
 ---------------------------
 This application has failed to start because MSVCR80.dll was not found. Re-installing the application may fix this problem.
 ---------------------------
 OK
 ---------------------------

Bollox! How can it be missing msvcr80.dll? That’s the Visual C++ 2005 runtime DLL, which is installed when Visual C++ is.

If the extent of your Visual C++ 2005 experience is with using the IDE to build stuff, you probably have no fucking idea what’s wrong. If, like me, you’ve had to deal with nmake makefiles written by old make curmudgeons who refuse to use a GUI, then you totally feel my pain.

You see, Microsoft thought it was bad that it was so easy to install the wrong version of a DLL and break a bunch of apps (so-called DLL Hell). To fix it, they borrowed an idea from the .NET world; dynamic linking based on a cryptographic hash of a file, not just a file name. This way, you can have multiple versions of fuckit.dll installed side-by-side, and apps automatically load the right one. Brilliant. We’ll call it…side-by-side installation. Brilliant.

The catch is that apps that link to DLLs need to have some additional metadata descriptors to point to these side-by-side DLLs. This metadata is called a manifest, and it’s embedded in each executable as a resource. When you do a build with the Visual C++ IDE, it generates a manifest to link the C++ runtime DLL, and embeds it in the executable automatically. When you build something with nmake, well, you’re on your own.

The Ruby extension build tools don’t know/care about this, because they’re written by a guy who sees no problem building software in 2007 using a compiler that was EOLed last century. You, however, do care about this, because you insist on building Ruby extensions with a compiler that was released after you were born. So, after you do your little nmake, and before nmake install, you need to do this:

 mt.exe -manifest rcapdissector.so.manifest -outputresource:rcapdissector.so;2

(Ignore the .so extension; the Ruby developer responsible for the Ruby extension build system is a cultural imperialist who seems intent upon forcing his vision of the world upon users of other operation systems, like for example operating systems that use .dll to denote dynamically linked libraries).

Now, do a nmake install and try the shit out in irb:

 irb
 irb(main):001:0> require 'rcapdissector/rcapdissector'
 => true
 irb(main):002:0> shitoutofluck=CapDissector::CapFile.new('like i give a fuck')
 => #<CapDissector::CapFile:0x2e7ca50>
 irb(main):003:0>

Fuckin’ A. There you have it. How’s that for ‘Just Works’.

NOTE: I’d probably still be scratching my head and kicking my cat over this if it weren’t for Al Hoang’s two posts on the subject. Thanks Al.

Comments (9) Trackbacks (0)
  1. What a spot on rant. I’m

    What a spot on rant. I’m trying to build the ibm db2 adaptor and luckily found this post. It was very helpful :-)

  2. This is an ancient post, but here’s a thought: why not contribute your work back? I’m sure the other Windows users would appreciate it.

  3. I have contributed it inasmuch as the code is available on my SVN repository. I can’t contribute it in to the Ruby code because 1.8.6 is built with the ancient Visual C++ 6.0, and 1.9 is build with mingw, neither of which I support.

  4. I would definitely suggest considering mingw or cygwin. Since we windows users have been abandoned we need something closer to unix-y :)
    http://programming-gone-awry.blogspot.com/2009/05/how-to-setup-complete-mingw-msys-build.html
    Re: performance: you could try 1.9 I suppose http://rubyinstaller.org/downloads/
    GL to all us windows users out there.
    =r

  5. Mingw/cygwin are your only options most of the time on Windows, but neither would work for me because I need a 21st century debugger like Visual Studio’s. GDB is not the same, no matter what the graybeards say. Since Ruby 1.9 on Windows has gone the mingw route, it looks like we’ll never get relief.

  6. This isn’t still an issue is it? I recently built 1.9.1 on windows with VCExpress 2008. wasn’t too difficult as I recall. I ran the configure.bat file in [src]/win32 which generated config.h and the Makefile. Then it was just nmake, nmake test, nmake install. A few extensions got skipped, but I’m going back and fixing them now (zlib, iconv)

  7. awesome. great real world solution with swears. thanks, you are the only guy who solved my fucking problem. Fuckin A. (seriously, thanks). fuck.

  8. What’s a real world solution without swears? Fuck that!

  9. cl.exe said, “cl : Command line warning D9002 : ignoring unknown option ‘-G6′”

    ANelson said, “‘-G6′? What the fuck does that mean? Screw it, I’m ignoring it”

    Well, I wondered.. so I did a search. Finally found a reference under the old Visual Studio .NET 2003 docs:

    -G6 (or /G6) = “Optimizes code to favor the Pentium Pro, Pentium II, and Pentium III processors”
    http://msdn.microsoft.com/en-us/library/fwkeyyhe(v=VS.71).aspx

    Great.. does anyone really run those machines any more?


Leave a comment

No trackbacks yet.