In the previous episode, I was struggling with the Ruby extension build environment on Windows. I finally gave up and created a Visual C++ 2005 project that built the extension, and wrote a post-build step to copy the files into the Ruby install directory. Obviously this is a short-term hack; I’ll need to get something that will build on *NIX, but I don’t want to spend any more time on the fucking build environment right now.

The next clusterfuck was not unlike the previous one. See, I’m wrapping some Wireshark functionality in a Ruby extension so I can extract dissected packets from a capture file as Ruby objects for my own nefarious purposes. To make this work, I take a Wireshark source tree, build it, then link to some of the resulting Wireshark DLLs in order to implement packet dissection. I built Wireshark with VC2k5 because I refuse to build anything with Visual C++ 6.0.

Unfortunately, I’m using the Ruby one-click installer for Windows, which is built with Visual C++ 6.0 (don’t even get me started!) This presents a problem, as the following dependency chain will (hopefully) illustrate:

 my stuff => wiretap.dll => zlib1.dll
 ruby => zlib.so => zlib1.dll

Now, my stuff, wiretap.dll, and the zlib1.dll that wiretap.dll depends on were all built with VC2k5, and use its runtimes. Ruby, zlib.so, and the zlib1.dll that zlib.so depends on where all built with VC6 and use its runtimes. Do you see the problem yet?

My stuff runs within Ruby’s process as a DLL, which means that the zlib1.dll that I want and the zlib1.dll that Ruby wants can’t both be loaded; it’s one or the other. I was running Ruby’s version, but either one has the same problem as illustrated below.

Here’s a bit of code from a file within wiretap.dll, snipped and macros expanded for brevity:

 wth->fd = _wopen(filename, O_RDONLY|O_BINARY, 0000);
 wth->fh = gzdopen(wth->fd, "rb");

Here’s what’s happening: The C runtime function _wopen is being called to open a file, and it returns a file descriptor, which is just an integer that identifies that open file. Then the zlib1 function gzdopen is called, passing in the file descriptor that it is to operate on. Remember, wiretap.dll is linked against the VC2k5 runtimes, so the integer returned by _wopen identifies the file to the vc2k5 runtime library functions. Then, this descriptor is passed into a zlib1 function, which is linked against the vc6 runtime, which keeps a separate list of file descriptors. Depending on what’s happened up to this point, the FD returned by _wopen might be a valid FD to vc6’s runtimes, but it certainly won’t refer to filename as we expect. Thus, gzdopen fails strangely.

Sure, I could use the zlib1 that is built with vc2k5 and get around this problem, but what happens when ruby, built with vc6, passes one of its FDs to zlib1? That’s right, the same damn thing.

What to do? If I controlled all the sources involved, I’d just build them all with vc2k5 and be done with it. Alas, I do not. Another option is to capitulate and use VC6, however I’d sooner port my code to Visual Basic than take that giant leap backward. I could build Wireshark without the HAVE_LIBZ define, but that would require users do a custom build of Wireshark, plus it removes compression functionality. I could build Ruby with VC2k5 (assuming it even has an option for that), but then the one-click installer version of Ruby won’t work.

To be honest, I don’t know the solution yet. I just know if I meet the Ruby build engineer who thought it would be a good idea to use an ancient compiler to build the latest Ruby, one of us will be walking away with a brutal wedgie.