I can’t tell you how many times I’ve been using a System assembly which is misbehaving, and been forced to use Lutz Roeder’s Reflector tool to reverse-engineer to uncommented C# code in the hopes of figuring out that is wrong. It was equivalent to the days of old when the C Runtime Library source code wasn’t shipped with the compiler.

All I can say is, “what took so long!?”

UPDATE I’ve read some bitching about the nature of the license under which this code is to be released; specifically, some Microsoft license which is no-modify and no-distribute. Those who find this unacceptable are, imho, missing the point. 80% of the value of a unrestricted GPL-type .NET Framework source release would be easier diagnosis and debugging anyway, since even if you could fork the MS framework, that would be like modifying the MFC code that shipped with VC 4 and shipping the modified DLLs with your product; you could do it, but you’d have to be a fucking moron.

Yes, yes, I know, binaries are included with the release. But what if you want to debug, or modify the code, or copy-paste it and get it to build in your own code? You’re screwed to the wall, that’s what. Here’s the secret handshake to get it building.

To start with, you go through the usual conversion wizard stuff. You build the XP version of the tool, which is missing some features. That builds fine, albeit with a torrent of scary security warnings. If you’re not willing to rewrite the code to use the new more-securer security APIs, add _CRT_SECURE_NO_WARNINGS to the list of preprocessor definitions.

That’s all fine, but let’s say you want to build the ‘Server’ version, which contains VSS features available only in Windows Server 2003. Well, in addition to the warnings you fixed above, you get this steamer:

1>revert.obj : error LNK2019: unresolved external symbol "long __stdcall ShouldBlockRevert(wchar_t const *,bool *)" (?ShouldBlockRevert@@YGJPB_WPA_N@Z) referenced in function "public: void __thiscall VssClient::RevertToSnapshot(struct _GUID)" (?RevertToSnapshot@VssClient@@QAEXU_GUID@@@Z)
1>C:\Program Files\Microsoft\VSSSDK72\TestApps\vshadow\bin\Debug-Server/vshadow.exe : fatal error LNK1120: 1 unresolved externals

Not. Helpful.

As it happens, I ran dumpbin /exports on vssapi.lib, and found that it does export ShouldBlockRevert, but thanks to C++ name mangling the mangled name is different. Why is it different? Because in vssapi.lib, the first argument to ShouldBlockRevert isn’t wchar_t, it’s unsigned short. “So what”, you’re thinking, “they’re equivalent”. And I don’t disagree, but the compiler treats them as different types for name manging purposes. What’s the fix? Well, disable the intrinsic wchar_t type in the C/C++ Language property page in the project properties (equivalent to the /Zc:wchar_t- switch if you’re one of the two people on the planet who build Visual C++ projects with makefiles).

Once that’s done, the LPCWSTR macro is defined to unsigned short, name mangling matches, planets align, and you can link. QED.

However, this does lead to some unfortunate complexity. One of the most significant of these is the need to use the debugger on my host machine, to debug code on the virtual machine.

Since Visual C++ 6.0 at least, I’ve known of the Remote Debug Monitor, which is a lightweight executable shipped with Visual C++ that you run on a target machine to enable you to debug processes thereon, from a remote host across the network. It’s gone through various stages of shittiness, and now operates primarily via DCOM (presumably, using sockets was too reliable and straightforward).

If you’ve ever coded with DCOM, you probably sense I’m about to relate a tale of misery and woe. DCOM sucks for a few reasons:

• Its security model is only fully understood by Keith Brown, and even he’s forgotten alot of it now that no one uses DCOM anymore.
• When something goes wrong (and it always does with DCOM) the error message you get is typically something like “Error -8234286245: You’re shit out of luck”.
• The parts of it that matter are virtually undocumented, leaving you with few options but to scrounge ten year old USENET postings for stupid hacks that no one understands but that seemed to work for a guy in Brazil once.

Add to this the fact that you’re trying to get a remote debugger to work on top of DCOM, and it’s fairly obvious that your day is about to get a whole lot worse.

Installation

To start, you need to get the Remote Debug Monitor installed on the machine running the process you want to debug. The MSDN article Howto: Setup Remote Debugging, is fairly comprehensive, albeit a bit dense.

The gist is, either install Visual Studio 2005, or install the RDM separately from the Visual Studio 2005 installation media. Since I’m running on a VM that is supposed to roughly approximate a real-world environment, I’m not about to install VS2k5, so I opted for the separate install. It’s in the Remote Debugger folder, with subfolders for the x86, x64, and ia64 platforms.

The installer gives you the option to run the monitor as a service; I declined, and run it explicitly from a shortcut I put on my desktop, when I need remote debugging.

XP SP2 and Firewalls

If you have the misfortune of running XP SP2 or a personal firewall, you’ve some work to do. The aforementioned article enumerates all the ports you have to open. Port tcp/135 on the machine you debug from; tcp/135, tcp/139, tcp/445, udp/137, and udp/138 on the machine running the debug monitor. There are also a couple of IPsec ports if you use IPsec, which I don’t.

Intuitively, you might expect only the remote machine would need firewall rule changes, but the magic of DCOM means there are comms in both directions. Forget this at your peril.

KB 833977 has some additional steps to try on XP SP2.

Running

When you’re ready to do some debugging on the remote host, startup the debug monitor. By default it will run with Windows authentication, which also seems to mean DCOM. You can switch it out of this mode, but the resulting dumbed-down TCP-based mode doesn’t support managed debugging, which makes it a non-starter for a .NET programmer such as myself. Accept the defaults.

How you configure Visual Studio to use the remote debugger depends on what kind of project you’re trying to debug. Visual C++ projects have a drop-down box in the Debugging section of the project Properties dialog, which defaults to ‘Local Windows Debugger’; change that to ‘Remote Windows Debugger’, and fill out the debug fields as appropriate, taking care to enter the name or IP address of the machine running the debug agent, in the ‘Remote Server Name’ field. ‘Connection’ and ‘Debugger Type’ leave at their defaults.

.NET projects have a completely different project properties screen; go to the Debug tab, check the ‘Use remote machine’ checkbox, and enter the remote machine name.

A few things to note, regardless of which type of project you’re debugging:

• The command/external program to run is evaluated on the remote machine, so the path you specify must be valid on that machine. Specifying the path to the executable on your Visual Studio machine won’t work, unless by some happy coincidence they’re the same.
• Ditto for any paths or file names given as command line arguments
• It’s imperative that the remote machine have the exact binaries you just built with Visual Studio. If not, the debugger will detect that its symbols don’t match the binaries, and breakpoints won’t work, not to mention any changes you made since you last updated the remote host won’t be there. Do not mess this up.

At this point, kick off the debugger with F5. If you’re very lucky, things will Just Work. If not, consider the troubleshooting steps below:

Gotchas

The real problem with the Visual Studio’s remote debugging support is that it doesn’t provide much in the way of useful error messages; it either works or claims there’s no remote debug monitor running on the target.

To that end, try the following steps:

• If at all possible, make sure the username and password are the same on both the host and the remote machine.
• According to Remote Debugging Permissions you can remotely debug a process owned by the account you authenticate with to the remote debug agent. I run without admin privs on my host machine, but I’ve never tried to run the debug monitor without admin rights, so ymmv.
• DCOM privs may get in the way. Adjust the DCOM privs as specified in this article on the remote machine.
• KB 908099, though ostensibly for XP SP2, is mostly applicable to W2k3 server as well (sans the firewall stuff, obviously). Pay particular attention to the dcomcnfg steps
• Sometimes the Visual Studio remote debug code doesn’t use the right username to find the remote debug monitor instance. In my case, the host and remote machines were both running as administrator, but the remote machine was running under a domain administrator account for a domain controlled by the remote machine itself, and Visual Studio reported:

   Unable to start debugging.

  The Microsoft Visual Studio Remote Debugging Monitor (MSVSMON.EXE) does not appear to be running on the remote computer. Please see Help for
  assistance.

  This despite the fact that the agent was indeed running. In this case, try explicitly specifying the user name running the debug monitor. So, if your remote machine is ‘buggy’, run the remote debug monitor, and look at the first line of output it produces. It should look something like:

  Msvcmon started a new server named 'WHATEVER\someone@buggy'. Waiting for new connections

  Back in the Visual Studio debug settings, specify that entire quoted string (without quotes, obviously) as the hostname. Everything before the ‘@’ will be interpreted as the username.

• Finally, if it just won’t work, try to simplify the situation as much as possible. Make sure the usernames and passwords match, that both ends are admins, etc. If all else fails, you can switch the debug monitor and Visual Studio to use the native mode without authentication (See Tools|Options), but this won’t allow managed debugging, so if you’re writing .NET code you’re screwed.

Good luck, and godspeed. You’ll need both.

The gist of it? If you have a solution with two or more DLL projects, one dependent upon the other, and you use the “Link Library Dependencies” linker option to automatically link against the lib produced by the project being dependened upon, the IDE will compile and link find but vcbuild (and msbuild) will fail to link. Upon inspecting the build log, you see that the library that should’ve been linked in is not.

So, add that to the growing pile of shit that never should’ve made it into RTM and will be fixed in the next version of Visual Studio.

The object is currently in use elsewhere.

Upon dismissing the box, the designer remains active, but the form itself is displayed mostly as a white blob, and it clearly in some sort of corrupted state.
A Google yields KB 838463, uselessly entitled ""System.InvalidOperationException" error message when you use a program that changes the display orientation in Windows XP Tablet PC Edition". Needless to say, I’m neither changing the display orientation or running XP Tablet PC Edition. Next.
From a Google Groups hit:

This is the common response of the system when it finds a problem with a control at design time. Your grid control build may be out of date or may have failed or the control may be active in the designer toolbox so that it cannot be written to by the build process.

I’m not really buying that; none of those apply to me. I’ll try restarting VS2k3.
Interestingly, VS2k3 crashed when I closed it, further supporting my theory that VS2k3 is just confused.
Aha, after a restart all is well. Oh how I love VS2k3.