Gmail IMAP wait is over
No longer am I impatiently awaiting IMAP; IMAP support for my apocryph.org domain was turned on just now. I know this because I was checking obsessively throughout the day 
Interestingly, nullpointer.net was a domain I set up with Google Apps somewhat later; apocryph.org has been running Google Apps since the early days of the closed beta.
Great paper on Amazon's Dynamo distributed storage system
I just read this great paper describing the architecture and implementation details of Amazon’s Dynamo system, the distributed fault-tolerant storage architecture that drives Amazon’s computing system. The paper’s author, Dr. Werner Vogels, is Amazon’s CTO and thus is clearly an excellent authority on the subject.
Some of the details which appeal to me are the surprising (to me) design tradeoffs they made. Rather than accept the poor availability and high consistency of relational databases, they opted for a highly asynchronous, fault-tolerant ‘eventually consistent’ architecture, abstracted at the high level as a key/value container. The paper has much more detail, and it makes for a very interesting peek behind the scenes at one of the world’s largest distributed computing systems.
Impatiently waiting IMAP support on my Google Apps domains
Ever since Google announced IMAP support for Gmail accounts on the 24th, I’ve been waiting for them to enable it for my Google Apps domains, nullpointer.net and, more importantly, apocryph.org. Yesterday afternoon I logged out and back in to my mail accounts, and lo and behold, my nullpointer.net domain had the IMAP option in the Settings window. Still waiting for apocryph.org, though. How long must I wait? It’s killing me!
Unfortunate gotcha in Visual Studio 2005 Source Server support
My team lives and dies by uses the Symbol Server and Source Server features in Visual Studio 2005 to help us with remote debugging and crash dump analysis of our numbered release builds. Together these two features let us open a dump or a debug session and have VS2k5 automatically find and download the symbols for our product DLLs from our symbol server, then go to our SVN repository and pull down the source files that went into those symbols. Apart from the iffy-at-best variable inspection abilities that come with release builds, it’s a pretty sweet feature.
Today I was working on a problem and hit two snags, one after the other.
The first one is a problem using VS2k5 on a 32-bit Windows install to debug a crash dump of a 32-bit process running on 64-bit Windows. For some reason VS can’t seem to fetch the kernel32.dll symbols for the SysWOW64 folder in 64-bit Windows installs. I assume this is related to the fact that you can’t debug 64-bit processes or dumps on a 32-bit machine, though how exactly I don’t know.
I know the MS public symbol server has the kernel32.dll symbols, because if I analyze the dump on a 64-bit Windows machine it works fine.
Knowing this, I moved to a 64-bit Windows box to analyze my minidump. I fired it up only to discover that SVN wasn’t installed, so when VS2k5 tried to run the SVN command to fetch the source files from our repository, it failed because the SVN binary wasn’t in the path. I quickly installed SVN and made sure it was in the path, then restarted VS2k5 and tried again. Annoyingly, VS2k5 would not retrieve source files, nor even put anything in the Output window suggesting it was trying to.
I made sure SVN had a cached copy of my credentials, thinking that was the problem, but no change. Then I tried to analyze another memory dump, and VS pulled down the source files without a problem. Hmm.
Turns out, the Source Server stuff must cache the results of its attempts to retrieve specific source files, presumably so it doesn’t repeatedly slam its head into a wall for the same ungettable file. I deleted the Source Server folder in ~\Local Settings\Application Data, restarted my debug session, and was back in business.
That’s a shitty little gotcha MS really needs to get on.
Solution to undrinkable office coffee problem
At work we have a communal coffee maker (a Cuisinart piece-of-shit, for what it’s worth), with all the associated perverse incentive problems:
- No one wants to make the coffee
- Everyone wants to drink the coffee
- Those that do make the coffee do so according to their tastes not others
- The company-supplied coffee is the caffeine equivalent of skunk weed
Last week I decided I’d had my last cup of Ass Blend. After hearing good things about the AeroPress on BoingBoing, I ordered it from ThinkGeek, and picked up a pound of StarBuck’s House Blend ground coffee while I was at it. Though the mechanics of brewing coffee with the AeroPress are a little awkward at first, I must say it lives up to its reputation: rich, flavorful brew lacking the bitterness and taste of used motor oil I have come to associate with coffee.
As a bonus, since my AeroPress makes only single servings (yes, you can ostensibly make up to four espresso servings, but that’s one cup for me), and it’s not available for use by others, the tragedy of the commons problems go away entirely. Now, whenever one of my coworkers gripes about the dick who took the last cup of coffee or the unique flavor of dirty feet and cabbage in the latest pot, I chuckle smugly and sip my flawless House Blend.
Transcoding AVI to FLV with ffmpeg
I recently needed to convert from XviD AVIs ripped from a DVD to Flash Video format (FLV), which my Gallery software has better embedded player support for. Initially I tried ripping the DVD again with ImTOO DVD Ripper Platinum, which also supports ripping directly to FLV. However, no matter how high I cranked up the bitrate the resulting video was very blocky and shitty looking.
Next, I tried using MediaCoder, which is supposed to be a solid open-source transcoder, but damned if I could get it to work with Flash; it would just hang. I think the FLV support in the version of ffmpeg used by MediaCoder is broken, since I couldn’t even get FLV conversion working with the ffmpeg command line below.
Finally I tried getting ffmpeg running on my Ubuntu Feisty box.
Initially I used the ffmpeg version obtained with sudo apt-get install ffmpeg. I ran the following:
ffmpeg -i in.avi -y -b 800 -r 25 -f flv -vcodec flv -ab 128 -ar 44100 out.flv
That worked great, except for one thing. out.flv had no audio.
After some Googling about I ran across a helpful post by someone else with the problem. Like me, he wasn’t able to get an audio stream in the output file.
I suspected the ffmpeg binary from the Ubuntu package was missing the MP3 encoding support, hence no audio. Using instructions on this page I built the latest ffmpeg from code, then used the following command line:
ffmpeg -i in.avi -y -b 800 -r 25 -f flv -vcodec flv -acodec mp3 -ab 128 -ar 44100 out.flv
And that did the trick. I can’t believe what a pain in the ass video work is. Seriously, is the command line parser from hell really the best we can do?
Nasty Winsock Overlapped I/O Gotcha
Today at work I ran into a nasty gotcha with network socket I/O in Windows.
To achieve maximal network throughput in Replay, we use I/O Completion Ports (IOCPs), which are kernel objects which aggregate asynchronous queuing of multiple I/O operations. Due to the way we’ve written our network transfer code, when doing a transfer we don’t know in advance how many read operations will be needed to complete the transfer, so I keep some n number of socket reads outstanding at all times, then once we get the EOF, I use CancelIO on the socket object to cancel any reads left outstanding. That way, they still come back from GetQueuedCompletionStatus and thus the memory associated with them can be freed, plus we maximize throughput by keeping plenty of read buffers available to the TCP/IP stack at all times.
This works fine except in two isolated, non-reproducible cases in which the call to CancelIO resets the TCP socket, resulting in a RST packet being sent back to the source, and both sides failing subsequent send/recv calls with WSAECONNRESET. I can only conjecture that the CancelIO implementation has some sort of failure case wherein it must reset the connection and give up. Highly fucking lame.
As a result, I had to rearchitect our transfer code to make sure we do all the writes we need to do before CancelIO, since I now have to assume CancelIO is a death sentence for whatever socket I call it on. Lame.
Tip: Use cifs instead of smbfs to mount Windows shares on Linux
Now that I have Azureus mostly running on Ubuntu, I’ve been focusing on integrating the machine into my download/watch/archive pipeline.
Back when I ran BT on a Windows XP box, I kept a drive mapped to my media share on nemes, my 2TB NAS box. Whenever a download finished, I’d copy it over to that share. Now that BT is running on Linux, I want much the same functionality.
Initially I went with NFS, since NFS is the closest thing to a native network filesystem technology on Linux, and my NAS box, being a Linux derivative itself, also supports it. However, I had alot of trouble with the UIDs not matching between nemes and aenea, such that I could copy files to the media share, but then they’d be read-only when I accessed them from Windows. Lame.
Rather than fuck about, I just switched to using Samba on Ubuntu to get at the Windows share. Lame, but I don’t give a shit.
Anyway, since it’s been at least six months since I last had to set up Samba, I forgot how to do it and Goggled about. I found a post somewhere about setting up an fstab entry to mount the Windows share with the smbfs file system driver. Something like this:
# Mount nemes via Samba
//nemes/media /usr/local/nemes/media smbfs auto,credentials=/etc/samba/nemes_creds,rw,uid=anelson 0 0
//nemes/warez /usr/local/nemes/warez smbfs auto,credentials=/etc/samba/nemes_creds,rw,uid=anelson 0 0
I tried it, it worked, yay.
But then I started getting really annoying problems. If I kept the box up for very long I’d start to get crap like this in syslog:
syslog.0:Oct 24 22:04:11 aenea kernel: [86867.358441] smb_add_request: request [ffff81000dd4de00,mid=23932] timed out!
syslog.0:Oct 24 22:10:30 aenea kernel: [87245.396073] smb_add_request: request [ffff8100205e0e00,mid=23934] timed out!
syslog.0:Oct 24 22:10:30 aenea kernel: [87245.396095] smb_add_request: request [ffff8100025d2e00,mid=23935] timed out!
And timeouts trying to copy files into the mount point for the share. If I tried to umount the volume, I’d get device busy errors.
Turns out, smbfs is deprecated in favor of cifs. I then reworded fstab to:
# Mount nemes via Samba
//nemes/media /usr/local/nemes/media cifs auto,credentials=/etc/samba/nemes_creds,rw,uid=anelson 0 0
//nemes/warez /usr/local/nemes/warez cifs auto,credentials=/etc/samba/nemes_creds,rw,uid=anelson 0 0
That still wasn’t working, though. Something about an error -13 from cifs. Turns out whitespace is now significant in the credentials file, so I edited /etc/samba/nemes_creds and removed the whitespace to the left and right of the =, and that took care of it.
Now I have stable Windows share mounts, with decent transfer performance.
Must 'Good Catholics support government-run health care'?
Via InstaPundit I ran across this piece on NRO, wherein some Catholics in Detroit are arguing that government-run healthcare in general, and the SCHP expansion in particular, are social justice/right to life issues which good Catholics must support, and that to do otherwise, like the politicians the group has targeted, is not consistent with the Church’s “right to life” doctrine.
I like to think of myself as a ‘good Catholic’, to the extent an adherent to a religion which emphasizes man’s imperfection and his need to seek redemption from God can be ‘good’. At the same time, I strongly oppose the expansion of government healthcare, even when it’s to help the wealthy white kids with their medical bills.
As far as I know, support for socialized medicine is not a requirement of Canon Law, and if it were, it would likely be lumped with opposition to the death penalty, which is an ‘authentic’ teaching against which believers can legitimately reason, unlike opposition to abortion, which is a definitive church teaching carrying the pain of excommunication for disobedience. To liken opposition to abortion with the expansion of a government entitlement which will include middle-class and wealthy children and some adults, funded by American smokers, is the kind of intellectual dishonesty I’ve sadly come to expect from my rank-and-file coreligionists.
UK police have yet another power over British subjects
The British didn’t invent strong abusive central government, but they seem intent on perfecting it. I just read this Cryptogram post about a new UK law that allows police to demand decryption keys in the course of an investigation, on pain of five years imprisonment. Not a moment too soon, apparently, as terrorists, child pornographers, drug dealers, and naughty dissident subjects are escaping police oversight using encryption technology. We can’t have subjects communicating without government supervision; who knows what they might say.
We Americans are hardly impervious to this kind of authoritarian stupidity ourselves. As Scheier reminded me, FBI director Louis Freeh made similar noises in ’93, and Clinton attorney general/boogeyperson Janet Reno often defended the then-current US policy of regulating the export of encryption technology as a munition. While we don’t have any current laws providing prison time for failure to disclose decryption keys, it’s most likely because Congress hasn’t found someone willing to pay them enough, or a tragic bit of misbehavior suitable for political exploitation (and anyway, refusal to furnish encryption keys for data subject to a warrant can easily earn you contempt of court and open-ended jail time).
For you Britons with something to hide from your benevolent masters, might I recommend TrueCrypt, which has a neat little feature whereby you can create double-encrypted disks such that, without knowing the second layer decryption key, it’s impossible to determine if a second layer of encrypted data even exist. So, you’re safe, at least until the police are given the power to electrocute the genitals of suspected enemies of the state (so-called ‘rubber hose cryptanalysis’, the most successful kind of cryptanalysis there is), at which time you’d be well advised to flee, or at least don’t get caught.