Pulled trigger on Verizon FiOS
Today I finally submitted the work order to have Verizon FiOS installed at my house. For the last several years I’ve had a 1.6Mbps down/768Kbps up ADSL account from Speakeasy, which I love, but I just can’t stand the slowness anymore. I currently pay nearly $90/mo for my Speakeasy account (including wide-open TOS, two static IPs, and other Speakeasy-specific goodness), while Verizon will sell me 30Mbps down/5Mbps up fiber for $55/mo. Sure, it’s Verizon, Evil Empire and all that, but at the end of the day it’s the performance that matters to me.
The shitty thing about FiOS is that it’s an all-or-nothing proposition. Part of the installation process is disabling your copper lines, so you can’t have DSL side-by-side with FiOS, and once you go w/ FiOS it’s hard/impossible to go back. So, needless to say, I really hope it works out okay.
I’ll lose my static IPs, of course, but I use FreeDNS anyway, so I’ll just set up dynamic updates and not give it a second thought.
Fog Creek Copilot is pretty cool
Fog Creek Software’s Copilot service is one of those might-come-in-handy-someday services I’ve had filed in the back of my mind for a while now, but never actually had to use. Yesterday I had to help my grandfather with a problem on his computer, and knew I didn’t want to spend hours having him read me dialog boxes and click buttons in the hopes things would just work out.
If your grandfather is anything like mine, walking him through setting up an SSH tunnel through his router to a VNC server on his desktop is about as unlikely to succeed as running Visual Studio 2005 for eight contiguous hours, and just as frustrating. Fortunately, I remembered Copilot, and the awesome usability courtesy the influence of Joel Spolsky, who may be self-righteous and preachy at times, but definitely knows how to make software.
I bought a $5 day pass (it should be noted at no point was I required to register, create a password, or jump through any of the other bullshit hoops I otherwise would have if the service was run by any other company), got a 12-digit invitation code, and downloaded a ~750KB client w/ my invitation code hard-coded in it. I fired up the client, which connected to Copilot’s servers and waited for Grandpa to connect.
On his end, I directed him to copilot.com (thank God his Internet connection was working), read him the invite code, and walked him through the one-click download and run (no install; another Spolsky innovation); automatically, just like that, he was in, prompted to allow me to connect, and I had remote control of his system. He has dual monitors, and I saw them as one large contiguous desktop, and could drag windows between screens without difficulty. When I was done, the client software offered to delete itself. Slick.
This is definitely worthwhile for cases when you need to do a little remote support, and for reasons of user sophistication, firewalls, or both, you can’t set up your own SSH and VNC/Remote Desktop connection. Currently at AppAssure where I work we use GoToMeeting for remote support tasks, and I think Copilot would make for a much better solution (plus the new version has some basic branding features, so we could even brand it with our logo and domain name).
__declspec(thread) bites me in the ass yet again
Sometimes I find myself writing code that needs thread-local variables; that is, variables that have a separate value in every thread in the process. This is handy when guarding against recursion while preserving thread-safety, for example.
In the latest case, I have some logging code, and if the logging fails, it uses the same logging code to report the failure (the code is based on log4cplus, where each log attempt writes to multiple outputs, so the logic isn’t as stupid as it sounds). Obviously, if the attempt to log the failure itself fails, I don’t want the code to keep logging meta-failures forever, so I need a bool that says “You’re logging an error in logging, so if you fail don’t try to log that failure”.
However, I’m writing code that runs in a server with multiple concurrent threads, and I’d hate to miss log events written from other threads that happen to take place while my bool is set, so I want the bool to be thread local, so each thread has its own copy of it.
Windows provides the TLS API for this purpose, but Visual C++ makes it so much easier with the __declspec(thread) modifier, which automatically does the TLS stuff under the covers. It’s so easy that I often forget it has a huge caveat.
If a DLL declares any nonlocal data or object as _declspec( thread ), it can cause a protection fault if dynamically loaded. After the DLL is loaded with LoadLibrary, it causes system failure whenever the code references the nonlocal _declspec( thread ) data. Because the global variable space for a thread is allocated at run time, the size of this space is based on a calculation of the requirements of the application plus the requirements of all the DLLs that are statically linked. When you use LoadLibrary, there is no way to extend this space to allow for the thread local variables declared with __declspec( thread ). Use the TLS APIs, such as TlsAlloc, in your DLL to allocate TLS if the DLL might be loaded with LoadLibrary.
Did you get that? If you’re writing code that might be compiled to a DLL, and that DLL might want to someday be loaded with LoadLibrary, don’t even think about using __declspec(thread). As it happens, that’s exactly the case I’m facing. Fuck!
Review of Infrant ReadyNAS NV+
As promised, I’ve got my hands on an Infrant ReadyNAS NV+, which is a small NAS appliance capable of holding up to four SATA hard drives, each up to 750GB large.
Background
For those of you who don’t know, a NAS, or Network Attached Storage, is a device on a network that provides storage space for other nodes on the network. If you’ve ever used a file share, you probably used NAS.
Before the ReadyNAS NV+, my NAS box was a regular PC (aenea) with five 250GB hard drives and a RAID card, adding up to 1TB of storage which, through the magic of RAID, could survive the failure of one of those disks without losing any data. I was fine w/ this for a while, but I recently filled aenea to capacity with my movies, photos, music, software collection, and documents. If 1TB wasn’t enough, what is?
I found my answer in the ReadyNAS NV+. With four 750GB hard drives and its proprietary X-RAID, I get 2.25TB of redundant storage, just as I would with standard RAID 5, plus the unique benefits of X-RAID, which include the ability to automatically plug in bigger drives to grow the volume size, and easy recovery from a bad disk.
Sourcing
I bought the NV+ itself (specifically the RNV2-S4-0000 model with no pre-installed disks and 1GiB of RAM) from eAegis for $740.00 including shipping. I got four 750 GB Seagate Barracuda ES drives (which is one of two 750GB variants on Infrant’s hardware compatibility list) from Newegg.com for $1069.00, with free shipping. At the same time I picked up a NetGear 8 port GigE switch, since the five-port Linksys I had didn’t support Jumbo Frames, a key performance enhancement for bulk file transfers over GigE LAN segments. Total cost for the project was thus under $2k, compared to well over $2k for my previous 1TB NAS project.
Unpacking
My first impression of the NV+ when I opened the box was ‘damn, that _is_ small!’. The box really is about the size of a toaster, which is remarkable considering its storage capacity potential. The case has a clean industrial design, with a polished chrome look and a minimalist front panel, as befits a plug-and-play storage appliance.
Setup
While the box was in transit to me I had read the docs and already decided I would allow it to use the default X-RAID mode rather than force a standard RAID 5 (it’s a black-box storage appliance, so what do I care what kind of proprietary RAID mode it uses?), so I just installed the four SATA drives in the provided empty drive trays using the provided drive screws, plugged the box into GigE and power, and turned it on.
When the NV+ boots for the first time, it installs its firmware into flash memory, then initializes the disks. There is a brief window in which you can override its default selection of X-RAID across all installed disks, but I ignored it as X-RAID is fine with me. The NV+ goes through a roughly two-hour volume creation process during which it is not accessible via the web-based management console, and the RAIDar utility used to locate the NV+ on the network and invoke the setup utility refuses to launch setup.
After the volume is created, the NV+ then starts a resync, which like normal RAID takes forever (roughly 18 hours in my case). However, unlike volume creation, the NV+ is operable during resync, though obviously at considerably reduced performance. I immediately christened the new addition to the adamnet family as nemes (if you picked up the dual Hyperion quartet references, kickass), deleted the two default shares (media and backup) and set up my own.
The share security options is one area in which my custom-built 1TB NAS was superior. I had my old NAS set up rather simply, with a anelson$ hidden Windows share for my personal stuff, then a filestor share with stuff like photos, movies, music, and warez. However, since aenea ran Samba, I was able to setup two access levels on filestor: a guest level that would allow anyone read-only access to the share, and an authenticated level that would allow only users with my credentials to write to the share. That way, I could let anyone access my stuff, but I didn’t have to worry about them (or malware on their machines) deleting my precious stuff.
The NV+ doesn’t have this flexibility. You can use user-level security, wherein you create users and passwords on the NV+ and assign them access permissions, or share-level security wherein you can optionally set a share-level password but otherwise cannot control share access. This isn’t a deal-breaker for me, but I do with there were more granularity here.
UPDATE: I figured out how to achieve the above arrangement with the NV+. Though the docs don’t make it obvious, when enabling user-mode security there is still a guest access option, and for each share you can set the default access level, and give specific users/groups read or write access. With user security turned on, each user gets their own home directory as well, so I can delete the anelson share I created and use the home directory for anelson instead. Too bad I already copied 110GB of stuff up to anelson.
rsync
While the NV+ was still resyncing, I enabled rsync on a couple of the shares (anelson for my files and video for my movies) and started to rsync over the relevant directories from my old 1TB NAS. Initially I ran into trouble because I expected the NV+ to expose the shares as directories off the root, so I used an rsync command like this:
rsync -avz /usr/local/filestor/video nemes:/video
However this wasn’t too well received:
ssh: nemes: hostname nor servname provided, or not known
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(443)
The first problem was that the NV+ doesn’t sent a host name when it requests an IP address via DHCP sends a DHCP lease request on boot only, so if you boot it with the default hostname (which is derived from the MAC address), then change the hostname, it won’t update its DHCP lease again until you reboot. So when I set the hostname, then found I couldn’t resolve that hostname to an IP, at first I thought the ReadyNAS wasn’t sending a hostname; it wasn’t until I noticed it’s old hostname in my DNS server that I figured out what happened, rebooted the NV+, and was able to resolve its hostname to an IP.
Anyway, perhaps this will work:
rsync -avz /usr/local/filestor/video 192.168.1.126:/video
Hmm, no.
anelson@192.168.1.126's password:
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(443)
It got further this time, in that I was prompted for a password, but it still failed.
Some rummaging around on the Infrant support forum turned up the problem, which may be hidden somewhere in the docs but not obvious enough for me: the NV+ exposes shares as rsync modules, not directories on a filesystem. rsync modules are a little like web aliases in web servers: you access them by their module names, and under the covers rsync maps them to an actual location on the filesystem. The correct syntax is thus:
rsync -avx /usr/local/filestor/video 192.168.1.126::video
Note that instead of :/video, I use ::video, where :: is the module delimiter and video is the name of the module/share name.
Once I got that figured out, rsync was easy, though it ran like a three-legged dog since most of the NV+’s resources were dedicated to the resync.
Share setup
As noted above, the first thing I did w/ the NV+ setup was delete the default media and backup shares. Once I spent 48 hours rsyncing my 250GB of videos to my cleverly-named video share on nemes, I began to understand the depth of my fuckup.
You see, the NV+ supports UPnP AV, iTunes, Network DVD, and SlimServer streaming services, to expose photos, movies, and music to media appliances elsewhere on your network. I obviously want to take advantage of this (I don’t have any such devices yet, but I surely will before I retire nemes), but I found out too late how the NV+’s functionality is limited. When enabling the various streaming services, you can configure exactly one share as the source of the media. So, if I want to expose my music, photos, and video via UPnP AV, they must all be on the same share, not scattered across multiple shares. Given this, it’s rather obvious why the NV+ defaults to a single media share. Doh!
This exposes another shortcoming in the NV+’s design: there’s no (obvious) way to move files from one share to another. If this were a UNIX box, I’d just ssh in and do a mv, but the NV+ doesn’t provide an interactive shell environment. (NB: After I’d already deleted my video share and resigned myself to copying my videos back up a second time to the media share, I found this post which says you can use the backup functionality to copy between shares. That will still take alot longer than moving files since it involves a copy, and takes up double the disk space until the copy completes, but it’s almost certainly faster than re-uploading everything a second time).
Anyway, as penance I set up a media share with folders for video, music, and photos. Talk about the school of hard knocks.
Shitty rsync gotcha
Once I figured out I could have the security functionality I wanted by switching to user security, I deleted the anelson share I’d created for my files, and created an anelson user instead, which automatically generates a private share called anelson that only I can access. Then I find this post, which hit me with one helluva ‘oh by the way’:
Currently, home shares are only exported over CIFS. The next release will make them available over NFS, however.
I call bullshit! Why the hell can’t I rsync to my home share!? Fuck! Now I have to copy all of my home directory files via slow-and-shitty Windows file sharing. Uncool! Dammit Infrant, fix this! I’ve posted a feature request, but I’m not holding my breath.
Clunky Web Admin Interface
If you were a late 90′s sysadmin suddenly transported forward through time to 2007 and made to use the NV+ web admin console, you’d feel right at home. No AJAX goodness, no rich UIs, just tons of page refreshes and barely-responsive controls. It works, and it’s more or less straightforward, but it’s no pleasure to use.
Overall
The NV+ is easy and cheap and makes for a ton of storage space. It has some warts, some of which will hopefully be addressed in a future firmware release, but for the price it’s still a good value. Get one, and never again be forced to choose between keeping your Alias season 1 DVDrips and downloading Star Trek: Deep Space Nine season four.
Bullshit Visual C++ 2005 regression bug
I’m working on building a Win32 static library version of zlib, which uses some .asm files. One of these files, inffas32.asm, contains this line (line 647):
movd mm4,[esp+0]
This line fails to assemble with the following error:
error A2070: invalid instruction operands
I only found one mention of this problem with zlib, an incoherent blog post here. It links to a Microsoft bug report which has this to say:
Microsoft Macro Assembler 8.0, included with Visual C++ 2005 Express Beta 1, refuses to assemble some instructions that were valid under MASM 7.1 included with Visual Studio .NET 2003. Specifically, it refuses to assemble a MOVD instruction with a memory operand with an implied size, and requires that “dword ptr” prefix the memory operand. Although this is ambiguous under AMD64, it is not ambiguous in IA32 and should not be an error, particularly as this behavior is a regression from 7.1 and breaks existing valid source code.
The resolution?
this isnt a bug, rather its syntax that was previously allowed that should not have been. The “fix” here is to use the correct operand sizes in the syntax.
Goddamit, that’s helpful. Sure, the code is technically wrong. But it’s been technically wrong since at least Visual Studio 6.0 some eight years ago, and has worked all this time. You’d think there’d be at least a ‘heads up, we’re about to fuck you’ note somewhere. So now I guess I have to go through and change all the movd instructions to use explicit pointer sizing. Fuck.
UPDATE
It seems there are already Visual Studio 2k3 and 2k5 project files in the contrib\vc7 and contrib\vc8 folders respectively in the zlib source tarball. They seem to work quite a bit better. Still, that doesn’t get MS off the hook for a real dick move.
Incoming: 2.25TB of RAID storage!
I’ve previously noted with horror that I’ve exhausted most of the 1TB of RAID 5 storage on aenea, with a deluge of video and picture files.
This week I’m in Colorado for my brother’s graduation, and have taken the time to select and order a new NAS solution.
I ended up selecting the Infrant ReadyNAS NV+. I ordered the 1GB RAM version with no disks from eAegis for about $700, and four 750GB Baracuda SATA drives from newEgg for under $1200. Once I put it all together, I’ll have 2.25 TB of available storage capacity.
I like the ReadyNAS coz it’s a small appliance, it supports UPnP AV, SlimServer, and Network DVD, does jumbo frames, and has external USB ports for USB backup. It does rsync so I can easily migrate my existing store over, and perform backups to my dreamhost account.
I’ll post again when I get it going.
WTF!? I'm out of disk space already
Just as I was going to copy season 3 of Highlander to aenea, my 1TB NAS system, I ran out of free space on the /usr volume, to which the bulk of the storage had been allocated. Nearly 850GB, gone in a little over a year. Sure, I could clean house a bit and make a little room, but the sad fact is, I need more storage.
I haven’t made a build vs buy decision yet, but I will point out that there seems to be a blind spot in NAS offerings between 1TB and 3TB; plenty go up to 1TB, plenty pick up at around 3TB, but where’s the midrange?
Annoying gotcha w/ _m128
I ran into a nasty gotcha w/ the Visual C++ intrinsic type __m128 today. __m128 is a 128-bit integer which is used with the Intel SSIMD extensions, which allow fast integer math on 128-bit registers. SSIMD was created for games and video, but the code I was using (not written by me) was just taking advantage of the additional bits to compute a faster ECC.
The first problem is that 128-bit integers must be aligned on a 16 byte memory boundary. This is intuitively obvious, as the x86 architecture likes (but in some cases doesn’t require) all the multi-byte integer types to be aligned on a memory boundary equal to their size. You wouldn’t have a 32-bit integer at memory address 0x3, so why would you expect a 128-bit integer at memory address 0x9 to work?
You wouldn’t, and neither would I, but it’s not that simple. The code I was dealing with had __m128 members nested deep within multiple levels of structs. My problem was I was creating a BYTE array, reading in some data from a file, casting it to this struct, then operating on the struct. Ouch.
What threw me off is the exception I got when accessing the __m128 variables. I would’ve expected an exception code indicating an alignment problem, which would’ve immediately clued me in. What did I get instead? Exception 0xc (access violation) at memory address 0xffffffff. WTF!? I was hunting for a pointer to 0xffffffff in vain for several minutes before I dropped to the disassembly view and saw a SSIMD equivalent of a mov, which led me to look up the __m128 type in the first place.
In the end I was able to add padding to my BYTE array to ensure I started the read at the proper memory boundary, but that was crufty so I just changed my I/O code to use the struct itself, and the compiler took care of the alignment.
Dammit, why is programming so hard all the time?
AppAssure Software Replay 1.5 Released
This past Monday, my company released the first version of our product, AppAssure Replay(tm). It’s been a long hard slog to get it out the door, and we were tweaking right up until the end.
In a nutshell, Replay protects Microsoft Exchange servers by taking snapshots of them at intervals (15 minutes by default), and allows you to restore to any of those points in time. What’s unique about Replay is granularity: you can roll back the entire machine (using our customized Windows PE boot environment), specific volumes, specific Exchange message stores, specific mailboxes, or even specific messages. Soup to nuts, as they say.
My dev team has already started work on the next release, which will incorporate additional bits of goodness, plus inevitable bug fixes. A dev’s work is never done.