"Gem::SourceIndex#search support for Regexp patterns is deprecated" from Rails after upgrading to RubyGems 1.3.0
In my attempts to get Ruby on Rails running on my FutureHosting VPS I ran into the following warning/error attempting to run the RoR script/generate controller command:
Gem::SourceIndex#search support for Regexp patterns is deprecated
/usr/lib/ruby/gems/1.8/gems/rails-2.1.1/lib/rails_generator/lookup.rb:211:in `each' is outdated
Apart from this Rails seems to work fine. This didn’t start until I upgraded to RubyGems 1.3.0, which (unbeknownst to me at the time) was just released a few days ago. From the release notes:
Other Changes Include:
…
- lib/rubygems/source_index.rb
- Deprecate options to ‘search’ other than Gem::Dependency instances and issue warning until November 2008.
I assume the above deprecation is causing the Gem::SourceIndex#search support for Regexp patterns is deprecated warning. So, is Rails using a deprecated Gem function? If so, why hasn’t Rails been updated accordingly? I’ve seen nothing about this on Google, and I’m running the latest Rails as of now, so I must assume I have to suffer in (relative) silence until an RoR update brings Rails into compliance. Since the message doesn’t seem to impact Rails’ functionality it’s a mere nuisance, but nuisance nonetheless.
If anyone has more info or a fix (apart from upgrading to the new RoR when it becomes available), please leave a comment.
Moving to Future Hosting, Part IV
This installment was supposed to be about getting ffmpeg built with all the necessary libraries, but I ran into something else that pre-empted that topic. As I attempted to do an svn checkout of the ffmpeg source tree on lio, SVN kept failing with a timeout. I recall Yousif claiming to have this problem on his FutureHosting VPS box, which I wrote off as User Error, but this was too much of a coincidence. I tried the same svn checkout on my personal machine, and it worked fine, confirming the ffmpeg SVN server was indeed up.
I immediately suspected a firewall issue, since I did opt for the free security lock-down service on setup of my account. I know outbound HTTP requests work as I’ve done wget alot already, but maybe the firewall allows outbound connections on known ports only. To answer the question, I sought out the firewall configuration, which was easier said than done.
At first I looked at /etc/sysconfig/iptables-config, but it was just the stock configuration. Then I remembered reading in one of the welcome docs from FutureHosting that they installed and configured the APF firewall. I looked into that a bit, and found that it’s config files are in /etc/apf. However, the config files are much more complicated, and I couldn’t find any rules passing outbound http, or blocking other outbound traffic.
Then I went to the Virtuozzo Power Panel for lio, which had a ‘Firewall’ link on it. Unfortunately, the firewall configuration left a bit to be desired. Nothing in the output indicated that rules were in force at the time, so I couldn’t tell what the problem was. Therefore, I ran the firewall setup, and chose to use the ‘Normal’ firewall rules, and checked the box to reset the firewall configuration. After that I got a list of stock rules that were in place, like ‘Allow Outbound Connections’ (this was enabled after the reset), ‘Allow DNS’, ‘Allow POP3′, etc. I deleted the rules for services I’m not hosting on lio, but left ‘Allow Outbound Connections’ enabled.
After that, lo! and behold, svn checkout worked. The lesson there is FutureHosting’s locked-down firewall config is locked down, to the point of uselessness, and there’s no obvious way to tune the lockdown params; you just reset them all. Nice.
I’m sure I could’ve opened a support ticket to get this resolved, but I want to know how this stuff works on my VPS box. I must admit to being no closer to an understanding, I still can’t match the firewall rules displayed in the Power Panel to the /etc/apf/ config files. I guess the important thing is that it’s working now.
Moving to Future Hosting, Part III
In this episode of Moving to FutureHosting, I’ll be installing and configuring Apache2, the mpm-worker multiprocessing module, passenger (aka mod_rails), Ruby, Ruby on Rails, and all associated cruft. I’ve done this before on Ubuntu using aptitude, but since FutureHosting runs CentOS, which uses the yum package manager, this will be a new experience for me.
Apache2
My FutureHosting box came with Apache2. /usr/sbin/httpd -v reports Server version: Apache/2.2.9 (Unix), which seems fine.
Which MPM?
Apache2 supports multiple concurrency models for handling more than one connection at once. Each model is implemented in a Multi-Processing Model (MPM). prefork-mpm causes Apache to fork off a number of processes on startup, each of which will process some requests. worker-mpm creates multiple worker threads within a process to handle requests. I’ve read that the latest versions of Passenger work well with worker-mpm, which has the added advantage of being less memory-hungry than a bunch of forked processes.
I don’t know how to tell what MPM the existing install is using. I grep‘d all the LoadModule directives in the HTTP configuration files in /etc/httpd/conf/ but I don’t see anything pertaining to an mpm. /usr/sbin/httpd -l lists prefork.c but no worker.c. This tells me the version of Apache that comes with CentOS 5 is built without support for worker-mpm. Ubuntu has handy aptitude packages for both MPMs, so you install the one you want and it configures Apache accordingly. CentOS seems to be missing that. I’ll go with prefork for now until I can figure out how to change it.
Ruby
The latest version of Ruby available as a yum package is 1.8.5-5. Um, no. That’s ancient. That means I have to build from source. Out-fucking-standing. Why does anyone run CentOS anyway?
No matter, the forum thread here pointed me to a custom repository some guy is running, with the latest Ruby 1.8.6 bits. Pathetic that it came to that. I installed ruby, ruby-devel, ruby-irb, ruby-mysql, and ruby-rdoc.
Ruby Gems
I installed Ruby Gems from the same source as above, which then pulled in a dizzying dependency chain of additional packages, nearly 60 (!!) in all. The installation seemed to work, though, so I can’t complain.
Ruby on Rails
Rails is easy. sudo gem install rails. QED.
Passenger
Phusion Passenger runs Rails apps in Apache fast and reliably. What more is there to say?
It’s distributed as a Ruby gem, so
sudo gem install passenger
And Passenger is installed. But it needs to be configured as a module in Apache2. For that there’s
sudo passenger-install-apache2-module
Which builds the Apache module from source, and spits out the appropriate verbiage for httpd.conf. The httpd configuration stuff is in /etc/httpd/conf, which includes files in /etc/httpd/conf/extras. In keeping with this idiom, I created a new file, /etc/httpd/conf/httpd-passenger.conf, to contain the Passenger-related configuration:
LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-2.0.3/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.0.3
PassengerRuby /usr/bin/ruby
which I include in /etc/httpd/conf/httpd.conf using an Include directive.
After that, sudo apachectl restart and the module was loaded.
Ruby Enterprise Edition
Phusion (makers of Passenger) offer an ‘enterprise’ edition of the Ruby interpreter that works with Passenger to reduce memory load and increase performance. I’ve never used it before, but I’d like to give it a try.
After downloading and extracting the source tarball, there’s no familiar configure/make/make install cycle. Instead there’s an installer script that takes you through the installation.
I accepted the default install directory (/opt/ruby-enterprise-1.8.6-20080810), and off it went building from source. I don’t have any Postgres components installed, so it choked trying to install Postgres support, which I don’t give a shit about. I can always install it later with /opt/ruby-enterprise-1.8.6-20080810/bin/ruby /opt/ruby-enterprise-1.8.6-20080810/bin/gem install postgres.
And that was it. To cause Passenger to use Ruby EE, I changed the PassengerRuby line in the httpd-passenger.conf file to:
PassengerRuby /opt/ruby-enterprise-1.8.6-20080810/bin/ruby
Too easy!
Testing
To verify Passenger and Ruby EE are working, I need a Rails app to serve. I’ll create one in my home directory and make Apache use that as the root for the default VirtualHost:
rails passtest
to create the project.
script/generate controller itworked
failed with:
Rails requires RubyGems >= 1.1.1 (you have 0.9.4). Please `gem update --system` and try again.
Yikes! I had no idea I was running ancient gems. sudo gem update --system made quick work of that problem.
Now, as I was saying,
script/generate controller itworked
It spit out an annoying error:
/usr/lib/ruby/gems/1.8/gems/rails-2.1.1/lib/rails_generator/lookup.rb:211:in 'each' is outdated
but otherwise seemed to work.
Now ~anelson/passtest/public is the public root for my rails app, which I need to point Passenger to.
For now I’ll just use one of the VirtualHost entries in /etc/httpd/conf/extras/httpd-vhost.conf, since I’m just testing.
And…it worked! I pointed DocumentRoot to the public folder in my Rails project, and got the default Rails screen. It seems to be working. Yay.
Absolutely bullshit Ruby HTTP client situation
For one of my self-edification projects I’ve been trying to implement some very simple HTTP client code in Ruby, which I want to use to transfer hundreds or thousands of megabytes of data from an HTTP server. Since I need to transfer large blocks of data, I need a streaming HTTP API that allows me to read the data in small chunks so as not to exhaust available memory buffering reads. This is a very common idiom, and I was not surprised to find that Ruby’s built-in HTTP implementation, Net::HTTP, had just such a function.
Imagine my surprise, then, when I ran the following code:
#!/usr/bin/ruby -w
require 'net/http'
require 'logger'
BLOCK_SIZE = 1024*128 #128 K
REMOTE_URL = "http://wdc01.futurehosting.biz/test100.zip"
logger = Logger.new(STDOUT)
logger.debug("Parsing URL #{REMOTE_URL}")
url = URI.parse(REMOTE_URL)
logger.debug("Starting HTTP session with host #{url.host}, port #{url.port}")
Net::HTTP.start(url.host, url.port) do |http|
logger.debug("Sending HTTP request for path #{url.path}")
http.request_get(url.path) do |response|
logger.debug("Processing response")
logger.debug("Response headers: #{response.to_hash.to_s}")
logger.debug("Rendering response")
File.open('foo', 'wb') do |file|
response.read_body do |body|
logger.debug("Sending back #{body.length} bytes of response data")
if (file.respond_to?(:syswrite))
file.syswrite(body)
else
file.write(body)
end
end
end
end
end
What’s supposed to happen here is the 100MB test file is read in manageable chunks, thus memory and CPU very minimally used as the transfer proceeds. What actually happens is the CPU redlines, and the data are transferred in 1024 byte chunks. That’s right, there’s hard coded 1K chunk size in Ruby’s HTTP implementation, which means there’s tons of CPU overhead executing Ruby methods and socket system calls.
Don’t believe me? Think I’m some Ruby n00b, or maybe a Ruby hater? Read this thread. The combined might of the ruby-talk list was not sufficient to solve this guy’s problem. He came up with a hack that monkeyed with the built-in Net::HTTP classes to force a larger buffer size, but that hack broke in Ruby 1.8.5, and there’s no fucking way I’m going to go through that sort of contortion to make a craptastic library usable.
I continue to be amazed at the various areas in which Ruby is immature and vastly inferior to alternatives like Python. Still, every time I swear it off, I end up coming back after being reminded how much I hate using Python, and how refreshing Ruby’s dynamic duck-typing idioms are after a hard day of C# and C++. So, with fuck it no longer an option, I had to find an alternative.
There are other HTTP clients, but they built on top of Net::HTTP, which is not helpful. This led me to rfuzz, which is a fuzz testing tool for web apps. One nice feature of this tool is very light HTTP client library written atop the low-level TCPSocket object, and thus suffering none of Net::HTTP‘s dreadful performance. This client was made for a very different purpose, and had no streaming response abilities that I could find, but by subclassing the HttpClient class I was able to adapt it to my needs with minimal effort.
Now, I can stream a large file with a single-digit CPU hit, and I can control the buffer size to my liking.
The longer I’m a software developer, it seems the more I find myself irritated as the kind of low-level wheel-reinventing I’m constantly stuck with, for no better reason than software isn’t as far along as it should be by now. Maybe I was born too early after all.
UPDATE: In a shockingly uncharacteristic attempt to light a candle, I’ve done a detailed analysis of Ruby 1.8 HTTP client performance, complete with pretty pictures.
Moving to Future Hosting, Part II
The saga continues. In the last installment, I talked about the signup and initial use experience. This quick post is about performance.
I ran the UnixBench 4.1.0 WHT variant on both of my VPS boxes at various times, to see how they perform.
My two VPS boxes, lio and raz (Yes, I’m on an Anathem kick right now), are both brand-new FutureHosting ‘Elite’ VPS accounts. I’ve not done any optimization, nor are either of them hosting any sites with any traffic.
I ran the test with no arguments, then with 8 to make it utilize all eight virtual cores. I repeated this for both boxes. Then I ran the test simultaneously in both boxes, so see if they’re on the same node and if that degraded performance. The results are below.
Summary
lio turns in a WAY better score with 8 threads than raz, though they’re supposed to be identical. As in, 150 for raz, 250 for lio. Why the difference? I don’t know. Still, both of these scores are respectable for $35/mo, imho.
Raz, default thread count
System -- Linux raz.bulshytt.com 2.6.9-023stab046.2-enterprise #1 SMP Mon Dec 10 15:22:33 MSK 2007 i686 i686 i386 GNU/Linux
/dev/vzfs 41943040 1300448 40642592 4% /
Start Benchmark Run: Thu Sep 25 19:11:34 CDT 2008
19:11:34 up 9:45, 1 user, load average: 0.13, 0.03, 0.01
End Benchmark Run: Thu Sep 25 19:23:12 CDT 2008
19:23:12 up 9:57, 1 user, load average: 17.11, 6.92, 2.98
INDEX VALUES
TEST BASELINE RESULT INDEX
Dhrystone 2 using register variables 376783.7 13403469.3 355.7
Double-Precision Whetstone 83.1 1345.6 161.9
Execl Throughput 188.3 3856.0 204.8
File Copy 1024 bufsize 2000 maxblocks 2672.0 55015.0 205.9
File Copy 256 bufsize 500 maxblocks 1077.0 150.0 1.4
File Read 4096 bufsize 8000 maxblocks 15382.0 303300.0 197.2
Pipe Throughput 111814.6 604792.1 54.1
Pipe-based Context Switching 15448.6 178394.1 115.5
Process Creation 569.3 12508.3 219.7
Shell Scripts (8 concurrent) 44.8 1044.8 233.2
System Call Overhead 114433.5 1207185.3 105.5
=========
FINAL SCORE 107.7
Raz, with 8 threads
BYTE UNIX Benchmarks (Version 4.1-wht.2, 8 threads)
System -- Linux raz.bulshytt.com 2.6.9-023stab046.2-enterprise #1 SMP Mon Dec 10 15:22:33 MSK 2007 i686 i686 i386 GNU/Linux
/dev/vzfs 41943040 1300468 40642572 4% /
Start Benchmark Run: Thu Sep 25 19:26:07 CDT 2008
19:26:07 up 10:00, 1 user, load average: 0.95, 3.88, 2.48
End Benchmark Run: Thu Sep 25 19:37:34 CDT 2008
19:37:34 up 10:11, 1 user, load average: 26.39, 10.88, 5.65
INDEX VALUES
TEST BASELINE RESULT INDEX
Dhrystone 2 using register variables 376783.7 22294025.3 591.7
Double-Precision Whetstone 83.1 1187.7 142.9
Execl Throughput 188.3 4524.2 240.3
File Copy 1024 bufsize 2000 maxblocks 2672.0 60357.0 225.9
File Copy 256 bufsize 500 maxblocks 1077.0 1513.0 14.0
File Read 4096 bufsize 8000 maxblocks 15382.0 286290.0 186.1
Pipe Throughput 111814.6 1282962.2 114.7
Pipe-based Context Switching 15448.6 179541.3 116.2
Process Creation 569.3 12956.3 227.6
Shell Scripts (8 concurrent) 44.8 1321.5 295.0
System Call Overhead 114433.5 1134031.5 99.1
=========
FINAL SCORE 152.9
lio, with default threads
BYTE UNIX Benchmarks (Version 4.1-wht.2)
System -- Linux lio.bulshytt.com 2.6.9-023stab046.2-enterprise #1 SMP Mon Dec 10 15:22:33 MSK 2007 i686 i686 i386 GNU/Linux
/dev/vzfs 34207160 1300636 5728564 19% /
Start Benchmark Run: Thu Sep 25 22:37:29 CDT 2008
22:37:29 up 13:08, 1 user, load average: 0.13, 0.04, 0.01
End Benchmark Run: Thu Sep 25 22:48:21 CDT 2008
22:48:21 up 13:19, 1 user, load average: 14.59, 6.19, 2.72
INDEX VALUES
TEST BASELINE RESULT INDEX
Dhrystone 2 using register variables 376783.7 21966322.3 583.0
Double-Precision Whetstone 83.1 1394.1 167.8
Execl Throughput 188.3 4554.2 241.9
File Copy 1024 bufsize 2000 maxblocks 2672.0 342.0 1.3
File Copy 256 bufsize 500 maxblocks 1077.0 18094.0 168.0
File Read 4096 bufsize 8000 maxblocks 15382.0 27651.0 18.0
Pipe Throughput 111814.6 1277497.8 114.3
Pipe-based Context Switching 15448.6 384725.3 249.0
Process Creation 569.3 13990.8 245.8
Shell Scripts (8 concurrent) 44.8 1006.5 224.7
System Call Overhead 114433.5 1249871.1 109.2
=========
FINAL SCORE 104.2
lio, with 8 threads
BYTE UNIX Benchmarks (Version 4.1-wht.2, 8 threads)
System -- Linux lio.bulshytt.com 2.6.9-023stab046.2-enterprise #1 SMP Mon Dec 10 15:22:33 MSK 2007 i686 i686 i386 GNU/Linux
/dev/vzfs 34354440 1300704 5875776 19% /
Start Benchmark Run: Thu Sep 25 23:44:28 CDT 2008
23:44:28 up 14:15, 1 user, load average: 0.00, 0.02, 0.39
End Benchmark Run: Thu Sep 25 23:55:50 CDT 2008
23:55:50 up 14:26, 1 user, load average: 28.57, 11.10, 4.88
INDEX VALUES
TEST BASELINE RESULT INDEX
Dhrystone 2 using register variables 376783.7 35652954.9 946.2
Double-Precision Whetstone 83.1 1362.3 163.9
Execl Throughput 188.3 4617.4 245.2
File Copy 1024 bufsize 2000 maxblocks 2672.0 92159.0 344.9
File Copy 256 bufsize 500 maxblocks 1077.0 30916.0 287.1
File Read 4096 bufsize 8000 maxblocks 15382.0 705176.0 458.4
Pipe Throughput 111814.6 2252648.2 201.5
Pipe-based Context Switching 15448.6 404533.9 261.9
Process Creation 569.3 12242.2 215.0
Shell Scripts (8 concurrent) 44.8 1070.5 239.0
System Call Overhead 114433.5 1826626.0 159.6
=========
FINAL SCORE 277.4
raz and lio, running at the same time, with 8 threads
raz
BYTE UNIX Benchmarks (Version 4.1-wht.2, 8 threads)
System -- Linux raz.bulshytt.com 2.6.9-023stab046.2-enterprise #1 SMP Mon Dec 10 15:22:33 MSK 2007 i686 i686 i386 GNU/Linux
/dev/vzfs 41943040 1300592 40642448 4% /
Start Benchmark Run: Thu Sep 25 23:52:53 CDT 2008
23:52:53 up 14:26, 2 users, load average: 0.01, 0.02, 0.00
End Benchmark Run: Fri Sep 26 00:04:31 CDT 2008
00:04:31 up 14:38, 2 users, load average: 30.74, 11.70, 4.89
INDEX VALUES
TEST BASELINE RESULT INDEX
Dhrystone 2 using register variables 376783.7 29430207.6 781.1
Double-Precision Whetstone 83.1 1298.8 156.3
Execl Throughput 188.3 4788.6 254.3
File Copy 1024 bufsize 2000 maxblocks 2672.0 4551.0 17.0
File Copy 256 bufsize 500 maxblocks 1077.0 26223.0 243.5
File Read 4096 bufsize 8000 maxblocks 15382.0 15476.0 10.1
Pipe Throughput 111814.6 1254858.6 112.2
Pipe-based Context Switching 15448.6 185965.8 120.4
Process Creation 569.3 13295.8 233.5
Shell Scripts (8 concurrent) 44.8 772.5 172.4
System Call Overhead 114433.5 1508631.3 131.8
=========
FINAL SCORE 122.5
lio
BYTE UNIX Benchmarks (Version 4.1-wht.2, 8 threads)
System -- Linux lio.bulshytt.com 2.6.9-023stab046.2-enterprise #1 SMP Mon Dec 10 15:22:33 MSK 2007 i686 i686 i386 GNU/Linux
/dev/vzfs 34328540 1300728 5849852 19% /
Start Benchmark Run: Thu Sep 25 23:56:24 CDT 2008
23:56:24 up 14:27, 1 user, load average: 16.16, 9.93, 4.72
End Benchmark Run: Fri Sep 26 00:09:12 CDT 2008
00:09:12 up 14:39, 1 user, load average: 29.57, 12.20, 6.97
INDEX VALUES
TEST BASELINE RESULT INDEX
Dhrystone 2 using register variables 376783.7 33713810.3 894.8
Double-Precision Whetstone 83.1 1357.9 163.4
Execl Throughput 188.3 4615.5 245.1
File Copy 1024 bufsize 2000 maxblocks 2672.0 21267.0 79.6
File Copy 256 bufsize 500 maxblocks 1077.0 33052.0 306.9
File Read 4096 bufsize 8000 maxblocks 15382.0 755979.0 491.5
Pipe Throughput 111814.6 2226506.9 199.1
Pipe-based Context Switching 15448.6 386106.1 249.9
Process Creation 569.3 11791.2 207.1
Shell Scripts (8 concurrent) 44.8 1232.2 275.0
System Call Overhead 114433.5 1487546.9 130.0
=========
FINAL SCORE 240.9
Moving to Future Hosting, Part I
Lately I’ve been having tons of problems with my cheap web hosting service from DreamHost. When I first moved to DreamHost, I couldn’t believe the boundless disk space I was able to get for practically nothing. Of course, I allowed myself to forget the old adage about anything too good to be true.
I should’ve known something was up after their devastating outage, but the hosting was still cheap and moving is a major pain.
However, last week I logged into my shell account to work on a Ruby on Rails idea I had. It was a quick-and-dirty thing so I just worked over SSH within vim. However, performance was so shitty that a w command (write changes to my little 25 line file) would take five seconds to complete, and things like cd, ls, and especially tar ran an order of magnitude slower than they should have. I opened a ticket with support who assured me they didn’t see anything wrong. I just sent them a response suggesting they run the ffmpeg configure script and see what happens; I gave up after ten minutes of waiting.
Anyway, I’ve already decided this is bullshit, and it’s time to move on. Sadly, the awesome $/GB deal offered by DreamHost is still pretty exceptional. I now know they achieve this by overselling, hence their incessant suckyness, so I was looking at shared hosting plans that come with bandwidth and disk resources allocated just to me, and the markup was huge. MediaLayer is a good example of this. 2500MB of storage, 500GB of bandwidth, and 12 domains, for a whopping $40/mo! That’s double what I pay at DreamHost for 500GB of storage and 10TB of bandwidth!
So, I started looking into dedicated hosting, which is, um, fucking insane. Then I started looking into Virtual Private Servers (VPS), which is a trend in hosting in recent years that leverages virtualization to allow you to have root on a box that runs in a VM, which allows the hosting provider to capitalize on the economies of scale offered by virtualization, so you get a quasi-dedicated account for (relatively) cheap.
After much research on WebHostingTalk and other useful sources of information, I decided to case my lot with Future Hosting. This is the first part of what should be a series documenting my experiences.
My first impression of Future Hosting was their domain name. I’ve been conditioned over long years on the Internet that .biz == shady. To even consider a hosting provider on the .biz TLD took alot of convincing. But, the following were in their favor:
- They have a Corporate Sponsor account on WebHostingTalk, which had the good sense to ban such shit hosts as CI Host. This constitutes evidence they suck less than C I Host; faint praise to be sure, but praise nonetheless
- Numerous positive reviews on WebHostingTalk and elsewhere, including explicit reports of excellent VPS performance
- A post on WebHostingTalk attempting to explain away the shady .biz domain (I can’t find it now but I recall reading it somewhere)
- A long-running Sticky in the VPS Hosting Offers forum on WHT, with a pretty compelling value
- Month-to-month plans only. They won’t let you prepay even if you wanted to
- Options of managed or unmanaged, cPanel or Plesk or DirectAdmin
What sold me was a spreadsheet I put together with about twenty VPS plans from around the ‘net, ranked by value in terms of $/MB of RAM, $/GB of bandwidth, $/GB of storage, and the arithmatic mean of all three. FutureHost’s Elite VPS plan came in third for overall value, behind two very-shady-seeming hosts, SmokyHosts (whose main page could easily be coming through a time warp leading back to 1994) and FortBox (whose ad on WHT uses the name ‘ForBox’; great attention to detail).
So, determined not to go penny-wise pound-foolish again, but also seeking a good deal, I decided FutureHost was a good balance of cheap and seemingly reputable.
Initially I emailed their Sales department with a question about upgrading storage. Their top-tier VPS plan maxes out at 50GB of storage, and my gallery alone is 35GB, so I wanted more breathing room. I submitted the inquiry sometime late at night, and woke up the next day with an answer. The response time was good; the answer was not. They don’t offer storage upgrades; you must upgrade to the next service tier, which in this case was their ‘hybrid’ VPS at $130/mo! Shit! So, that sucks right away. FutureHosting, if you’re reading this, start offering storage upgrades on your VPS accounts. Storage is cheap, and your $/GB of storage numbers are way higher than other plans I was looking at, even with the 40% discount I got.
Next I emailed their Sales department to confirm my interpretation of the upgrades and discounts listed on the WHT posting, to make sure they were combined and without any extra costs. I never heard back from that inquiry, which is disturbing, but my buddy Yousif went ahead and ordered with them anyway so I knew from his experience that my interpretation of the pricing structure was indeed accurate.
Having decided to give them a try, I decided I could work around the storage limitation by getting two of the Elite VPS plans, which have 40GB of storage each, and come in at $35/mo after a 40% discount. I wanted to be sure their 40% discount could be applied to both plans as long as I ordered them together, so I submitted another inquiry to sales, at 6:19 PM ET. At 7:37 PM ET I received a response from Vik Patel, the CEO, confirming they would allow me to use the discount on two instances.
With this in hand, I placed an order for one Elite VPS (their ordering system doesn’t permit you to order more than one server at a time). I then immediately ordered another one, but the 40% off coupon code wouldn’t work (presumably because I was no longer a ‘new’ customer). I place the order anyway, and emailed Vik requesting he apply the discount to my second order. He opened a ticket with Jim and it was taken care of promptly.
I placed the orders around 10PM ET, and by 11 AM ET the next day, the boxes were provisioned. I would’ve liked them sooner, but I assume they provision during business hours, so that’s roughly two hours into the work day. A two hour turnaround is not bad.
At the time of my first order I had to create an account on their system. I used KeePass to generate a long and complex password, which their system accepted. The ordering process seemed to indicate that the password I had used would become the root password on the two boxes. However, once they were provisioned this password would not work with root on either box. I know I had the password correct as I could log in to their Parallels Business Automation system using that password. Furthermore, my buddy Yousif had the exact same problem with his order.
I opened a ticket requesting help with this. They did reset my root passwords within an hour, after I verified some information, but no explanation was offered as to the cause of the problem. I would’ve liked a bit more insight into it, but it’s not worth pushing.
Once the boxes were provisioned, I began to explore my new hosting service. Maybe I’m just a VPS hosting n00b, but I found the way Future Hosting manages this service to be unnecessarily complicated. I have six accounts to keep track of:
- Parallels Business Automation account
- Support Portal
- DirectAdmin account on Server A
- DirectAdmin account on Server B
rootaccount on Server A (and the Virtuozzo Power Panel, with the same username and password)rootaccount on Server B (ditto)
Obviously the two root accounts would be there anywhere; I don’t mind that. But why must I have a separate DirectAdmin account for each box, which is separate from the support portal, which is separate from the Parallels BA account?
It was also not obvious at first what these various things were for. For example, you can add domains in Parallels, but also in DirectAdmin. Only after reading in the Knowledgebase does it become clear that you must add a domain in Parallels for their DNS infrastructure to service the domain, while adding the domain in DirectAdmin configures Apache to service HTTP requests for that domain. Not complicated once you know it, but not obvious if you don’t.
To add further complication to the domain stuff, when I first signed up my account included one domain, the domain I specified upon signup. So, when I went to add more domains in Parallels, it failed, saying I didn’t have any domains left on my account. I had to go to the ‘Store’ and ‘buy’ more domains. Since there is no domain limit on my account, new domains cost $0.00 each, but I still had to go through with the ‘purchase’, and even got a receipt. I bought 20 domains to start with, though the quantity is obviously irrelevant. This strikes me as a clumsy and confusing way to handle accounts with unlimited domain hosting, and I imagine it costs them alot of help desk calls.
I haven’t done any perf testing yet, nor have I moved any of my sites over. Those results in the next installment…
UPDATE: Part II is up.
Quotes from Heinlein's "Time Enough for Love"
A few months back I finally finished reading Robert Heinlein’s sci-fi opus, Time Enough for Love. It’s a long, sprawling, ambitious book which made me glad my first exposure to Heinlein was the life-changing Starship Troopers, for I fear Time Enough would have put me off Heinlein indefinitely.
Don’t get me wrong, the book is classic Heinlein, full of contradictory character traits and the most un-PC of dialog. Free love and quasi-anarchistic distrust for authority blends seamlessly with a very authoritarian streak which seems antithetical to the rest. That, to me, is one of the reasons why Heinlein’s work is so compelling: he has so many disagreeable positions, and so many agreeable ones, you must reflect on which is which, unlike a less unconventional author who would be easily placed in the ‘like’ or ‘hate’ bucket.
I don’t much enjoy writing book reviews, so here are some quotes from the book that struck me as quintessentially Heinlein:
- “Having any trouble with the democrats these days?”
“Democrats? Oh–you must mean ‘equalitarians.’ I thought at first you meant the Church of the Holy Democrat. We leave that church alone; they don’t meddle. There is an equalitarian movement every few years, certainly, under various names. The Freedom Party, the League of the Oppressed–names don’t matter as they all want to turn the rascals out, starting with me, and put their own rascals in. We never bother them; we simply infiltrate, then some night we round up the ringleaders and their families, and by daylight they are headed out as involuntary migrants. Transportees. ‘Living on Secundus is a privilege, not a right.’”
- “…I took it to mean ‘Sleep whenever you can; you may have to stay awake a long time.’ Early rising may not be a vice, Ira, but is it certainly no virtue. The old saw about the early bird just goes to show that the worm should have stayed in bed. I can’t stand people who are smug about how early they get up.”
“I didn’t mean to sound smug, Grandfather. I get up early from long habit–the habit of work. But I don’t say it’s a virtue.”
“Which? Work? Or early rising? Neither is a virtue. But getting up early does not get more work done…any more than you can make a peice of string longer by cutting off one end and tying it onto the other. You get less work done if you persist in getting up yawning and still tired. You aren’t sharp and make mistakes and have to do it over. That sort of busy-busy is wasteful. As well as unpleasant. And annoying to those who would sleep late if their neighbors weren’t so noisily active at some ungodly cow-milking hour. Ira, progress doesn’t come from early risers–progress is made by lazy men looking for easier ways to do things.”
- Before and after school he had to do chores on his family’s farm, which he hated, as they were what was known as “honest work”–meaning hard, dirty, inefficient, and ill-paid–and also involved getting up early, which he hated even worse.
- “But a reform politician has no such lodestone. His devotion is to the welfare of all the people–an abstraction of very high order and therefore capable of endless definitions. If indeed it can be defined in meaningful terms. In consequence your utterly sincere and incorruptible reform politician is capable of breaking his word three times before breakfast–not from personal dishonesty, as he sincerely regrets the necessity and will tell you so–but from unswerving devotion to his ideal.
“All it takes to get him to break his word is for someone to get his ear and convince him that it is necessary for the greater good of all the peepul. He’ll geek.
“After he gets hardened to this, he’s capable of cheating at solitaire. Fortunately he rarely stays in office long–except during the decay and fall of a culture.”
- Place your clothes and weapons where you can find them in the dark.
- An elephant: A mouse built to government specifications.
- A woman is not property, and husbands who think otherwise are living in a dream world.
- A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a well, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects.
- Beware of altruism. Is is based on self-deception, the root of all evil.
- Dear, don’t bore him with trivia or burden him with your past mistakes. The happiest way to deal with a man is never to tell him anything he does not need to know.
- If you are part of a society that votes, then do so. There may be no candidates and no measures you want to vote for, but there are certain to be ones you want to vote against. In case of doubt, vote against. By this rule you will rarely go wrong.
If this is too blind for your taste, consult some well-meaning fool (there is always one around) and ask his advice. Then vote the other way. This enables you to be a good citizen (if such is your wish) without spending the enormous amount of time on it that truly intelligent exercise of franchise requires.
- To stay young requires unceasing cultivation of the ability to unlearn old falsehoods.
- This sad little lizard told me that he was a brontosaurus on his mother’s side. I did not laugh; people who boast of their ancestry often have little else to sustain them. Humoring them costs nothing and adds to happiness in a world in which happiness is always in short supply.
- The difference between science and the fuzzy subjects is that science requires reasoning, while those other subjects merely require scholarship.
- Secrecy is the beginning of tyranny.
- Does history record any case in which the majority was right?
- The greatest productive force is human selfishness.
- Be wary of strong drink. It can make you shoot at tax collectors–and miss.
- A whore should be judged by the same criteria as other professionals offering services for pay–such as dentists, layers, hairdressers, physicians, plumbers, etc. Is she professionally competent? Does she give good measure? Is she honest with her clients?
It is possible that the precentage of honest and competent whores is higher than that of plumbers and much higher than that of lawyers. And enormously higher than that of professors.
- Expertise in one field does not carry over into other fields. But experts often think so. The narrower their field of knowledge the more likely the are to think so.
- Never try to outstubborn a cat.
- Tilting at windmills hurts you more than the windmills.
- Yield to temptation; it may not pass your way again.
- Waking a person unnecessarily should not be considered a capital crime. For the first offense, that is.
- Political tags–such as royalist, communist, democrat, populist, fascist, liberal, conservative, and so forth–are never basic criteria. The human race divides politically into those who want people to be controlled and those who have no such desire. The former are idealists acting from highest motives for the greatest good of the greatest number. the latter are surly curmudgeons, suspicious and lacking in altruism. But they are more comfortable neighbors than the other sort.
- Sin lies only in hurting other people unnecessarily. All other “sins” are invented nonsense. (Hurting yourself is not sinful–just stupid.)
- Being generous is inborn; being altruistic is a learned perversity. No resemblance–
- Pessimist by policy, optimist by temperament–it is possible to be both. How? By never taking an unnecessary chance and by minimizing risks you can’t avoid. This permits you to play out the game happily, untroubled by the certainty of the outcome.
- A committee is a life form with six or more legs and no brain.
- “In my early days it was an article of faith among a self-styled ‘intellectual elite’ that they could teach calculus to a horse…if they started early enough, spent enough money, supplied special tutoring, and were endlessly patient and always careful not to bruise his equine ego. They were so sincere that it seems downright ungrateful that the horse always persisted in being a horse. Especially as they were right..if ‘starting early enough’ is defined as a million years or more.”
Only a few of the above are quoted from actual dialog. Large chunks of the book consist of one or two line entries in Lazarus’ Notebook, which makes me think Heinlein got stuck somewhere and just started brain-dumping sage advice until he picked up the thread again. Time Enough is twice as long as it needs to be, but then if it were a minimalist tale, it wouldn’t be Heinlein.
Generating memorable passphrases from mad-libs
After my last experiment with Markov chains for passphrase generation, I thought I’d try something different. Rather than go crazy with statistical trickery, why not take a list of words organized by part of speech, and use templates with placeholders for different parts of speech (effectively, simple mad-libs) to generate memorable passphrases?
The POS list from the word lists collection contains a nice selection of English words and the part or parts of speech to which they belong. The README doesn’t have a breakdown of the word counts per POS, so I used a quick Ruby script to produce the following:
Noun has 128674 words
Noun Phrase has 71806 words
Adjective has 57900 words
Verb (usu participle) has 18443 words
Verb (transitive) has 16265 words
Adverb has 14782 words
p has 8554 words
Verb (intransitive) has 5141 words
Interjection has 430 words
Preposition has 162 words
Pronoun has 117 words
Definite Article has 103 words
Conjunction has 93 words
e has 1 words
p and e are not listed in the README file as parts of speech so I assume they’re some sort of secondary tag.
Anyway, that’s a goodly portion of nouns and verbs to work with. Enough to populate a simple template like:
<definite article> <adjective> <noun> <verb> <adverb>.
Using Excel and Shannon’s theory of information, I computed the entropy of such a template applied to the above wordlist, assuming each word in a given POS has equal probability of selection:
POS Noun Verb Adverb Adjective Total
Num words (size of X): 128674 18443 14728 57900
p(x) for any x in X 7.77158E-06 5.42211E-05 6.78979E-05 1.72712E-05
p(x) log p(x) -0.00013191 -0.000768356 -0.000940133 -0.000273252
SUM of above for all x in X -16.97336105 -14.17078573 -13.84627391 -15.82127573
Negative sum (entropy) 16.97336105 14.17078573 13.84627391 15.82127573 60.81169641
So using the POS word list and the above template, I can generate 60-bit passphrases. That’s a far cry from 128-bit AES keys, but it’s still not bad for an easy to implement and easy to understand solution. You could also boost the entropy by randomly applying permutations like upper casing random letters or replacing letters with numbers (L is 1, O is 0, etc).
So, what sort of passphrases would this produce? Here’s a sample:
fifty interplacental ugliness sleaving lunatically.
such uretic apertometer straddled unimpulsively.
eighteen overground inflationism funneled treacherously.
twelve quasi-military aquamanile rummage curvilinearly.
nineteen annunciative Malanje light presumptively.
hundred amaranthaceous hindi reinspired nonbodingly.
those unfirm lah do work moveably.
another chockful miracle-worship stencilling certes.
two humourless half-jack outsmart sweepingly.
eighty vaulting pediculus counterclaim nonpredicatively.
Clearly these would make more sense if the POS tags included things like plural vs. signular nouns and so on, but doing so would reduce the possible words for each template position and thus lower the entropy. It’s possible a more grammatically correct template could be made longer while remaining equally memorable, but I’m not familiar with any experimental evidence as to the extent grammatical correctness improves recall, and I suspect it wouldn’t be a big enough improvement to overcome the need for more words in the template.
New Picasa 3 Beta Awesomeness
Last weekend I was playing with the new beta of Google Picasa 3, including the new Picasa Web Albums.
I’ve used the Picasa 2 desktop photo management software for a couple years now, and been mostly happy with it. It can upload directly to Facebook, and it’s not too hard to upload to my Gallery 2 site either. However, I’ve never seen the point in paying for the Picasa Web Albums photo hosting service, when I can host my own photos on my own site for no extra charge.
That changed last weekend, when I discovered the Name Tags feature in Picasa Web 3. This is what I thought Facebook was doing a while back. The idea is simple: after you upload your photos to Picasa Web, Google detects all the faces in the photos, and asks you to name them. Google then uses facial recognition technology to guess the identities of future occurrences of the same faces.
Once you’ve tagged your photos thusly, all the cool Facebook features related to photo tagging are available, like sharing photos of people with the subjects therein, grouping photos by the people in them, and (most important to me) capturing for posterity the identities of the people in the photos.
It’s still beta, and some of the face detection and recognition stuff is weak, but when that is ironed out I’ll gladly pay $25/yr for 10GB of storage so I can migrate my photo collection into Picasa.
I do have one feature request for the devs: let me use my existing Gmail contacts as possible labels for my photos. Having to type names and email addresses a second time blows.
More on the bullshit Windows video experience
I’ve written before about the unreasonable difficulty associated with watching DVD-quality video on a Windows machine. Yesterday I finally figured out how to get the x64 version of Vista Media Center to play back HD video without the video lagging significantly behind the audio.
As I’ve noted before, video on Windows is hard because:
- Microsoft does not ship the codecs needed to play the relevant high-resolution video formats
- There is no standard source for such codecs. Multiple codec packs exist, while others install the codecs they need manually
- 64-bit editions of Vista run the 64-bit Vista Media Center, which can only use 64-bit codecs, but for some reason the world of codec developers has not caught up with the last five years of processor architecture advancements, and ships only 32-bit binaries
None of these problems are insurmountable, but if you’re just a regular user and you’re trying to set up a media center PC with which to play your video collection, it’s unlikely you’ll have a good time of it. It’s also hard to find answers to these questions because it seems everyone posting on TheGreenButton and related forums find a slightly different way to solve the problem. It also doesn’t help that Microsoft are shipping ‘updates’ to VMC that break it substantively.
So, here’s my secret handshake for HD playback in x64 VMC:
- Install the Vista Codec Package
- Install the Vista Codec x64 Components
- From the Start menu, go to VistaCodecPack, 64-bit Tools, Video Decoder Configuration. Check the check box to enable OSD (on-screen display). You can muck w/ the font settings to give the OSD an alpha channel value of 0 (meaning it’s fully transparent and you can’t see it). This seems absurd, but there really is a bug in the FFDshow stuff such that HD content, at least in MKV files, will playback with the video lagging the audio by several seconds, making the result unwatchable. For some reason, turning on the OSD, even if it’s alpha-channeled into invisibility, makes the problem go away.
- Uninstall the God-forsaken June 2008 Cumulative Update (KB950126). This was a delightful little best-ever update from Microsoft that made Vista Media Center super-awesome, but broke the following things:
- If video playback is paused or stopped, it will resume when the screensaver starts, when the Media Center window is minimized or resized, or in response to other random events
- When you stop playback, you may or may not get the menu with the ‘Done’, ‘Resume’, ‘Delete’ options.
- When you pause video and then resume, you may or may not resume where you left off, or playback may skip an hour ahead
And there you go. With this done, assuming your box has the juice for HD video, VMC will play HD content without the damnable lag. If you’re like me, you’re wondering how the learning-disabled monkeys that run the Vista Media Center QA department could possibly have missed the above bugs, so glaring that the Media Center forum sites are abuzz with wrath for this fucking update. I have no answer for you, except to note that Microsoft continues to not be serious about the Media Center convergence concept, despite BillG’s protestations to the contrary.