Installing Rails 2.2 on Mac OS X - MySQL problem

Rails 2.2 is out and you want to install it - but you may run into this issue on Mac OS X.

1: Make sure you have upgraded to rubygems 1.3.1
$ sudo gem update --system

If that barfs try this which does the same thing a different way:
$ sudo gem install rubygems-update
$ sudo update_rubygems

2: Install Rails
This should install just fine
$ sudo gem install rails

3: One important change in 2.2 is that the Mysql database driver is no longer bundled and you have to install yourself. But the obvious command may well fail like this:

$ sudo gem install mysql
Password:
Building native extensions. This could take a while...
ERROR: Error installing mysql:
ERROR: Failed to build gem native extension.
[...]

The issue is that the gem needs more information about your MySQL installation, so do this instead:

$ sudo gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
Building native extensions. This could take a while...
Successfully installed mysql-2.7
1 gem installed

4: That looks good but you may not be out of the woods yet... I got this when I tried a rake db:migrate

$ rake db:migrate
(in /Users/jones/Documents/myapp)
!!! The bundled mysql.rb driver has been removed from Rails 2.2. Please install the mysql gem and try again: gem install mysql.
rake aborted!
dlsym(0x1c71570, Init_mysql): symbol not found - /usr/local/lib/ruby/gems/1.8/gems/mysql-2.7/lib/mysql.bundle


After some poking around I found reference to the version of mysql that is installed.

I installed mine as the packaged disk image from mysql.com. For Mac OS X 10.5 you have the choice of these two:
mysql-5.0.67-osx10.5-x86.dmg
mysql-5.0.67-osx10.5-x86_64.dmg

Both work fine on my MacBook but the mysql gem wants the '_x86' version, not the '_x86_64'

Look in /usr/local to see which you have symlinked in:

$ ls -l /usr/local
[...]
lrwxr-xr-x 1 root wheel 24 Nov 24 09:35 mysql -> mysql-5.0.67-osx10.5-x86

Install the correct version, reinstall the gem just to be safe, and try your rake again - it should be fine!

IMPORTANT

If you do have the same issue and install the non _64 version on top of the _64 one then mysql will NOT copy over your data files. To fix this:

1: Shut down MySQL
2: In the new (non _64) version
$ sudo mv data data.bak
$ sudo cp -pr ../mysql-5.0.67-osx10.5-x86_64/data .
3: Start up MySQL

Wrapping Scripts with Platypus on Mac OS X

Platypus is a Mac OS X tool, written by Sveinbjorn Thordarson, that lets you create Mac OS X applications that wrap up command line scripts, create installers, etc. It is a great way to package Ruby, Perl, etc. scripts for a broader user audience.

Platypus is easy to figure out if you just have a single script. But I often times develop data analysis pipelines of various sorts where I have one master Ruby script calling a bunch of others.

Wrapping the whole thing up with Platypus can work really well but to do that you need to understand how the tool works.

At first glance you might think that it just calls the target script in its original location, but in fact it copies the script into the folder that embodies the Mac OS X application. So if I have a primary script calling a secondary one then I need to either refer to it by an absolute path, which is not a good idea for portability, or I need to include the secondary script in the Mac application.

Here is an example that shows how you handle this - I'll assume that you have tried Platypus with some simple scripts already.

The primary script (hello.rb) is what the Platypus generated application will execute. This in turn will execute world.rb and echo its output.

hello.rb

#!/usr/bin/ruby
puts "Hello ..."
world = File.join(File.dirname(__FILE__), 'world.rb')
puts `#{world}`

world.rb

#!/usr/bin/ruby
puts "... World"

First of all, specify the path to your Ruby (or whatever) interpreter explicitly. The convention in the Ruby world is to use '#!/usr/bin/env ruby' but doesn't work directly with Platypus. I'm sure you can mess with some of the Platypus options to make it work but I haven't bothered.

The third line in hello.rb specifies where to find the script 'world.rb', namely in the same directory as hello.rb. For Perl coders, this is the same as using FindBin.

Package hello.rb with Platypus, sending output to a Text window and keeping the window open after the script completes, then run the script. You should get something like this:

/Users/jones/Documents/Platypus/hello_world.app/Contents/Resources/script:7: command not found: /Users/jones/Documents/Platypus/hello_world.app/Contents/Resources/world.rb
Hello ...

It has found and run hello.rb but it can't find world.rb. The error message tells us that the script it has run is actually called 'script', not hello.rb, which seems odd.

Platypus copies your script into a new Mac application, which as you should know, is really a directory. 'cd' to the application directory (hello_world.app in my case), then into Contents -> Resources.

Look at the file 'script' and you'll see that is actually 'hello.rb'

To get the result you want you have to add 'world.rb' to the application as a 'Resource' and you do this in the 'Advanced Options' panel in Platypus. Just add the secondary file to this panel, rebuild the application and re-run it.

Hello ...
... World

With more complex applications you can add additional scripts and data files. Test things out by running them from the command line in the platypus generated application directory.

If you want to get around this you can specify an absolute path to the secondary script, or create a symlink to it from within the application directory. But down this path lies madness. A better alternative would be to use an environment variable to show where something lies, such as the PATH variable or something custom. Platypus will let you pass that into the app.

I'm very impressed with Platypus, compared to earlier attempts at doing this, like DropScript. When I get the chance I want to try it in combination with CocoaDialog which allows you to create simple GUIs in your scripts.

Memory Leak in Ruby 1.8.6

Just spent way too much time trying to track down a memory leak in a Ruby script that parsed out information from a massive text file. It would just gradually consume all available memory until the swap daemon went nuts.

I can't say that I've had a memory leak that caused me pain before using Ruby, but this one stopped the show.

None of the suggestions for possible causes was of any help, other than a general opinion that garbage collection in Ruby 1.8.6 had some issues.

So I downloaded ruby 1.8.7 (2008-08-11 patchlevel 72) from http://www.ruby-lang.org/en/downloads/, compiled it and reran the code.

Sorted!

1.8.6 is still the default on the systems I work with. If you run into this problem, the first thing to try is an upgrade to 1.8.7.

To help debugging, this line of code will spit out the current usage (Sorry, I forget where I got it from...)

STDERR.puts 'processBlock ' + `pmap #{Process.pid} | tail -1`[10,40].strip

Tutorial on the Ruby gem aws-sdb - an interface to AWS SimpleDB

Tim Dysinger's aws-sdb is a Ruby interface to the SimpleDB service from Amazon Web Services.

I've written a tutorial on how to use this interface to access AWS SimpleDB from your Ruby code.

 

Ruby Hpricot Tip - Extracting Arbitrary Blocks of HTML

Hpricot is a HTML parser for Ruby, written by 'whytheluckystiff', and is a great tool for extracting information from web pages.

If the target page uses divs with unique ids or classes then this task is especially easy, but most of the pages I care about are not as well designed as they might be. I often come across pages where the distinct sections are delimited by some arbitrary feature, such as a horizontal rule or simply a title in plain text.

Hpricot uses CSS selectors (as well as XPath) to pull out specific elements but that approach is not a great match for this class of arbitrary pages.

Here is one way to solve this problem. I've set up a simple web page with four sections, separated by <hr> tags. You can find that here and you can find the Ruby code to parse it here.

Basically you get the first Hpricot element on the page contained in the Body, then step through the elements in turn adding each to a new Hpricot::Elements object until either a hr tag or the end of the document is encountered. Every time it finds a delimiter it pushes the current Elements structure into an array and starts a new one.

Once done, you have an array of Hpricot::Elements objects, one for each section of your page. Each of these can be processed further using Hpricot.

The short version of the code, with comments removed, is here:

el = doc.search("body > *").first
blocks = Array.new
block = Hpricot::Elements.new
while el = el.next
  if el.to_html =~ /\<hr/
    blocks << block
    block = Hpricot::Elements.new
  end
  block << el
end
blocks << block

Let me know if you have other solutions to this.

EC2, SSH and Capistrano

The various ways that you can set up SSH keys for secure remote access to a machine confuse the heck out of me.

Amazon Web Services use SSH keypairs for connecting to EC2 nodes, like this:
$ ssh -i mykeypair root@ec2-75-101-234-79.compute-1.amazonaws.com

But a more common way to use keys is to create a private/public key pair and copy the public key to the remote machine. The default location for storing these is ~/.ssh and the file name of the public key is id_rsa.pub. So to set up a key for ssh between two 'regular' machines you would do this:
$ ssh-keygen
$ cat ~/.ssh/id_rsa_pub | ssh user@yourdomain "cat >> .ssh/authorized_keys2"
$ ssh user@yourdomain

With EC2 nodes you have to use a 'keypair' and that involves a different type of private key and a different key stored on the remote host. You can find that on the EC2 node in ~/.ssh/authorized_keys -- NOTE the filename - this is not authorized_keys2 - the two versions relate to the SSH1 and SSH2 versions.

Using the EC2 flavor of SSH login is not a problem, until you want to use Capistrano, the powerful Ruby software for deploying Rails applications and other things on remote hosts. Capistrano uses SSH to connect to remote machines and by default will use the current user and the regular private/public keys.

Try to use Capistrano with its defaults to connect to an EC2 node and you'll get nowhere. To get it to work you need to do two things:

1: Set up a SSH private/public keypair as above and copy to the EC2 node, putting it in ~root/.ssh/authorized_keys2 (That's keys*2* !!). So you now have two keys for EC2.

2: Create a Capistrano capfile and include these two lines that tell it the remote user and where the key lives:
set :user, 'root'
ssh_options[:keys] = [File.join(ENV["HOME"], ".ssh", "id_rsa")]

Run Capistrano and everything should work.

You would think you could just use the EC2 keypair in the capfile but that did not work in my hands. Capistrano has minimal documentation but it looks like the SSH options are the same as those for the Ruby SSH library.

Now you still have to enter your SSH key passphrase. You can avoid that by registering your keys with ssh-agent, but that is another can of SSH worms...

Common AWS EC2 mistake

Well, it's a common mistake for me...

You start a new instance in Amazon Web Services (AWS) Elastic Compute Cloud (EC2) using the ec2-run-instances command. You should include a keypair in the command line like this:
$ ec2-run-instances ami-2b5fba42 -k mykeypair

But if you forget to include the keypair an instance will still start up and appear in ec2-describe-instances. When you try to ssh into that node you get this error message:

$ ssh -i mykeypair root@ec2-75-101-234-79.compute-1.amazonaws.com
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
b0:b8:fa:f6:f2:c5:e8:2f:7b:9c:e8:44:b7:ff:a3:70.
Please contact your system administrator.
Add correct host key in /Users/jones/.ssh/known_hosts to get rid of this message.
Offending key in /Users/jones/.ssh/known_hosts:67
RSA host key for ec2-75-101-234-79.compute-1.amazonaws.com has changed and you have requested strict checking.
Host key verification failed.
lost connection

Confusing, until you realize your mistake...

You have to terminate that instance and then create a new one with a keypair.