EC2, SSH and Capistrano

The various ways that you can set up SSH keys for secure remote access to a machine confuse the heck out of me.

Amazon Web Services use SSH keypairs for connecting to EC2 nodes, like this:
$ ssh -i mykeypair root@ec2-75-101-234-79.compute-1.amazonaws.com

But a more common way to use keys is to create a private/public key pair and copy the public key to the remote machine. The default location for storing these is ~/.ssh and the file name of the public key is id_rsa.pub. So to set up a key for ssh between two 'regular' machines you would do this:
$ ssh-keygen
$ cat ~/.ssh/id_rsa_pub | ssh user@yourdomain "cat >> .ssh/authorized_keys2"
$ ssh user@yourdomain

With EC2 nodes you have to use a 'keypair' and that involves a different type of private key and a different key stored on the remote host. You can find that on the EC2 node in ~/.ssh/authorized_keys -- NOTE the filename - this is not authorized_keys2 - the two versions relate to the SSH1 and SSH2 versions.

Using the EC2 flavor of SSH login is not a problem, until you want to use Capistrano, the powerful Ruby software for deploying Rails applications and other things on remote hosts. Capistrano uses SSH to connect to remote machines and by default will use the current user and the regular private/public keys.

Try to use Capistrano with its defaults to connect to an EC2 node and you'll get nowhere. To get it to work you need to do two things:

1: Set up a SSH private/public keypair as above and copy to the EC2 node, putting it in ~root/.ssh/authorized_keys2 (That's keys*2* !!). So you now have two keys for EC2.

2: Create a Capistrano capfile and include these two lines that tell it the remote user and where the key lives:
set :user, 'root'
ssh_options[:keys] = [File.join(ENV["HOME"], ".ssh", "id_rsa")]

Run Capistrano and everything should work.

You would think you could just use the EC2 keypair in the capfile but that did not work in my hands. Capistrano has minimal documentation but it looks like the SSH options are the same as those for the Ruby SSH library.

Now you still have to enter your SSH key passphrase. You can avoid that by registering your keys with ssh-agent, but that is another can of SSH worms...

Common AWS EC2 mistake

Well, it's a common mistake for me...

You start a new instance in Amazon Web Services (AWS) Elastic Compute Cloud (EC2) using the ec2-run-instances command. You should include a keypair in the command line like this:
$ ec2-run-instances ami-2b5fba42 -k mykeypair

But if you forget to include the keypair an instance will still start up and appear in ec2-describe-instances. When you try to ssh into that node you get this error message:

$ ssh -i mykeypair root@ec2-75-101-234-79.compute-1.amazonaws.com
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
b0:b8:fa:f6:f2:c5:e8:2f:7b:9c:e8:44:b7:ff:a3:70.
Please contact your system administrator.
Add correct host key in /Users/jones/.ssh/known_hosts to get rid of this message.
Offending key in /Users/jones/.ssh/known_hosts:67
RSA host key for ec2-75-101-234-79.compute-1.amazonaws.com has changed and you have requested strict checking.
Host key verification failed.
lost connection

Confusing, until you realize your mistake...

You have to terminate that instance and then create a new one with a keypair.