Using Opscode Chef to start up a node on AWS EC2 - A Simple Example

Chef from opscode.com is a suite of tools for managing computing infrastructure, from spinning up new nodes to installing Ruby gems to any custom operation you care to code up. It gets great reviews from those who manage numbers of unix (and other) systems and it has support for Amazon AWS and other cloud vendors baked right in.

I'm just getting started with it and am eager to go further but, like many projects, the documentation is far from clear. It has a lot of options and that typically means complexity. Perhaps because they are targeting folks who do systems administration for a living, their docs skim over some of the crucial basics.

My main interest in Chef is using it to spin up AWS EC2 nodes, configure them to my tastes and then spin them down as needed.

Here are the steps I took to get a node up and running:

0: I'm assuming you know how to fire up EC2 nodes from the management console or the command line on your desktop machine, that you have a keypair set up and that you are familiar with AMIs and the other terminology. If not, then get comfortable with EC2 before going any further.

1: I'm also assuming that you have a basic chef setup on your desktop and an account on Opscode's chef server (it's free for up to 5 nodes) and that you have written and tested a simple recipe on your desktop. If not, get comfortable with that before going further.

2: Add your AWS credentials to your knife.rb file (typically in ~/.chef)
The docs say to enter them in this format, which is different from the other configuration parameters that are generated from the opscode server... I don't know why... but this works:

# EC2 access keys
knife[:aws_access_key_id]     = "<your access key id>"
knife[:aws_secret_access_key] = "<your secret access key>"

3: Figure out your EC2 parameters for the node you want to spin up.

The AMI ID - I'm using a 32 bit Ubuntu AMI (ami-480df921)

The Instance Type - I'm testing with a Micro instance (t1.micro)

The Keypair name (mine is called 'craic-ec2-keypair') and the location of the SSH identity file that is linked with this (mine is in ~/.ssh/craic-ec2-keypair.pem).

The User that you (and chef) will ssh in to the node with (Ubuntu 10.04 wants me to use 'ubuntu' instead of 'root')

The Security Group for the server (I just use 'default' which is what I set up when I first started with EC2)

The Chef Recipe(s) and/or Role(s) that you want to run on the new EC2 server (I'm just going to run a simple one that I created called 'craic_test' that writes a file in /tmp) - Make sure this works on a non-EC2 client first!

4: Let's do it...
The chef command that you use is 'knife' with some EC2 specific options. You can find more information HERE in their FAQs, which is MUCH more complete than their Wiki documentation on knife...

The command will be 'knife ec2 server create' followed by a bunch of options. That's pretty self explanatory - but then we get into the options... Here is the full command I used split across multiple lines, which I shall explain below:

$ knife ec2 server create "recipe[craic_test]" \
      -i ami-480df921 \
      -f t1.micro \
      -x ubuntu \
      -S craic-ec2-keypair \
      -I ~/.ssh/craic-ec2-keypair.pem \
      -G default

The first line is the command itself followed by the recipes and/roles in the standard chef format ("recipe[craic_test]").

The EC2 AMI is specified with the -i option, then the Instance Type is specified with the -f option (aka --flavor - I don't know why they chose that...).

Next comes the ssh user with the -x option, the SSH Keypair name with -S (upper case S) and the location of the matching identity file with the -I option (upper case I) - then finally the security group with the -G option (upper case G).

I am really hoping that there is some way to store some of these options in knife.rb or someplace convenient as most of these will not change for my uses.

All being well, what happens next is that the node gets spun up on EC2, chef installs the prerequisites it needs to run on the node and downloads the chef client over there. Chef will create the new node record on the chef server, download your recipes and then execute them.

What you will see is a TON of output as everything runs. I won't subject you to all of that here, but the first few line will look like this:

Instance ID: i-08da8865
Flavor: t1.micro
Image: ami-480df921
Availability Zone: us-east-1b
Security Groups: default
SSH Key: craic-ec2-keypair

Waiting for server............
Public DNS Name: ec2-184-72-190-66.compute-1.amazonaws.com
Public IP Address: 184.72.190.66
Private DNS Name: domU-12-31-39-0F-29-E8.compute-1.internal
Private IP Address: 10.193.42.22

Waiting for sshddone
INFO: Bootstrapping Chef on
0% [Working]90-66.compute-1.amazonaws.com
[...]

This shows the IP info, etc for your new instance and is followed by pages of messages reporting the downloading and installation of all the prerequisites for Chef. At the end you should see your recipes being run and finally a summary once again of the instance information:

Instance ID: i-08da8865
Flavor: t1.micro
Image: ami-480df921
Availability Zone: us-east-1b
Security Groups: default
SSH Key: craic-ec2-keypair
Public DNS Name: ec2-184-72-190-66.compute-1.amazonaws.com
Public IP Address: 184.72.190.66
Private DNS Name: domU-12-31-39-0F-29-E8.compute-1.internal
Private IP Address: 10.193.42.22
Run List: recipe[craic_test]

5: Now you can ssh into your node and you should see that your recipes have run and done whatever you asked of them.

One good thing about all the prerequisites getting installed, for an Ubuntu system at least, is that you now have a node with a lot of development libraries and tools already in place (especially if you work in Ruby).

That's it for the basic installation.

Remember to terminate your node when you are finished as you are being charged for it AND remove any test nodes from your Opscode server account as they will count against your account limit.

There is obviously a lot more that you can, and need to, do to set up your nodes but this walk through gets you to a functioning node.

Happy cooking