Simple Example of Customizing AWS EC2 Instances at Launch

With Amazon Web Services EC2, you fire up a new Instance (compute node) from a given AMI (machine image) and you can build custom AMIs to suit your specific needs. But there are many things that you only want to specify when you fire up your instance. And in my case there is always some stupid detail that I forgot to specify when I built the AMI.

So having a way to customize the instance at launch time is important.

One guide to doing this is by PJ Cabrera on the AWS site. This is great but it is relatively complex for many needs. Here is a simpler tutorial on the steps needed for basic customizations:

I'm using one of the EC2-tailored Ubuntu AMIs as my base. Eric Hammond provides a valuable service to the EC2 community maintaining these. Specifically I'm using an Amazon EC2 Ubuntu 9.04 jaunty AMI that I've added lots of custom packages to.

In /etc/init.d you'll find several init scripts that start with 'ec2-'. ec2-run-user-data is the one that matters here, written by Eric Hammond. This will run automatically on start up and look for a user-supplied script at the URL http://169.254.169.254/2008-02-01/user-data. This is not a generally accessible URL and has a special role in EC2.

If ec2-run-user-data finds a file at this URL that begins with the characters '#!' it assumes that this is an executable script and executes it. You can create this script and have it set environment variables, run other scripts, etc.

This custom script actually comes from your desktop machine and you specify it when you start up a new EC2 instance. Somehow in the innards of EC2 the file is uploaded and is made available to your instance only via this special URL.

Here is how I fire up a new instance from my desktop:

$ ec2-run-instances -k mykeypair -f ec2_custom_script.sh -t c1.medium -z us-east-1a ami-12345678

You specify your file after the -f flag and it needs to be an executable script file. Bash, Perl, Ruby, etc. should all be fine as long as your AMI has that interpreter installed.

So what goes into a launch script? One common use is to set up your AMAZON_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID keys that you need for accessing S3, etc. You don't want to hard wire these into your AMI, but you do need these to be available when you login to an instance. So in my launch script I add these to my .bashrc file which is loaded when I actually login. I also create a couple of custom mount points this way where I can mount EBS volumes.

Here is a simple launch script:

#!/bin/bash
# Simple custom EC2 launch script
mkdir /mnt/craic
mkdir /mnt/data
BASHRCFILE=/root/.bashrc
AMAZON_SECRET_ACCESS_KEY=<yourkeygoeshere>
AWS_ACCESS_KEY_ID=<yourkeygoeshere>
echo "export AMAZON_SECRET_ACCESS_KEY=${AMAZON_SECRET_ACCESS_KEY}" >> ${BASHRCFILE}
echo "export AMAZON_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}" >> ${BASHRCFILE}
# Need this for Rails on Ubuntu
echo "export PATH=/var/lib/gems/1.8/bin:$PATH" >> ${BASHRCFILE}

The great thing about Eric's ec2-run-user-data script is that it is all set up and ready to go. Pass it a valid script and it should just work.

Note that there is a limit of 16kb the size of launch files using this mechanism. That is quite a lot for a script but you still want to be frugal. If you need more than this then put additional steps into one or more secondary scripts and have your primary launch script fetch these from S3 and execute them.

Also note that you cannot specify a file when you launch an instance using the AWS Management Console. You should be able to cut and paste in the contents of your script into the user-data field in the advanced options, but this is an ugly way to do it. You may be best off to use the command line ec2-run-instances command as shown above. I'm not sure if other interfaces like ElasticFox can handle this.

You can get fancy by rolling your own /etc/init.d scripts and passing a compressed zip file, etc., to it via this mechanism. But I've found that a pain in the neck to troubleshoot when I've had issues. As long as you are using Ubuntu the simple approach is the way to go.