Simple Example of Customizing AWS EC2 Instances at Launch

With Amazon Web Services EC2, you fire up a new Instance (compute node) from a given AMI (machine image) and you can build custom AMIs to suit your specific needs. But there are many things that you only want to specify when you fire up your instance. And in my case there is always some stupid detail that I forgot to specify when I built the AMI.

So having a way to customize the instance at launch time is important.

One guide to doing this is by PJ Cabrera on the AWS site. This is great but it is relatively complex for many needs. Here is a simpler tutorial on the steps needed for basic customizations:

I'm using one of the EC2-tailored Ubuntu AMIs as my base. Eric Hammond provides a valuable service to the EC2 community maintaining these. Specifically I'm using an Amazon EC2 Ubuntu 9.04 jaunty AMI that I've added lots of custom packages to.

In /etc/init.d you'll find several init scripts that start with 'ec2-'. ec2-run-user-data is the one that matters here, written by Eric Hammond. This will run automatically on start up and look for a user-supplied script at the URL http://169.254.169.254/2008-02-01/user-data. This is not a generally accessible URL and has a special role in EC2.

If ec2-run-user-data finds a file at this URL that begins with the characters '#!' it assumes that this is an executable script and executes it. You can create this script and have it set environment variables, run other scripts, etc.

This custom script actually comes from your desktop machine and you specify it when you start up a new EC2 instance. Somehow in the innards of EC2 the file is uploaded and is made available to your instance only via this special URL.

Here is how I fire up a new instance from my desktop:

$ ec2-run-instances -k mykeypair -f ec2_custom_script.sh -t c1.medium -z us-east-1a ami-12345678

You specify your file after the -f flag and it needs to be an executable script file. Bash, Perl, Ruby, etc. should all be fine as long as your AMI has that interpreter installed.

So what goes into a launch script? One common use is to set up your AMAZON_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID keys that you need for accessing S3, etc. You don't want to hard wire these into your AMI, but you do need these to be available when you login to an instance. So in my launch script I add these to my .bashrc file which is loaded when I actually login. I also create a couple of custom mount points this way where I can mount EBS volumes.

Here is a simple launch script:

#!/bin/bash
# Simple custom EC2 launch script
mkdir /mnt/craic
mkdir /mnt/data
BASHRCFILE=/root/.bashrc
AMAZON_SECRET_ACCESS_KEY=<yourkeygoeshere>
AWS_ACCESS_KEY_ID=<yourkeygoeshere>
echo "export AMAZON_SECRET_ACCESS_KEY=${AMAZON_SECRET_ACCESS_KEY}" >> ${BASHRCFILE}
echo "export AMAZON_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}" >> ${BASHRCFILE}
# Need this for Rails on Ubuntu
echo "export PATH=/var/lib/gems/1.8/bin:$PATH" >> ${BASHRCFILE}

The great thing about Eric's ec2-run-user-data script is that it is all set up and ready to go. Pass it a valid script and it should just work.

Note that there is a limit of 16kb the size of launch files using this mechanism, so you need to be frugal. If you need more than this then put additional steps into one or more secondary scripts and have your primary launch script fetch these from S3 and execute them.

Also note that you cannot specify a file when you launch an instance using the AWS Management Console. You need to use the command line ec2-run-instances command as shown above. I'm not sure if other interfaces like ElasticFox can handle this.

You can get fancy by rolling your own /etc/init.d scripts and passing a compressed zip file, etc., to it via this mechanism. But I've found that a pain in the neck to troubleshoot when I've had issues. As long as you are using Ubuntu the simple approach is the way to go.

 

Installing Rails on Ubuntu - Path Problem

There is an annoying issue when installing recent versions of Rails on Ubuntu (and possibly other Linux variants).

You install the necessary gems but when you try to run rails you get this:

# rails -v
The program 'rails' is currently not installed.  You can install it by typing:
apt-get install rails
-bash: rails: command not found

You can see that the gem is installed:

# gem list rails
*** LOCAL GEMS ***
rails (2.3.2, 2.2.2)

So where is it? Turns out you need to add this link to your PATH variable (in your .bashrc file for example):

# export PATH=/var/lib/gems/1.8/bin:$PATH

That does the trick:

# rails -v
Rails 2.3.2

You will find mention of this on page 25 of the 3rd Edition of 'Agile Web Development with Rails' but it needs to be more widely known. Gotchas like this that just prevent you from doing anything with a piece of software are BAD NEWS - someone make it go away...

 

Passing Arguments to Rake with a Rails application

In order to pass arguments to a Rake task you specify them as named arguments in the task definition and then pass the actual values in square brackets following the rake task on the command line.

Here is a simple example:

task :mytask, [:word0, :word1] do |t, args|
    puts "You typed: '#{args.word0}' and '#{args.word1}'"
end

You run this from the command line thus:

$ rake mytask[hello,world]
You typed: 'hello' and 'world'

Interacting with a Rails app involves adding a dependency of :environment. Add this after the array of arguments like this:

task :mytask, [:word0, :word1] => :environment do |t, args|
    puts "You typed: '#{args.word0}' and '#{args.word1}'"
end

The syntax looks weird but it works.


 

Sending Email from a Linux machine

Sending a notification email from a server to a remote address is a very common requirement. If all you want to do is send email and not receive any (from local or remote sources) then installing good old sendmail is overkill.

A simple alternative is to install ssmtp. On ubuntu:

$ sudo aptitude install ssmtp

and then edit /etc/ssmtp/ssmtp.conf. Here is an example configuration that uses a Gmail account for relaying:

#
# Config file for sSMTP sendmail
mailhub=smtp.gmail.com:587
AuthUser=youraccount@example.com
AuthPass=yourpassword
UseSTARTTLS=YES
# The full hostname of this machine
hostname=yourhost.example.com
# Allow users to set their own From: address?
FromLineOverride=YES

You send mail using /usr/sbin/ssmtp just as you would with sendmail, but without any of the sendmail.cf configuration business.

As a bonus, ssmtp sets up symlinks for /usr/lib/sendmail and /usr/sbin/sendmail such that if you have an existing script that calls sendmail it will probably work as is one a system with ssmtp installed.

Here is a ssmtp man page.



 

Erasing Disks on Linux Machines

I'm decommissioning a couple of Linux boxes and want to erase the disks before passing them on to a recycler. There are a couple of favored options:
1: Boot from a Knoppix LiveCD and use the 'shred' utility
2: Use DBAN - Darik's Boot and Nuke

Using Knoppix ought to give you more control over the process but I found it to be less than ideal. This is how you would use it:
1: Boot the machine from a burned copy of the Live CD
2: Open a console in the desktop that appears
3: Run shred with suitable parameters on each partition on your machine, e.g.:

# shred -n 10 -z -v /dev/hdb

This would run 10 rounds of writing random date to the specified disk (-n 10), followed by one round of writing zeroes (-z).

The problem I had was that I didn't remember the disk ids on that machine. Normally you could just run 'fdisk -l' and see what was there... BUT the Knoppix version of fdisk doesn't give you that - dang it - so I had to reboot the machine under its own linux and see what was there. That was annoying - must be another way to do that but I don't know it.

DBAN does what it says on the tin. It boots a machine and then nukes the contents of every disk it can find. It is designed for disk erasing and gives you various degrees of erasure, including the Dept. of Defense standards.

1: Boot your machine with a CD version of the DBAN iso (only 2MB!)
2: Look at the options or just type 'autonuke' at the boot prompt (this gives you the default 'DoD short' level of erasing.
3: Go away and do something else - it takes ages (as in an hour or more)
4: Done...

One thing to note - the current version (as of June 2009) is 1.0.7 - the DBAN site mentions that for Core 2 Duo processors you need version 2 which is a pre-release at this stage.


 

Removing a Linux machine from LDAP

You'll find loads of guides to setting up LDAP authentication, etc. on a network and loads of information about Linux and LDAP. But I want to convert a Linux node that gets its users from a LDAP server into a standalone system with one or two local users and no NFS mounted filesystems. I can't find any information on how to do that. So here is what I came up with...

I have a 'mature' Linux system probably pushing 9 years old (Red Hat 7.3). It gets its user accounts from another Linux system set up as a LDAP server. At the moment I'm trying to simplify my network and as I'm the only user I really don't need LDAP (it is convenient but the systems overhead is not worth it right now.)

The LDAP server handles the user accounts, passwords and the mounting of user home directories via NFS from a third server. I just want one user account on the client with a local home directory.

Before making any changes I can log into the client as 'jones' and get all my home directory files mounted via NFS. If I look in /etc/passwd there is no line for 'jones', but there is one for local user 'root'.

1: Edit /etc/nsswitch.conf (you need to be root) and remove the ldap option from the following lines.

passwd:     files nisplus ldap
shadow:     files nisplus ldap
group:      files nisplus ldap

So your line will look like this:

passwd:     files nisplus
shadow:     files nisplus
group:      files nisplus

These options define the search order for each item. So for a password the order is the password file on the local machine (files), nisplus (if that is still used these days?) and finally LDAP. Removing the 'ldap' option means that if the system can't find the requested user in the local password file it will give up.

2: You also need to rename /etc/ldap.conf to something else

# mv /etc/ldap.conf /etc/ldap.conf.bak

3: Reboot the machine.

Now try logging in as root (root should always be a local user). Now try changing to a user account that was previously valid (e.g. jones in my case). The user should be unknown as we've broken the connection to the LDAP server.

To recreate that user on this client machine do the regular steps:

# /usr/sbin/adduser jones
# passwd jones

Now if I look in /etc/passwd there is a line for 'jones' and I can 'cd' to /home/jones, where I will find an empty directory.

That seems to be all there is to it. There are probably other lines in /etc/nsswitch.conf with 'ldap' in them. Try removing the ldap options, rebooting and verifying that everything still works the way you expect.

You might also want to check /etc/fstab, /etc/auto.master and /etc/auto.misc to make sure you're not mounting any other filesystems from remote machines.

At this point your system should be completely standalone (perhaps save for DHCP). Try unplugging the network cable, rebooting and verifying that it functions as expected.


 

Finding the Address of your DHCP Server

Firewalls and routers are often configured as DHCP servers by default. That can cause issues if you already have a server on your internal network. How do tell which DHCP server any given client is using?

Mac OS X:

$ ipconfig getpacket en0

Where en0 is your Ethernet interface (you might want en1 if you are using a wireless connection)

Windows:

ipconfig /all

Linux:

$ more /var/lib/dhcp3/dhclient.eth0.leases

Where eth0 is your Ethernet interface.