A collection of computer systems and programming tips that you may find useful.
Brought to you by Craic Computing LLC, a bioinformatics consulting company.

Tuesday, December 21, 2010

Per Page Inclusion of JavaScript in Rails view pages

I use various jQuery plugins in my Rails views but I don't want to download all that code in pages that don't use them. Furthermore I want to minimize the chance of namespace conflicts between different blocks of code.

A simple way to do this is to add these lines to your application.html.erb file after the main JS includes:
<%= yield :custom_javascript_includes %>
and then add a block similar to this in each view template that needs custom JS.
<% content_for :custom_javascript_includes do %>
<%= javascript_include_tag "your_custom_plugin.js" %>
<script type="text/javascript">
// Your custom JavaScript code
<% end %>
In this example I'm including a specific file from the public/javascripts directory and then adding some custom JS code that might define some events or trigger some action on page load.

Here is a real example that uses the facebox jQuery plugin which provides 'light box' functionality. This code is placed at the top of my 'show' template.
<% content_for :custom_javascript_includes do %>
<%= javascript_include_tag "facebox.js" %>
<script type="text/javascript">
jQuery(document).ready(function($) {
loadingImage : '/images/facebox_loading.gif',
closeImage : '/images/facebox_closelabel.gif'
<% end %>
You can take this further by also including custom stylesheets linked to these plugins in the same block. That can result CSS and JS includes being interspersed in the resulting HTML file which some people may not like, but I don't think there are any practical drawbacks to it.

The big win is that you keep your custom JS code right there in the same file as your HTML. If that JS code is substantial you could move it into a partial that resides in your views directory, as opposed to having it separated in the public/javascripts directory.

For high performance production sites you should weigh the benefits of this approach in terms of clarity with the performance benefit of stashing all your JS code in a single file and minifying it.


Tuesday, December 14, 2010

MySQL Data export/import probelm with SQL SECURITY DEFINER

Importing a MySQL database dump from a client into my system I got this error:
$ mysql -u root -p  example_db < example_db_dump.sql 
Enter password:
ERROR 1449 (HY000) at line 5172: The user specified as a definer ('smith'@'localhost') does not exist
Running 'grep' for that user turned up a bunch of lines like this:
/*!50013 DEFINER=`smith`@`localhost` SQL SECURITY DEFINER */
These are created on the 'donor' MySQL system when creating one or more views of the data. That user does not exist on my system and so MySQL complains.

I don't care about those views so the easiest way to deal with this issue is to remove these '50013' lines. You can do that with 'sed':
$ sed '/\*\!50013 DEFINER/d' example_db_dump.sql > example_db_dump_clean.sql
You need to drop the new database as the import process partially worked, re-create it and then you can reimport:
$ mysqladmin -u root -p drop example_db
$ mysqladmin -u root -p create example_db
$ mysql -u root -p example_db < example_db_dump_clean.sql

Now, if you need to use the Views in your copy of the database then you need to either create that user locally and leave the lines, or change the user in those lines to one that does exist locally.


Sunday, December 12, 2010

Installing Amazon EC2 tools on Ubuntu 10.04

The Amazon EC2 ami and api tool packages are not found by apt-get using the default sources list. You need to add 'multiverse' to the list. Here is how you install the tools:
$ sudo perl -pi -e 's%(universe)$%$1 multiverse%'  /etc/apt/sources.list
$ sudo apt-get update
$ sudo apt-get install ec2-ami-tools
Of course, you also need to set up your environment variables as per usual so that the tools can access your keys.


Sunday, December 5, 2010

Using Bundler with a non-Rails Ruby project

Bundler is a Ruby Gem to help you manage the various gem dependencies in your application.

It has gotten a lot of attention as it is ,er, bundled with Rails3 and represents a welcome addition to that framework. But you can use Bundler in any Ruby application to help make deployments go smoothly.

For a regular Ruby application the steps involved are simple:

1: Install Bundler on your system
$ gem install bundler
2: Create a file called 'GemFile' at the top level of your application directory
Add gem sources to this and each gem that your application relies on. This example includes the main gem sources and a single gem
source :rubygems
source :rubyforge
source :gemcutter

gem "json"
3: Run 'bundle install'
This will now install any missing gems. It does harm to run it more than once.
$ bundle install
Fetching source index for http://rubygems.org/
Fetching source index for http://rubygems.org/
Fetching source index for http://rubygems.org/
Using json (1.4.6)
Using bundler (1.0.3)
Your bundle is complete! It was installed into /Users/jones/.rvm/gems/ruby-1.9.2-p0
If you look in your application directory you will see a new file called Gemfile.lock. Bundler uses this and you want to keep it, adding it to your git repository (or equivalent). You can also find a .bundle directory containing a config file. I guess you can use this for fancier configurations, but a basic app you can ignore it.

4: Deploy your app
Now when you deploy your app in a new location (with the bundler gem installed) you can simply run 'bundle install' in your app directory and you'll be good to go.


Deleteing a file with a name that starts with a hyphen

Here's a simple fix to a tricky problem (simple with hindsight, of course)...

By accident I created a file called '--latest' and wanted to delete it. But unix thinks that '--' signifies a program option. Escaping the hyphens didn't work either.
$ rm '--latest'
rm: illegal option -- -
usage: rm [-f | -i] [-dPRrvW] file ...
unlink file
$ rm '\-\-latest'
rm: \-\-latest: No such file or directory
$ rm "\-\-latest"
rm: \-\-latest: No such file or directory

The simple solution is to give a more complete path specification that avoids having the hyphens as the first characters.
$ rm ./--latest
Problem solved!


Friday, December 3, 2010

Using Opscode Chef to start up a node on AWS EC2 - A Simple Example

Chef from opscode.com is a suite of tools for managing computing infrastructure, from spinning up new nodes to installing Ruby gems to any custom operation you care to code up. It gets great reviews from those who manage numbers of unix (and other) systems and it has support for Amazon AWS and other cloud vendors baked right in.

I'm just getting started with it and am eager to go further but, like many projects, the documentation is far from clear. It has a lot of options and that typically means complexity. Perhaps because they are targeting folks who do systems administration for a living, their docs skim over some of the crucial basics.

My main interest in Chef is using it to spin up AWS EC2 nodes, configure them to my tastes and then spin them down as needed.

Here are the steps I took to get a node up and running:

0: I'm assuming you know how to fire up EC2 nodes from the management console or the command line on your desktop machine, that you have a keypair set up and that you are familiar with AMIs and the other terminology. If not, then get comfortable with EC2 before going any further.

1: I'm also assuming that you have a basic chef setup on your desktop and an account on Opscode's chef server (it's free for up to 5 nodes) and that you have written and tested a simple recipe on your desktop. If not, get comfortable with that before going further.

2: Add your AWS credentials to your knife.rb file (typically in ~/.chef)
The docs say to enter them in this format, which is different from the other configuration parameters that are generated from the opscode server... I don't know why... but this works:
# EC2 access keys
knife[:aws_access_key_id] = "<your access key id>"
knife[:aws_secret_access_key] = "<your secret access key>"
3: Figure out your EC2 parameters for the node you want to spin up.

The AMI ID - I'm using a 32 bit Ubuntu AMI (ami-480df921)

The Instance Type - I'm testing with a Micro instance (t1.micro)

The Keypair name (mine is called 'craic-ec2-keypair') and the location of the SSH identity file that is linked with this (mine is in ~/.ssh/craic-ec2-keypair.pem).

The User that you (and chef) will ssh in to the node with (Ubuntu 10.04 wants me to use 'ubuntu' instead of 'root')

The Security Group for the server (I just use 'default' which is what I set up when I first started with EC2)

The Chef Recipe(s) and/or Role(s) that you want to run on the new EC2 server (I'm just going to run a simple one that I created called 'craic_test' that writes a file in /tmp) - Make sure this works on a non-EC2 client first!

4: Let's do it...
The chef command that you use is 'knife' with some EC2 specific options. You can find more information HERE in their FAQs, which is MUCH more complete than their Wiki documentation on knife...

The command will be 'knife ec2 server create' followed by a bunch of options. That's pretty self explanatory - but then we get into the options... Here is the full command I used split across multiple lines, which I shall explain below:

$ knife ec2 server create "recipe[craic_test]" \
-i ami-480df921 \
-f t1.micro \
-x ubuntu \
-S craic-ec2-keypair \
-I ~/.ssh/craic-ec2-keypair.pem \
-G default
The first line is the command itself followed by the recipes and/roles in the standard chef format ("recipe[craic_test]").

The EC2 AMI is specified with the -i option, then the Instance Type is specified with the -f option (aka --flavor - I don't know why they chose that...).

Next comes the ssh user with the -x option, the SSH Keypair name with -S (upper case S) and the location of the matching identity file with the -I option (upper case I) - then finally the security group with the -G option (upper case G).

I am really hoping that there is some way to store some of these options in knife.rb or someplace convenient as most of these will not change for my uses.

All being well, what happens next is that the node gets spun up on EC2, chef installs the prerequisites it needs to run on the node and downloads the chef client over there. Chef will create the new node record on the chef server, download your recipes and then execute them.

What you will see is a TON of output as everything runs. I won't subject you to all of that here, but the first few line will look like this:
Instance ID: i-08da8865
Flavor: t1.micro
Image: ami-480df921
Availability Zone: us-east-1b
Security Groups: default
SSH Key: craic-ec2-keypair

Waiting for server............
Public DNS Name: ec2-184-72-190-66.compute-1.amazonaws.com
Public IP Address:
Private DNS Name: domU-12-31-39-0F-29-E8.compute-1.internal
Private IP Address:

Waiting for sshddone
INFO: Bootstrapping Chef on
0% [Working]90-66.compute-1.amazonaws.com
This shows the IP info, etc for your new instance and is followed by pages of messages reporting the downloading and installation of all the prerequisites for Chef. At the end you should see your recipes being run and finally a summary once again of the instance information:
Instance ID: i-08da8865
Flavor: t1.micro
Image: ami-480df921
Availability Zone: us-east-1b
Security Groups: default
SSH Key: craic-ec2-keypair
Public DNS Name: ec2-184-72-190-66.compute-1.amazonaws.com
Public IP Address:
Private DNS Name: domU-12-31-39-0F-29-E8.compute-1.internal
Private IP Address:
Run List: recipe[craic_test]

5: Now you can ssh into your node and you should see that your recipes have run and done whatever you asked of them.

One good thing about all the prerequisites getting installed, for an Ubuntu system at least, is that you now have a node with a lot of development libraries and tools already in place (especially if you work in Ruby).

That's it for the basic installation.

Remember to terminate your node when you are finished as you are being charged for it AND remove any test nodes from your Opscode server account as they will count against your account limit.

There is obviously a lot more that you can, and need to, do to set up your nodes but this walk through gets you to a functioning node.

Happy cooking

System-wide rvm install on Ubuntu

Here are the steps I used to install RVM system wide on a Ubuntu 10.04.1 LTS node on Amazon EC2, using one of the Canonical AMIs. These set up the primary user as 'ubuntu', requiring you to sudo for system commands. I could install rvm under that user but installing System-Wide is recommended.

This HOWTO is based on the RVM page http://rvm.beginrescueend.com/deployment/system-wide/. But I find the RVM site documentation makes a few assumptions with regard to commands.

1: Install git if you don't already have it
$ sudo apt-get install git-core
2: Download RVM
$ sudo bash < <( curl -L http://bit.ly/rvm-install-system-wide )
This creates /usr/local/rvm and the group 'rvm'

3: Add each user to group rvm
$ sudo usermod -aG rvm ubuntu
4: Add startup lines to each user's .profile
[[ -s '/usr/local/lib/rvm' ]] && source '/usr/local/lib/rvm'
5: Add this line to /etc/rvmrc (may not be necessary)
$ echo "export rvm_path=/usr/local/rvm" > /etc/rvmrc
6: Use rvm as a user - best to logout/login first to make sure you get all the paths etc setup
$ rvm list
$ rvm package install zlib ( I needed this to get install to work )
$ rvm install 1.9.2 --with-zlib-dir=/usr/local/rvm/usr
Note that you use rvm as a regular user, not via sudo, even though this is a system wide install.


Archive of Tips