Craic Computing Tech Tips

Getting absolute paths with the Unix ls command

2012-01-27T11:16:00.000-08:00

The standard Unix command ls lists filenames and directories in the specified directory. The default behaviour is to list just the filenames as including the full pathname would clutter the screen.

But sometimes you want the absolute paths. I need this all the time if I want to create a file containing a list fo filenames. The obvious command to get all the YAML files, for example, is:
$ ls -1 *yml
A.yml
B.yml

In order to get the full pathnames you need to use this:
$ ls -1 -d $PWD/*yml
/home/jones/A.yml
/home/jones/B.yml

Wolfram CDF Player and Chrome Browser

2012-01-25T12:23:00.000-08:00

Wolfram Computable Document Format (CDF) is a way to embed interactive documents into web pages, in particular those that perform calculations in response to a user changing the parameters. For example you can create graphs of functions that will change as the function is modified. It is an extension of the Wolfram Mathematica software.

It looks really promising for some applications and you should check out their demonstrations - some impressive, some not so much.

You 'play' CDF files using a browser plugin - just like Flash - and these are available for all the current browsers.

I'm running it on Google Chrome on Mac OS X on a fairly recent laptop. It performs OK depending on the specific application and the amount of data it is asked to push around. But when you close that window or move to another page the CDF player process continues to run. In my case that was taking 5% of my cpu and 66MB of memory and it continued to do so for perhaps 10 minutes after the page was closed.

This sort of plugin drain on your cpu is fairly common - just look at everything going on in Activity Monitor when you are browsing an 'active' web page with ads, etc.

In Chrome you can go to Window -> Task Manager, select a process and End it - but that didn't appear to do anything in my case.

CDF looks very interesting but if it requires too many resources, and then fails to release them properly, then it is not likely to be broadly adopted. It is something to keep an eye on, for sure.

Disabling Spotlight (mds) on Mac OS X (Snow Leopard)

2012-01-04T08:59:00.000-08:00

I run a lot of command line scripts on my laptop - some of which can run for hours. I want to continue using the machine for reading mail, etc., but I don't want any other intensive task sucking up the cpu cycles. So I shut down iTunes, don't watch any videos, etc.

But sometimes I see some other process taking all my cycles. The odds are that it is either something to do with Flash or it is a process called mds.

mds is the indexing software that powers Spotlight - the built in Mac search facility.

I suspect that when I'm generating gigabytes of data and hundreds of files in one of my compute jobs, mds is responding by trying to index them at the same time.

I don't use spotlight at all, so let's turn it off and see if that helps.

This turns it off:
$ sudo mdutil -a -i off

This turns it back on:
$ sudo mdutil -a -i on

Turning it back on will presumably trigger a big mds run as it plays catch up, so run this command only when you can afford the cycles.

strsplit in R

2011-11-17T08:48:00.001-08:00

The strsplit function in the R statistics package splits a string into a list of substrings based on the separator, just like split in Perl or Ruby. The object returned is a List, one of the core R object types. For example:
> a <- strsplit("x y z", ' ')
> a
[[1]]
[1] "x" "y" "z"
> class(a)
[1] "list"

If you are not that familiar with R, like me, the obvious way to access an element in the list will not work:
> a[1]
[[1]]
[1] "x" "y" "z"
> a[2]
[[1]]
NULL

So what do you do? There seem to be two options:

1: You can 'dereference' the element (for want of a better word) by using the multiple sets of brackets

> a[[1]][1]

[1] "x"

> a[[1]][2]

[1] "y"

... but I'm not going to write code that looks like that !!

2: You can unlist the List to create a Vector and then access elements directly

> b < unlist(a)

[1] FALSE FALSE FALSE

> b <- unlist(a)

> b

[1] "x" "y" "z"

> class(b)

[1] "character"

> b[1]

[1] "x"

> b[2]

[1] "y"

Much nicer !

Running R scripts on the Command Line

2011-11-16T13:26:00.001-08:00

Using the R statistics package via a GUI is great for one off tasks or for learning the language, but for repeated tasks I want the ability to create and run scripts from the UNIX command line.

There are several ways to do this:

R CMD BATCH executes R code in a script file with output being sent to a file.
$ R CMD BATCH myscript.R myoutputfile
If no output file is given then the output goes to myscript.Rout. There is no way that I know of to have it go to STDOUT. Passing parameters to your script with this approach is a pain. Here is an example script:
args <- commandArgs(TRUE)
for (i in 1:length(args)){
print(args[i])
}

This is invoked with this command:
$ R CMD BATCH --no-save --no-restore --slave --no-timing "--args foo=1 bar='/my/path/filename'" myscript.R tmp
In particular, note the strange quoting of the arguments, preceded by --args inside the outer quotes - it's not a typo!

That command produces this output in file 'tmp':
[1] "foo=1"
[1] "bar='/my/path/filename'"
All those '--' options are necessary ! Try leaving out --slave and --no-timing and you'll see why.

Thankfully there is a better option ...

Rscript is an executable that is part of the standard installation

You can add a 'shebang' line to the file with your R script, invoking Rscript, make the file executable and run it directly, just like any other Perl, Python or Ruby script.

You don't need those extra options as they are the default for Rscript, and you pass command line options directly without any of that quoting nonsense.

Here is an example script:
#!/usr/bin/env Rscript
args <- commandArgs(TRUE)
for (i in 1:length(args)){
print(args[i])
}

Running this with arguments:

$ ./myscript.R foo bar

produces this output on STDOUT (which you can then redirect as you choose)

[1] "foo"
[1] "bar"

Much nicer - but we've still got those numeric prefixes. If you are passing the output to another progran these are a major pain.

The way to avoid those is to use cat() instead of print() - BUT you need to explicitly include the newline character as a separate argument to the cat() function

#!/usr/bin/env Rscript
args <- commandArgs(TRUE)
for (i in 1:length(args)){
cat(args[i], "\n")
}
results in this output:

$ ./myscript.R foo bar
foo
bar

For the sake of completeness, you can run R scripts with a shebang line that invokes R directly. But Rscript seems to be the best solution.

If you want to pass command line arguments as attribute pairs then you need to parse them out within your script. I haven't got this working in a general sense yet. What I want is to pass arguments like this:
$ ./myscript.R infile="foo" outfile='bar'
But I'm not quite there yet...

Deleting a File that starts with '-' on UNIX

2011-11-16T12:35:00.001-08:00

Filenames that begin with 'special' characters like '-', '.' or '*' cause problems on Unix. Standard commands like ls or rm view the characters as signifying command options.

You don't typically create files with names like this but they can arise through errors such as cut and pasting text into your command line.

Simply escaping the character or quoting the filename does not work.

The solution is to use a longer path to the file - the easiest being a relative path to the same directory.

If the filename is '--myfile' you will get an error like this:

$ ls --myfile

ls: illegal option -- -

usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]

But this works just fine:

$ ls ./--myfile

./--myfile

Plotting a simple bar plot in R

2011-11-16T10:53:00.001-08:00

Here is my cheat sheet for loading data from a CSV file into the R statistics package, plotting one column of data as a bar plot and saving it as a PNG image.

My input file is simple CSV file with two columns :
Position,Entropy
1,0.2237
2,0.4051
3,0.1312
4,0.1312

[...]

I want to load this into R as a data frame, then plot the values in the second column (Entropy) as a bar plot, using the values in the first column as the labels for the bars.

First step is to use read.csv (a shortcut version of read.table)

> d <- read.csv('<your path>/myfile.csv')

I can use barplot directly on the data frame (d)

> barplot(d[,'Entropy'])

But the default plot options are not great, so I can add custom options to the call, such as a main title and labels for X and Y axes. I set the lower and upper limits for the Y axis to be 0.0 and 1.0 and use the values in the first column of the data frame as the labels for the bars on the X axis

> barplot(d[,'Entropy'], main="Entropy Plot", xlab="Position",
ylab="Entropy", ylim=c(0.0,1.0), names.arg=d[,'Position'])

The plot is displayed on my screen and looks the way I want it. To save it out to an image file, I specify the plotting device ('png') and the output filename, repeat the plot and then close/detach the plotting device.
> png('<your path>/myfile.png')
> barplot(d[,'Entropy'], main="Entropy Plot", xlab="Position",
ylab="Entropy", ylim=c(0.0,1.0), names.arg=d[,'Position'])
> dev.off()

This produces the following image:

There are endless configuration options to play with but this works for a quick, simple plot.

Here are the steps without the prompts for you to cut and paste as needed:

d <- read.csv('<your path>/myfile.csv')
png('<your path>/myfile.png')
barplot(d[,'Entropy'], main="Entropy Plot", xlab="Position", ylab="Entropy", ylim=c(0.0,1.0), names.arg=d[,'Position'])
dev.off()

You could put these into a text file and run it from your system command line like this:

$ R CMD BATCH myfile.R

R is an incredibly useful system but as an occasional user I find the syntax and command names/options hard to learn. Hopefully this simple example helps you with the learning curve.

... and always remember - arrays in R start at 1, not 0 ...

Captain Beefheart Song Titles

2011-11-14T10:34:00.001-08:00

Here is a silly project that I knocked out last week - a generator of Fake Captain Beefheart Song Titles

It was inspired by a comment on Gideon Coe's radio show on BBC 6 Music where he wondered if such a thing existed - it didn't - so I wrote one!

The 'algorithm', if you can call it that, combines words from real Beefheart song titles and a list of others that sound (to me) like they could be. The structure of the real titles is relatively simple with most of them following a few simple patterns. Some of the generated titles don't work but every so often it'll spit out a good one.

The site is built with Ruby and Sinatra, with some jQuery thrown in for the scrolling effect. It serves as a nice example if you are learning Sinatra. You can find the code at Github and the application is hosted on Heroku.

It's just a bit of fun, but writing a small, self-contained application like this is a great way to learn new technologies. Giving yourself a short time frame in which to develop and deploy a complete application is a great exercise in efficiency.

When jQuery $.ajax() keeps returning the same value on Internet Explorer

2011-11-11T13:55:00.001-08:00

I just mentioned this in my last post, but it is important enough that I'm going to give it its own post!

If you have jQuery Ajax calls, such as $.ajax() , $.get(), etc., that are continually fetching data from a remote server, you may find this works on Firefox, Chrome, etc. but in Internet Explorer it keeps returning the same value.

IE has cached the first return value and thinks that you are making identical requests so it just uses the cached value and never contacts the server.

Turn off caching by adding a call to ajaxSetup at the start of your script block and set cache to false.

<script>

$.ajaxSetup ({
cache: false
});
[...]
</script>

The fix is simple once you realize it but it took me a while to figure it out this morning.

Debugging JavaScript is painful at the best of times. It's worse when you throw AJAX into the mix. So take small steps and test on multiple platforms right from the start. I need to learn that lesson.

Updating a DIV with jQuery and Sinatra

2011-11-11T10:09:00.001-08:00

Here is a minimal example of how to update a DIV in a web page with content from a remote Sinatra server using jQuery AJAX.

On the Server end you want an action that generates some new content and just returns it as a string. Let's just get the current time:

get '/update' do
Time.now.to_s
end

In the client web page you need to include the jQuery library:
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.7/jquery.min.js"></script>

You need a named DIV that will receive the content

<div id='update-div'></div>

Then you need to create your script block. You need a function that updates the DIV. update_time() calls the jQuery get function with two arguments. The first is the URL of the server action. The second is a function that is called on the returned data and the third is a type of data that the server will return, which is 'text' in our case.

Here we update the named DIV with the string returned from the server. Then we use the setTimeout function to wait 5 seconds, and then we call the function itself to repeat the process. That instance fetches content from the server, waits 5 seconds, calls itself, and so on.

At the start of the script block we add a call to ajaxSetup and set cache to false. This seems to be necessary for Internet Explorer which would otherwise cache the first value returned and keep using that instead of fetching new values.

Finally we make the initial call to the function when the document is first loaded and ready.
<script>

$.ajaxSetup ({
cache: false
});

$(document).ready(function() { update_time(); });

// fetch text from the server, wait 5 secs and repeat
function update_time() {
$.get("/update",
function(data) {
$("#update-div").html(data);
window.setTimeout(update_time, 5000);
},
'text');
}
</script>

The jQuery get function is a simplified wrapper around the ajax call.

That is all you need to have a DIV continually updated with remote content, in this case coming from Sinatra.

Latest version of jQuery on Google APIs

2011-11-11T09:48:00.001-08:00

If a web page uses the jQuery library it usually makes sense to link to copy hosted on an external content delivery network (CDN). You don't have to keep an updated copy on your site, the client may already have this copy cached, and the CDN has high bandwidth connections. Google, Microsoft and jQuery themselves offer CDN hosted jQuery.

I use the Google version and call it with this line:

<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.0/jquery.min.js"></script>

That is fine, but what happens when a new version comes out?

In these cases some sites will offer a 'latest' url that automatically points to the most recent version.

Google does not do that - but they offer something similar, and which might be slightly better.

If I want version 1.7.0 and nothing else then I use the link given above.

If I want the latest version in the 1.7 series then I use:

And if I want the latest version in the 1 series then I use:

Note that you don't always want to use the most general form. New releases are not necessarily back compatible. But you have the choice.

And note that you no longer need to include type="text/javascript" in the script tag anymore - it's the default.

Sorting on multiple String keys in Ruby

2011-11-10T13:08:00.001-08:00

Ruby gives you two ways to sort arrays of 'complex' objects - sort and sort_by

Consider this array of hashes

array = [{:key0 => 'foo', :key1 => 'bar'},{:key0 => 'hot', :key1 => 'cold'}, ... ]

sort_by is the most compact form. To sort on :key0 in ascending order you write:

array.sort_by{ |a| a[:key0] }

sort is a little more verbose, but offers more flexibility:

array.sort{ |a, b| a[:key0] <=> b[:key0] }

If you want to sort strings in descending order then you switch the a and b assignments:

array.sort{ |a, b| b[:key0] <=> a[:key0] }

You can use sort_by with multiple keys if you include them in an array:

array.sort_by{ |a| [ a[:key0], a[:key1] }

But if you are using strings and you want one of the keys sorted in descending order then you need to use sort.

In this example the desired ordering is to first sort on :key0 in descending order and then on :key1 in ascending order:

array.sort do |a,b|
(b[:key0] <=> a[:key0]).nonzero? ||
(a[:key1] <=> b[:key1])
end

Not the most elegant or concise piece of code, but it does the job.

Manipulating Model/Table/Controller Names in Rails

2011-11-09T15:43:00.000-08:00

There are some very useful methods that operate on the names of your models, tables and controllers in Rails. I use them infrequently enough that I have to relearn them every time - so I'm putting them in here:

Given a string with the name of a model (e.g. Project), how can I create an ActiveRecord call on the fly using that model?

str = 'Project'
project = (str.constantize).find(1)

How can I get the name of the corresponding controller ?

controller = str.tableize

Given a string with the name of a controller, how can I get the name of the Class/Model ?

str = 'sales_contact'
model_name = str.camelize

I the string is plural:

str = 'sales_contacts'
model_name = str.classify

How do I do the reverse operation?

str = 'SalesContact'
name = str.underscore

Parsing 96 / 384 well Plate Maps

2011-11-09T13:16:00.000-08:00

In molecular biology and related fields, we use rectangular plastic plates with 96 or 384 wells for many purposes, allowing us to run many experiments in parallel. In particular, they are the standard format for tasks involving laboratory robotics, such as fluid dispensing, DNA sequencing, mass spectrometry, etc.

In setting up a plate of samples, it is common for a researcher to use Excel to create a 'plate map' - a table of 8 rows and 12 columns, for a 96 well plate, with each cell holding information on the sample in that well.

Converting this into a list of individual wells for input into an instrument or a piece of software should be straightforward but variability in manually created plate maps can be an issue. Extraneous text and the placement of the table in the spreadsheet are just two of them.

I have to deal with this problem every year or two and so I decided to write a utility script that implements this conversion and offers several options for the output file.

It is not intended as a complete solution for any one application. Rather it is a starting point that I can customize as needed.

I know other people have to deal with the same type of data, so I've made the script as general as possible and put it up on github under the MIT license. Please adapt this as you see fit for your needs.

The script can take CSV or Tab delimited files as input, containing 96 or 384 well plate maps. The output is a table with one well per row and the columns : plate, row, column, value. Options allow you to specify CSV or Tab delimited as the output format, whether to parse the input in row or column major order, and whether or not to output empty cells in the plate map.

The code is written in Ruby and you can find it at https://github.com/craic/plate_maps

Let me know if you find it useful

Unzipping multiple zip files on Unix

2011-08-10T08:10:00.001-07:00

To unzip multiple .zip files with one command command on a UNIX system the obvious command to try is 'unzip *.zip' - but that doesn't work...

$ unzip *zip

Archive: ipab20080103_wk01.zip

caution: filename not matched: ipab20080110_wk02.zip

caution: filename not matched: ipab20080117_wk03.zip

[...]

UNIX is expanding the wildcard and passing that string to unzip, which thinks that you are asking for specific files from the zip archive given as the first argument.

You need to escape the wildcard with a backslash so that 'unzip' sees it and expands it correctly.

$ unzip \*.zip

Archive: ipab20080103_wk01.zip

inflating: ipab20080103.xml

inflating: ipab20080103lst.txt

inflating: ipab20080103rpt.html



Archive: ipab20080110_wk02.zip

inflating: ipab20080110.xml

inflating: ipab20080110lst.txt

inflating: ipab20080110rpt.html

[...]

Glad I stumbled on that - saves me a lot of time...

Default Auto Capitalization in HTML Forms with Safari on iPad2

2011-07-26T20:20:00.000-07:00

Ran into a nasty gotcha testing out one of my web sites on an iPad2 with Safari.

By default the iPad will capitalize the first letter in a sentence. But with my application the login page requires an email address and these are typically in al lower case.

Safari forces the first letter of the email address into upper case, e.g. Jones@example.com - but this does not match the text in the application's databases (jones@example.com).

My app doesn't force text into lower case on its end (and it shouldn't !), which means that I could not login to my site. And in this case using the SHIFT key on the keyboard does not let you enter the first letter in lower case.

The solution is for the User of the iPad to go to Settings -> General -> Keyboard and set Auto-Capitalization to Off.

This is a really bad default setting - yes, it can be useful if you are entering regular text - but it's not that useful...

I can't expect my users to deal with this on their end. I could force all email address into lower case in my application and that is probably what I'll end up doing.

HTML5 forms can define that a text entry box represents an email address. This adds convenience when entering from a mobile device (the keyboard can display a '.com' key, for example). Whether or not using this has any effect on auto-capitalization I have yet to test.

Firefox Security and file:/// URLs cause problems when testing HTML5

2011-07-21T10:03:00.000-07:00

I'm experimenting with some of the great features in HTML5 and specifically with Web Storage. The way I typically try out things like this is to write a simple HTML page with the relevant JavaScript and CSS code and then open that in the browser.

With my Web Storage example, things work great in Chrome and Safari but nothing happens in Firefox 5 on the Mac - even though I know the browser supports this feature.

The issue lies in the security model behind Firefox. It is quite restrictive in what it will let you do with file:/// urls. The intention is to avoid any chance of a remote page being able to access a local file.

That is fine, but there is no obvious alert or notification that Firefox is doing this - instead your code just doesn't work. If I hadn't run it on Chrome first, my reaction would have been to question my own code.

So bear that in mind if you work the same way that I do - perhaps use Chrome as your first choice for development - and if something simple doesn't work, try it in a different browser before questioning your own code.

Rails meta_search - sorting on custom columns

2011-06-10T11:17:00.000-07:00

I use Ernie Miller's excellent meta_search gem for searching the contents of tables in my Rails 3 Apps.

I have a lot of index views that are tables of data and each has a search form at the top of the page that uses meta search and the column headers use the sort links provided by meta_search to reorder the rows.

Typically I also have custom columns, such as the number of objects that are associated with a given row via some relationship. Some of those relationships can be quite complex and in some cases there is no simple way to express them in a way that works with meta_search.

I handle the sorting in the controller. It's outside the database but with my current application the amount of data makes this perfectly manageable.

I've come up with a way to let my custom column sorting coexist with meta_search and I reckon it might be useful to others.

All the code can be found at this gist on Github :https://gist.github.com/1019358

It works by adding another parameter called 'custom_sort' to the existing search[] parameters. The index action in the controller sees this, performs the custom sort and passes a custom_sort value to the view.

The view calls the custom_sort_helper with a set of arguments and that creates the column header link with the correct parameters to pass back to the controller.

So far it works well and allows me to sort on my custom columns with a set of rows selected by meta_search searches.

I'm sure the code could be more elegant - leave me a column and tell my how.

Rails send_data and UTF-8

2011-06-07T08:58:00.000-07:00

I often use send_data to output plain text from a Rails action, typically as a simple way to download, say, a CSV format file.

I'll have something like this to output the text to the browser:

send_data(@output, :type => 'text/plain', :disposition => 'inline')

But if the text is UTF-8 and contains non-ASCII characters then you need to specify the character set.

Confusingly, to me, you don't do this with a separate argument at the Rails level. Instead you add it as a modifier on the :type argument like this:

send_data(@output, :type => 'text/plain; charset=utf-8', :disposition => 'inline')

Rails migration add_column and Postgresql

2011-06-03T11:04:00.000-07:00

Just ran into a difference between MySQL and Postgresql when running a migration that added a new column to an existing table.

The intent of the original line was to add a new boolean column called 'active' and set the default value to true:

add_column :people, :active, :boolean, :default => true

This works fine on MySQL but with Postgresql the default value was not used on existing records. It appears that Postgresql sees that NULL is an acceptable value for the column and uses that as the value.

The way to fix this is to add a 'null' attribute to the call like this:

add_column :people, :active, :boolean, :default => true, :null => false

Now, running rake db:migrate sets all existing records to true

Google Chrome Browser, Integers and HTML5 forms

2011-06-03T10:21:00.000-07:00

In HTML5 forms you can specify the type of input that is expected with the 'type' attribute.

For example, type = "number" specifies that the input tag expects a number. In general this is great as it permits client side validation, but it can have some side effects in some cases.

The specific case that is causing me problems occurs in the Google Chrome browser when it handles integers entered into an input element with type = "number".

Rather than simple displaying the integer, Chrome inserts commas into the number.

For example, 123456789 becomes 123,456,789

The commas are not included when the form is submitted but they are there if I cut and paste the text.

I don't want these - if I give it an integer all I want to see if the integer. But there is no readily accessible option in Chrome to disable this feature.

The workaround is to explicitly set the type to text (type = "text"). You lose the client side validation but you avoid the commas.

Now, I write Rails applications and use the simple_form gem to help create forms. This knows about the types of input each form input is going to accept and so it liberally uses the 'type' attrbutes. Fortunately you can override these as follows:

<%= f.input :myinteger, :input_html => { :type => 'text'} %>

The real solution, in my opinion, is for the browser to act very conservatively in interpreting user input. If the server wants the user to see commas then it should tell the browser explicitly.

Distinct Logins to a Rails App using Google Chrome Browser

2011-06-03T10:07:00.000-07:00

If you are developing a Rails app that requires user login, then it can be really helpful to have more than one browser window open at a time, with a separate user logged into each.

For example, I typically have an Administrator user with special privileges as well as regular users. I want to test my app from both perspectives at the same time.

But because my app uses cookies to handle session information, I can't simply have two logins from the same browser. Until now I've dealt with this by having one user in Firefox and one in Safari, or Chrome.

Google Chrome allows you open Incognito Windows that store cookies, history, etc. in a sandbox that is destroyed when you close the window. This allows me to manage two active logins from Chrome.

Open up your first account in a normal window, then open an Incognito window and login as the second user... Simple!

Cookies, etc are shared between all Incognito windows and so you only get to have two active users, but even so this is really useful.

Firefox and other browsers have similar modes - Firefox has 'private browsing' - these may well give you similar functionality.

Craic Therapeutic Antibodies Database

2011-06-01T13:52:00.000-07:00

Craic Computing LLC is pleased to announce the launch of the Tabs Therapeutic Antibody Database

Tabs is a unique database focused solely on Therapeutic Abs under development by the Biotechnology industry.

As of 1st June 2011, Tabs contains data on more than 950+ antibodies, targeting 400+ antigens, being developed by 300+ companies.

Antibody records are linked to a wide range of associated data including:

Patents
Papers
Clinical Trials
Antigens
Companies
Conditions/Indications
Regulatory Actions
Protein Sequences
Protein Structures
Press Releases
Development Timelines
Conference Abstracts

The database is intended for Biotechnology industry staff - especially those in R&D and in Business Development.

Research staff have direct access to relevant patents and papers for each antibody.

BizDev staff can see the big picture of developments against a target antigen across companies.

Users can define custom Antibody Sets and download data in to Excel, bioinformatics software and popular Reference management tools. Whenever new data related to Antibody Sets are added to the database, users can be alerted by email.

Tabs is offered as a web based subscription service to biotech companies. The annual subscription offers unlimited access to unlimited users for a given site.

Tabs can be evaluated with a 30 day Free Trial. Sign up for an account here.

Tips for poking around a Win XP machine

2011-04-01T13:15:00.001-07:00

Thankfully I encounter Win XP PCs rarely enough that I forget how to find my way around them.

Here are few of the tricks that I am using as I try and recover a heavily infected PC for my in-laws.

1. Reboot the PC while holding down the F8 key will let you select Safe Mode, Safe Mode with Network and other non-standard boot modes.

2. Windows hides a lot of files from regular users. To reveal everything go to a Folder and then Tools, Folder Options and View.
In the list of options that are shown, you want to Check 'Hidden files and filders' -> 'Show hidden files and folders' and 'Display the contents of system folders' and you want to UnCheck 'Hide protected operating system files'

3. Start up a Command Shell either by the Start Menu -> Accessories -> Command Prompt, or Start Menu -> Run and enter cmd.exe.

The cmd shell is a basic DOS (yes, really) shell. Type 'help' for options. 'dir' lists the contents of a directory, 'cd' moves you around. 'del' deletes a file.

'cd' has a pseudo-auto-complete function. Enter 'cd "' (cd space double quote) and then tab will cycle through the options.

4. From the command shell:
'ipconfig' shows your IP address etc.
'netstat -am' shows the network services in use.

5. Ctrl-Alt-Del brings up a window with the running applications and processes, which is useful for spotting rogue processes.

I really hope that you, and more importantly I, never have to use this knowledge again...

Getting rid of McAfee antivirus products on a Win XP PC

2011-04-01T07:57:00.000-07:00

McAfee antivirus products are widely installed on PCs. I'm sure they work fine but they have a reputation of being intrusive when you let your subscription lapse. The company is not alone in this. With a number of products you will get intrusive popups and warnings if you let your subscription lapse.

What you should be able to do is go to 'Add/Remove Programs' in your Control Panel and uninstall the software - just like most other professional software allows you to do.

For some reason, at least in their older products, McAfee has chosen to make this difficult. In order to remove most (not all) traces of McAfee from your system you should get the MPCR.exe program from McAfee and run it.

http://download.mcafee.com/products/licensed/cust_support_patches/MCPR.exe

Download this onto your PC and run it - it will take a while and it will popup black 'command prompt' windows while it runs, each with cryptic text indicating the individual scripts that are being executed. Just let it do it's thing.

Restart you machine when it is done.

When you are trying to fix an infected PC, these remnants of old antivirus software clutter up your PC's registry and other directories.

Hope this helps

Most efficient way to remove the XP Home Security Malware from a PC

2011-04-01T07:42:00.000-07:00

My in-laws Win XP PC became infected with malware about a week ago. It was the XP Home Security malware that pops up windows warning you about being infected and appears to scan real files on your PC that it says are infected. In addition the browsers on the machine were hijacked such that you could get to Google OK but when you clicked on any other link you would be redirected to another seemingly random sites. It was a mess...

Over the past week I've put in at least 11 hours work on the problem and run up 50 miles of driving back and forth to their house. I still don't have a complete fix... I'm going to post a few insights here over the next few days but I wanted to start out with my advice if you have the same problem.

1: Unplug the PC's network cable
2: Reboot the machine and hold down the F8 key in oder to see the boot options menu
3: Select Safe Mode, or Safe Mode with Networking
4: Pull off your user files onto a USB flash drive or disk drive
5: Turn the machine off
6: Go out and buy a Mac

I'm not trying to be funny (even though it is April 1st) - this is really the most effective way of dealing with this problem.

The time and frustration involved in sorting out a mess like this is simply not worth it.

Cut your losses, go buy a nice new Mac - you'll love it and you won't that these problems

Rails 3 ActionMailer - defining default host

2011-03-25T15:12:00.000-07:00

In Rails 3 ActionMailer is really easy to set up and use. Ryan Bates has a great Railscast on the topic that you should check out.

But I ran into one issue when creating emails that contain links back to my server.

Because you are sending an email to a remote machine, you need to include absolute URLs in your links.

You can define that in a config file but the example given in the Railscast is now deprecated, plus when I first tried it, my app was not picking it up and all my links were still relative.

You two things for this to work:

1: In your application.rb or environments/development.rb add a line like this in the configuration block, replacing craic.com with your domain

    config.action_mailer.default_url_options = { :host => "craic.com" }

The application.rb file applies to all environments. Putting this in development.rb or production.rb lets you use different hosts for each environment.

2: There are several ways to specify a url to pass into a link_to call in Rails - but only one seems to work.
Specify the urls for your links using <your_model>_url(<your_object>) - not <your_model>_path(<your_object>) and not url_for(...). I tend to use mymodel_path in links in regular web pages so I can't just cut and paste code into a mailer view.

Here is an example that works:

<%= link_to object.name, mymodel_url(object) %>

...produces...

<a href="http://craic.com/mymodels/1">My Object</a>

Tandem Select jQuery Plugin

2011-03-18T16:04:00.000-07:00

A standard HTML multiple Select menu allows the user to select multiple options using Command-Click. But when the list of options is long, requiring the user to scroll, it becomes cumbersome and prone to errors.

An effective solution is to use two select menus in tandem, with buttons allowing options to be swapped between the two.

I'm pleased to announce the release of a new jQuery plugin that makes tandem selects simple to set up and easy to customize.

Tandem Select consists of a JavaScript function that leverages the jQuery library, a CSS file and template HTML code.

More information and live demos can be found on the Project Page and the software can be downloaded from the GitHub repository.

Adobe AIR Application Installer running flat out on MacBook

2011-03-14T17:00:00.001-07:00

My MacBook (model 5,1 - few years old) running Mac OS X 10.6.5 has had a few incidents recently where it's responsiveness takes a nosedive. That typically means that some process is pegging the cpu at 100%.

I run into fairly frequent problems with Firefox, especially if a browser window is running Flash - say, for a video feed or an advert. Depending on the level of ambient noise in my office, I can even hear the fan and/or disk in the laptop running at full speed.

You can use the UNIX 'top' command in a shell to find out who the culprit is, plus other tools like 'lsof', etc. But I find the Apple desktop utility 'Activity Monitor' (in your Applications -> Utilities folder) is more useful. Open it up and select 'All Processes' in the select menu at the top of the window. Then click on the CPU tab down below and then on the cpu column to sort the processes in descending order based on % cpu utilization. Look for something that is up near 100 %.

If there is nothing striking then look for something with very high Real Mem and Virt Mem and check out the DiskActivity tab as you might have a process that is so memory hungry that it has to page data out to disk.

The example in the screenshot shown above is a straightforward CPU hog. It is the Adobe AIR Application Installer. I figure this is involved in installing the Adobe AIR runtime libraries that are used by Flash.

I have no idea why this is running in the first place - and why is it using 100% of my cpu? Flash and associated technologies can be great but they do seem to be around when I have problems...

If you have the same issue, just select the AIR Installer line in Activity Monitor and click the big red 'Quit Process' button.

Sorting Dates and Times in Ruby

2011-03-14T14:39:00.000-07:00

Sorting an array of objects on a field that represents a time or date can be a bit tricky.

Take an example of a Rails ActiveRecord object with a standard 'created_at' column.

You can sort these with code like this:

myobjs.sort_by{|a| a.created_at}

This works but it is really sorting on the String representation of those dates. That may be good enough for some uses but not if you want to sort down to the minute and second and in particular, if you want to sort in correct descending order.

The best approach is to convert the date or time to the number of seconds since the epoch, which is an integer, and do a numeric sort on that.

A 'created_at' column has the class 'ActiveSupport::TimeWithZone' and this will output a String under most uses. Convert this to an integer with 'myobj.created_at.to_i' and then sort on that.

myobjs.sort_by{|a| a.created_at.to_i}

If you are working with Ruby Date objects, such as 'Date.today' this will not work. In most uses the Data will be converted to a string like this "Mon, 14 Mar 2011". If you try to convert the Date object directly to an integer with 'to_i' you will get an 'undefined method' error.

Here you need to explicitly convert to the epoch seconds format using 'strftime' and THEN convert that to an integer.

> today = Date.today
 => Mon, 14 Mar 2011
> today.strftime("%s").to_i
 => 1300060800

If you only have date strings, such as '2011-03-14', then you will need to convert these to Date objects using 'Date.parse' and then convert those to integers.

Working with dates and times can get messy but converting to integers is the best way to avoid complications.

simple-tooltip Ruby Gem for tooltip help

2011-03-04T15:14:00.000-08:00

I'm pleased to announce the release of my first Ruby Gem: simple-tooltip

Simple Tooltip helps you make tooltip help available in your Rails 3 applications.

This is often done using a JavaScript plugin which produces a small popup window containing a short text string. That works great in many applications but often the help content that you want to provide is more extensive and you many want to include richer formatting, such as links to other web pages.

Simple Tooltip acts as a Rails Generator which will create a Tooltip model in your application along with a database table, a controller and views. You create help records using a web interface using plain text, html, Textile or Markdown formatting. The help records are stored in the database and retrieved using the unique, informative titles that you give them.

In your web pages you include a Tooltip helper that refers to the title of the relevant tooltip. Depending on how you set it up, when you hover over a tooltip icon or click on it, a popup window will appear displaying your help.

This is much richer and more flexible than the basic solutions available from JavaScript alone.

More than that, if the audience for your application is not just English-speaking, you can create multiple instances of each tooltip with content in each target language, identified by a two letter locale string. If a user has a non-English locale defined in your application, Simple Tooltip will look for content in the matching locale and display that. For content longer than single sentences, this is a more practical internationalization solution than I18n functionality built into Rails 3, which is more focused on shorter strings.

The gem is available at https://rubygems.org/gems/simple-tooltip.

The code is available at https://github.com/craic/simple_tooltip and is made freely available under the MIT license.

Passing Environment Variables to the nginx web server

2011-03-01T14:22:00.000-08:00

I need to write this down for future reference...

With the Apache web server you can pass environment variables to Rails applications, etc by specifying them using SetEnv in your configuration file. But with nginx you don't do this - or at least you don't need to.

When you fire up the server, simply tell sudo to preserve your environment. Without that, sudo will reset all your variables. With it, they are available to nginx and your applications - nice and simple...

$ sudo -E /usr/local/sbin/nginx

Rails, UTF-8 and Heroku

2011-02-24T14:59:00.000-08:00

I've had problems with Ruby character encodings over the years, especially when pulling text with non-ASCII characters in from remote sites. I thought I had it mostly sorted but the past few days showed me that was not the case.

I have a MySQL database on my local machine and a Rails 3 app that pulls in text from remote sources, stores it in the database and does stuff with it. I am deploying the application on Heroku prior to public release. Heroku uses PostgreSQL exclusively.

I was under the belief that all the components in my system were set up to use UTF-8 encoding and therefore moving text with non-ASCII characters around should be fine. But in practice that was not the case - characters like 'α' that looked fine on my local machine showed up as 'Î±' on Heroku, etc. So clearly I was doing something wrong. Rather than go into all the gory details, this is the way to do it right...

Bottom-line: Make everything use UTF-8 explicitly ... EVERYTHING

0: Backup your current database!

1: MySQL
Here are the contents of my /etc/my.cnf file:

[mysqld]
datadir=/usr/local/mysql/data
bind-address = 127.0.0.1
character-set-server = utf8
max_allowed_packet = 32M
[client]
default-character-set = utf8
[mysql]
max_allowed_packet = 32M

Even if your tables are held in utf8, you should add these lines. You want the following mysql command to look as shown:

mysql> SHOW VARIABLES LIKE 'character\_set\_%';
+--------------------------+--------+
| Variable_name            | Value  |
+--------------------------+--------+
| character_set_client     | utf8   |
| character_set_connection | utf8   |
| character_set_database   | utf8   |
| character_set_filesystem | binary |
| character_set_results    | utf8   |
| character_set_server     | utf8   |
| character_set_system     | utf8   |
+--------------------------+--------+

2: Rails
I'm using Rails 3 - can't tell you how this works in Rails 2.x
a: In config/application.rb make sure this line is uncommented:

    # Configure the default encoding used in templates for Ruby 1.9.
    config.encoding = "utf-8"

MySQL and Rails use different variants of utf8/utf-8 - make sure you are using the right one. And note the comment above this line - this sets up utf-8 encoding for templates ONLY.
b: In your database.yml, specify the encoding for the databases - for example:

development:
  adapter:  mysql2
  host:     localhost
  encoding: utf8
  [...]

Here you are telling the database adapter that the database uses utf-8.
c: mysql2
Notice that I am using the mysql2 adapter instead of mysql. At this point (Feb 2011) the mysql gem is NOT encoding aware. Replace mysql with mysql2 in your Gemfile and run bundle.
d: In each Model that uses text add this line at the very top of the file:

# encoding: UTF-8

This tells Ruby that we're using utf-8 in this model. I don't see a way to set this at the application level so you have to have to add it in all relevant model .rb file. I also don't like defining something with a comment line. I can't see how to define this in, say, an irb interactive session.

With all those components in place, you should be all set. Try entering non-ASCII characters into a form - such as accented characters or greek/math symbols. These should be displayed correctly in the browser and in the mysql command line client.

With regards to Heroku, assuming you have your app already set up, you should be able to do a 'heroku db:push' to copy the database into PostgreSQL on Heroku and the characters should display correctly on the remote pages. You will see reference to using 'heroku db:push' with explicit database URLs that include an encoding option, such as '?encoding=utf8'. If your MySQL is set up correctly then this should be unnecessary.

A critical part of running apps on Heroku is the ability to pull the database back to your local database using 'heroku db:pull'. Before getting all my components set up with utf-8, this step failed for me. With everything using utf-8, and after adding the 'max_allowed_packet' lines to my my.cnf file, this process works fine.

But because I was working with data in before everything was truly utf-8, I had some instances of text in the database that had been incorrectly encoded - and thereby effectively corrupted. I could see what the 'corrupt' characters looked like and I knew what the correct versions should be. Because everything is now using utf-8 I could simply do a substitution on the text. For example:

str.sub!(/ÃŸ/, 'β')

I gathered up the character mappings that I needed (which were not may in my case) and wrote up a class method that I cloned in each model with the issue. I then ran those in the Rails console to correct the bad characters. The method is:

  def self.make_utf8_clean
    mappings = [  ['Î±', 'α'],
                  ['ÃŸ', 'β'],
                  ['Î²', 'β'],
                  ['â€™', '’'],
                  ['â€œ', '“'],
                  ['â€\u009D;', '”'],
                  ['â€', '”'],            
                  ['Ã¶', 'ö'],
                  ['Â®', '®']
              ]
    # Get the list of String columns
    columns = Array.new
    self.columns.each do |column|
      if column.type.to_s == 'string'
        columns << column
      end
    end
    
    # Go through each object -> column against all mappings
    self.all.each do |obj|
      columns.each do |column|
        mappings.each do |mapping|
          value = obj.attributes[column.name]
          if value =~ /#{mapping[0]}/
            s = value.gsub(/#{mapping[0]}/, "#{mapping[1]}")
            obj.update_attribute(column.name.to_sym, s)
          end      
        end
      end
    end
  end

This looks at your model and figures out which columns are of type String. It goes through all records and all character mappings, replacing text and updating the database as needed. Your mappings array could be much larger. There may be a better source of these, but this is a start.
You run this in a rails console like this:

Loading development environment (Rails 3.0.4)
ruby-1.9.2-p0 > YourModel.make_utf8_clean

It's a hack but it helped my 'fix' quite a few records that would have been a pain to recreate.

Character encodings are HARD - Yehuda Katz wrote a nice article on the issues. For most purposes (unless you work with Japanese text) UTF-8 is your best choice for encoding and so I'm using it exclusively. Java and Python both made the same choice and things are probably easier to set up in those worlds. Ruby has it's roots in Japan and so it is not surprising that it could not go down that path.

From now on, I'm going to make sure everything I touch is configured for UTF-8. There are fews reasons not to at this stage and it allows you to handle most languages.

Tables named with reserved words in MySQL

2011-02-23T10:39:00.000-08:00

Trying to fix a string encoding issue in a MySQL database I realized that one of my tables was called the same as a MySQL reserved word - specifically I have a table called 'references'.

This was created from a Rails app and I have been using this with no problems for a few months, so you can use reserved words, or at least that one.

Problem is that direct SQL statements in the MySQL client like this don't work:

mysql> describe references;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that 
corresponds to your MySQL server version for the right syntax to use near 'references' at line 1

The solution is to prefix the table name with the database name like this:

mysql> describe mydb.references;
+------------------+--------------+------+-----+---------+----------------+
| Field            | Type         | Null | Key | Default | Extra          |
+------------------+--------------+------+-----+---------+----------------+
| id               | int(11)      | NO   | PRI | NULL    | auto_increment |

Setting up PostgreSQL on Mac OS X 10.6

2011-02-22T08:50:00.000-08:00

I needed to set up PostgreSQL on my Mac in order to troubleshoot a problem with a Rails application.

Here are the steps that I followed:

1: Using Homebrew, install and build PostgreSQL
Homebrew will give you commands to create a Launch Agent that starts the server on a reboot

$ brew install postgresql
$ cp /usr/local/Cellar/postgresql/9.0.3/org.postgresql.postgres.plist ~/Library/LaunchAgents
$ launchctl load -w ~/Library/LaunchAgents/org.postgresql.postgres.plist

2: Setup PostgreSQL
You should look into setting administration user - I skipped that for my purposes

$ initdb /usr/local/var/postgres
The files belonging to this database system will be owned by user "jones".
This user must also own the server process.
[...]

3: Create a specific database and verify that it is running

$ createdb mydb
$ psql mydb
mydb-# select version();
mydb-# \h

4: Install the PostgreSQL ruby gem
You'll see different names for this gem - just use 'pg'. Using the ARCHFLAGS env variable is important. I did not need to specify the explicit path to the Homebrew installation of the PostgreSQL software.

$ env ARCHFLAGS="-arch x86_64" gem install pg

Problem with sqlite when installing the taps ruby gem on Mac OS X

2011-02-21T15:59:00.000-08:00

Just worked my way through one of those installation problems where trying to install A fails because B is missing and trying to install B fails because... and so on.

1: Trying to push a database to the Heroku hosting service failed because I did not have the 'taps' gem installed
2: 'gem install taps' failed because it couldn't compile a component that interacts with sqlite databases. I'm not using sqlite but there is no way to skip this part of the code - hmm...
3: Mac OS X has sqlite installed by default (or at least with Xcode installed)
4: Explicitly telling taps where to find the lib and include files didn't fix it

So the problem lay in my sqlite installation

Lots of mentions of the issue on Google - some of which said to install a new version with MacPorts

In the past I have found MacPorts to be a very frustrating experience - just wanting to install a single library has led to literally hours of watching it fetching and installing all sorts of apparent dependencies. So I tend to just get the source code for whatever I want and compile it manually, but that can be a pain in and of itself.

I've heard very good things about Homebrew as a MacPorts replacement so I figured I should take a look.

1. Set up a 'staff' group so that you can install code without having to sudo everything
2. Get Homebrew
3. Install your library

$ sudo dscl /Local/Default -append /Groups/staff GroupMembership $USER
$ ruby -e "$(curl -fsSL https://gist.github.com/raw/323731/install_homebrew.rb)"
$ brew install sqlite

Simple... Brilliant... I'm sold...

Homebrew installs code in /usr/local/Cellar by default so I need to tell the ruby gems where to find that - and somewhere along the line I saw a sqlite ruby gem. So I figured I should install that first as a check that things are working before trying 'taps'.

$ gem install sqlite3 -- --with-sqlite3-include=/usr/local/Cellar/sqlite/3.7.5/include \
                                            --with-sqlite3-lib=/usr/local/Cellar/sqlite/3.7.5/lib
$ gem install taps

It worked...
You may see mention of a sqlite3-ruby gem. That is now called sqlite3 - it's the same thing.

And finally I can run '$ heroku db:push' and send my database to my Heroku app.

Phew...

Rails, Devise and custom User models

2011-02-18T14:50:00.000-08:00

Devise is an excellent solution for user authentication in a Rails application.

Ryan Bates has done two great Railscasts episodes on Devise - #209 Introducing Devise and #210 Customizing Devise.

The default Devise configuration uses a simple sign up process - you give it an email address, a password and password confirmation. Follow the installation instructions and it should just all work.

But in my current application I need a bit more. I want the user to enter their first and last names, I want to assign a role and I want to link that individual user to a company account. Devise is capable of handling all this but the README on github doesn't really explain how and I, for one, get a bit nervous messing with the code of my authentication solution.

It turns out to be incredibly easy as long as you don't try and be too clever.

Be going into the steps given below, I recommend trying out a basic off the shelf Devise installation in a test application first just so you know that it works on your machine and you can see what files it creates, etc.

In these steps I'm going to use Devise with a User model that contains some custom fields.

1. Before creating your own User model, do a basic Devise installation into your app

$ gem install devise  # or in Rails 3 add it to your Gemfile, and 'bundle'
$ rails generate devise:install  # follow the instructions given
$ rails generate devise User
$ rails generate devise:views # this generates sign_in, etc views under 'app/views/devise' - not user!

2. Modify your User model by adding custom fields to attr_accessible
Here I'm adding :first_name, :last_name, :active, :role

attr_accessible :email, :password, :password_confirmation, :remember_me,
        :first_name, :last_name, :active, :role

Add your own validations, etc to the model. For testing, at least, require the presence of at least on of your custom fields.

3. Modify your Migration for the User table and run the migration
In my case I added these lines:

     t.string :first_name
      t.string :last_name
      t.boolean :active, :default => true
      t.string :role, :default => 'user'

Run the migration with 'rake db:migrate'

4. Modify your Sign_up form
This lives in app/views/devise/registrations/new.html.erb
NOTE: You can have Devise install its views under app/views/user but I prefer to keep Devise specific views in their own directory
Add fields to the form for your custom fields e.g.

<%= f.text_field :first_name %>
etc.

5. Try it out - sign up a new user
Go to the URL /users/sign_up
Your input to the custom fields should go into the database and any validations against custom fields which fail should give you the proper error messages and highlighting in the sign_up form
In my experience this 'just worked'

6. But the whole reason you want custom fields in the User model is to work directly with them...

For this you need a User controller and views. Devise does not give you either of these.
Either run a scaffold generator and skip the model or copy over another controller and set of views.
Now you have the regular set of actions for your model.
Go to /users and you should see the user(s) that you added, /users/1 will show you that user with whatever columns you choose to display.

In my case I display my custom fields and the email in my show and index actions and just ignore the rest of the Devise specific fields.

You want to be careful with the User new/create/edit/update actions. If you create a new user via that path then they will have no password, etc so you might want to remove new/create. The edit/update actions are useful if a user want to change their name and other 'profile' information, but don't mess with the Devise-specific fields via this route.

Basically, the Devise side of things and your custom User model can coexist quite happily. Make sure you don't mess with the fields that Devise requires and don't use the same column names.

I would also avoid using virtual attributes in the custom fields. I tried this and couldn't get it to work. Not a big deal for my case.

When I started integrating Devise into my app I had a sinking feeling that the custom fields would be a real problem. Quite the opposite - this turned out to be really easy.

Great kudos to the folks at Platforma Tec - Jose Valim and colleagues for a really nice piece of work.

Basic image rollover effect in jQuery

2011-02-18T08:17:00.001-08:00

There are so many fancy image effects that you can write in jQuery that it is easy to overlook the basics. Here is basic image rollover script.

I have two images 'logo.png' and 'logo_highlight.png'. I want to display 'logo' by default and then replace it with 'logo_highlight' when I roll the mouse over it.

Here is my image tag in the html:

<img id="logo" src="images/logo.png" alt="Logo" />

And here is the script (assuming that you have the jQuery loaded)


<script>
$(document).ready(function() {
    $('#logo').hover(function(e) { 
  this.src = this.src.replace('logo', 'logo_highlight');
 },
    function(e) { 
  this.src = this.src.replace('logo_highlight', 'logo');
 });
});
</script>

The script attaches a 'hover' event handler to the DOM element with ID 'logo'. This has two functions that are applied when the mouse enters and leaves the element respectively.

On entry, the image 'src' attribute is updated. The new one is derived by replacing the string 'logo' in the filename of the original image with 'logo_highlight' in the new version. In other words, the image tag now sources the 'logo_highlight' image.

When the mouse leaves the element, the second function is executed and that replaces the highlighted image with the original.

Short and sweet...

Authentication in Mongo and Mongoid

2011-02-16T09:22:00.000-08:00

Mongo has primitive authentication - just basic user/password authentication per database.

It's preferred mode of operation is no authentication in a trusted environment. That's fine, but it's not always possible. I want to run mongo on a Amazon EC2 node and access it from remote clients so I need to use authentication. On top of that, I already have the database running without authentication on a node.

Here are the steps you need to make the migration to a server with authentication...

1. Create an admin user on the database
Open up a mongo shell on the machine running the server

$ mongo
> use admin
> db.addUser("your_admin_user", "your_password")
> exit

2. Restart your Mongo server with --auth
It is CRITICAL that you restart with the --auth option. Users and passwords are simply ignored without this option.

$ mongod --auth

3. Set up database specific users

$ mongo
> use admin
> db.auth("your_admin_user", "your_password")
> show dbs
> use your_db
> db.addUser("your_db_user", "your_password")
> db.system.users.find()
> exit

4. Set up authenticated access from your application
I work in Ruby and use Mongoid as the Object Document Mapper to access Mongo. Mongoid, in turn uses the Ruby Mongo Driver. If you are using Mongoid outside of Rails then you will need a configuration block along the lines of this;


Mongoid.configure do |config|
  name = "your_db"
  config.database = Mongo::Connection.new.db(name)
  config.database.authenticate("your_db_user", "your_password")
end

Note that you are authenticating with the Ruby Mongo driver - not with Mongoid.

If you are working with Rails then you'll need to add username and password into your config/database.yml file. I see that the Devise authentication gem can work with Mongo to handle authentication of individual users but I've not explored that yet.

5. Clearly there is an issue having your password in plain text in your code
The bottom line is that you probably don't want to trust Mongo authentication for critical data. In that case, you really need to set up Mongo access in a secure environment and perhaps handle interfacing this with the outside work through a separate gateway application, say a Sinatra app that handles all authentication itself.

For my needs I have non-critical data - I just want to prevent access to arbitrary users (i.e. port scanning scripts) and only access from a few defined scripts on specific machines. So for now this will work for me.

With mongo authentication in place, how do you handle backing up and restoring the database?

On the machine hosting the server you can use these two variants of the dump and restore commands:

$ mongodump -d your_db -o . -u your_db_user -p your_password
[...]
$ mongorestore -u your_db_user -p your_password your_db

To work with all databases you would use the admin user

In order for someone to break into your database someone has to
1: Guess/crack your admin username and password
or
2: Guess/crack your specific database, your db username and password.

You have to evaluate the chances of this along with the value of the data in the database before going down this path.
You can also configure the database to use a non-standard port. There is no harm in this but it offers minimal to no additional security as many malicious scripts will scan across all ports on a machine looking for one that responds.

Caveat emptor...

Always index columns that you want to sort on in Mongo

2011-02-15T14:30:00.000-08:00

I'm using Mongo as a non-relational database for a few projects. In general it's working out great. MySQL would work too but I like not having to explicitly create a database or run migrations. Plus I figure you can't really understand the strengths and weaknesses of a technology unless you build a real application with it.

I work in Ruby and use the MongoMapper and Mongoid Object Data Mappers to talk to Mongo.

One issue that I do not like is the requirement that you explicitly create an index for every column that you think you will want to sort on. If you don't then all the data gets loaded into memory for the sort and you get an error like this:

[...]/gems/mongo-1.2.1/lib/mongo/cursor.rb:86:in `next_document': 
too much data for sort() with no index (Mongo::OperationFailure)

And if you want to sort on two columns then you need an index on the combination of the two.

You can add indexes at any point - it takes some action but it's not that big a deal. But it doesn't 'just work'... in MySQL it does - an index might give you better performance but it doesn't blow up without one.

You'll hear people claim that the NoSQL databases are schema-free, giving you a lot of flexibility. I don't really buy that argument - in most applications you want a clear schema.

Where I do see the benefit is that, with NoSQL databases, your schema resides your Model - not in the DB itself - and that is where it belongs. When you want to change the schema you just change the Model - no database migrations - very flexible.

But, with Mongo at least, if you have to define indexes ahead of time in order to sort even relatively small numbers of objects then that nullifies some of that benefit.

Using Mongoid in Ruby applications outside of Rails

2011-02-15T11:32:00.000-08:00

Mongoid and MongoMapper are two Ruby ODM (Object Document Mapper) gems for the Mongo database.

I've used both to a limited extent and they seem comparable for my needs. Mongoid seems to be getting a bit more traction than MongoMapper and it certainly has better docs.

My current project uses Mongo in a standalone Ruby application - no Rails in sight - but the docs are almost totally focused on Rails. Here is how you use Mongoid outside of Rails.

I'm storing relevant RSS entries in the database. My model looks something like this (heavily truncated):


class RssEntry
  include Mongoid::Document

  field :entry_id
  field :title
  field :authors, :type => Array
  field :timestamp, :type => Time

  index :timestamp
  index :title
end

Be sure and define your indexes carefully for fields that you want to search on, otherwise Mongo will run out of memory when searching even modest datasets. I see this as a weakness of the database. See important note on creating indexes below!

and the application looks a bit like this (edited):


#!/usr/bin/env ruby
require 'mongoid'
$:.unshift File.dirname(__FILE__)
require 'mongoid_test_model'

Mongoid.configure do |config|
  name = "mongoid_test_db"
  host = "localhost"
  port = 27017
  config.database = Mongo::Connection.new.db(name)
end
[...]
entry = RssEntry.create({
    :title => title,
    :entry_id => id,
    :authors => authors,
    :timestamp = Time.new
})

And if you are using the defaults of localhost and 27017 then you can leave those definitions out.

NOTE: Simply defining an index in your model is NOT enough. You have to explicitly create the index. When you use Mongoid with Rails it sets up a rake task so you can run 'rake db:create_indexes' but outside of that environment you need to do this yourself.

You'll want to write a simple script/rake task to set this up, in which you call create_indexes on EACH class in your model that uses Mongoid. For example:


#!/usr/bin/env ruby
require 'mongoid'
$:.unshift File.dirname(__FILE__)
require 'mongoid_test_model'

Mongoid.configure do |config|
  name = "mongoid_test"
  host = "localhost"
  port = 27017
  config.database = Mongo::Connection.new.db(name)
end

# Call on each of the relevant models
RssEntry.create_indexes()

Previously, you could specify auto-indexing within your models but this has now be deprecated or removed, so ignore any references to that.

Using Twitter for System Notifications

2011-02-14T14:12:00.000-08:00

I finally figured out something that many, many people have been doing for quite a while - using Twitter as a way to deliver notification of system events.

Twitter is a great way to deliver short messages to many people via many forms of media and devices. The default is that any message is available to anyone in the world. But you can also configure a Twitter account to be private, requiring the owner to explicitly allow access to other users. In the extreme case the owner can deny access to none but him or her self.

Twitter handles all the messaging, all you need to do is have your server, web application or whatever, send a message to your private account whenever some event takes place. For example, I run long calculations on servers at work and I want to be notified when a job completes.

You can find a load of UNIX command line twitter clients and libraries in all the main languages. So finding or building a suitable client is straightforward.

I'll show you how to build a simple client in Ruby.

If you want to send tweets to a private account then you will need proper authentication credentials.

For this you need to use OAuth - username/password authentication has been deprecated.

1: Sign in to Twitter as the owner of the private account
2: Go to http://dev.twitter.com - you'll still be signed in
3: Click on 'Register an App' - now you're not really creating a new twitter application but pretend that you are - give it a name - and you want to select that it is a 'client' application and that it should have 'read write' access to the account.
4: Now go to 'your apps' and click on the new dummy app.
5: Scroll down and get the 'Consumer key' and 'Consumer secret' - you'll need these in your code.
6: Those are required for your application, but in addition you need a key and secret for the actual twitter account that you will want to write to.
7: On your app settings page, on the right sidebar, click on 'My Access Token' and get 'Access Token (oauth_token)' and 'Access Token Secret (oauth_token_secret)'.

Now we can write some code.

8: Get the 'twitter' Ruby gem

$ gem install twitter

9: Write a small ruby app. This simple example takes a message on the command line, configures the client with the FOUR OAuth tokens/strings and then updates the private twitter account with the message:

#!/usr/bin/env ruby
require 'twitter'
abort "Usage: #{$0} message" if ARGV.length == 0
# Hard-wired to my private twitter account
Twitter.configure do |config|
  config.consumer_key = 'your-app_key'
  config.consumer_secret = 'your_app_secret'
  config.oauth_token = 'your_account_token'
  config.oauth_token_secret = 'your_account_secret'
end
client.update(ARGV[0])

10: It's that simple...
11: chmod a+x your script and run it with a message - check your private twitter account and you should see it.

It's easy to think up (and code) custom notification scripts for this. As long as you have a network and as long as Twitter is up (OK, it has had some issues) then you don't need to worry about anything to do with distributing your messages. You can get them on your phone or your desktop, and you can leverage the work of others to display popup windows on your desktop, play tunes, flash lights, etc, etc.

Just remember that when you create your private Twitter account that you go into the settings and make sure that it is indeed set to private.

One extension that I've thought about is having my script take an optional URL, say pointing to the results from a computational run, and using a URL shorting service like http://bit.ly or http://goo.gl to let me include that in the tweet. Unfortunately none of the 'big name' services allow you to have private URLs so that might be a problem in some applications. But it's worth considering for some applications.

PATSY - a web service that makes patents easier to read

2011-02-14T11:36:00.000-08:00

I've just launched PATSY - a new web service that reformats US patents to make them much easier to read than their original format.

The text of patents is typically very dense and difficult to read.

They are written as legal documents and inevitably this results in verbose and sometimes arcane text. Every component of invention will have all possible variants enumerated and this can result in sentences of ridiculous length with these variants delimited by commas. On top of that, the US patent office still prints patents as two narrow columns of text of each page - a format that might work in newspapers but which in technical patents is nonsensical.

The underlying problem is that the patent offices should define and enforce a modern way of text formatting that is both easy to read and easy to parse in software.

But as this is not likely to happen any time soon, I decided to write an application that reformats the text of patents into something more palatable.

You enter a patent number into PATSY and it fetches the web page from the US patent office web site. It scans the text and splits up paragraphs into component sentences. Furthermore it splits sub-sentences by punctuation such as semi-colons. Simply adding this spacing makes a big difference.

But PATSY goes much further. It highlights a series of phrases that are typically of interest - such as 'preferred embodiment' and 'SEQ ID NO'. It recognizes references to other patents and hyperlinks these to either their patent office site or to PATSY directly. In some cases, references to scientific publications can be identified and links are added that will take the user to the NIH PubMed site of abstracts, and from there the original publication can be accessed in most cases.

PATSY only works with US patents right now and some of its features are geared towards biotechnology patents. The text parsing is not perfect but even at this early stage in its development, it can really make dense blocks of text much easier to read. In cases where the result is unclear, you can click the head of each text block to see the original text before any processing.

While it is in this early stage, PATSY is completely free. If it turns out to be useful to a lot of people then I may offer it via subscription to heavy users, while retaining free access to occasional users.

Please try PATSY out and send me feedback at info @ craic.com.

Technical aspects:

PATSY is written in Ruby using Sinatra as a lightweight web application framework. It runs on Heroku which is a hosting service for Ruby web applications that sits atop Amazon web services. My steps involved in setting it up are described here.

My experience with Heroku for this application thus far has been great. They allow you to set up applications with limited resources at no charge. If and when PATSY starts to get some traction then I can scale it up by adding more of what they call 'dynos'. That will incur some cost but there is no commitment or up front payment, plus the process of scaling is incredibly easy.

JavaScript Bookmarklet that can create a New Window

2011-02-11T10:37:00.000-08:00

The popup-blocking features of current browsers can be a problem if you are writing a JavaScript Bookmarklet that wants to open a new window. For example, I want to select text in an arbitrary window and then have a remote server operate on the text and return its results to a separate window, or tab. A bookmarklet is a great way to do this.

One approach to writing these is to make the bookmarklet simply call a JavaScript script on a remote server, which does the real work. This results in a simple bookmarklet and lets you perform arbitrary operations in the remote script.

But when the end result is the creation of a new browser window this approach will fail...

Modern browsers view this a potential exploit and will only allow the creation of new windows as the result of direct user interaction - i.e. the user clicks something.

All is not lost - it just means that you need to put all your code in the bookmarklet itself. This is messy but for many scripts this should not be a problem.

Here is my example. It gets the currently selected text and adds that to the URL of a remote service. It opens that URL in a new browser tab or window, depending on the specific parameters. If no text has been selected then it prompts to user to enter some. This first version will open the new page as a new tab in most browsers.

<a href="javascript:(function(){
// Get the current selection
var s = ''; 
if (window.getSelection) { 
s = window.getSelection(); 
} else if (document.getSelection) { 
s = document.getSelection(); 
} else if (document.selection) { 
s = document.selection.createRange().text; 
} 
// Prompt for input if no text selected
if (s == '') {
s = prompt('Enter your text:');
}
// Open the target URL in a new tab
if ((s != '') && (s != null)) {
window.open('http://example.com/yourapp?id=' + s);
}
})();">BOOKMARKLET</a>

You would want to remove the comments from the bookmarklet, but you don't need to strip the newlines or minify the code.

The default of most current browsers is to create new tabs instead of new windows. Users can set their preferences to override this but most will not.

You may want to force the creation of a separate window. Think this through carefully - it may annoy some users if you start generating loads of windows. In some cases it is appropriate. It used to be that you could force this by passing '_blank' as the name of the new window but this does not appear to work in all browsers. Instead you need to explicitly specify one or more window properties, like width and height.

This is a messy solution but it works. In my application I just replaced the window.open call with this form:

window.open('http://patsy.craic.com/patsy?id=' + s, '_blank', 
'height=600,width=1024,status=1,toolbar=1,directories=1,menubar=1,location=1');

The options string in the third argument specifies what the new window should look like. You may need to experiment with these. With Google Chrome on the Mac these do not give me the expected result - the address is not editable and there is no bookmarks bar. I also found that simply using 'status,toolbar,etc' without the '=1' did not work, although you will see this listed as a valid syntax.

Running a Ruby 1.9 Sinatra app on Heroku

2011-02-04T12:39:00.000-08:00

I just got my first significant application running on Heroku. It uses Sinatra instead of Rails and uses Ruby 1.9 - as a result the steps to get the application up and running were slightly different from the Heroku Quickstart Guide, which is tailored towards Rails apps.

1. Setup
Setup a Heroku account and set up the SSH keys
Create your app and make sure it runs correctly on your local machine.
Make sure that all paths are relative to the application root.
Run the Sinatra app from a config.ru file.

2. Setup Bundler
I had an issue with this originally but this is what worked. I only need the sinatra gem so my Gemfile is:

source :gemcutter
gem 'sinatra'

and my config.ru file is:

require 'bundler'
Bundler.require
require 'sinatra'
require './my_app.rb'
run MyApp.new

Note that although you need to have the heroku gem installed on your system in order to upload to Heroku, you do not 'require' it in your app.

3. Setup git and commit the project.

4. Create the Heroku app
You want to specify an application name, otherwise Heroku will give you an arbitrary one. You also want to specify the run environment at Heroku that will be used. They refer to this as the 'stack' and for Ruby 1.9 you want to specify this directly and currently the correct option is 'bamboo-mri-1.9.2'. The create command is:

$ heroku create my_app_name --stack bamboo-mri-1.9.2

5. Push the git repository to Heroku with:

$ git push heroku master

The messages that follow should look something like this (some lines removed):

-----> Heroku receiving push
-----> Sinatra app detected
-----> Gemfile detected, running Bundler version 1.0.7
       Unresolved dependencies detected; Installing...
       Fetching source index for http://rubygems.org/
       Installing rack (1.2.1) 
       Installing tilt (1.2.2) 
       Installing sinatra (1.1.2) 
       Using bundler (1.0.7) 
       Your bundle is complete! It was installed into ./.bundle/gems/
       Compiled slug size is 500K
-----> Launching... done
       http://my_app_name.heroku.com deployed to Heroku

Now go to that URL and your app should be running.

If there was a problem and the app failed to start then look at the logs:

$ heroku logs -n 100

You can set up apps with minimal (or no) database storage for free on Heroku. This is a great service as it lets you experiment to your heart's content.

The idea behind Heroku is to remove from you the burden of server configuration. For my simple application this seems to work remarkably well.

Ruby 1.9 and incompatible character encodings

2011-02-01T13:53:00.000-08:00

I run into issues pulling remote text data into a Ruby 1.9 / Rails 3 app, which is using utf-8 encoding by default. The problem apparently comes from non-Ascii characters in binary or so-called ASCII-8BIT encoded text. I don't have a proper way to translate the offending characters as yet but my workaround is to strip them out and/or replace them with an ASCII character.

This regex implements the workaround. Be sure to use the 'n' modifier on the regex. This specifies that the encoding of the text should be ignored and thus multibyte characters are treated as separate bytes.

    str.gsub!(/[^\x00-\x7F]/n,'?')

Far from perfect, but it gets the job for me right now.

rvm and shell scripts

2011-01-04T15:27:00.000-08:00

If you want to use a specific version of Ruby via RVM inside a shell script then you need to make sure that the environment is setup correctly.

The RVM site addresses that HERE for the case where RVM is installed in a user's home directory.

But on my server I have a system-wide installation. In that case you should have a /etc/rvmrc file with a default setup and you need to source this.

Your scripts should look something like this:

#!/bin/bash                                                                                                                                                 
source "/etc/rvmrc"

rvm use 1.9.2

# mycommands_go_here

Per Page Inclusion of JavaScript in Rails view pages

2010-12-21T11:20:00.000-08:00

I use various jQuery plugins in my Rails views but I don't want to download all that code in pages that don't use them. Furthermore I want to minimize the chance of namespace conflicts between different blocks of code.

A simple way to do this is to add these lines to your application.html.erb file after the main JS includes:

<%= yield :custom_javascript_includes %>

and then add a block similar to this in each view template that needs custom JS.

<% content_for :custom_javascript_includes do %>
  <%= javascript_include_tag "your_custom_plugin.js" %>
  <script type="text/javascript">
    // Your custom JavaScript code
  </script>
<% end %>

In this example I'm including a specific file from the public/javascripts directory and then adding some custom JS code that might define some events or trigger some action on page load.

Here is a real example that uses the facebox jQuery plugin which provides 'light box' functionality. This code is placed at the top of my 'show' template.

<% content_for :custom_javascript_includes do %>
  <%= javascript_include_tag "facebox.js" %>
  <script type="text/javascript">
 jQuery(document).ready(function($) {
       $('a[rel*=facebox]').facebox({
         loadingImage : '/images/facebox_loading.gif',
         closeImage   : '/images/facebox_closelabel.gif'
       });
 });
  </script>
<% end %>

You can take this further by also including custom stylesheets linked to these plugins in the same block. That can result CSS and JS includes being interspersed in the resulting HTML file which some people may not like, but I don't think there are any practical drawbacks to it.

The big win is that you keep your custom JS code right there in the same file as your HTML. If that JS code is substantial you could move it into a partial that resides in your views directory, as opposed to having it separated in the public/javascripts directory.

For high performance production sites you should weigh the benefits of this approach in terms of clarity with the performance benefit of stashing all your JS code in a single file and minifying it.

MySQL Data export/import probelm with SQL SECURITY DEFINER

2010-12-14T14:58:00.000-08:00

Importing a MySQL database dump from a client into my system I got this error:

$ mysql -u root -p  example_db < example_db_dump.sql 
Enter password: 
ERROR 1449 (HY000) at line 5172: The user specified as a definer ('smith'@'localhost') does not exist

Running 'grep' for that user turned up a bunch of lines like this:

/*!50013 DEFINER=`smith`@`localhost` SQL SECURITY DEFINER */

These are created on the 'donor' MySQL system when creating one or more views of the data. That user does not exist on my system and so MySQL complains.

I don't care about those views so the easiest way to deal with this issue is to remove these '50013' lines. You can do that with 'sed':

$ sed '/\*\!50013 DEFINER/d' example_db_dump.sql > example_db_dump_clean.sql

You need to drop the new database as the import process partially worked, re-create it and then you can reimport:

$ mysqladmin -u root -p drop example_db
$ mysqladmin -u root -p create example_db
$ mysql -u root -p example_db < example_db_dump_clean.sql

Now, if you need to use the Views in your copy of the database then you need to either create that user locally and leave the lines, or change the user in those lines to one that does exist locally.

Installing Amazon EC2 tools on Ubuntu 10.04

2010-12-12T10:06:00.000-08:00

The Amazon EC2 ami and api tool packages are not found by apt-get using the default sources list. You need to add 'multiverse' to the list. Here is how you install the tools:

$ sudo perl -pi -e 's%(universe)$%$1 multiverse%'  /etc/apt/sources.list
$ sudo apt-get update
$ sudo apt-get install ec2-ami-tools

Of course, you also need to set up your environment variables as per usual so that the tools can access your keys.

Using Bundler with a non-Rails Ruby project

2010-12-05T10:28:00.000-08:00

Bundler is a Ruby Gem to help you manage the various gem dependencies in your application.

It has gotten a lot of attention as it is ,er, bundled with Rails3 and represents a welcome addition to that framework. But you can use Bundler in any Ruby application to help make deployments go smoothly.

For a regular Ruby application the steps involved are simple:

1: Install Bundler on your system

$ gem install bundler

2: Create a file called 'GemFile' at the top level of your application directory
Add gem sources to this and each gem that your application relies on. This example includes the main gem sources and a single gem

source :rubygems
source :rubyforge
source :gemcutter

gem "json"

3: Run 'bundle install'
This will now install any missing gems. It does harm to run it more than once.

$ bundle install
Fetching source index for http://rubygems.org/
Fetching source index for http://rubygems.org/
Fetching source index for http://rubygems.org/
Using json (1.4.6)
Using bundler (1.0.3)
Your bundle is complete! It was installed into /Users/jones/.rvm/gems/ruby-1.9.2-p0

If you look in your application directory you will see a new file called Gemfile.lock. Bundler uses this and you want to keep it, adding it to your git repository (or equivalent). You can also find a .bundle directory containing a config file. I guess you can use this for fancier configurations, but a basic app you can ignore it.

4: Deploy your app
Now when you deploy your app in a new location (with the bundler gem installed) you can simply run 'bundle install' in your app directory and you'll be good to go.

Deleteing a file with a name that starts with a hyphen

2010-12-05T09:21:00.000-08:00

Here's a simple fix to a tricky problem (simple with hindsight, of course)...

By accident I created a file called '--latest' and wanted to delete it. But unix thinks that '--' signifies a program option. Escaping the hyphens didn't work either.

$ rm '--latest'
rm: illegal option -- -
usage: rm [-f | -i] [-dPRrvW] file ...
       unlink file
$ rm '\-\-latest'
rm: \-\-latest: No such file or directory
$ rm "\-\-latest"
rm: \-\-latest: No such file or directory

The simple solution is to give a more complete path specification that avoids having the hyphens as the first characters.

$ rm ./--latest

Problem solved!

Using Opscode Chef to start up a node on AWS EC2 - A Simple Example

2010-12-03T15:54:00.000-08:00

Chef from opscode.com is a suite of tools for managing computing infrastructure, from spinning up new nodes to installing Ruby gems to any custom operation you care to code up. It gets great reviews from those who manage numbers of unix (and other) systems and it has support for Amazon AWS and other cloud vendors baked right in.

I'm just getting started with it and am eager to go further but, like many projects, the documentation is far from clear. It has a lot of options and that typically means complexity. Perhaps because they are targeting folks who do systems administration for a living, their docs skim over some of the crucial basics.

My main interest in Chef is using it to spin up AWS EC2 nodes, configure them to my tastes and then spin them down as needed.

Here are the steps I took to get a node up and running:

0: I'm assuming you know how to fire up EC2 nodes from the management console or the command line on your desktop machine, that you have a keypair set up and that you are familiar with AMIs and the other terminology. If not, then get comfortable with EC2 before going any further.

1: I'm also assuming that you have a basic chef setup on your desktop and an account on Opscode's chef server (it's free for up to 5 nodes) and that you have written and tested a simple recipe on your desktop. If not, get comfortable with that before going further.

2: Add your AWS credentials to your knife.rb file (typically in ~/.chef)
The docs say to enter them in this format, which is different from the other configuration parameters that are generated from the opscode server... I don't know why... but this works:

# EC2 access keys
knife[:aws_access_key_id]     = "<your access key id>"
knife[:aws_secret_access_key] = "<your secret access key>"

3: Figure out your EC2 parameters for the node you want to spin up.

The AMI ID - I'm using a 32 bit Ubuntu AMI (ami-480df921)

The Instance Type - I'm testing with a Micro instance (t1.micro)

The Keypair name (mine is called 'craic-ec2-keypair') and the location of the SSH identity file that is linked with this (mine is in ~/.ssh/craic-ec2-keypair.pem).

The User that you (and chef) will ssh in to the node with (Ubuntu 10.04 wants me to use 'ubuntu' instead of 'root')

The Security Group for the server (I just use 'default' which is what I set up when I first started with EC2)

The Chef Recipe(s) and/or Role(s) that you want to run on the new EC2 server (I'm just going to run a simple one that I created called 'craic_test' that writes a file in /tmp) - Make sure this works on a non-EC2 client first!

4: Let's do it...
The chef command that you use is 'knife' with some EC2 specific options. You can find more information HERE in their FAQs, which is MUCH more complete than their Wiki documentation on knife...

The command will be 'knife ec2 server create' followed by a bunch of options. That's pretty self explanatory - but then we get into the options... Here is the full command I used split across multiple lines, which I shall explain below:

$ knife ec2 server create "recipe[craic_test]" \
      -i ami-480df921 \
      -f t1.micro \
      -x ubuntu \
      -S craic-ec2-keypair \
      -I ~/.ssh/craic-ec2-keypair.pem \
      -G default

The first line is the command itself followed by the recipes and/roles in the standard chef format ("recipe[craic_test]").

The EC2 AMI is specified with the -i option, then the Instance Type is specified with the -f option (aka --flavor - I don't know why they chose that...).

Next comes the ssh user with the -x option, the SSH Keypair name with -S (upper case S) and the location of the matching identity file with the -I option (upper case I) - then finally the security group with the -G option (upper case G).

I am really hoping that there is some way to store some of these options in knife.rb or someplace convenient as most of these will not change for my uses.

All being well, what happens next is that the node gets spun up on EC2, chef installs the prerequisites it needs to run on the node and downloads the chef client over there. Chef will create the new node record on the chef server, download your recipes and then execute them.

What you will see is a TON of output as everything runs. I won't subject you to all of that here, but the first few line will look like this:

Instance ID: i-08da8865
Flavor: t1.micro
Image: ami-480df921
Availability Zone: us-east-1b
Security Groups: default
SSH Key: craic-ec2-keypair

Waiting for server............
Public DNS Name: ec2-184-72-190-66.compute-1.amazonaws.com
Public IP Address: 184.72.190.66
Private DNS Name: domU-12-31-39-0F-29-E8.compute-1.internal
Private IP Address: 10.193.42.22

Waiting for sshddone
INFO: Bootstrapping Chef on
0% [Working]90-66.compute-1.amazonaws.com
[...]

This shows the IP info, etc for your new instance and is followed by pages of messages reporting the downloading and installation of all the prerequisites for Chef. At the end you should see your recipes being run and finally a summary once again of the instance information:

Instance ID: i-08da8865
Flavor: t1.micro
Image: ami-480df921
Availability Zone: us-east-1b
Security Groups: default
SSH Key: craic-ec2-keypair
Public DNS Name: ec2-184-72-190-66.compute-1.amazonaws.com
Public IP Address: 184.72.190.66
Private DNS Name: domU-12-31-39-0F-29-E8.compute-1.internal
Private IP Address: 10.193.42.22
Run List: recipe[craic_test]

5: Now you can ssh into your node and you should see that your recipes have run and done whatever you asked of them.

One good thing about all the prerequisites getting installed, for an Ubuntu system at least, is that you now have a node with a lot of development libraries and tools already in place (especially if you work in Ruby).

That's it for the basic installation.

Remember to terminate your node when you are finished as you are being charged for it AND remove any test nodes from your Opscode server account as they will count against your account limit.

There is obviously a lot more that you can, and need to, do to set up your nodes but this walk through gets you to a functioning node.

Happy cooking

System-wide rvm install on Ubuntu

2010-12-03T11:37:00.000-08:00

Here are the steps I used to install RVM system wide on a Ubuntu 10.04.1 LTS node on Amazon EC2, using one of the Canonical AMIs. These set up the primary user as 'ubuntu', requiring you to sudo for system commands. I could install rvm under that user but installing System-Wide is recommended.

This HOWTO is based on the RVM page http://rvm.beginrescueend.com/deployment/system-wide/. But I find the RVM site documentation makes a few assumptions with regard to commands.

1: Install git if you don't already have it

$ sudo apt-get install git-core

2: Download RVM

$ sudo bash < <( curl -L http://bit.ly/rvm-install-system-wide )

This creates /usr/local/rvm and the group 'rvm'

3: Add each user to group rvm

$ sudo usermod -aG rvm ubuntu

4: Add startup lines to each user's .profile

[[ -s '/usr/local/lib/rvm' ]] && source '/usr/local/lib/rvm'

5: Add this line to /etc/rvmrc (may not be necessary)

$ echo "export rvm_path=/usr/local/rvm" > /etc/rvmrc

6: Use rvm as a user - best to logout/login first to make sure you get all the paths etc setup

$ rvm list
$ rvm package install zlib   ( I needed this to get install to work )
$ rvm install 1.9.2 --with-zlib-dir=/usr/local/rvm/usr

Note that you use rvm as a regular user, not via sudo, even though this is a system wide install.

Extracting Text from PDF documents on Mac OS X

2010-11-12T13:31:00.000-08:00

There are various ways to extract the text from a PDF document. In Mac OS X there are two effective ways that cost nothing.

1: Open the PDF in Preview or Adobe Acrobat Reader, select all the text, copy and paste into your editor of choice.

This is a quick and easy solution for one or a few documents. But for a large number of documents it would quickly become tedious.

Neither application allows you to Save As Text from the menu, which is unfortunate.

I also see a significant difference between the pasted text when the original PDF has columns of text. Compare the same original text copied from Acrobat Reader:

MOLECULAR FORMULA C6492H10060N1724O2028S42
MOLECULAR WEIGHT 146.0 kDa
TRADEMARK None as yet
MANUFACTURER MedImmune
CODE DESIGNATION MEDI-563
CAS REGISTRY NUMBER 1044511-01-4

and Mac OS X Preview:

MOLECULAR FORMULA MOLECULAR WEIGHT TRADEMARK MANUFACTURER CODE DESIGNATION CAS REGISTRY NUMBER
C6492H10060N1724O2028S42 146.0 kDa None as yet MedImmune
MEDI-563 1044511-01-4

The first version is the right one. So copying from Acrobat Reader is the better solution.

2. To handle many documents you want a script of some sort that just extracts the text. You can do this with the Mac OS X Automator application. I'd not used this before but it gives you access to a whole load of functions/services. Well worth checking out. You'll find Automator in your applications folder. Open it up and...

- Select Application in the template window that appears on opening.
- In the Automator window you have two vertical panels on the left and a grey workspace on the right. Click the 'Actions' button in the top left.
- In the left hand column, under Library, click 'PDFs' to bring up a list of PDF related actions in the second column.
- Select, drag and drop 'Extract PDF Text' onto the workspace. A dialog will appear.
- Select the appropriate options - plain text or RTF, folder in which to output the text, etc.
- Save As... and give your new app a suitable name
- Quit Automator
- Drag and drop a PDF onto the icon for the app and it will extract the text and output that in a file in the directory you configured in the app

Simple! Now you can just drag and drop PDFs and they will be converted automatically.

Automator can do a whole lot more than this, especially if you have a series of steps that must applied to files (e.g. image processing). If you are into scripting, take a look at the way this all works. Right click on the app icon and 'Show Package Contents'.

You can also learn about this via this article by Mathias Bynens on a simple way to convert UNIX scripts into apps ... it's the same mechanism.

So now I've got an automated way to extract text... BUT... the script uses Preview for the extraction, and that still gives me the incorrect ordering of text that I showed in the above example... dang it... not a deal breaker but frustrating all the same

Javascript Date.parse browser compatibility issue

2010-11-12T08:28:00.000-08:00

Just got burned with a Javascript incompatibility between Firefox and Safari on the Mac...

Date.parse() takes a string representation of a Date and return the number of milliseconds since the epoch. It has always been able to take dates in IETF format, such as "Jan 1, 2010" but as of Javascript 1.8.5 it can handle ISO8601 format as well, such as "2010-01-01".

I use ISO8601 in all my applications and in Firefox 3.6.12 (on the Mac) this works fine:

> Date.parse("Jan 1, 2010 GMT");
1262304000000
> Date.parse("2010-01-01");
1262304000000

Note that the ISO8601 form assumes the GMT timezone, but the IETF form assumes local timezone unless you specify GMT.

The problem for me arose in Safari (5.0.2) on the Mac:

> Date.parse("Jan 1, 2010 GMT");
1262304000000
Date.parse("2010-01-01");
NaN

Hmm...

And just to muddy the waters further, here is the output on Google Chrome 7.0.517.44:

> Date.parse("Jan 1, 2010 GMT");
1262304000000
Date.parse("2010-01-01");
1262332800000

The ISO8601 date is handled OK but is returned in the local timezone.

I need a fail safe way to parse ISO8601 dates - what to do...? Here's what I came up with:

var date_str = '2010-01-01';
var iso8601_regex = /(\d{4})[\/-](\d{2})[\/-](\d{2})/;
var match = iso8601_regex.exec(date_str);
var date = new Date(match[1], match[2] - 1, match[3]);
var milliseconds = Date.parse(date);    // -> 1262332800000 (local time)
milliseconds = Date.UTC(match[1], match[2] - 1, match[3], 0, 0, 0);  // -> 1262304000000 (UTC/GMT)

Note the '- 1' with the month (match[2]) - the first month is 0 - go figure.
This produces the same results in Mac Firefox, Safari and Chrome - consistent behaviour - yes, really!

What a palaver...

Rails has_many :through associations and edit/update actions

2010-11-09T13:06:00.000-08:00

Consider a basic has_many :through association

class Drug < ActiveRecord::Base
  has_many :indications
  has_many :diseases, :through => :indications, :uniq => true

Here a drug can be used to treat multiple diseases and any disease can be treated with multiple drugs (that side of the association is not shown). The 'indications' is a basic linking table with drug_id and disease_id.

In the drug#new and drug#edit forms you might use a select menu that allows you to select multiple diseases. I use the 'simple_form' gem for my forms and the 'association' method makes this trivial.

<%= f.association :diseases,
           :collection => Disease.all(:order => 'name') %>

In order for this to work you need to add an attr_accessible called :disease_ids to your model. With that, simple form should handle all details needed to create and update the association.

But there is a problem with the edit/update actions when you want to deselect ALL diseases. If you do this in the form then no disease_ids parameter will get passed to your controller and so this column will not get updated. It is a classic issue with HTML form updates and applies to checkboxes as well.

The solution is to add a line to the updater action in your controller that sets the disease_ids parameter to an empty array if it does not exist:

  def update
    params[:drug][:disease_ids] ||= []
    @drug = Drug.find(params[:id])
    [...]

This works fine - adds a bit of clutter to the controller but there you go...

However, this will break a basic functional test for the update action and you will get an error similar to this:

test_update_valid(DrugsControllerTest):
NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.[]

The problem stems from the basic stub for your object not passing a params[:drug] hash. Now, I'm not an expert on stubs/mocking so there may be a much cleaner way of fixing this, but I fix this by explicitly creating the needed parameters and passing them in the 'put' method.

Here is an example of a basic update test

  def test_update_valid
    Drug.any_instance.stubs(:valid?).returns(true)
    put :update, :id => Drug.first
    assert_redirected_to drug_url(assigns(:drug))
  end

and here is the modified one that will work

  def test_update_valid
    Drug.any_instance.stubs(:valid?).returns(true)
    put :update, { :id => Drug.first, :drug => { :disease_ids => [] } }
    assert_redirected_to drug_url(assigns(:drug))
  end

Of course I should be building out the tests to provide truly useful tests, but if you can't get beyond this step, the others don't really matter.

Hope this helps....

Disabling spell check in HTML forms

2010-11-09T07:56:00.001-08:00

I work with DNA and protein sequences and I often have HTML forms with a textarea for entering sequence. Unfortunately my browser sees that text (e.g. 'agctagagctcgatagc') and decides that this is misspelled and underlines all the sequence text with a red dotted line... ugly...

In HTML5 you can now disable spell checking on textarea and text inputs using the option 'spellcheck' = 'false' - EASY!

Note that this is a HTML attribute, NOT CSS - so you have to set it in the form itself.

Browser support for the feature may vary. It works on Firefox and Safari on the Mac for sure.

A related attribute is 'contenteditable' that allows you to control whether specific parts of a textarea's content can be modified - like 'readonly' but with much more control.

Rails, Factory_girl and Bioinformatics - a gotcha

2010-11-02T08:11:00.000-07:00

I've been using factory girl as a replacement for Fixtures in a new Rails application. It has been working well but I just stumbled across a BIG gotcha for my application.

I work on bioinformatics applications with DNA and Protein sequences and having a model with a column called 'sequence' is a natural choice.

The problem is that Factory Girl allows you to create objects in which a specific field is given a sequential id. For example 'user_1', 'user_2', etc. You set that up in your Factory definition like this:

Factory.define :mymodel do |f|
  f.sequence(:name) {|n| "name_#{n}" }
end

When you have a model with a field called sequence you would define that like this:

Factory.define :mymodel do |f|
  f.sequence(:name) {|n| "name_#{n}" }
  f.sequence 'acgtacgtacgt'
end

You see the problem... Running a test with this in it brings up an error message like this:

  1) Error:
test: A Sequence instance should be valid. (SequenceTest):
NoMethodError: undefined method `call' for nil:NilClass
    /Users/jones/.rvm/gems/ruby-1.9.2-p0/gems/activesupport-3.0.1/lib/active_support/whiny_nil.rb:48:in `method_missing'

The same problem will arise if you have a column called 'association'.

This is a bad design choice to mix up field names and methods. A better choice would have been to use a 'verb' such as f.generate_sequence(:name), which is much less likely to be used as a column name.

I can't see a way around this other than changing the name of the column in my model, which I am very reluctant to do.
Machinist is an alternative to Factory Girl, so that might be a solution - or hacking the factory_girl gem to change the name of the methods...

UPDATE: Factory Girl will allow alternate syntaxes - see the "Alternate Syntaxes" section. Not sure if its the best path for me but might get me over the immediate hurdle.

UPDATE 2: Just tried Machinist and that solves the first problem... however it fails if you have a column called 'alias' in a blueprint - the trick here is to precede the column name with 'self', i.e. 'self/alias' works.

Searchlogic versus MetaSearch and Rails3

2010-10-27T15:13:00.000-07:00

Searchlogic is a great Rails gem for basic searching of your tables. I've used it for quite a while now. Problem is that at the time of writing (October 2010) it does not work with Rails3.

You can install it just fine but starting up your server will produce and error like this:

/Users/jones/.rvm/gems/ruby-1.9.2-p0/gems/activesupport-3.0.1/lib/active_support/core_ext/module/aliasing.rb:31:in `alias_method': undefined method `merge_joins' for class `Class' (NameError)

What to do? Well there is an alternative search gem called MetaSearch that has a very similar interface to Searchlogic and which does work with Rails3. In fact it may be a little better.

In my code I have search code in three places - the index action of my controllers, a search form at the top of my index view pages and column headers on my index view pages that will reorder the matching rows when I click on them.

So here are the three code blocks in Searchlogic and then MetaSearch (with a very simple model where I'm simple searching the name column). Note that I'm using will_paginate in both cases.

Controller Index Action

Searchlogic

  def index
    @search = Product.search(params[:search])
    @search.order ||= :ascend_by_name
    @products = @search.all.paginate :page => params[:page], :per_page => 20
  end

MetaSearch

  def index
    @search = Product.search(params[:search])
    @search.meta_sort ||= 'name.asc'
    @products = @search.all.paginate :page => params[:page], :per_page => 20
  end

View Index Page Search Form (omitting some of the html formatting)

Searchlogic

<% form_for @search do |f| %>
<p>SEARCH</p>
<p><%= f.label :name_like, 'Name' %>
<%= f.text_field :title_like, :size => 15 %>
<%= f.submit "SEARCH" %>
<%= @search.count %>Matches</p>
<% end %>

MetaSearch - NO CHANGE!

View Index Page Column Headers (just showing a single column header)

Searchlogic

<%= order @search, :by => 'name', :as => 'name'.humanize %>

MetaSearch

<%= sort_link @search, 'name', 'name'.upcase %>

All in all, a pretty straightforward replacement.

But there is at least on additional bonus, over and about working with Rails3. 'attr_searchable' and 'assoc_searchable' allow you to specify in your models exactly which fields can and cannot be searched. Searching in either Searchlogic or MetaSearch uses GET requests which display the search parameters in the URL in the browser. That opens the door to people trying out other likely field names and searching otherwise private data. This mechanism provides a way to limit that problem.

I have no doubt that searchlogic will get updated soon, as so many people use it. But until then MetaSearch is the way to go.

Passenger 3, nginx and rvm on Mac OS X 10.6

2010-10-22T11:14:00.000-07:00

I wanted to set up the web server nginx, an alternative to Apache, along with Phusion Passenger on my Mac. I've become a convert to RVM as a way to manage multiple versions of Ruby so I needed to setup Passenger (a Ruby gem) through RVM. Whenever you have a combination like that you can run into installation problems.

Here are the steps that I used (after a couple of false starts):

1. Make sure your rvm setup is working correctly. In particular I've found it best to ignore your system Ruby installation completely and in fact I install the system version (currently 1.8.7) separately under rvm and make that my rvm default. You have to reinstall your gems, a pain, but I think it the way to go - everything lives under rvm.

2. Do not install nginx! Passenger will do that for you. If you already have an installation then rename it or just ignore it.

3. Turn off Apache or Nginx if you have either of them running.

4. Put yourself into your default Ruby and install the Passenger gem

$ rvm default
$ gem install passenger

5. Run passenger-install-nginx-module with rvmsudo

Very important - don't use regular sudo - if you do it will complain about not finding the current gemset.

$ rvmsudo passenger-install-nginx-module

The script will ask you if you want a default installation or a custom/advanced on. I just did the default (option 1).

The script downloads and compiles nginx. It will ask you where you want it installed. I suggest /usr/local/nginx. Don't just say /usr/local as that will create conf, html and logs directory right in /usr/local. You really want these in a nginx specific directory.

All being well the compilation and installation will go smoothly and your fresh installation of nginx will be modified for use with passenger.

If I remember the installation correctly it actually fires up nginx and leaves it running. Check that with 'ps ax | grep nginx'

6. Go to your browser.
'http://localhost' should give a 'welcome to nginx' page. Look in /usr/local/nginx/logs/ for logging output.

7. Configuration

I set up a symlink in /usr/local/sbin to the nginx binary so that it can be found in my regular PATH

$ sudo ln -s /usr/local/nginx/sbin/nginx /usr/local/sbin/nginx

You start nginx with 'sudo nginx' and shut it down with 'sudo killall nginx'. It needs to be run via sudo.

Configuring nginx is done in /usr/local/nginx/conf/nginx.conf and if you are used to Apache config files this will be a breath of fresh air. The main changes I made from the default was to set myself as the 'user' and set the worker_processes to 2.

To set up a Rails application simply add a server block like this:

server {
listen 80;
server_name myapp.local;
root /Users/jones/Documents/myapp/public;
passenger_enabled on;
rails_env development;
}

Be sure to set the rails_env unless you are in production (the default) otherwise it will not work. Restart nginx and with any luck you'll be able to access your application.

I made a few missteps doing my installation - such as thinking I needed to install nginx myself - but overall this went very smoothly.

8. Extra credit

If you want to use rvm and passenger to run multiple apps with multiple versions of Ruby then you will want to look at this post by Phusion:
http://blog.phusion.nl/2010/09/21/phusion-passenger-running-multiple-ruby-versions/

As you see it can get really complicated, but as a way to migrate a Rails app from 2.x to 3.x and/or Ruby 1.8.x to 1.9.x, this seems like the way to go.

Setting Environment Variables for Sudo on Mac OS X

2010-10-21T08:20:00.000-07:00

On Mac OS X you typically set up Unix environment variables in your ~/.bashrc or ~/.bash_profile files. To perform actions that need Root privileges you use 'sudo' instead of 'su'.

But when you 'sudo' a command your environment does not pick up your entire user environment and can cause problems in some cases.

To illustrate the issue, compare the output of these two commands in a Terminal (Unix shell):

$ printenv
$ sudo printenv

So how do you make a custom Environment variable visible to sudo? Rather than trying to modify system wide files like /etc/profile (which may not work anyway...) you want to modify the file /etc/sudoers.

Now, to modify this file you want to use the special command 'visudo', which as you can surmise, is a version of the editor 'vi'. Specifically it knows which file to edit and how to set the permissions on it. NOTE: If you don't know how to edit a file with 'vi' then learn the basics BEFORE you try the following steps!

$ sudo visudo

will bring the editor with the file opened. Look for this section:

# Defaults specification
Defaults        env_reset
Defaults        env_keep += "BLOCKSIZE"
Defaults        env_keep += "COLORFGBG COLORTERM"
[...]

The env_keep lines indicate that the named environment variable should be made available to sudo. So you want to add your custom variable to this list by adding lines like this at the end of this section:

Defaults        env_keep += "APXS2"

And I would suggest adding a comment line (preceded by a # character) that makes it clear that this is a custom change. Save the file and look at the output of sudo printenv:

$ sudo printenv
[...]
APXS2=/usr/sbin/apxs

Your custom environment variable should now be available.

Backtick operator in Jruby 1.5.3 not working

2010-10-20T14:55:00.000-07:00

The backtick operator in Ruby (e.g. `date`) executes a system command.

In Jruby 1.5.3 this is not working and in my example caused some odd side effects. This is a known issue (http://jira.codehaus.org/browse/JRUBY-5133) that I'm sure will be fixed in the next release.

For now, use Jruby 1.4.1 as backticks seem to work fine here.

rvm on Mac OS X Snow Leopard

2010-10-20T09:09:00.000-07:00

rvm, the Ruby Version Manager, is a great way to manage multiple Ruby versions. The documentation is extensive but I find it to be fragmented, such that if you do run into problems you have to look in multiple locations on the site to find relevant guidance.

On Mac OS X 10.6 (Snow Leopard) I had not problem installing rvm or ruby 1.9.2 but it blew up on my when I tried to list the gems under 1.9.2. The solution was to install zlib under the rvm tree:

$ ruby -v
ruby 1.8.7 (2009-06-12 patchlevel 174) [i686-darwin10.2.0]

$ rvm package install zlib
$ rvm install 1.9.2 -C --with-zlib-dir=$HOME/.rvm/usr

$ rvm use 1.9.2
$ gem list
$ gem install rails --pre

I had a similar problem when installing ruby 1.8.7 under rvm. This time it was the readline library that was the problem. Again, if you poke around in the docs you can find a fix:

$ rvm package install readline
$ rvm install 1.8.7 -C --with-readline-dir=$HOME/.rvm/usr

Although rvm makes it easy to switch back to your system ruby (with 'rvm system') I would strongly suggest that you do not use this. Instead, install the same release as your system ruby under rvm (1.8.7 in my case) and make that the default ('rvm --default 1.8.7').

You will have to reinstall gems under that ruby installation, which is a pain, but once you have all your rubies under rvm then you should have no issues with confusion over where specific libraries are located, which has been an issue for me more than once.

rvm also has a notion of gemsets, which I have not explored as yet, which allow you to isolate specific gem versions for specific applications. This could be extremely useful in migrating Rails applications to newer versions of Rails.

One area where I have not yet been successful is getting rvm and passenger to work. The rvm docs explain how to do this - but no luck for me as yet.

!! in Ruby

2010-10-19T15:45:00.000-07:00

I noticed this odd looking construct in Rick Olsen's restful_authentication plugin a while back:

!!current_user

At face value it appears to be a 'double not' operation on the variable current_user. But that didn't make much sense to me at the time. I wondered if it was some special Ruby operator, but I could find no reference to it.

After a couple of experiments I see now that it is indeed a double not operator. It serves here as a very concise way to return true or false depending on whether the variable is nil or not. Here is how it works in an irb shell:

>> foo = nil
=>> nil
>> !foo
=>> true
>> !!foo
=>> false
>> foo = 1
=>> 1
>> !foo
=>> false
>>> !!foo
=>> true

Concise is good, but not at the cost of being unclear. I'm on the fence about this one.

Stripping non-ASCII characters from text in Ruby

2010-10-13T12:32:00.002-07:00

I need to get rid of occasional non-ASCII characters in otherwise plain ASCII text, such as 'curly quotes' like “ and ”. I don't know the real encoding of my source text but I can tell that the characters are encoded as hexadecimal characters such as \x94

Here is the regular expression I use to remove them:

str.gsub!(/[\x80-\xff]/, '')

I'm sure this won't work in many cases but with my text it does the job just fine.

/etc/paths.d as a way to configure Paths in Mac OS X

2010-10-11T09:49:00.000-07:00

Just found out about a simple way to manage paths to installed software in Mac OS X

As well as, or instead of, setting PATH environment variables in startup files, you can simply add a path as a file to the directory /etc/paths.d

For example, here is a command that sets up a path to a custom installation of MongoDB

$ sudo sh -c 'echo "/Users/jones/Documents/mypath/mongodb/bin" > /etc/paths.d/mongodb'

Start up a new shell and you will find that path added to your PATH environment variable.

Note that this is a system wide setting and so it gets set early in the process of shell creation. If you need to override PATH settings for specific users then you still need to set paths in .bashrc, etc. I don't know the order in which /etc/path.d files are sourced but I would assume alphabetical.

Also note that you need the 'sh -c ' construct in order to create the file as shown above. Simple 'sudo echo' will not work.

Mouse and Trackball Setup with MacPymol

2010-09-08T07:26:00.000-07:00

I was having problems configuring a Kensington Expert Mouse Trackball to work correctly with the MacPymol protein modeling software on a Mac OS X 10.6.3 Intel Mac.

The Kensington mouse drivers (and specifically their MouseWorks software) has not been updated for several years and they are not supporting the trackball on Mac OS X Snow Leopard. But in fact the software works... MouseWorks is a 32bit version of a Preference Pane and so Preferences will have to shift to that version before you can interact with it.

But with just the trackball on that machine I was not able to configure the buttons correctly. That was the only mouse I had on that machine.

I noticed references in the MacPymol docs saying that the Apple Mighty Mouse with the little ball works well with the software. Even though I don't like that mouse, I did have one lying around so I plugged that it and configured it. With following setup it works as expected with MacPymol:

Left Button = Primary Button
Ball Button = Button 3
Right Button = Secondary Button

In 3-Button Viewing Mode this setup gives you Rotation on Left Button down and drag, Zoom on Right Button down and drag and XY translation on Ball Button down and drag.

Interestingly, once I had the Mighty Mouse setup like this, I could then get the Trackball working correctly. In Kensington MouseWOrks I have it set up as follows:
Left Lower button = Click
Right Lower Button = Right-Click
Left Upper Button = Middle-Click

MouseWorks knows about MacPymol (somehow) and you can setup MacPymol application settings - but I don't have this figured out. Unfortunately I'm not finding any more info in the MacPymol docs/wiki.

Hope this is useful

Printing Colored Text in UNIX Terminals from Ruby

2010-08-26T12:14:00.000-07:00

Outputting text in color in a UNIX console/Xterminal is kind of old school - some might say obsolete - but once in a while it is very useful. One example is the Red/Green coloring of test results. You don't get a lot of options but enough to highlight relevant text.

The cryptic escape codes are explained in this Wiki page.

This Ruby script outputs fore and background colors and a couple of the other useful modifiers.

#!/usr/bin/ruby

puts "\033[1mForeground Colors...\033[0m\n"
puts "   \033[30mBlack (30)\033[0m\n"
puts "   \033[31mRed (31)\033[0m\n"
puts "   \033[32mGreen (32)\033[0m\n"
puts "   \033[33mYellow (33)\033[0m\n"
puts "   \033[34mBlue (34)\033[0m\n"
puts "   \033[35mMagenta (35)\033[0m\n"
puts "   \033[36mCyan (36)\033[0m\n"
puts "   \033[37mWhite (37)\033[0m\n"
puts ''

puts "\033[1mBackground Colors...\033[0m\n"
puts "   \033[40m\033[37mBlack (40), White Text\033[0m\n"
puts "   \033[41mRed (41)\033[0m\n"
puts "   \033[42mGreen (42)\033[0m\n"
puts "   \033[43mYellow (43)\033[0m\n"
puts "   \033[44mBlue (44)\033[0m\n"
puts "   \033[45mMagenta (45)\033[0m\n"
puts "   \033[46mCyan (46)\033[0m\n"
puts "   \033[47mWhite (47)\033[0m\n"
puts ''
puts "\033[1mModifiers...\033[0m\n"
puts "   Reset (0)"
puts "   \033[1mBold (1)\033[0m\n"
puts "   \033[4mUnderlined (4)\033[0m\n"

Note that the exact colors will vary between different terminals. So try it and see.

The script will show you enough to create your own colored terminal output. But what if you sometimes want your output to go to a terminal (TTY) and other times to a file? In the latter, all the escape codes will give you garbage characters when you view the file. The trick here is to test whether our not your script is writing to a TTY with 'STDOUT.tty?'. If so then add the escape codes, if not then leave them out.

SSL Certificates and Apache2

2010-08-11T10:06:00.000-07:00

The process of generating a new or renewal SSL certificate for a web site can appear quite daunting given the terminology involved. In reality it is pretty simple. Here are the steps I used for renewing a Wildcard SSL cert, using rapidssl.com as the vendor.

A wildcard certificate costs more than an single fully qualified domain name, but it lets me secure any FQDN under my main domain.

The basics steps are:
1: Generate a Key
2: Generate a CSR (Certificate Signing Request)
3: Send that the vendor and give them some money
4: Receive an SSL Certificate back via email
5: Paste that into a file on your host
6: Restart Apache and verify operation

In Detail, here are my specific steps:

1: Start the new or renewal certificate process on the vendor's web site

2: Create a SSL Key
You should only need to do this once and then you can reuse it multiple times.
Create a directory in which to create and save your SSL files. For the purposes of this tutorial I'm using 'example.com as the domain. You can put the key into any file but give it one that makes sense, i.e. example.com.key

# openssl genrsa 1024 > example.com.key
Generating RSA private key, 1024 bit long modulus
.................................++++++
.........++++++
e is 65537 (0x10001)

If you want to protect the key with a passphrase then add the -des3 option to the command. If you do then you need to enter the passphrase whenever your server starts up. if you do NOT then set the permissions on your files correctly (see below).

3: Create a CSR (Certificate Signing Request)
This process will ask you several questions, of which the MOST IMPORTANT is the COMMON NAME. This the Domain Name that you want the certificate for. For a single domain this might be www.example.com. For a Wildcard certificate use an asterisk in the name like *.example.com.

The prompt from the openssl program is unclear when it asks for YOUR name - it wants the domain name. Clear on that, right?

It also asks for a challenge password - with rapidssl.com this wasn't used, so don't worry about it. Enter the other info as appropriate for your organization.

# openssl req -new -key ./example.com.key > example.com.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Washington
Locality Name (eg, city) []:Seattle
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Craic Computing LLC
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:*.example.com
Email Address []:youraddress@example.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []: 
An optional company name []:

4: Send the CSR to your Vendor
Cut and paste the contents of the file in full into the Vendor's web form. With rapidssl.com there is a phone verification process that you have to go through after they receive it. Basically an automated call comes through and you have to enter a confirmation code that they put on their web site. Plus you need to do the whole credit card and contact information thing on their site. It all ends with several emails back from them, one of which includes the Certificate.

5: Paste the certificate into a file on your host
Once again, choose an appropriate name, e.g. example.com.crt
Verify the contents of the certificate file with:

# openssl x509 -text -in example.com.crt

You will see all sorts of information.

6: CHANGE THE FILE PERMISSIONS - CHANGE THEM NOW!

# chmod 400 example.com.key
# chmod a-w example.com.crt

7: Setup Apache2
Modify your configuration files so that Apache knows where to find the certificate. In my case I'm using something like this:

<VirtualHost *:443>
  ServerName www.example.com
  ServerAlias *.example.com
  DocumentRoot "/path/to/my/site"
  SSLEngine on
  SSLOptions +StrictRequire
  SSLCertificateFile /mnt/ssl_certs/example.com.crt
  SSLCertificateKeyFile /mnt/ssl_certs/example.com.key
</VirtualHost>

You can put the certificate and key in any files you want, you just need to tell Apache where they. Your Apache configuration might be different from this.

Restart apache!

8: Test it out
Go to the site in your browser. You should see a lock icon. Click on that and view the certificate details. It should all be good.

9: COPY THE KEY AND CERTIFICATE FILES TO A BACKUP MACHINE - COPY THEM NOW!

Maximum Fixnum value in Ruby

2010-08-06T14:10:00.000-07:00

The Maximum Fixnum value that can be represented in Ruby is calculated as: (2**(0.size * 8 -2) -1)

Rails Routes and Models with 'Uncountable' Names

2010-06-30T14:54:00.000-07:00

Rails follows the convention of using singular and plural forms of model names for various purposes. For example user.rb for a Model name and users_controller.rb for the controller.

Every once in a while you need a model where the singular and plural forms are the same. I just ran into that with a model called 'species'. This can cause you all sorts of hurt but here is a way round it.

First of all, decide if you really need to use that name. If you can figure out an equivalent where singular and plural forms are different then use it - much simpler... but if like me there is no good alternative then you do this:

1: Tell the built-in pluralization functions to skip your name.

In config/initializers/inflections.rb add this:

ActiveSupport::Inflector.inflections do |inflect|
  inflect.uncountable %w( species )
end

2: Create your model, controller, table etc in the normal way using 'species' as the model name.

3: In your routes.rb give the :species resource an artificial singular form:

ActionController::Routing::Routes.draw do |map|
  map.resources :species, :singular => :species_instance

Doing this allows the routing magic to distinguish between singular and plural. But the downside is that you need to change a bunch of controller and view paths.

4: run 'rake routes' to see what Rails expects your new routes to look like.

5: In your controller, change any 'redirect_to @species' lines to 'redirect_to species_instance_path(@species)' (Should be just create and update actions).

6: In your index.html.erb view change:

link_to "Show", species
to
link_to "Show", species_instance_path(species)

In doing so you make it explicit that you want the singular form.
Similarly, change the following:

"Edit", edit_species_path(species)
becomes
"Edit", edit_species_instance_path(species)

link_to "Destroy", species ...
becomes
link_to "Destroy", species_instance_path(species) ...

and finally
link_to "New Species", new_species_path
becomes
link_to "New Species", new_species_instance_path

7: In your show, new and edit view pages, make similar changes. Links back to the index page can stay as 'species_path' but edit, delete and new links should be updated as shown above.

It's messy, for sure, but it solves the problem and allows you to retain 'natural' names in the user interface and in the URLs that users will see.

Compiling Ghostscript on Intel 64 bit Mac OS X

2010-05-11T13:58:00.000-07:00

Ran into problems getting Ghostscript to compile on an Intel Mac running Snow Leopard.

The error messages were telling me it was trying to compile a 32 executable, and then after fixing that it complained that libraries in /opt/local/lib were compiled for 32 bit. The thing is, except for some old mac ports stuff, I don't use /opt/local!

I was able to fix compile successfully with three steps.

1: Take the old /opt/local/lib code out of the picture by renaming the directory to a temporary name. I can bring it back if it turns out something actually relies on it.

2: Rub ./configure with CFLAGS and LDFLAGS defined. These end up getting passed to 'make'

$ ./configure CFLAGS='-arch x86_64' LDFLAGS='-arch x86_64'

3: Compile it 'make' (with no flags) and install with 'sudo make install'

Not sure yet if it can pick up all the fonts that it needs - we'll soon see...

Image Resizing with Rails Paperclip and ImageMagick

2010-05-11T11:14:00.000-07:00

The Rails Paperclip plugin uses ImageMagick for creating thumbnail images, etc.

You specify the size of images using ImageMagick's Geometry syntax which can be a little confusing. Here are the aspects of that which are most useful for Paperclip users.

The basic specification is <width>x<height> in pixels, optionally followed by a modifier. In some cases you can omit either width or height.

256x256

256x256!

256x

x256

256x256^

256x256>

256x256<

Other options exist within the syntax. See the ImageMagick docs for more details.

Rails, Paperclip and ImageMagick

2010-05-10T16:37:00.000-07:00

Ran into trouble generating Thumbnail images using the Paperclip plugin, which delegates the image manipulation to ImageMagick.

The error was "Paperclip::NotIdentifiedByImageMagickError: /tmp/stream,33194,0.jpg is not recognized by the 'identify' command".

A LOT of people have run into this - here is my solution.

In my setup I'm on MacOSX with a binary download of ImageMagick in /usr/local/ImageMagick-6.6.1/bin and I'm running Rails under Apache/Passenger. I've got Paperclip installed as a plugin.

There are 3 steps needed to get this working:

1: Make sure you have ImageMagick working at the UNIX command line level. This involves adding it to your path and exporting these environment variables (pointing to your ImageMagick installation, of course)

MAGICK_HOME=/usr/local/ImageMagick-6.6.1
DYLD_LIBRARY_PATH=/usr/local/ImageMagick-6.6.1/lib

Check that identify works with your images at the command line level. If not then fix those problems first.

2: Tell Paperclip where to find the ImageMagick executables
In config/environment.rb add this at the bottom of the file

Paperclip.options[:command_path] = "/usr/local/ImageMagick-6.6.1/bin"

At this point, after restarting Passenger, you would see that 'identify' is run from within Paperclip but is not able to identify the file... the final step is...

3: Identify needs those two exported environment variables - and Apache/Passenger (or other web servers probably) does not pass those through by default!
In your passenger vhost file add these lines:

SetEnv MAGICK_HOME /usr/local/ImageMagick-6.6.1
SetEnv DYLD_LIBRARY_PATH /usr/local/ImageMagick-6.6.1/lib

Restart apache/passenger and it should work fine.

For me, the binary installation of ImageMagick has worked fine so far. Compiling it yourself is a real pain (make sure you compile for x86-64 if you are on an Intel Mac) - but you don't need to do it. Likewise, you do not need the Rmagick gem if you are using Paperclip, at least for basic image resizing, etc.

Suppressing newlines in Ruby ERB documents outside of Rails

2010-04-20T09:13:00.000-07:00

In ERB files the standard <% ... %> tag pair will result in a newline being added to the output.

In Rails you can add a minus sign before the closing tag to suppress this:

<% ... -%>

But if you use ERB outside of Rails, this will produce an ERB compile error.

The way to handle this is to leave out the minus signs and instead specify the trim_mode when you create a new ERB object. There are 3 options for this parameter:

    %  enables Ruby code processing for lines beginning with %
   <> omit newline for lines starting with <% and ending in %>
   >  omit newline for lines ending in %>

Setting the trim_mode requires that you also set a safe_level, but this is normally set to 0. So to suppress newlines on all lines with ERB tag pairs create the ERB object like this:

erb = ERB.new(template_file, 0, '>')

This solves the newline issue for me but it would be nicer to have the per-line option available in Rails.

You can also set the global trim_mode in Rails if you want to. in environment.rb:

config.action_view.erb_trim_mode = ">"

Script aliases in the style of git

2010-03-31T11:04:00.000-07:00

The Git version control software lets you run its commands either through as a single program name followed by the command as an argument (such as git status), or as individual scripts (such as git-status).

The advantage of the second form is that you can use the text completion feature of your shell to save you some typing. It's a personal preference...

It is implemented by a series of symlinks from the longer command names to a single git executable. git itself gets the name of the executable it was called as and breaks that down to get the name of the command.

It is easy to create your own version of this. Here it is in Ruby...

The 'primary' script is called myscript.

#!/usr/bin/env ruby
script = File.basename($0)
if script =~ /^\S+?_(.*)$/
  command = $1
else
  command = ARGV.shift
end
puts "command #{command}"

if ARGV.length > 0
  puts "args  #{ARGV.join(', ')}"
end

Create an alias to it by adding a suffix (the sub command name), separated by some delimiter and symlinking this to the primary script:

$ ln -s myscript myscript_cmd_0
$ ln -s myscript myscript_cmd_1

The script looks at how it was called ($0 in Ruby) and sees if it can split off a command. If it does then it takes any other arguments as they are presented. If you call the script as the primary script followed by a separate command then it shifts ARGV to get the command. These examples show how it works.

$ ./myscript_cmd_0 foo bar
command cmd_0
args  foo, bar
$ ./myscript_cmd_1 foo bar
command cmd_1
args  foo, bar
$ ./myscript cmd_1 foo bar
command cmd_1
args  foo, bar

You don't want to use a technique like this all the time - you end up with loads of symlinks in your bin directory, but in the right situation it can be very useful.

Merging PDF Documents in Preview on Mac OS X Snow Leopard

2010-03-18T12:51:00.000-07:00

Preview in Mac OS X is not only a viewer for PDF (and other) documents, it allows you to merge multiple PDF documents into one. This is useful for a lot of reasons, especially when you have scanned several pages of a documents into individual files and you want to combine them.

In Snow Leopard the way this works has changed and, as there is no menu item for merging, it can be a little confusing.

Open your first 'page' or document in Preview and open up the sidebar.

If you drag a new document into the sidebar and drop it in a blank region you will see that appear in the viewing window. But this has not added this page to the first. Preview is simply allowing you to view two separate documents.

To combine pages, drag and drop the second page on top of the first. The second page will appear as thumbnail in the sidebar below the first AND the two pages will appear in the same document in the main viewing window.

This is confusing as both scenarios look the same in the sidebar. You can see the true document structure in the sidebar by picking one of the pages and moving it slightly as though you were reordering it. All pages in the same document will become surrounded with a border and shaded background.

You can reorder pages within a document by dragging and dropping as needed and 'Save As' will save the merged document as a single file.

It is a great feature of Preview but the user interface means that it is effectively hidden unless you know about it.

Raphaël Live

2010-03-11T10:14:00.000-08:00

Raphaël is an amazing JavaScript Library for creating Vector Graphics in browsers. It was created by Dmitry Baranovskiy. It is goes further than HTML Canvas in that any object is accessible in the DOM and so can be made in to buttons, dragged around the canvas, etc. You need to know about it!

To help my exploration of the library I built a simple in-browser environment with a drawing canvas and the CodeMirror code editor so that I could try out Raphaël calls and see the results immediately. That worked out really well for me and so today I've released a more developed version of the tool, along with a range of code examples.

Raphaël Live allows you to load in code examples into the editor, run them, see the results, modify the attributes, etc., re-run them and thereby learn how to use the library.

The tool is freely distributed. You can use it on the craic.com site, or download you own version from GitHub.

Hope that you'll check it out...

Rails searchlogic and confusing column names

2010-03-10T15:58:00.000-08:00

Searchlogic is a great Rails gem from Ben Johnson for adding model search capabilities to your Rails app with a minimum of effort.

You can build complex queries very easily such as Company.name_like_or_address_like.

But if you have column names that contain the Model name you can run into problems.

For examples, let's say my Company model has columns 'company_name' and 'company_address', then my query becomes:

Company.company_name_like_or_company_address_like

It gets worse is my Person model has_one :company and I want to search companies through that model - now my query becomes

Person.company_company_name_like_or_company_company_address_like()

Not only is that ugly as sin, it may cause searchlogic to barf when you use it in a search form.

My experience is that it can handle ugly queries like this when run in script/console but for some reason they may fail in the context of a real Rails app.

So what can you do about it?

The best solution is to change the names of your columns to remove the model names, but that is not always possible, especially in legacy databases.

Failing that, you can create a named scope in your model that performs the same query but uses a shorter, more sensible name. Searchlogic will use your named scopes quite happily.

In my example I would create a named scope in the Company model like this:

named_scope :company_name_address_like, lambda { |name|
  { :conditions => ['company_name like ? or company_address like ?', "%#{name}%", "%#{name}%"] }
}

I can then call it as Company.company_name_address_like().

Don't be tempted to include 'or','and', etc. in your custom named scope names. Searchlogic appears to try and split up the scope into components based on these 'operator' words.

Nokogiri and Snow Leopard

2010-03-10T07:56:00.000-08:00

I'm not alone in having problems installing the nokogiri ruby gem on a Mac that has been upgraded to Snow Leopard. The problem lies in the gem not being able to find a suitable version of libxml2, despite Snow Leopard having a recent version of that library installed. (Not sure if this is by default or only if you have the developer tools installed...).

People have tried various things but the place to look first is /opt/local/lib/xml2.*

If you have the libxml2 installed there then remove (or move) those files, along with /opt/local/libz.*

Try installing the gem again (sudo gem install nokogiri).

If it succeeds, you're good - if not then try removing the libxml2 include files in /opt/local/include - try again - and if still no luck then try trawling through other directories under /opt/local.

You should not have to specify the explicit libxml2 file locations with options to the gem install.

Git - push and pull between repositories

2010-03-03T17:12:00.000-08:00

I've been trying to improve my git skills beyond the basics. But I ran into real confusion when trying to clone one repo into another and then trying to keep them in sync using git pull and git push.

In principle this should work fine but when I pushed changes back to the origin and then ran git status in the origin I would see the 'old' versions of files in the origin marked as having changed - no merge conflicts, just that they were not up to date.

Turns out that you can't (easily) keep two repos in sync where the origin is a regular working copy. What you need to do is create a third repo that is a so called 'bare' repository. Bare repos only contain the contents of the .git directory and their role is simply to track changes.

So here is a simple example of how to set this up:

(I'm just using repos on my local machine to keep things simple)

1: Create your original working directory under git and check your code in. I'll call this repo_original.

2: Clone this into the bare repo with 'git clone --bare repo_original repo_bare'
(Take a look at the contents of repo_bare - it's just like a .git directory)

3: Clone the bare repo into a working copy with 'git clone repo_bare repo_1'

4: Make a second working copy with 'git clone repo_bare repo_2'

5: Try making some changes in repo_1, commit them and push back to the origin (repo_bare) with 'git push'

6: Now go to repo_2 and pull from repo_bare with 'git pull'. Look at your files and you should see the changes you made in repo_1. Try making changes in repo_2, push them, go to repo_1, pull changes and see that they came across.

So far so good.

7: Make conflicting changes in the same file in repo_1 and repo_2 and commit in both repos.

8: Push the changes from repo_1

9: Push the changes from repo_2 and you should be rejected with something like this

$ git push
To /Users/jones/tips/repo_bare
! [rejected]        master -> master (non-fast forward)
error: failed to push some refs to '/Users/jones/tips/repo_bare'

That's OK - there is a conflict but you can't resolve those on a bare repo, so it won't let you proceed.

10: Instead, on repo_2, pull the current origin and you will see something like this:

$ git pull
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From /Users/jones/tips/repo_bare
0d97e7a..494885c  master     -> origin/master
Auto-merged README
CONFLICT (content): Merge conflict in README
Automatic merge failed; fix conflicts and then commit the result.

That's what we want to see - there is a real conflict (we created it) and we can resolve it by editing the file and committing in the normal way.

So it's not difficult once you figure it out but it is not well explained in the guides that I've seen. Hope this helps.

Bonus Section - Branches

There are two ways to handle branching in repository setups like this - not sure of the official terms but I think of them as private and shared.

A 'private' branch is something I setup in my clone of a repo which will not get passed back to the origin repo. I create it, work in it and eventually merge it back into my repo's master. That branch will never get copied back to the origin and therefore any collaborators with their own repos will never know about it.

A 'shared' branch is one that I want to share with collaborators, so I need a way for them to access it. To do this, go to the origin repo, 'repo_bare' in the above example and create the branch there: 'git branch dev'. When you go to a cloned repo and do a pull you will see that new branch pulled over and 'git branch -a' will how it listed as 'origin/dev' but when you try to check it out you'll get an error.

$ git co dev
error: pathspec 'dev' did not match any file(s) known to git.
Did you forget to 'git add'?

What you need to do is track this branch in each of the cloned repos. To do this, in the cloned repo, run 'git branch --track dev origin/dev'. Now you can see a local branch called dev and you can checkout the branch. push and pull will now keep the tracked branch in sync as well as the master. You need to setup the tracked branch on each of the cloned repos in order to work with it.

What I haven't figured out yet is how to 'promote' a private branch in a cloned repo up to a shared branch. I suspect you have to use 'git stash' to stash the contents of that branch, delete the branch and then create a shared branch, track it and then put the stash back into it.

Apple Numbers and CSV Files

2010-03-01T14:51:00.000-08:00

The Mac OS X spreadsheet program 'Numbers', from Apple and part of the iWork suite, is a competitor to Microsoft Excel. There are some things about it I prefer to Excel, others where I prefer Excel.

But one glaring omission in Numbers is that it will not open a Comma Separated Values (CSV) file from the Open menu. CSV files are a standard way to exchange spreadsheet datasets and not being able to load them into Numbers makes no sense.

In fact there is a way to do this by Drag and Dropping the file into a worksheet.

1: Your CSV file MUST have a .csv suffix - you will just copy the filename otherwise.
2: Drag and Drop the file into a single cell of an open worksheet - choose the cell that will take the 'top left' value of your dataset.
3: That's it - simple once you know the trick - seemingly impossible until you do...

Numbers will accept tab delimited files via the Open menu option with no problems.

Rails check_box_tag and Rails 2.3.x

2010-02-15T13:22:00.000-08:00

Basic HTML checkboxes have an inherent problem in that there is no explicit value passed to the server when the box is unchecked.

Under Rails 2.2.2 (and thereabouts) you could circumvent this limitation with an ugly hack.

If you wanted to use a check_box_tag helper in a form you had to use a had to follow it with a hidden_field_tag that contained the unchecked value. For instance:

<%= check_box_tag :approval_check_box, '1' %>
<%= hidden_field_tag :approval_check_box, '0' %>

If the check box was not checked then value in the hidden field would be passed. Ugly, but it worked fine.

But at some point in the move to Rails 2.3.x this hack was turned on its head and the construct show above will now always pass the unchecked value to the server.

The fix is simple - just reverse the two lines:

<%= hidden_field_tag :approval_check_box, '0' %>
<%= check_box_tag :approval_check_box, '1' %>

The real solution if to use a check_box helper where you can explicitly specify the unchecked value, rather than a check_box_tag.

I stumbled across this while updating an older app from Rails 2.2.2 to 2.3.5. It took a while to narrow the issue down to this problem.

Rails, Time Zones and the current local time

2010-02-10T14:06:00.000-08:00

Since version 2.1 it has been easy to handle user-specific time zones in Rails. Take a look at Railscast #106 for details.

Rails takes care of converting times in your models back and forth from local timezones and UTC in the database and it works great.

But I wanted to put the current local time at the bottom of each page, so that if they printed out the page they had a time stamp right there.

My server is set up to with UTC as its time zone and Rails picks up on that, such that if I simply print out Time.now I get it in UTC, regardless of having set Time.zone to 'US Pacific...', which is not what I want.

Because I'm not accessing the time from a database record, it doesn't apply the user specific conversion.

The way to do this is to add 'in_time_zone' which forces Time to use the current Time.zone.

Bottom Line:

To display the current time in a user's local time zone, use Time.now.in_time_zone

Rails, searchlogic and will_paginate

2010-02-03T16:17:00.000-08:00

The searchlogic gem, for active record searching, integrates well with will_paginate, for pagination of the results in index pages, and it comes with simple helpers for setting up sortable column headers.

But I had trouble setting the default sort order for the results if not filtering had taken place or if no column header had been clicked. What I found on the web was a little confusing. Here's how it works (as of Feb 2010).

The model has a list of companies and I want the default search order to use their names.

In your controller:


 @search = Company.search(params[:search])
 @search.order ||= :ascend_by_name
 @companies = @search.all.paginate :page => params[:page], :per_page => 20

Note that you set the order in @search and leave out the :order parameter to paginate.

Simple, elegant, love it...

Rails, Searchlogic and Dates

2010-02-02T14:55:00.000-08:00

I've been using the searchlogic gem from Ben Johnson and its a great way to build complex search forms for your models. Railscast #176 is a great introduction to the topic by Ryan Bates (You can't go wrong with Railscasts!).

One topic missing from the documentation is how searchlogic handles Dates, but the functionality is in there. Here is how you use it.

You treat a date effectively as a number and you can use 'equals', 'gte', 'lte', etc. as comparison operators.

Assuming you have a column named 'date', of type 'date' in your model, you can put something like this in your form:


<%= f.label :date, "Date" %>
<%= f.text_field :date_gte, :size => 12 %> - 
<%= f.text_field :date_lte, :size => 12 %>

You can then enter dates into one or both of the fields to retrieve records that fall into that date range.

But what date format should you use? This is where it gets clever. It wants to use YYYY-MM-DD format but if you enter MM/DD/YYYY it accepts is and converts it for you. Brilliant! It even takes text like 'Dec 1 2009' and converts that. Interestingly it won't accept invalid dates like '2009-11-31' and just gives you back the form with the field cleared out.

With a column of type 'Datetime' it converts your entries as well but this time it converts '2010-01-01' into 'Fri Jan 01 00:00:00 -0800 2010'. I'm sure it does something similar with columns of type 'Time'.

Undoubtedly the code is leveraging the Ruby time and date libraries to make the conversions, which can be lifesavers. Incorporating the functionality into searchlogic makes your forms much more tolerant of user inputs and it does so in a totally unobtrusive way.

Creating a Mail Merge from Contacts in Highrise

2009-12-17T11:14:00.000-08:00

I've started using Highrise from 37 Signals to manage sales prospects for the SQIP Patent Sequence Database. My requirements for a CRM system are pretty basic. Highrise fits the bill for now, but it's OK - not great.

One glaring problem for me is that I want to create a file of all contacts and their addresses for a physical mailing. In Highrise I record the address for each Company and then just record the company for each Person. When you dump your contacts out as a CSV file 'Person' records do not inherit the address of the linked Company... really, it doesn't...

So I wrote a Ruby script to create what I want from the CSV file. You can find that here: http://gist.github.com/258950

1. Dump your contacts from Highrise
2. Run the script on that file
$ ./merge_highrise_contacts.rb contacts.csv > mail_merge.csv
3. Load the output file into Excel or Numbers (09)
4. Save as a Worksheet
5. Follow the regular Mail Merge instructions for MS Word/Excel or Apple Pages/Numbers

You need the '09 version of Pages/Numbers to do a proper Mail Merge. I prefer Numbers to Excel for this as it handles international characters properly, which I need.

In Numbers you need to designate the Header line as such by selecting 'Convert to Header Row' from the pull down menu on the row 1 label.

I'm all for the 37 Signals 'Keep things simple' approach but there are features that are just too widely used to ignore. Not being able to build a mailing list is one. Not having Prefix and Suffix for Person records is another.

Many of my contacts have doctorates and I want to record the prefix 'Dr.' for those that do. The only way I can do that in Highrise is to include it in the First Name. This is a pain.

Screencasting on Mac OS X 10.6 Snow Leopard

2009-12-16T13:47:00.000-08:00

I've been making several screencasts as tutorials for our SQIP Patent Sequence Database service.

If, like me, you're not that familiar with video technology and terminology, the process can be confusing - which compression settings? how many frames per second? etc.

With Mac OS X 10.5 (Leopard) you could do simple video editing in Quicktime Player if you upgraded to the Pro version. Ironically in Snow Leopard you can trim video clips in regular Quicktime Player, but without the flexibility and controls of the previous version - and there is no Pro version... so Snow Leopard is a step backwards. The version of iMovie in iLife 09 is great when it comes to editing clips, but is confusing in terms of exporting the finished movie.

Here are the steps that I use to get from my screen to a finished screencast on a hosting service that I can embed in a web page. Its not perfect, but it works for me...

Here are the main steps:
1. Set browser to 1024 x
2. Record screen activity and audio in iShowU
3. Load the movie into iMovie
4. Cut and splice together clips in iMovie
5. Export as MPEG-4 movie (.mp4)
6. Load into QuickTime Player and re-export as a streaming MPEG-4 file (.m4v)
7. Upload to screencast.com

I'm not going into all the details but here are the settings that work for me in each of these components:

1. Set my browser to a defined size (1024 x 768)

Create a bookmarklet with this string: 'javascript:window.resizeTo(1024,768)'

2. Record screen activity and audio in iShowU

Here are my settings - 'Apple Animation' for compression, capture size and frame rate are the most important.

3. Import in to iMovie09

Go to File -> Import... -> Movies... -> Optimize video: Full - Original Size

Not sure if you need to optimize video - not tried it unchecked. Importing takes a while, probably due to whatever 'optimize' involves.

4. Cut clips from your raw video and add to the new movie.

I'm not going into detail with this - look at the iMovie Docs. I find iMovie to be pretty easy to use once you get the hang of it. I would really like to see real time codes for the start and stop of clips and be able to set those directly, but I guess that is what Final Cut Pro is for.

5. Export as MPEG-4 movie (.mp4)

Here is where it can get tricky... pick the wrong settings and the output looks bad, or it takes *hours* to export, or the output file is huge... or ALL of the above!

I don't have this totally figured out but here is what works for me. Note that I'm going from 1024x768 raw footage to 640x480 output.

The important settings are:
- MP4 file format
- H.264 video format
- image size (640 x 480 VGA)
- IMPORTANT - frame rate - Custom - 5 frames per second (this is what you recorded the footage at)
- don't worry about the data rate and don't worry about the Streaming tab
- set the audio to mono

In my hands a 12 minute screencast is exported in a few minutes and yields a file around 25 MB.

6. Load into QuickTime Player and re-export as a streaming MPEG-4 file (.m4v)

Without this step, your users have to wait to download the entire file before they can start viewing. You must be able to do this within iMovie but I haven't figured it out yet.

First of all, view your .mp4 movie in Quicktime Player to make sure it works the way you want.

Then go File -> Save for Web... and Export versions for Computer (and iPhone if you want) - then 'Save'.

This will create a folder with several files - all I care about is the .m4v, but view the .html file for links and more info. Open the .m4v file in Quicktime Player to check that it is the same as the .mp4. Compare the files sizes - the .m4v may actually be larger than the .mp4.

7. Upload to screencast.com

I use screencast.com to host my screencasts. You can get a free account, which has been fine for my needs or pay a modest amount and get a much higher bandwidth limit.

It works out pretty well for me, but I do run into odd issues from Firefox on the Mac - things freezing in the admin interface - seems fine from Safari.

Once uploaded to screencast, look under the 'share' icon to get the code needed to embed a flash player in your own web page and try viewing the video. Having converted it to 'streaming' in the previous step you should see the video start pretty much right away while the rest of the data downloads.

If you skip this step then you will be stuck with a black screen for as long as it takes to download - no progress indicator! Not sure if this is a Flash problem or a Screencast one, either way it is not good.

BONUS - Title slides in iMovie

I make title slides in OmniGraffle that are the size of a frame (1024 x 768) and save these as PNG files. You can add these to you iMovie project by dragging and dropping and then changing the length of time they should be visible for.

But a big Gotcha for me, is that by default iMovie will try and apply a silly 'Ken Burns' transition to the slide making it grow (or shrink) slightly. This makes the text look rubbish in my examples.

Click on that 'clip' and use the small 'cog wheel' pull down menu to go to 'Cropping, Ken Burns and Rotation'. Now in the video preview panel (top right), click on the 'Fit' button to make the title slide fit the frame exactly - then click 'Done'

Here is an example of my screencasts embedded in a web page: Creating a New Account in SQIP

Good Luck

Announcing the Launch of the SQIP Patent Sequence Database

2009-12-16T08:15:00.001-08:00

Craic is pleased to announce the launch of a web service that we've been working on for the past couple of years.

SQIP (pronounced 'skip') is a database of DNA, RNA and Protein sequences derived from issued patents and published patent applications, along with a sophisticated interface that allows users to load their own sequences, run searches against the database, evaluate matches and then view the patents associated with those matches.

It is targeted not only at biotech patent agents and attorneys, but also at business development and research staff in biotech companies.

SQIP has been built on many of the technologies that this blog has touched on. It has been a lot of work but by leveraging best-in-class software we've been able to build something that we think will change patent sequence searching.

You can sign up for a free account at http://sqipdb.com. You can explore all the features of SQIP, including searches with pre-loaded sequences, at no charge. You simply pay for searches with your own searches - no annual commitment, no per-user charges, no surprises.

Take a look at SQIP.

Ruby and Rails issues on Snow Leopard Upgrade

2009-12-11T11:33:00.000-08:00

I had been holding off upgrading my MacBook to Snow Leopard as I figured there would be issues with the Ruby installation and some of the gems. I decided to go for it yesterday... yup, there were issues...

Snow Leopard comes with Ruby 1.8.7 already installed so it is tempting to use that but I follow Dan Benjamin's advice and install my own versions of MySQL, Ruby and Ruby gems in /usr/local. Dan has produced a series of excellent guides on installing Rails, etc on the Mac. HERE and HERE for example. However, his instructions do not always work for me. Here are the steps I needed to do get my setup running on my upgraded laptop.

The main issue is that you want Ruby and MySQL compiled as 64 bit applications. If you have Ruby as 32 bit and MySQL as 64 then things will break with cryptic messages.

I don't like MacPorts and, although I like installing binary packages, I prefer to compile Ruby etc. That way I retain full control on installing new versions.

MySQL:
1. Follow Dan Benjamin's advice on XCode, paths, etc
2. Dump your existing MySQL contents.
3. Download and compile MySQL as per Dan's instructions.
4. Load your data and check that MySQL is working as expected.

Ruby:
1. Don't compile the code as root/sudo - use sudo to run make install and gem install, but nothing else.
2. Make a list of your currently installed gems (gem list --local > gem.list)
3. Download and compile readline-6.0 (http://ftp.gnu.org/gnu/readline/). Supposedly you no longer need to do this, but I found it necessary.
4. Download Ruby into a user writeable directory in /usr/local/src or ~/src
5. Move the existing /usr/local/bin/ruby and /usr/local/lib/ruby (if you have them) - you shouldn't need to but it will make it easier to validate that the new version has been installed
6. Run ./configure with these options
./configure --enable-shared --enable-pthread CFLAGS=-D_XOPEN_SOURCE=1 --with-readline-dir=/usr/local
7. Follow with make, sudo make install
8. Check that you now have /usr/local/bin/ruby and check that it is compiled as as 64 bit by running 'file /usr/local/bin/ruby'. You should see '/usr/local/bin/ruby: Mach-O 64-bit executable x86_64'.
9. Now download and install rubygems as per Dan Benjamin.
10. Any gems that have C extensions need to be recompiled, but to be safe you should install all of them (You could copy them from your old installation but don't). I have a script for doing this which I should post.
11. The MySQL gem is different - you want to pass the path of the mysql dir to the gem install, like this:
sudo gem install mysql -- --with-mysql-dir=/usr/local/mysql
Note that you should not need to specify ARCH_FLAGS or anything else, which is recommended elsewhere - not needed.
12. Install Phusion Passenger if you are using that.
13. Check out your Rails apps. Use 'script/console' and try some simple finds on your models.

If you see this: NameError (uninitialized constant MysqlCompat::MysqlRes) it is most likely that your Ruby and/or gems are not compiled as 64 bit. You will see all sorts of posts where people have seen that error. I think most of them are the result of this mismatch.

To check any executable/library just do 'file ' and see if it says 32-bit or 64-bit.

Source Code for Mozilla Jetpack Features

2009-11-11T14:14:00.000-08:00

Mozilla Labs have released Jetpack, an environment for writing Firefox extensions using JavaScript. It looks pretty neat and I'm interested in trying writing one myself.

The best way to learn a new environment is to look at working examples and you can find a growing number of these in the Jetpack Gallery.

You just install Jetpack and then install the 'Features' that interest you and try them out.

But where is the source code? It's not under the Tools menu and you can't do something simple like right-click the Feature icon in your status bar.

1: Enter 'about:jetpack' in the URL box
2: This brings up a page with links to various things including a tutorial and the API reference.
3: Click on 'Installed Features' to see a list of everything you have installed.
4: Click on 'view source' next to each Feature to bring up the JavaScript source in a new window.

I'm not a fan of too many browser extensions but some of them like Firebug, YSlow and S3Fox are invaluable. Jetpack will make it easier for folks like me to contribute new extensions.

Of course this only applies to the Firefox browser...

Remote Desktop Software

2009-10-15T12:54:00.000-07:00

In the past I've used VNC as a remote desktop solution for viewing, say, a Linux desktop on a remote Mac, but I've now switched to NoMachine NX.

VNC is widespread and I think most Linuxes come with it already installed. But when I tried to use it recently with a Fedora VNC server and Mac OS X VNC client (Chicken of the VNC) I was seeing performance that made it unusable.

I think the problem stems from a variant of the VNC protocol or software called TightVNC and I believe the default Linux implementation uses this. Running a Tight VNC client on a Windows machine gave reasonable performance and doing the same on Mac OSX might solve the problem.

Instead I decided to check out NoMachine NX, which I had heard about in the past. This is commercial software but there are free versions of the server and client available for all the main platforms.

Installing the server (on Fedora) involves downloading RPMS for the client, node and server (you need all 3) and installing them:

# sudo rpm -i nxclient-3.4.0-5.x86_64.rpm
# sudo rpm -i nxnode-3.4.0-6.x86_64.rpm
# sudo rpm -i nxserver-3.4.0-8.x86_64.rpm

It installs into /usr/NX by default and you start/stop the server with
# /usr/NX/bin/nxserver --status|--start|--stop|--restart

You don't need to bother with any other NX options right now. At least, not if you are accessing machines in a private network. You may need to tweak your Fedora to allow the remote client to access your desktop (I didn't have to).

On the client end (Mac OS X in my case), download and install then fire up the application. It will give you a 'Connection Wizard' which is self explanatory.

The session will open up an X window on your Mac and I find the performance (on a local network) to be more than adequate. With remote desktops there can be issues with the 'scope' of keystrokes - for example you can't Cmd-C some text in the Mac world and expect to Cmd-V it into a Linux app - that sort of thing.

So if you want an open source solution then play around with VNC, otherwise go with NoMachine NX.

Fixing Broken Macs

2009-10-14T07:11:00.000-07:00

If and when you need to resuscitate a Mac that won't boot up all the way, there are some commands that will save you a lot of time.

If your problem is not a true hardware issue then chances are there are disk errors and so the first task is to check and repair the disks. You can boot off an install CD/DVD or one with repair tools on it, but if you are comfortable with booting UNIX machines then skip that and...

Boot in Single User Mode

Hold down Command-S when you turn the machine on. You should get a gray screen which will quickly turn into a classic UNIX console with all sorts of cryptic text. At the bottom of which should be some instructions to either check the disks or to reboot in 'full' Single User Mode.

Disk repair with fsck

Your first step should be to run fsck, the UNIX disk checking program, with the options -fy. The 'f' forces fsck to check all available filesystems and 'y' answers Yes whenever it would ask you to approve a fix. If you skip the 'y' then you'll be sat there for hours responding to prompts.

# fsck -fy

That will likely produce lots of scary messages about different files. Don't sweat the details, just let fsck do its thing. Any files that it breaks were already broken by the underlying disk issue. fsck won't compound the damage, only repair most or all of it.

When it finishes, run it again! There can be dependency issues with disk fixes so to play it safe just run it multiple times until it reports no new errors. Typically the first run will get them all.

If you want to poke around on the filesystem then you can mount it with:
# mount -uw /

Reboot the System

If you are feeling lucky then just type 'reboot' at the prompt to see if your system comes back all the way. Fingers crossed...

If it doesn't then just restart the machine with Command-S as before and get back to Single User Mode.

'Full' Single User Mode

At this point you are in Single User Mode but with only the basic services. You can start up a bunch more stuff with

# sh /etc/rc.d

You'll see a bunch more verbage on the console. Keep an eye on it to see if any services are not able to start up - important clues to your problem. You'll either get a system prompt or an error. Hit return to get the prompt if you don't see one.

Now you have access to pretty much all the command line programs, man pages etc. an you are free to do untold damage to your system...

diskutil

One useful program to know about is diskutil, which is the Mac tool for messing with disks. Look at the man page for details.

To see what disks the system knows about:
# diskutil list

To eject a CD/DVD in the drive (your device name may be different)
# diskutil eject /dev/disk1

Zapping the PRAM

You'll see mention about zapping the PRAM memory in the machine by starting it up while holding down Option + Command + P + R (don't worry about Shift for the P and R). Hold them until your hear two chimes.

This is easy to do - and has never had any effect on any system that I've played with...

Booting from a CD/DVD

To force a boot from a CD with recovery tools on it, turn the machine on, pop the disk in and hold down the C key. You should get to a gray screen with the spinning indicator. Let go of the C key and you should get an OS install menu, or whatever you have on your CD.

If a simple disk repair doesn't fix the issue then you may need to reinstall the OS from disk. Not a problem but be sure to select 'Archive and Install' in order to keep your existing files. Some might say just go for a reinstall regardless of the problem - but I would strongly suggest doing a disk repair first.

Good luck with your fix.

Rails Production Environments

2009-10-13T09:27:00.000-07:00

Two things that I always forget when working with a Rails app in production mode - without them you get the development environment.

Migrations:
$ rake db:migrate RAILS_ENV=production

Console
$ script/console production

SHA1 Digest of an Empty String

2009-09-15T08:53:00.000-07:00

The use of SHA1 digests of data as unique identifiers is the answer to everything.

I know that appears to be ridiculous hyperbole, but I'll go into why I think that is the case in a longer post at some point. For now you'll have to take my word for it that using them as identifiers has simplified the inner workings of a big project tremendously. It's not just me either, the Software revision control system Git uses them as IDs for all its commits.

Anyway... one issue that can arise with them is that you notice that one specific digest string occurs more frequently than others in this uniformly distributed hash space... yikes... what's going on?

That digest would be da39a3ee5e6b4b0d3255bfef95601890afd80709 - make a note of it!

It represents the SHA1 digest of an Empty String

You might want to consider adding an explicit test for that string somewhere in your code.

Ideally you want a direct test for an empty string being used as an ID, but checking for this digest is a useful catch-all test.

Once I realized that this digest was being generated, and realized what it represented, then I could focus in on the root cause very quickly.

Capistrano and Environment Variables

2009-08-19T16:47:00.001-07:00

When it works well, Capistrano is a great way to deploy Rails applications and take care of any remote server operations that are needed for that app.

But if you run into problems it can be a bear to troubleshoot. And the documentation is not great.

One issue that can catch you (or me!) unawares is the 'run' command that runs a Unix command on the remote host. I use these to symlink various files and to start up a daemon on the new deployment once that is complete.

You run into a problem if any of your remote commands require access to your Unix Environment Variables. The 'run' command is executed in a minimal environment without any of these.

The solution is to specify the variables that you need, such as PATH, as key/value pairs in the 'default_environment' hash in your cap deploy.rb file.

For example, here are two variables defined near the top of my file:

default_environment['AMAZON_ACCESS_KEY_ID'] = "YourAwsKeyHere"
default_environment['AMAZON_SECRET_ACCESS_KEY'] = "YourSecretAccessKeyHere"

Easy enough, once you know... not at all easy if you don't know!

Lightbox2 in a Rails App

2009-08-19T09:28:00.000-07:00

Lightbox2, by Lokesh Dhakar, is an excellent JavaScript script for displaying images and slide shows as overlays on a web page, triggered by clicking on a thumbnail.

I wanted something like this to display screenshots in the online help for a Rails application. Overall it was pretty simple to set up, but here are some of the tweaks I needed to make.

1: Lightbox2 uses prototype.js and Scriptaculous. I had those in my Rails app but I found I had to upgrade those scripts to the latest versions in order to get Lightbox2 to work.

2: I had included the prototype and scriptaculous libraries using the Rails helper 'javascript_include_tag :defaults', but that alone is not sufficient. Lightbox2 needs the builder and effects scripts loaded. Lightbox2 says to use this line:

<script type="text/javascript" src="/javascripts/scriptaculous.js?load=effects,builder"></script>

And I put that right after the javascript_include_tag. This is undoubtedly duplication... I should probably just include the supporting scripts directly and leave out the Rails helper.

3: As the Lightbox2 docs point out, make sure you have the four default gif image files in the right places and that they are defined correctly in lightbox.js and lightbox.css.

4: Add the calls to lightbox.js and lightbox.css in your Rails layout file. Note that you have to change the paths from the example to suit Rails:

<script type="text/javascript" src="/javascripts/lightbox.js"></script>
<link rel="stylesheet" href="/stylesheets/lightbox.css" type="text/css" media="screen" />

5: Now you're ready to add images and links that use Lightbox2. The idea is very simple. Take a thumbnail image img tag and place within a 'a' tag pair. Make the href of the 'a' tag point to the full size image. Importantly, in the 'a' tag add the attribute 'rel="lightbox"'. For example:

<a href="/images/fullsize.jpg" rel="lightbox" title="Picture of a cat"><img src="/images/thumbnail.jpg" style="border: 1px solid #808080;"></a>

The title attribute of the 'a' tag becomes the caption of the overlay image. In this example I have included a border on the thumbnail image. That should do it for basic operation.

6: To fine tune the script you can edit lightbox.js and lightbox.css. For me the animation is too slow. You can disable the animation entirely but I preferred to just speed it up with a maximum setting of 10. I also messed with the CSS for the 'imageData' set of CSS IDs to change the font and specified a different image for the 'close' icon in the .js file.

Lightbox2 is pretty slick. You can do slide shows with it as well. Hope this helps you get it integrated with Rails.

Configuring SSL on an AWS EC2 instance - Security Groups

2009-08-17T14:01:00.000-07:00

If you are trying to set up SSL with Apache (or any other server) on an AWS EC2 instance then before you do anything else, add port 443 to your Security Group.

Until you do that absolutely NOTHING will work!

You typically set up a Security Group when you create your EC2 account and then forget about them - I know I did...

You can see what ports you have open with this command:

$ ec2-describe-group

And you open up Port 443 with this command ('default' is the name of my security group):

$ ec2-authorize -p 443 default

I hope to write more on troubleshooting SSL set ups shortly.

AWS EBS Volumes and Snapshots

2009-08-12T13:53:00.000-07:00

AWS Elastic Block Store Volumes are invaluable for setting up EC2 nodes.

You can clone and/or backup Volumes into static Snapshots, also extremely useful. But be aware that the process of creating a Snapshot can be extremely slow and this can be a major problem if you are not expecting the delay.

For example, creating a snapshot from a 50GB volume took almost 2 hours for me today. Now, times will vary for all sorts of reasons. Most importantly, the first time you create a snapshot off a given volume will take the longest. Subsequent snapshots are just recording the changes since the previous one.

Being prevented from using your Volume for an extended period of time can be a huge problem, so plan ahead and make your snapshots overnight, for example.

Having said that, if your Volume contains Static or Read-Only data, then you have more flexibility:
- You can create a Snapshot from a mounted Volume, without having unmount or detach
- You can mount an unmounted Volume that is in the process of Snapshot creation.

This is convenient and does not appear to be explicit in the AWS docs, as far as I can see.
But this is limited to Volumes that will not change during snapshot creation.

WARNING: DO NOT try this with a Volume that is being written to, such as one holding a database.