Ruby serializable_hash removes ruby/object from YAML output

I use YAML as a convenient way to serialize data in a number of projects, most of which are written in Ruby and Rails

You generate the YAML representation with the .to_yaml method - really simple.

p = Paper.find(359)
puts p.to_yaml

But if the input is a ruby or rails object the output is prefixed with !ruby/object and that causes problems when you try and load that document in another script that does not know about this object

For example - here is an example of a Paper object from a Rails application.

--- !ruby/object:Paper
attributes:
  id: 359
  pmid: 7945531
  title: 'Pharmacokinetics of a new human monoclonal antibody against cytomegalovirus.
    Third communication: correspondence of the idiotype activity and virus neutralization
    activity of the new monoclonal antibody, regavirumab in rat serum and its pharmacokinetics'
[...]

If I try and read this file in a separate script I get this error because that script has no concept of a Paper object.

y = YAML.load_file('test.yml')
ArgumentError: undefined class/module Paper
from /Users/jones/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/psych-2.0.5/lib/psych/class_loader.rb:53:in `path2class'
from /Users/jones/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/psych-2.0.5/lib/psych/class_loader.rb:53:in `resolve'
from /Users/jones/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/psych-2.0.5/lib/psych/class_loader.rb:45:in `find'
[...]

The way to strip off the ruby/object 'header' is to use serializable_hash before to_yaml.

---
abstract: TI-23 consists of lyophilized regavirumab (monoclonal antibody C23, MCA
[...]
id: 359
pmid: 7945531
publication_date: 1994-07-01
title: 'Pharmacokinetics of a new human monoclonal antibody against cytomegalovirus.
  Third communication: correspondence of the idiotype activity and virus neutralization
  activity of the new monoclonal antibody, regavirumab in rat serum and its pharmacokinetics'

It looks like the keys in the yaml block are output in alphabetical order.

It's a simple fix but I had to hunt around to find it.

Web Apprentice - Tutorials on Web Technologies

The range of web technologies that are available to web designers and developers is just incredible right now - from maps and graphs to translation and speech recognition.

Many of these are easy to incorporate into your sites but getting started can be confusing. There are examples and tutorials that are obsolete or needlessly complex - I've written some of them myself !

I want to help people get up and running with simple examples that they can use in their work today.

So I have created Web Apprentice (http://apprentice.craic.com) - a site that contains clear, straightforward tutorials on web technologies across the spectrum.

Tutorials range from simple embedding of Twitter feeds or Maps through to Typography, Translation and Geolocation.

Each tutorial has a self-contained live demo and all the code is available and free to use. Tutorials walk you through all the steps and explains what is going on in the code.

There are three levels of tutorial:

Basic - embedding widgets into your pages - no need to know JavaScript
Intermediate - involve some JavaScript
Advanced - complex JavaScript and Server side programming - may involve new technologies that are not in all browsers.

The goal is to add new tutorials every couple of weeks at least and over time build the site into a destination for learning and applying web technologies.

Please check it out at http://apprentice.craic.com and let me know what you think.

Setting up a new Application on Heroku

You can set up a new web application on Heroku in several ways. Here is my preferred way:

1: Build and test your application locally

2: Set it up as a git repository and commit all changes

3: Go to your account on the Heroku web site and cerate your new app there with your preferred name.

4: In your local git repo run:

$ heroku git:remote -a <your-app-name>

The response should be 'Git remote heroku added'

5: Upload your code with:

$ git push heroku master

6: Finish up your Heroku configuration on the we b site

7: Import any existing database as directed by this article.

Fixing incorrect timezones in Ruby

Timezones can be tricky to manage and sometimes I just get them wrong - like now...

I had been parsing strings representing local times in Seattle and then storing these without an explicit timezone on a machine which uses UTC by default - so they got stored as that time in UTC (not converted to UTC - the exact time string in UTC.

For example: Fri, 05 Apr 2013 14:24:39 (the local time in Seattle) was stored as Fri, 05 Apr 2013 14:24:39 UTC +00:00 which is 7 hours earlier than it should be.

The easiest way to fix this, that I found is to convert back to a string, strip of any record of a timezone and then parse it back into a time, having temporarily set the TZ environment variable to the correct local timezone.

NOTE that this works on UNIX/Mac OS X systems - not sure about Windows...

puts ENV['TZ']         => nil
puts t                 => 2013-04-05 14:24:39 UTC
s = t.to_s
puts s                 => "2013-04-05 14:24:39 UTC"
s.sub!(/\s+UTC/, '')
puts s                 => "2013-04-05 14:24:39"
ENV['TZ'] = 'US/Pacific'
t1 = Time.parse(s)
ENV['TZ'] = nil
puts t1                => 2013-04-05 14:24:39 -0700

Remember to reset the TZ environment variable back to nil

Be very careful about using gmtime and localtime as these methods operate in place. So ALWAYS make a copy of your time before applying those conversions

t            => 2013-04-05 15:25:41 -0700
t1 = t.dup

t.gmtime     => 2013-04-05 22:25:41 UTC
t            => 2013-04-05 22:25:41 UTC
t1           => 2013-04-05 15:25:41 -0700

SODA - Socrata Open Data API

Socrata is a company that helps organizations make their data available to outside users. They work with people like the World Bank and a number of city governments, including the City of Seattle.

You can access these public data sources via SODA - the Socrata Open Data API which provides a SQL-like query language (SoQL) that you submit via HTTP request to an API endpoint for that resource. The default response is JSON but you can get CSV, XML and RDF.

They provide quite a bit of documentation but it is a bit short on examples and I had to poke around a bit to get the right syntax for the data sources that I am interested in.

A very useful resource is the 'live' query console they provide for a test dataset, which lets you try various queries.

I access their resources from Ruby. I generate the URL, use open-uri to fetch the data and then parse the JSON response. Here is an example fo the process:

The data source I want to query list the Seattle Fire Department 911 calls. That page lets you create custom views and download the data. If you click on 'Export' you can see the column names used in the database as well as the url for the API endpoint.

I want to get all records in the past hour, so I need a query along the lines of 'where datetime > my_timestamp'. Specifically I need to build a URL similar to this:

http://data.seattle.gov/resource/kzjm-xkqj.json?$where datetime > my_time_stamp

This is not a real query just yet but note a couple of things...

The API endpoint is http://data.seattle.gov/resource/kzjm-xkqj.json

This version ends in '.json' and so it will return the response in JSON. If you want CSV then replace '.json' with '.csv' - same goes for XML and RDF.

Also note the dollar sign before the 'where' - it is easy to overlook - that signifies the following word is a SoQL clause and not the name of a column.

A big problem I ran into was figuring out the format for timestamps. The web page showing the table formats them as '03/21/2013 07:35:00 AM -0700' whereas the JSON response uses "datetime" : 1363872960. But neither of these works in a query... and the documentation on data types does not describe what you should use. Trial and error with that console page led me to the correct format of '2013-03-21 07:00:00' which is similar to the ISO 8601 format.

Also note that the time needs to be quoted... again, apparently not in the documentation

So my query should now look something like this:

http://data.seattle.gov/resource/kzjm-xkqj.json?$where datetime > '2013-03-21 07:00:00'

But that URL needs to be escaped before submission. That was the other issue - URI.escape() handles the spaces, quotes, etc but for some reason it does not escape the dollar sign. So I had to do that for myself.

Here is a chunk of Ruby code that gets all the 911 calls in the past hour:

require 'open-uri'
require 'time'
require 'json'

endpoint = 'http://data.seattle.gov/resource/kzjm-xkqj.json'
timestamp = (Time.now - (60 * 60)).strftime("%F %H:%M:%S")

query = "$where=datetime > '#{timestamp}'"

url = "#{endpoint}?#{query}"
url = URI.escape(url)
url.gsub!(/\$/, '%24')

puts "query: #{query}
puts "url: #{url}

json = open(url).read
puts json

result = JSON.parse(json)
puts result.length

That will show you the query, the escaped URL, the JSON returned and the number of records in the parsed JSON - enough to get you started. Different data sources will use different column names and perhaps different formats for date/time (always be aware of time zone issues).

Having a common API to many government/public data sources is great but the Socrata-backed sources that I've looked a so far don't seem to be quite as consistent as one would hope. This is an issue for the different data providers - it's not a Socrata issue.

I would hope that one could query the endpoint in some way and get a listing of all the column names and data types. There is one for SODA v1.0 but this now deprecated... so perhaps there is a newer version but it has not been documented.

Part of the answer lies in two custom HTTP response headers (X-SODA2-Fields and X-SODA2-Types) that are returned with your data. Here they are for the data source used here:

X-SODA2-Fields: [":updated_at","address","longitude",":id","latitude","incident_number","datetime","type",":created_at","report_location"]
X-SODA2-Types: ["meta_data","text","number","meta_data","number","text","date","text","meta_data","location"]
X-SODA2-Legacy-Types: true

You can see the headers if you use 'curl -i "your_url"' on the command line with your escaped url.

Tool to help Count Features in Images

Craic has released a new tool to help count features in images. This can be used in a range of applications such as counting buildings in satellite images, bacterial colonies and cell types in histology images.


It takes the form of a Ruby Sinatra application that fetches remote images and Javascript in the client web page that implements the marking and counting functions.

The user can choose the shape and color of the marker. The current total is updated as features are clicked and the coordinates of each point are stored internally.

The image with the user's marks can then be saved to a PNG format file and the list of coordinate pairs can be displayed in a separate window.

The graphics and interaction are implemented with Canvas and Javascript and make use of the wonderful Canvas2Image code from Jacob Seidelin which allows the current state of a canvas element to be saved to an image file.

The live application is hosted at Heroku and can be accessed at http://counter.craic.com

The code is distributed freely under the terms of the MIT license and is archived at Github

Uploading contents of a PostgreSQL database to Heroku without db:push

I just created a new Rails app on my local machine and want to deploy it to Heroku. I've done this before and the uploading of the code typically works great - and it did today.

Then I wanted to upload the contents of a small PostgreSQL database to Heroku - and that's where I ran into issues.

Heroku has recently switched from a Heroku gem to Heroku Toolbelt which I installed onto my Mac OS X machine.

I should be able to run 'heroku db:push' from my Rails app and the data would get moved over using the taps gem. But it failed with this error:

$ heroku db:push
! Taps Load Error: cannot load such file -- taps/operation
! You may need to install or update the taps gem to use db commands.
! On most systems this will be:
!
! sudo gem install taps

I have taps installed, so what is going on? Well, I'm not sure but I run my rubies under rbenv which allows you to manage multiple versions - and Heroku's client doesn't seem able to figure out where the gems are located with rbenv.

I dug around in /usr/local/heroku and hacked /usr/local/heroku/lib/heroku/command/db.rb to tell me what it thought the load path for ruby gems was, using this line:

STDERR.puts "DEBUG: load path  #{$:}"

The paths given in that were all in /usr/local/heroku/vendor/gems and my gem paths were nowhere to be found. I tried installing the taps, sqlite3 and sequel gems to that directory and I was able to get a bit further but I still couldn't get all the way - so I gave up on that path.

And then I found mention of a page on Heroku's site that deals with the pgbackups addon. Part of that describes how to import from a PostgreSQL backup directly, without having to use the heroku clients or taps.

I followed that and it worked great - here are the steps involved. Your starting data must be in PostgreSQL and you need to have an account on Amazon S3 (and know how to use it)

1: On your machine create a backup of your PostgreSQL database - 'mydb' is your database name and 'myuser' is your database user.

$ pg_dump -Fc --no-acl --no-owner -h localhost -U myuser mydatabase > mydb.dump

2: Transfer that to an Amazon S3 bucket and either make it world readable or set a temporary access url.

3: Check that you can get the file back out of S3 using curl

$ curl 'https://s3.amazonaws.com/mybucket/mydb.dump' > tmp
$ ls -l tmp

4: Enable the pgbackups addon in Heroku

$ heroku addons:add pgbackups

5: Get the name of the Heroku database that you will restore to, by capturing a backup from the empty heroku db:

$ heroku pgbackups:capture

HEROKU_POSTGRESQL_CRIMSON_URL (DATABASE_URL)  ----backup--->  b001

Capturing... done
Storing... done

6: Restore to that database from the S3 URL

$ heroku pgbackups:restore HEROKU_POSTGRESQL_CRIMSON_URL 'https://s3.amazonaws.com/mybucket/mydb.dump'

7: Test it out - my app was already up so I just had to browse to the appropriate page and there were all my records.

8: Remove your S3 file

DONE !

Installing the OpenCV library on Mac OS X Lion using Homebrew

OpenCV (Open Source Computer Vision) is a remarkable library of functions for real time computer vision.

I'm interested in using it on to recognize buildings in satellite/aerial photographs - I'll explain more if I get the idea to work.

The first step is to get OpenCV up and running on my Mac (OS X Lion 10.7.4). The library has a lot of dependencies that need to be installed but thankfully there is a Homebrew recipe for doing all of the heavy lifting... almost... I ran into a few problems but here is what I needed to do.

In principle all you need is:

$ brew update
$ brew install opencv

But I got two errors:
1: brew wanted me to update my version of Xcode from 4.2 to 4.3
On Lion you update Xcode via the App store and it turns out that 4.4 is available. So I installed that and reran 'brew install opencv'
2: brew then complained that the Python package 'numby' (Numerical Python) was not installed. Brew does not install Python dependencies but suggested that I do this with:

$ easy_install numby

I did that but I still got the error.

I've got two versions of python installed (via homebrew)

$ python --version
Python 2.7.3
$ python3 --version
Python 3.2.3

For our purposes you only want to deal with Python 2 - version 3 should work but the homebrew package seems to want version 2

Given the problem with easy_install, I went with a manual installation
1: Download the tar file for the latest stable version of numby
http://sourceforge.net/projects/numpy/files/NumPy/1.6.2/
2: Unpack the file into a temporary directory and cd into it
3: Run the python setup script

$ python setup.py install

Sit back and relax as this takes a while. Everything went smoothly for me and hopefully for you too.


All being well you can now run

$ brew install opencv

Again this takes a while - about 9 minutes on my box. You will get a message at the end of the output telling you to add the following to your .bash_profile

export PYTHONPATH="/usr/local/lib/python2.7/site-packages:$PYTHONPATH"

OpenCV has an excellent interface to Python, but I work primarily in Ruby so the next step for me is to get an OpenCV ruby gem set up and start to do some image processing. The next post on this blog will cover that.

Variable substitution in Ruby sub/gsub replacement strings

With the sub and gsub methods in Ruby you can extract components of the matching string and use them in the replacement string.

For example, with input string 'Hello World' this replacement:

mystring.sub(/(Wo)rld/, '\1bble') -> Hello Wobble

Note the single quotes - if you use double quotes this happens:

mystring.sub(/(Wo)rld/, "\1bble") -> Hello \u0001bble

You can use double quotes if you double escape the 'sequence' \1

mystring.sub(/(Wo)rld/, "\\1bble") -> Hello Wobble

This is very important if you want to include a ruby variable in the replacement string

s = "Mr."
mystring.sub(/(Wo)rld/, "#{s} \\1bble") -> Hello Mr. Wobble

Just remember 'double quotes' means 'double escapes'....

logit - selective logging of commands from a Bash shell

I spend a lot of time running custom scripts on datasets in a unix bash shell - typically I'm running various tools on DNA and protein sequence datasets. I need to try different tools, edit files and change parameters.

I wanted a way to capture those commands that worked and ignore all the others that simply listed directories, ran the editor, etc. Cutting and pasting from the shell history was really inefficient and the output of script was too voluminous.

To solve this I have written logit - a tool that lets you selective log commands and comments to a file while you work in a shell.

logit consists of a Ruby script and an associated bash shell script. When you run 'logit start' it creates a custom bash sub shell that you do your work in.

You can log comments to the log file with:
$ logit 'here is a comment'
And you log the last command you ran with just:
$ logit
There are other options to help you log commands further back in your history.

The resulting log file contains only those commands and comments that you chose to log. As such it serves as a concise record of a work session and may form the basis for an executable script in itself.

The regular Bash history mechanism is does not allow you to record lines from your history in a files. I'm not clear why but the cleanest solution I could come up with was the pairing of a Ruby and a Bash script. In practice you only ever deal with the Ruby script.

The code is distributed freely under the MIT license and can be found on Github at https://github.com/craic/logit

Running Ruby executable in a world-writable directory

Just now I needed to run a Ruby executable that was located in a world-writable directory.

Normally this is something that you never want to do - anyone could put an executable into your directory  and/or replace your script with their own. BUT sometimes you just need to do this - in my case I am in a secure environment, working with collaborators but we have not been able to get IT to set us up as a 'group' as yet.

The problem is that Ruby will issue a warning about "Insecure world writable directory" every time you run your script.

The real solution is to get past the need for those permissions. But in the short term I just want to get rid of the warnings.

There are two options:

You can pass the -W0 (that's a zero) flag to ruby
$ ruby -W0

But I use this 'hash bang' line at the start of all my scripts:
#!/usr/bin/env ruby
and I can't figure out how to pass the flag correctly in this case

The alternative is to add this line right after the hash bang line
$VERBOSE = nil

That does work. It clearly silences any other warnings that I might want to see but at least it gets me past my current issue.

Captain Beefheart Song Titles

Here is a silly project that I knocked out last week - a generator of Fake Captain Beefheart Song Titles

It was inspired by a comment on Gideon Coe's radio show on BBC 6 Music where he wondered if such a thing existed - it didn't - so I wrote one!

The 'algorithm', if you can call it that, combines words from real Beefheart song titles and a list of others that sound (to me) like they could be. The structure of the real titles is relatively simple with most of them following a few simple patterns. Some of the generated titles don't work but every so often it'll spit out a good one.

The site is built with Ruby and Sinatra, with some jQuery thrown in for the scrolling effect. It serves as a nice example if you are learning Sinatra. You can find the code at Github and the application is hosted on Heroku.

It's just a bit of fun, but writing a small, self-contained application like this is a great way to learn new technologies. Giving yourself a short time frame in which to develop and deploy a complete application is a great exercise in efficiency.

Sorting on multiple String keys in Ruby

Ruby gives you two ways to sort arrays of 'complex' objects - sort and sort_by

Consider this array of hashes

array = [{:key0 => 'foo', :key1 => 'bar'},{:key0 => 'hot', :key1 => 'cold'}, ... ]

sort_by is the most compact form. To sort on :key0 in ascending order you write:

array.sort_by{ |a| a[:key0] }

sort is a little more verbose, but offers more flexibility:

array.sort{ |a, b| a[:key0] <=> b[:key0] }

If you want to sort strings in descending order then you switch the a and b assignments:

array.sort{ |a, b| b[:key0] <=> a[:key0] }

You can use sort_by with multiple keys if you include them in an array:

array.sort_by{ |a| [ a[:key0], a[:key1] }

But if you are using strings and you want one of the keys sorted in descending order then you need to use sort.

In this example the desired ordering is to first sort on :key0 in descending order and then on :key1 in ascending order:

array.sort do |a,b|
  (b[:key0] <=> a[:key0]).nonzero? ||
  (a[:key1] <=> b[:key1])
end

Not the most elegant or concise piece of code, but it does the job.

Parsing 96 / 384 well Plate Maps

In molecular biology and related fields, we use rectangular plastic plates with 96 or 384 wells for many purposes, allowing us to run many experiments in parallel. In particular, they are the standard format for tasks involving laboratory robotics, such as fluid dispensing, DNA sequencing, mass spectrometry, etc.

In setting up a plate of samples, it is common for a researcher to use Excel to create a 'plate map' - a table of 8 rows and 12 columns, for a 96 well plate, with each cell holding information on the sample in that well.

Converting this into a list of individual wells for input into an instrument or a piece of software should be straightforward but variability in manually created plate maps can be an issue. Extraneous text and the placement of the table in the spreadsheet are just two of them.

I have to deal with this problem every year or two and so I decided to write a utility script that implements this conversion and offers several options for the output file.

It is not intended as a complete solution for any one application. Rather it is a starting point that I can customize as needed.

I know other people have to deal with the same type of data, so I've made the script as general as possible and put it up on github under the MIT license. Please adapt this as you see fit for your needs.

The script can take CSV or Tab delimited files as input, containing 96 or 384 well plate maps. The output is a table with one well per row and the columns : plate, row, column, value. Options allow you to specify CSV or Tab delimited as the output format, whether to parse the input in row or column major order, and whether or not to output empty cells in the plate map.

The code is written in Ruby and you can find it at https://github.com/craic/plate_maps

Let me know if you find it useful

Script aliases in the style of git

The Git version control software lets you run its commands either through as a single program name followed by the command as an argument (such as git status), or as individual scripts (such as git-status).

The advantage of the second form is that you can use the text completion feature of your shell to save you some typing. It's a personal preference...

It is implemented by a series of symlinks from the longer command names to a single git executable. git itself gets the name of the executable it was called as and breaks that down to get the name of the command.

It is easy to create your own version of this. Here it is in Ruby...

The 'primary' script is called myscript.

#!/usr/bin/env ruby
script = File.basename($0)
if script =~ /^\S+?_(.*)$/
  command = $1
else
  command = ARGV.shift
end
puts "command #{command}"

if ARGV.length > 0
  puts "args  #{ARGV.join(', ')}"
end

Create an alias to it by adding a suffix (the sub command name), separated by some delimiter and symlinking this to the primary script:

$ ln -s myscript myscript_cmd_0
$ ln -s myscript myscript_cmd_1

The script looks at how it was called ($0 in Ruby) and sees if it can split off a command. If it does then it takes any other arguments as they are presented. If you call the script as the primary script followed by a separate command then it shifts ARGV to get the command. These examples show how it works.

$ ./myscript_cmd_0 foo bar
command cmd_0
args  foo, bar
$ ./myscript_cmd_1 foo bar
command cmd_1
args  foo, bar
$ ./myscript cmd_1 foo bar
command cmd_1
args  foo, bar

You don't want to use a technique like this all the time - you end up with loads of symlinks in your bin directory, but in the right situation it can be very useful.


 

Nokogiri and Snow Leopard

I'm not alone in having problems installing the nokogiri ruby gem on a Mac that has been upgraded to Snow Leopard. The problem lies in the gem not being able to find a suitable version of libxml2, despite Snow Leopard having a recent version of that library installed. (Not sure if this is by default or only if you have the developer tools installed...).

People have tried various things but the place to look first is /opt/local/lib/xml2.*

If you have the libxml2 installed there then remove (or move) those files, along with /opt/local/libz.*

Try installing the gem again (sudo gem install nokogiri).

If it succeeds, you're good - if not then try removing the libxml2 include files in /opt/local/include - try again - and if still no luck then try trawling through other directories under /opt/local.

You should not have to specify the explicit libxml2 file locations with options to the gem install.


 

Using Oracle as a Backend to Rails

In a recent project I was asked to use Oracle as the database for a Rails application rather than MySQL. I am somewhat of an Oracle newbie and so setting this up was a bit of a learning experience. This note describes the steps that it took to accomplish the task.

For testing purposes I installed a copy of Oracle Database Express Edition (XE) on one of my servers. This is a free download from Oracle that offers much of the core functionality, albeit with some restrictions. I've written up the steps needed to Install Oracle XE on Fedora Core 5 Linux in a separate note. The following steps assume that you have this up and running.

My installation had the Rails application and web server on one machine and the Oracle database on another.

Oracle has its own way of handling client/server comunication over a network. For a Rails application to talk to a remote database it needs to access all that machinery and the preferred way of doing that in Rails is to use a Ruby interface to the Oracle Call Interface. This resides on the machine that hosts the Rails application.

So you need to install an Oracle client interface, the Ruby OCI interface and tell Rails how to connect to the database.

The Ruby OCI interface is called ruby-oci8 and comes in two variants, depending on the Oracle client software that you installed. In the Oracle/Linux installation note I installed the Full Client, however I ran into problems getting ruby-oci8 to work with the libraries included with that client (it works fine with the Server libraries). So I have to recommend the other route using the Oracle Instant Client.

1. Install Oracle Instant Client

You can download the Instant Client packages for free from Oracle, although you will need to create an account for yourself first. Pick your platform, agree to their license and you'll get a page full of packages in different versions. All you need are these two from Version 10.2:
Instant Client Package - Basic
instantclient-basic-linux-x86-64-10.2.0.3-20070103.zip (36 MB)
Instant Client Package - SDK
instantclient-sdk-linux-x86-64-10.2.0.3-20070103.zip (0.6 MB)
The ruby-oci8 install guide tells you to install these into /opt/oracle but that is probably not mandatory. I had downloaded the files into /proj/downloads.

# mkdir /opt/oracle
# cd /opt/oracle
# unzip /proj/downloads/instantclient-basic-linux-x86-64-10.2.0.3-20070103.zip
# unzip /proj/downloads/instantclient-sdk-linux-x86-64-10.2.0.3-20070103.zip
# cd instantclient_10_2

Then create the following symbolic link

# ln -s libclntsh.so.10.1 libclntsh.so

Edit /etc/bashrc to include the directory in your LD_LIBRARY_PATH environment variable by adding these lines

LD_LIBRARY_PATH=/opt/oracle/instantclient_10_2:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH

The start a new shell, or source /etc/bashrc, in order to see the new value for the variable. Installing ruby-oci8 won't work without it!

2. Install ruby-oci8

Download version 1 of the interface from rubyforge:

# wget http://rubyforge.org/frs/download.php/16630/ruby-oci8-1.0.0-rc1.tar.gz

Unpack, make and install the package

# tar -zxvf ruby-oci8-1.0.0-rc1.tar.gz
# cd ruby-oci8-1.0.0-rc1
# make
# make install

The make calls ruby setup.rb to configure and then compile the code. It should all go smoothly provided the Oracle libraries are in place and it knows where to find them.

The official ruby-oci8 install guide may help if you have problems.

3. Test out the Connection to the Remote Database

This assumes that the remote database is running and that you have enable the test HR database that ships with the server. Test the connection with a simple Ruby one-liner. Note that this specifies the remote host using Oracle's Easy Connect naming scheme (bit like a URL). Alternatively you can set up a TNSNAMES.ORA files if you know about all that. Here the username is hr, the password is hr, the remote host is testbed.int.craic.com and the Oracle SID is XE. If the call is successful then you should get back a bunch of lines listing various jobs in the database.

# ruby -r oci8 -e "OCI8.new('hr', 'hr', '//testbed.int.craic.com/XE').exec(
     'select * from jobs') do |r| puts r.join(','); end"
AD_PRES,President,20000,40000
AD_VP,Administration Vice President,15000,30000
AD_ASST,Administration Assistant,3000,6000
FI_MGR,Finance Manager,8200,16000
[...]

If that works then you have finished the installation and can now move on to creating your Rails application. If, like me, you are not very familiar with setting up a database in Oracle here are some steps to get you started.

4. Create a 'Database' in Oracle

First of all, Oracle does not use that term in the way you might expect if you are coming from MySQL!

Oracle has an instance, identified by an SID (which in the above case is XE). Within that instance you need to create a new User and in doing so you create a new tablespace (I think that is the right term) which is equivalent to creating a database in MySQL.

You need to access the Oracle Server web site on your server, login as system and create the new user. Use the name of your Rails app for convenience. I use myapp in this example. Enter a password and check the boxes Connect and Resource in the Roles section. Don't worry about the Directly Granted System Privileges. Then click Alter User to set it up.

5. Create Your Rails Application

Back to you client machine. Assuming that you have Rails already setup, go to your target directory and create the subdirectories for your app.

# cd /proj/rails
# rails myapp

Then configure your database.yml file. You will actually want three Oracle users, one for the development, production and test databases, but just consider the development version right now. A suitable block for that file would be:

development:
adapter: oci
#  database: myapp_development
host: testbed.int.craic.com/XE
username: myapp
password: myapp

Note that the database: key is not relevant for Oracle, as I mentioned. The hostname is this Oracle-specific combination of the real hostname followed by a slash and the SID (which is always XE for Oracle Express Edition)

(Update: you can create your rails app with this option:

# rails myapp --database=oracle

Which specifies the adapter to be oracle instead of oci, but when doing so I got an error telling me that the TNS service name was not properly specified. But if you use the oci adapter as above it should work fine.)

From here on out you are working in regular Rails. You can create and modify your tables using migrations, you can CRUD your data and everything should just work.

If you are coming from MySQL, like me, then be aware that Oracle uses different underlying data types than MySQL and it appears that can be an issue with regard to dates and times in certain cases. Don't use the datatype :tinyint as it will barf - use :integer, :limit => 4 instead.

Note that in Rails 1.2.1 there is a bug that, with Oracle, will put the text empty_clob() in to empty textareas in a form. This has been noted and fixed but is not yet released. A simple work around is to set that field explicitly to blank string in the new action in your controller. For example:

def new
@blog_entry = BlogEntry.new
@blog_entry.text = ''
end