A collection of computer systems and programming tips that you may find useful.
 
Brought to you by Craic Computing LLC, a bioinformatics consulting company.

Tuesday, February 15, 2011

Using Mongoid in Ruby applications outside of Rails

Mongoid and MongoMapper are two Ruby ODM (Object Document Mapper) gems for the Mongo database.

I've used both to a limited extent and they seem comparable for my needs. Mongoid seems to be getting a bit more traction than MongoMapper and it certainly has better docs.

My current project uses Mongo in a standalone Ruby application - no Rails in sight - but the docs are almost totally focused on Rails. Here is how you use Mongoid outside of Rails.

I'm storing relevant RSS entries in the database. My model looks something like this (heavily truncated):

class RssEntry
include Mongoid::Document

field :entry_id
field :title
field :authors, :type => Array
field :timestamp, :type => Time

index :timestamp
index :title
end

Be sure and define your indexes carefully for fields that you want to search on, otherwise Mongo will run out of memory when searching even modest datasets. I see this as a weakness of the database. See important note on creating indexes below!

and the application looks a bit like this (edited):

#!/usr/bin/env ruby
require 'mongoid'
$:.unshift File.dirname(__FILE__)
require 'mongoid_test_model'

Mongoid.configure do |config|
name = "mongoid_test_db"
host = "localhost"
port = 27017
config.database = Mongo::Connection.new.db(name)
end
[...]
entry = RssEntry.create({
:title => title,
:entry_id => id,
:authors => authors,
:timestamp = Time.new
})

And if you are using the defaults of localhost and 27017 then you can leave those definitions out.

NOTE: Simply defining an index in your model is NOT enough. You have to explicitly create the index. When you use Mongoid with Rails it sets up a rake task so you can run 'rake db:create_indexes' but outside of that environment you need to do this yourself.

You'll want to write a simple script/rake task to set this up, in which you call create_indexes on EACH class in your model that uses Mongoid. For example:


#!/usr/bin/env ruby
require 'mongoid'
$:.unshift File.dirname(__FILE__)
require 'mongoid_test_model'

Mongoid.configure do |config|
name = "mongoid_test"
host = "localhost"
port = 27017
config.database = Mongo::Connection.new.db(name)
end

# Call on each of the relevant models
RssEntry.create_indexes()

Previously, you could specify auto-indexing within your models but this has now be deprecated or removed, so ignore any references to that.

 

Monday, February 14, 2011

Using Twitter for System Notifications

I finally figured out something that many, many people have been doing for quite a while - using Twitter as a way to deliver notification of system events.

Twitter is a great way to deliver short messages to many people via many forms of media and devices. The default is that any message is available to anyone in the world. But you can also configure a Twitter account to be private, requiring the owner to explicitly allow access to other users. In the extreme case the owner can deny access to none but him or her self.

Twitter handles all the messaging, all you need to do is have your server, web application or whatever, send a message to your private account whenever some event takes place. For example, I run long calculations on servers at work and I want to be notified when a job completes.

You can find a load of UNIX command line twitter clients and libraries in all the main languages. So finding or building a suitable client is straightforward.

I'll show you how to build a simple client in Ruby.

If you want to send tweets to a private account then you will need proper authentication credentials.

For this you need to use OAuth - username/password authentication has been deprecated.

1: Sign in to Twitter as the owner of the private account
2: Go to http://dev.twitter.com - you'll still be signed in
3: Click on 'Register an App' - now you're not really creating a new twitter application but pretend that you are - give it a name - and you want to select that it is a 'client' application and that it should have 'read write' access to the account.
4: Now go to 'your apps' and click on the new dummy app.
5: Scroll down and get the 'Consumer key' and 'Consumer secret' - you'll need these in your code.
6: Those are required for your application, but in addition you need a key and secret for the actual twitter account that you will want to write to.
7: On your app settings page, on the right sidebar, click on 'My Access Token' and get 'Access Token (oauth_token)' and 'Access Token Secret (oauth_token_secret)'.

Now we can write some code.

8: Get the 'twitter' Ruby gem
$ gem install twitter
9: Write a small ruby app. This simple example takes a message on the command line, configures the client with the FOUR OAuth tokens/strings and then updates the private twitter account with the message:
#!/usr/bin/env ruby
require 'twitter'
abort "Usage: #{$0} message" if ARGV.length == 0
# Hard-wired to my private twitter account
Twitter.configure do |config|
  config.consumer_key = 'your-app_key'
  config.consumer_secret = 'your_app_secret'
  config.oauth_token = 'your_account_token'
  config.oauth_token_secret = 'your_account_secret'
end
client.update(ARGV[0])
10: It's that simple...
11: chmod a+x your script and run it with a message - check your private twitter account and you should see it.

It's easy to think up (and code) custom notification scripts for this. As long as you have a network and as long as Twitter is up (OK, it has had some issues) then you don't need to worry about anything to do with distributing your messages. You can get them on your phone or your desktop, and you can leverage the work of others to display popup windows on your desktop, play tunes, flash lights, etc, etc.

Just remember that when you create your private Twitter account that you go into the settings and make sure that it is indeed set to private.

One extension that I've thought about is having my script take an optional URL, say pointing to the results from a computational run, and using a URL shorting service like http://bit.ly or http://goo.gl to let me include that in the tweet. Unfortunately none of the 'big name' services allow you to have private URLs so that might be a problem in some applications. But it's worth considering for some applications.



 

PATSY - a web service that makes patents easier to read

I've just launched PATSY - a new web service that reformats US patents to make them much easier to read than their original format.

The text of patents is typically very dense and difficult to read.

They are written as legal documents and inevitably this results in verbose and sometimes arcane text. Every component of invention will have all possible variants enumerated and this can result in sentences of ridiculous length with these variants delimited by commas. On top of that, the US patent office still prints patents as two narrow columns of text of each page - a format that might work in newspapers but which in technical patents is nonsensical.

The underlying problem is that the patent offices should define and enforce a modern way of text formatting that is both easy to read and easy to parse in software.

But as this is not likely to happen any time soon, I decided to write an application that reformats the text of patents into something more palatable.

You enter a patent number into PATSY and it fetches the web page from the US patent office web site. It scans the text and splits up paragraphs into component sentences. Furthermore it splits sub-sentences by punctuation such as semi-colons. Simply adding this spacing makes a big difference.

But PATSY goes much further. It highlights a series of phrases that are typically of interest - such as 'preferred embodiment' and 'SEQ ID NO'. It recognizes references to other patents and hyperlinks these to either their patent office site or to PATSY directly. In some cases, references to scientific publications can be identified and links are added that will take the user to the NIH PubMed site of abstracts, and from there the original publication can be accessed in most cases.

PATSY only works with US patents right now and some of its features are geared towards biotechnology patents. The text parsing is not perfect but even at this early stage in its development, it can really make dense blocks of text much easier to read. In cases where the result is unclear, you can click the head of each text block to see the original text before any processing.

While it is in this early stage, PATSY is completely free. If it turns out to be useful to a lot of people then I may offer it via subscription to heavy users, while retaining free access to occasional users.

Please try PATSY out and send me feedback at info @ craic.com.

Technical aspects:

PATSY is written in Ruby using Sinatra as a lightweight web application framework. It runs on Heroku which is a hosting service for Ruby web applications that sits atop Amazon web services. My steps involved in setting it up are described here.

My experience with Heroku for this application thus far has been great. They allow you to set up applications with limited resources at no charge. If and when PATSY starts to get some traction then I can scale it up by adding more of what they call 'dynos'. That will incur some cost but there is no commitment or up front payment, plus the process of scaling is incredibly easy.



 

Friday, February 11, 2011

JavaScript Bookmarklet that can create a New Window

The popup-blocking features of current browsers can be a problem if you are writing a JavaScript Bookmarklet that wants to open a new window. For example, I want to select text in an arbitrary window and then have a remote server operate on the text and return its results to a separate window, or tab. A bookmarklet is a great way to do this.

One approach to writing these is to make the bookmarklet simply call a JavaScript script on a remote server, which does the real work. This results in a simple bookmarklet and lets you perform arbitrary operations in the remote script.

But when the end result is the creation of a new browser window this approach will fail...

Modern browsers view this a potential exploit and will only allow the creation of new windows as the result of direct user interaction - i.e. the user clicks something.

All is not lost - it just means that you need to put all your code in the bookmarklet itself. This is messy but for many scripts this should not be a problem.

Here is my example. It gets the currently selected text and adds that to the URL of a remote service. It opens that URL in a new browser tab or window, depending on the specific parameters. If no text has been selected then it prompts to user to enter some. This first version will open the new page as a new tab in most browsers.
<a href="javascript:(function(){
// Get the current selection
var s = ''; 
if (window.getSelection) { 
s = window.getSelection(); 
} else if (document.getSelection) { 
s = document.getSelection(); 
} else if (document.selection) { 
s = document.selection.createRange().text; 
} 
// Prompt for input if no text selected
if (s == '') {
s = prompt('Enter your text:');
}
// Open the target URL in a new tab
if ((s != '') && (s != null)) {
window.open('http://example.com/yourapp?id=' + s);
}
})();">BOOKMARKLET</a>
You would want to remove the comments from the bookmarklet, but you don't need to strip the newlines or minify the code.

The default of most current browsers is to create new tabs instead of new windows. Users can set their preferences to override this but most will not.

You may want to force the creation of a separate window. Think this through carefully - it may annoy some users if you start generating loads of windows. In some cases it is appropriate. It used to be that you could force this by passing '_blank' as the name of the new window but this does not appear to work in all browsers. Instead you need to explicitly specify one or more window properties, like width and height.

This is a messy solution but it works. In my application I just replaced the window.open call with this form:
window.open('http://patsy.craic.com/patsy?id=' + s, '_blank', 
'height=600,width=1024,status=1,toolbar=1,directories=1,menubar=1,location=1');
The options string in the third argument specifies what the new window should look like. You may need to experiment with these. With Google Chrome on the Mac these do not give me the expected result - the address is not editable and there is no bookmarks bar. I also found that simply using 'status,toolbar,etc' without the '=1' did not work, although you will see this listed as a valid syntax.

 

Friday, February 4, 2011

Running a Ruby 1.9 Sinatra app on Heroku

I just got my first significant application running on Heroku. It uses Sinatra instead of Rails and uses Ruby 1.9 - as a result the steps to get the application up and running were slightly different from the Heroku Quickstart Guide, which is tailored towards Rails apps.

1. Setup
Setup a Heroku account and set up the SSH keys
Create your app and make sure it runs correctly on your local machine.
Make sure that all paths are relative to the application root.
Run the Sinatra app from a config.ru file.

2. Setup Bundler
I had an issue with this originally but this is what worked. I only need the sinatra gem so my Gemfile is:
source :gemcutter
gem 'sinatra'
and my config.ru file is:
require 'bundler'
Bundler.require
require 'sinatra'
require './my_app.rb'
run MyApp.new
Note that although you need to have the heroku gem installed on your system in order to upload to Heroku, you do not 'require' it in your app.

3. Setup git and commit the project.

4. Create the Heroku app
You want to specify an application name, otherwise Heroku will give you an arbitrary one. You also want to specify the run environment at Heroku that will be used. They refer to this as the 'stack' and for Ruby 1.9 you want to specify this directly and currently the correct option is 'bamboo-mri-1.9.2'. The create command is:
$ heroku create my_app_name --stack bamboo-mri-1.9.2

5. Push the git repository to Heroku with:
$ git push heroku master
The messages that follow should look something like this (some lines removed):
-----> Heroku receiving push
-----> Sinatra app detected
-----> Gemfile detected, running Bundler version 1.0.7
Unresolved dependencies detected; Installing...
Fetching source index for http://rubygems.org/
Installing rack (1.2.1)
Installing tilt (1.2.2)
Installing sinatra (1.1.2)
Using bundler (1.0.7)
Your bundle is complete! It was installed into ./.bundle/gems/
Compiled slug size is 500K
-----> Launching... done
http://my_app_name.heroku.com deployed to Heroku
Now go to that URL and your app should be running.

If there was a problem and the app failed to start then look at the logs:
$ heroku logs -n 100


You can set up apps with minimal (or no) database storage for free on Heroku. This is a great service as it lets you experiment to your heart's content.

The idea behind Heroku is to remove from you the burden of server configuration. For my simple application this seems to work remarkably well.



 

Tuesday, February 1, 2011

Ruby 1.9 and incompatible character encodings

I run into issues pulling remote text data into a Ruby 1.9 / Rails 3 app, which is using utf-8 encoding by default. The problem apparently comes from non-Ascii characters in binary or so-called ASCII-8BIT encoded text. I don't have a proper way to translate the offending characters as yet but my workaround is to strip them out and/or replace them with an ASCII character.

This regex implements the workaround. Be sure to use the 'n' modifier on the regex. This specifies that the encoding of the text should be ignored and thus multibyte characters are treated as separate bytes.
    str.gsub!(/[^\x00-\x7F]/n,'?')

Far from perfect, but it gets the job for me right now.

 

Tuesday, January 4, 2011

rvm and shell scripts

If you want to use a specific version of Ruby via RVM inside a shell script then you need to make sure that the environment is setup correctly.

The RVM site addresses that HERE for the case where RVM is installed in a user's home directory.

But on my server I have a system-wide installation. In that case you should have a /etc/rvmrc file with a default setup and you need to source this.

Your scripts should look something like this:
#!/bin/bash                                                                                                                                                 
source "/etc/rvmrc"

rvm use 1.9.2

# mycommands_go_here


 

Archive of Tips