JSON versus JSONP Tutorial

I recently found myself a bit confused on how to convert a JQuery $.getJSON() call to use JSONP (JSON with Padding).

JSONP is a way to get around the Same Origin Policy restriction when a script in one domain wants to fetch JSON data from a server in a different domain.

The approach is simple enough but I had trouble finding a clear explanation. So I wrote up a simple example in both JSON and JSONP with server and client code.

The live demo is at http://json-jsonp-tutorial.craic.com and all the code is on Github at https://github.com/craic/json_jsonp_tutorial

I hope that you find it useful...

Tool to help Count Features in Images

Craic has released a new tool to help count features in images. This can be used in a range of applications such as counting buildings in satellite images, bacterial colonies and cell types in histology images.


It takes the form of a Ruby Sinatra application that fetches remote images and Javascript in the client web page that implements the marking and counting functions.

The user can choose the shape and color of the marker. The current total is updated as features are clicked and the coordinates of each point are stored internally.

The image with the user's marks can then be saved to a PNG format file and the list of coordinate pairs can be displayed in a separate window.

The graphics and interaction are implemented with Canvas and Javascript and make use of the wonderful Canvas2Image code from Jacob Seidelin which allows the current state of a canvas element to be saved to an image file.

The live application is hosted at Heroku and can be accessed at http://counter.craic.com

The code is distributed freely under the terms of the MIT license and is archived at Github

Counting Temporary Shelters in Satellite Images using OpenCV

A number of areas of data analysis are dominated by sophisticated algorithms, intensive computation and, in some cases, limited access to raw data. Examples include the analysis of satellite imagery, feature extraction from video, protein structure analysis and language processing.

I am interested in how simple, approximate methods can be used to extend the application of these technologies. While simple approaches will clearly not match the accuracy and resolution of complex methods, they can, by nature of their simplicity, be implemented and deployed more easily and hence more widely.

I have just posted the code for one of these projects to the Craic Github site. The application involves image processing of satellite images downloaded from Google Maps in order to estimate the number of temporary dwellings or shacks in the slums of Port-au-Prince in Haiti.

The satellite images of Port-au-Prince show large areas covered with small white, blue and rust colored squares. These are the slums of the city in areas such as St.Martin, Cite Soleil and others. 

The blue features are very distinctive and most likely represent the ubiquitous blue plastic tarpaulins the you can find at any hardware store. 

Using the Python interface to the wonderful OpenCV image processing library, I wrote a simple application that identifies blue features below a cutoff size and then computes their area. By fetching adjacent squares that tile across the city, and calculating the area of blue features, I can come up with an matrix showing the relative density of the slums.

The approach is undoubtedly simplistic (the code is only about 35 lines of Python) but it demonstrates how simple approaches can be successfully applied to complex and sophisticated types of data. You do not always need access to expensive commercial software and proprietary datasets in order to work in these fields.

The code is made freely available at https://github.com/craic/count_shelters and you can read my write up of the work so far, with example images HERE.

Tutorial on JS/JQuery Bookmarklets

I recently needed to implement a JavaScript bookmarklet for a new project. There are many web resources available but even so I had a bit of a learning curve to contend with.

In the hope of lessening that burden on future users, I've written up a tutorial on two types of bookmarklets with simple, annotated examples and posted all the code on our Github page.


The simplest form of bookmarklet is a chunk of JS code contained in the Bookmarklet URL that invokes some action. This is a direct approach - you select some text, you click the bookmarklet and you get a result. If you want to repeat the process with some other text on the page then you have to click the bookmarklet link again.

However, for many applications you need to invoke an action multiple times on a single page - looking up words in a dictionary would be a good example. Having to click on a Bookmarklet link every time is not a good solution. Instead you want to modify the behaviour of the page by injecting a custom JS script. A bookmarklet can be used to initiate this. So you end up with three scripts, two of which are slightly modified, general purpose scripts.


The bulk of the code for this second approach comes from how-to-create-a-jquery-bookmarklet, written by Brett Barros of latentmotion.com, with some code input from Paul Irish.

To fully benefit from the tutorial, you should download the project, run 'bundle install' and 'rackup -p 4567' to set up a local Sinatra server, then point your browser to http://localhost:4567

Rails 3.2 Asset Pipeline - 'require_tree' is evil !

I have been battling the Asset Pipeline in Rails 3 for a couple of days - and finally I've got it working

An important part of the Asset Pipeline is the use of Manifest files in app/assets/javascripts and stylesheets which determine the specific files to include and, in some cases, the order in which they are processed.

The default javascripts Manifest file has these lines:

//= require jquery
//= require jquery_ujs
//= require_self
//= require_tree .

That reads as:
- process jquery, then jquery_ujs
- process anything in this file (self)
- process all the files in this directory tree (require_tree)

I don't have many JS files in my app - three custom JS files written by myself and a couple of third party libraries.

I try and follow defaults as far as possible but require_tree has two BIG problems.

First is that it processes ALL the files in the directory tree.
Second is that the ORDER is UNDEFINED.

It is common for JS libraries to distribute their code along with examples and minimized versions of the code. In the past I would just drop these distributions in the javascripts directory and then specify which of the files should be included in my app.

So one problem with require_tree is that it will process both a 'native' and minimized version of the same library is both are present. That results in the same functions being defined multiple times - and that seems to be a real problem if a library includes it's own copy of jQuery.

Not specifying the order of processing is the other BIG issue. For libraries with a 'core' JS file along with others will typically require that the core library is processed first.

The solution is to avoid require_tree at all costs. Put only the JS files that you absolutely need in app/assets/javascripts and specify them EXPLICITLY in the order they should be processed.

In my example, my application.js file becomes:

//= require jquery
//= require jquery_ujs
//= require ./tabs_timeline_simple
//= require ./tandem_select
//= require ./raphael-min
//= require ./facebox

I strongly recommend that you start out with everything clearly defined like this. Once you have it all working then you can remove unused files and simplify application.js - if not, you risk going through the same debugging misery that I just went through.

Upgrading from Rails 3.0 to 3.1 - Issues with Javascripts

I have been upgrading a Rails app from 3.0.10 to 3.1.5.

The BIG difference with 3.1 is the use of the Assets pipeline which requires that you move images, stylesheets and javascripts from public/* to app/assets/*. In the case of Stylesheets and Javascripts you also put these under the control of a Sprockets Manifest file: application.js or application.css.scss

That Manifest file caused me problems with my javascripts...

I followed the advice in the Railscasts Episode 282 (which I recommend you look at) and put these lines into my application.js

//= require jquery
//= require jquery_ujs
//= require_self
//= require_tree .

But when I ran my app none of my javascript functions were working. Looking at the source for one of the pages I saw this link

  <script src="/assets/application.js" type="text/javascript"></script>

Clicking on brought up a message, instead of a JS file,  saying that Sprockets was unable to find the file 'jquery'

Jquery is supposed to 'just be there' if you have the jquery-rails gem installed.

I think there were a two, maybe three, issues.

1: Sprockets requires that there are NO BLANK LINES in the Manifest file BEFORE the end of the lines that should be processed. In Rails 3.2 the default file says that explicitly but if you are creating your own that is an easy thing to overlook.

2: Look at the 4 lines shown above. The first two are 'require', followed by a SPACE and then the name of the javascript file.

The Third is 'require_self' - underscore, no space - that means process any JS code in this file.
The Fourth is 'require_tree' - underscore, no space - and that means process ALL JS files in this directory.

If you type these lines in and happen to use a space instead of the underscore then Sprockets will look for JS file called 'self' and 'tree' - and won't find them. So be very careful tying those in.

3: The main problem was the require_tree line

I have a number of JS files that are included as needed in certain views. At the top of each view page I specify those with a content_for block like this:

<% content_for :custom_javascript_includes do %>
  <%= javascript_include_tag "raphael.js" %>
  <%= javascript_include_tag "tabs_timeline_simple.js" %>
<% end %>

and in my application.html file I have

  <%= javascript_include_tag "application" %>
  <%= yield :custom_javascript_includes %>

That has worked just fine in the past and I kept all the custom JS files in the javascripts directory. But require_tree will load ALL of them on EVERY page. I assume that there are some variable/function names that are duplicated and/or conflicting and so with all that loaded, nothing worked.

The solution is to get rid of the require_tree line completely.

If I had JS files that should be included in each page then I would 'require' each of them explicitly. This is the same approach that Ryan Bates suggests for CSS files. It just gives you more flexibility.

Uploading contents of a PostgreSQL database to Heroku without db:push

I just created a new Rails app on my local machine and want to deploy it to Heroku. I've done this before and the uploading of the code typically works great - and it did today.

Then I wanted to upload the contents of a small PostgreSQL database to Heroku - and that's where I ran into issues.

Heroku has recently switched from a Heroku gem to Heroku Toolbelt which I installed onto my Mac OS X machine.

I should be able to run 'heroku db:push' from my Rails app and the data would get moved over using the taps gem. But it failed with this error:

$ heroku db:push
! Taps Load Error: cannot load such file -- taps/operation
! You may need to install or update the taps gem to use db commands.
! On most systems this will be:
!
! sudo gem install taps

I have taps installed, so what is going on? Well, I'm not sure but I run my rubies under rbenv which allows you to manage multiple versions - and Heroku's client doesn't seem able to figure out where the gems are located with rbenv.

I dug around in /usr/local/heroku and hacked /usr/local/heroku/lib/heroku/command/db.rb to tell me what it thought the load path for ruby gems was, using this line:

STDERR.puts "DEBUG: load path  #{$:}"

The paths given in that were all in /usr/local/heroku/vendor/gems and my gem paths were nowhere to be found. I tried installing the taps, sqlite3 and sequel gems to that directory and I was able to get a bit further but I still couldn't get all the way - so I gave up on that path.

And then I found mention of a page on Heroku's site that deals with the pgbackups addon. Part of that describes how to import from a PostgreSQL backup directly, without having to use the heroku clients or taps.

I followed that and it worked great - here are the steps involved. Your starting data must be in PostgreSQL and you need to have an account on Amazon S3 (and know how to use it)

1: On your machine create a backup of your PostgreSQL database - 'mydb' is your database name and 'myuser' is your database user.

$ pg_dump -Fc --no-acl --no-owner -h localhost -U myuser mydatabase > mydb.dump

2: Transfer that to an Amazon S3 bucket and either make it world readable or set a temporary access url.

3: Check that you can get the file back out of S3 using curl

$ curl 'https://s3.amazonaws.com/mybucket/mydb.dump' > tmp
$ ls -l tmp

4: Enable the pgbackups addon in Heroku

$ heroku addons:add pgbackups

5: Get the name of the Heroku database that you will restore to, by capturing a backup from the empty heroku db:

$ heroku pgbackups:capture

HEROKU_POSTGRESQL_CRIMSON_URL (DATABASE_URL)  ----backup--->  b001

Capturing... done
Storing... done

6: Restore to that database from the S3 URL

$ heroku pgbackups:restore HEROKU_POSTGRESQL_CRIMSON_URL 'https://s3.amazonaws.com/mybucket/mydb.dump'

7: Test it out - my app was already up so I just had to browse to the appropriate page and there were all my records.

8: Remove your S3 file

DONE !