Setting up a new Application on Heroku

You can set up a new web application on Heroku in several ways. Here is my preferred way:

1: Build and test your application locally

2: Set it up as a git repository and commit all changes

3: Go to your account on the Heroku web site and cerate your new app there with your preferred name.

4: In your local git repo run:

$ heroku git:remote -a <your-app-name>

The response should be 'Git remote heroku added'

5: Upload your code with:

$ git push heroku master

6: Finish up your Heroku configuration on the we b site

7: Import any existing database as directed by this article.

Adding extensions to Postgresql database with Rails on Heroku

This is a follow up to my previous post Adding extensions to Postgresql database with Rails

If your database is running on Heroku you need to access it slightly differently.

Using the 'rails db' command does not work:

$ heroku run rails db
Running `rails db` attached to terminal... up, run.6499
Couldn't find database client: psql. Check your $PATH and try again.

Instead access psql like this and then add the extension. \dx lists the current installed extensions

$ heroku pg:psql
psql (9.2.1, server 9.1.9)
[...]
SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)
Type "help" for help.
dftmhe7vpg4dhs=> create extension unaccent;
CREATE EXTENSION
dftmhe7vpg4dhs=> \dx
                         List of installed extensions
   Name   | Version |   Schema   |                 Description
----------+---------+------------+---------------------------------------------
 plpgsql  | 1.0     | pg_catalog | PL/pgSQL procedural language
 unaccent | 1.0     | public     | text search dictionary that removes accents

Note that this works just fine with a pg:basic database plan. There is no need to upgrade just to use postgres text search.

Web Text To Speech using Bing Translate - Demonstration Sinatra App

Given the variety of sophisticated web services available today, it is surprising that a good Text To Speech API (TTS) is not readily available.

Google has one as part of its Translation API but it is not publicized or actively supported. The Bing/Microsoft Translate API also has a TTS feature and this is supported. In my experience this works very well and allows you to specify the language of the text, which changes the voice and pronunciation that is used.

Accessing this API is easy enough using a wrapper library, such as https://github.com/CodeBlock/bing_translator-gem (disclaimer: I added the speak() function to this ruby gem) but it returns binary data that represents an MP3 file. The HTML5 audio tag allows you play audio files, but not, apparently, to play the data itself.

The result is that the TTS output must be first written to a file and then played via the audio tag.

To demonstrate how these different pieces fit together, I have written a TTS demo app that consists of a Javascript in a web page that sends the query text to a Sinatra app, using ajax. The server in turn sends this to Bing and get back the audio. The server then writes this to a file on Amazon S3 and returns the URL for this back to the web page where the audio is played.

The Live Demo for this is at http://bing-translate-tts-demo.craic.com/ and the code required to implement the whole thing is freely available at https://github.com/craic/bing_translate_text_to_speech

The demo has several moving parts and setting it up for yourself requires experience with Sinatra, S3, etc.

Tool to help Count Features in Images

Craic has released a new tool to help count features in images. This can be used in a range of applications such as counting buildings in satellite images, bacterial colonies and cell types in histology images.


It takes the form of a Ruby Sinatra application that fetches remote images and Javascript in the client web page that implements the marking and counting functions.

The user can choose the shape and color of the marker. The current total is updated as features are clicked and the coordinates of each point are stored internally.

The image with the user's marks can then be saved to a PNG format file and the list of coordinate pairs can be displayed in a separate window.

The graphics and interaction are implemented with Canvas and Javascript and make use of the wonderful Canvas2Image code from Jacob Seidelin which allows the current state of a canvas element to be saved to an image file.

The live application is hosted at Heroku and can be accessed at http://counter.craic.com

The code is distributed freely under the terms of the MIT license and is archived at Github

Uploading contents of a PostgreSQL database to Heroku without db:push

I just created a new Rails app on my local machine and want to deploy it to Heroku. I've done this before and the uploading of the code typically works great - and it did today.

Then I wanted to upload the contents of a small PostgreSQL database to Heroku - and that's where I ran into issues.

Heroku has recently switched from a Heroku gem to Heroku Toolbelt which I installed onto my Mac OS X machine.

I should be able to run 'heroku db:push' from my Rails app and the data would get moved over using the taps gem. But it failed with this error:

$ heroku db:push
! Taps Load Error: cannot load such file -- taps/operation
! You may need to install or update the taps gem to use db commands.
! On most systems this will be:
!
! sudo gem install taps

I have taps installed, so what is going on? Well, I'm not sure but I run my rubies under rbenv which allows you to manage multiple versions - and Heroku's client doesn't seem able to figure out where the gems are located with rbenv.

I dug around in /usr/local/heroku and hacked /usr/local/heroku/lib/heroku/command/db.rb to tell me what it thought the load path for ruby gems was, using this line:

STDERR.puts "DEBUG: load path  #{$:}"

The paths given in that were all in /usr/local/heroku/vendor/gems and my gem paths were nowhere to be found. I tried installing the taps, sqlite3 and sequel gems to that directory and I was able to get a bit further but I still couldn't get all the way - so I gave up on that path.

And then I found mention of a page on Heroku's site that deals with the pgbackups addon. Part of that describes how to import from a PostgreSQL backup directly, without having to use the heroku clients or taps.

I followed that and it worked great - here are the steps involved. Your starting data must be in PostgreSQL and you need to have an account on Amazon S3 (and know how to use it)

1: On your machine create a backup of your PostgreSQL database - 'mydb' is your database name and 'myuser' is your database user.

$ pg_dump -Fc --no-acl --no-owner -h localhost -U myuser mydatabase > mydb.dump

2: Transfer that to an Amazon S3 bucket and either make it world readable or set a temporary access url.

3: Check that you can get the file back out of S3 using curl

$ curl 'https://s3.amazonaws.com/mybucket/mydb.dump' > tmp
$ ls -l tmp

4: Enable the pgbackups addon in Heroku

$ heroku addons:add pgbackups

5: Get the name of the Heroku database that you will restore to, by capturing a backup from the empty heroku db:

$ heroku pgbackups:capture

HEROKU_POSTGRESQL_CRIMSON_URL (DATABASE_URL)  ----backup--->  b001

Capturing... done
Storing... done

6: Restore to that database from the S3 URL

$ heroku pgbackups:restore HEROKU_POSTGRESQL_CRIMSON_URL 'https://s3.amazonaws.com/mybucket/mydb.dump'

7: Test it out - my app was already up so I just had to browse to the appropriate page and there were all my records.

8: Remove your S3 file

DONE !

Rails, UTF-8 and Heroku

I've had problems with Ruby character encodings over the years, especially when pulling text with non-ASCII characters in from remote sites. I thought I had it mostly sorted but the past few days showed me that was not the case.

I have a MySQL database on my local machine and a Rails 3 app that pulls in text from remote sources, stores it in the database and does stuff with it. I am deploying the application on Heroku prior to public release. Heroku uses PostgreSQL exclusively.

I was under the belief that all the components in my system were set up to use UTF-8 encoding and therefore moving text with non-ASCII characters around should be fine. But in practice that was not the case - characters like 'α' that looked fine on my local machine showed up as 'α' on Heroku, etc. So clearly I was doing something wrong. Rather than go into all the gory details, this is the way to do it right...

Bottom-line: Make everything use UTF-8 explicitly ... EVERYTHING

0: Backup your current database!

1: MySQL
Here are the contents of my /etc/my.cnf file:

[mysqld]
datadir=/usr/local/mysql/data
bind-address = 127.0.0.1
character-set-server = utf8
max_allowed_packet = 32M
[client]
default-character-set = utf8
[mysql]
max_allowed_packet = 32M

Even if your tables are held in utf8, you should add these lines. You want the following mysql command to look as shown:

mysql> SHOW VARIABLES LIKE 'character\_set\_%';
+--------------------------+--------+
| Variable_name            | Value  |
+--------------------------+--------+
| character_set_client     | utf8   |
| character_set_connection | utf8   |
| character_set_database   | utf8   |
| character_set_filesystem | binary |
| character_set_results    | utf8   |
| character_set_server     | utf8   |
| character_set_system     | utf8   |
+--------------------------+--------+

2: Rails
I'm using Rails 3 - can't tell you how this works in Rails 2.x
a: In config/application.rb make sure this line is uncommented:

    # Configure the default encoding used in templates for Ruby 1.9.
    config.encoding = "utf-8"

MySQL and Rails use different variants of utf8/utf-8 - make sure you are using the right one. And note the comment above this line - this sets up utf-8 encoding for templates ONLY.
b: In your database.yml, specify the encoding for the databases - for example:

development:
  adapter:  mysql2
  host:     localhost
  encoding: utf8
  [...]

Here you are telling the database adapter that the database uses utf-8.
c: mysql2
Notice that I am using the mysql2 adapter instead of mysql. At this point (Feb 2011) the mysql gem is NOT encoding aware. Replace mysql with mysql2 in your Gemfile and run bundle.
d: In each Model that uses text add this line at the very top of the file:

# encoding: UTF-8

This tells Ruby that we're using utf-8 in this model. I don't see a way to set this at the application level so you have to have to add it in all relevant model .rb file. I also don't like defining something with a comment line. I can't see how to define this in, say, an irb interactive session.

With all those components in place, you should be all set. Try entering non-ASCII characters into a form - such as accented characters or greek/math symbols. These should be displayed correctly in the browser and in the mysql command line client.

With regards to Heroku, assuming you have your app already set up, you should be able to do a 'heroku db:push' to copy the database into PostgreSQL on Heroku and the characters should display correctly on the remote pages. You will see reference to using 'heroku db:push' with explicit database URLs that include an encoding option, such as '?encoding=utf8'. If your MySQL is set up correctly then this should be unnecessary.

A critical part of running apps on Heroku is the ability to pull the database back to your local database using 'heroku db:pull'. Before getting all my components set up with utf-8, this step failed for me. With everything using utf-8, and after adding the 'max_allowed_packet' lines to my my.cnf file, this process works fine.

But because I was working with data in before everything was truly utf-8, I had some instances of text in the database that had been incorrectly encoded - and thereby effectively corrupted. I could see what the 'corrupt' characters looked like and I knew what the correct versions should be. Because everything is now using utf-8 I could simply do a substitution on the text. For example:

str.sub!(/ß/, 'β')

I gathered up the character mappings that I needed (which were not may in my case) and wrote up a class method that I cloned in each model with the issue. I then ran those in the Rails console to correct the bad characters. The method is:

  def self.make_utf8_clean
    mappings = [  ['α', 'α'],
                  ['ß', 'β'],
                  ['β', 'β'],
                  ['’', '’'],
                  ['“', '“'],
                  ['â€\u009D;', '”'],
                  ['â€', '”'],            
                  ['ö', 'ö'],
                  ['®', '®']
              ]
    # Get the list of String columns
    columns = Array.new
    self.columns.each do |column|
      if column.type.to_s == 'string'
        columns << column
      end
    end
    
    # Go through each object -> column against all mappings
    self.all.each do |obj|
      columns.each do |column|
        mappings.each do |mapping|
          value = obj.attributes[column.name]
          if value =~ /#{mapping[0]}/
            s = value.gsub(/#{mapping[0]}/, "#{mapping[1]}")
            obj.update_attribute(column.name.to_sym, s)
          end      
        end
      end
    end
  end

This looks at your model and figures out which columns are of type String. It goes through all records and all character mappings, replacing text and updating the database as needed. Your mappings array could be much larger. There may be a better source of these, but this is a start.
You run this in a rails console like this:

Loading development environment (Rails 3.0.4)
ruby-1.9.2-p0 > YourModel.make_utf8_clean

It's a hack but it helped my 'fix' quite a few records that would have been a pain to recreate.

Character encodings are HARD - Yehuda Katz wrote a nice article on the issues. For most purposes (unless you work with Japanese text) UTF-8 is your best choice for encoding and so I'm using it exclusively. Java and Python both made the same choice and things are probably easier to set up in those worlds. Ruby has it's roots in Japan and so it is not surprising that it could not go down that path.

From now on, I'm going to make sure everything I touch is configured for UTF-8. There are fews reasons not to at this stage and it allows you to handle most languages.