A collection of computer systems and programming tips that you may find useful.
Brought to you by Craic Computing LLC, a bioinformatics consulting company.

Wednesday, April 17, 2013

Gotchas when working with HTML5 Audio

I've run into a few issues while building a demo web app that plays audio files, so I wanted to post my experiences...

1: Firefox does NOT play MP3 files (as of 2013-04-17)

This is because the MPEG format is proprietary and the Firefox/Mozilla folks insist on only open source formats. You need to use .wav, .ogg or .webm. See https://developer.mozilla.org/en-US/docs/HTML/Supported_media_formats for the allowed options.

Unfortunately they do not seem to produce a useful error message if you try and play an MP3 file.

Google Chrome plays anything - so your application may appear to work fine but may fail on Firefox.

2: Firefox is VERY strict in regard to Cross-Origin Resource Sharing (CORS)

Firefox will refuse to play an audio file on a host other than the one that served the original web page. So you can't store an audio file on, say, Amazon S3, and play it on your web site. The same goes for Ajax requests to another site. The Firefox console will, or may, display an error but this is not necessarily clear.

Google Chrome doesn't seem to care about this restriction.

See this page for more details https://developer.mozilla.org/en-US/docs/HTTP/Access_control_CORS

To work around this you can specify a header for the audio file which means that as the owner of the audio file you allow it to be referenced from any other site.
"Access-Control-Allow-Origin" = "*"

To work around the problem with Ajax requests you can use JSONP instead of JSON as I have described.

The special header ought to work with AWS S3... except....

3: AWS S3 does NOT allow the Access-Control-Allow-Origin header

You can set metadata headers with S3 files but not this one. The Amazon support forums are full of requests/complaints about this but it does not seem to have been resolved.

There does seem to be a way round this on Amazon Cloudfront, which pretty much exists to server assets for other sites, but from what I've seen this is pretty ugly.

My workaround for this was to add a proxy method to my server. The client requests a file that resides on S3 from the proxy. It fetches it from S3 and echoes it to the client, which sees it as coming from the originating server. This introduces an unnecessary delay but works file. Here is a snippet of code in Ruby for a Sinatra server. It uses the aws-sdk gem and leaves out set up for that.

  get '/proxy' do
    s3obj = bucket.objects[params['file']]
    response["Access-Control-Allow-Origin"] = "*"
    content_type 'audio/wav'

Apparently Google's equivalent of S3 does not have this issue - I've not tried it.

4: You cannot play audio files from localhost

This makes testing a pain. Your audio files must be hosted on a server other than localhost. I recommend Heroku as a way to get development servers up and running quickly and at no initial cost.

If you don't realize this, then there is no error message in the browser console - it simply doesn't work and you are left scratching your head until you figure it out. I hate stuff like that...

Tuesday, April 16, 2013

Web Text To Speech using Bing Translate - Demonstration Sinatra App

Given the variety of sophisticated web services available today, it is surprising that a good Text To Speech API (TTS) is not readily available.

Google has one as part of its Translation API but it is not publicized or actively supported. The Bing/Microsoft Translate API also has a TTS feature and this is supported. In my experience this works very well and allows you to specify the language of the text, which changes the voice and pronunciation that is used.

Accessing this API is easy enough using a wrapper library, such as https://github.com/CodeBlock/bing_translator-gem (disclaimer: I added the speak() function to this ruby gem) but it returns binary data that represents an MP3 file. The HTML5 audio tag allows you play audio files, but not, apparently, to play the data itself.

The result is that the TTS output must be first written to a file and then played via the audio tag.

To demonstrate how these different pieces fit together, I have written a TTS demo app that consists of a Javascript in a web page that sends the query text to a Sinatra app, using ajax. The server in turn sends this to Bing and get back the audio. The server then writes this to a file on Amazon S3 and returns the URL for this back to the web page where the audio is played.

The Live Demo for this is at http://bing-translate-tts-demo.craic.com/ and the code required to implement the whole thing is freely available at https://github.com/craic/bing_translate_text_to_speech

The demo has several moving parts and setting it up for yourself requires experience with Sinatra, S3, etc.

Friday, April 12, 2013

Zillow Neighborhood Boundaries in GeoJSON and KML for Seattle

The good people at Zillow have invested a lot of time defining the boundaries of around 7,000 neighborhoods in cities around the USA.
These are in the form of ESRI ARC Shapefiles that can be used as overlays on maps.
You can find these at Zillow Neighborhood Boundaries.
Zillow have kindly made these available to the rest of us under a Creative Commons license. This allows you to share and modify the data but you need to attribute Zillow and must distribute any derivative forms of the data under the same or similar licenses. Attribution should take the form of including their logo with a link back to their site.
ESRI shapefiles can be read by many types of mapping software, but one big exception to this is Google Maps. Overlays in Google Maps are loaded from KML format files. In addition GeoJSON is a JSON-based format that is gaining in popularity.
I needed neighborhood boundaries for the City of Seattle in KML format so I can use them in Google Maps. I couldn't find a convenient way to convert from Shapefile directly to KML (although these do exist in some GIS packages) and the solution I came up with involved converting first from Shapefile to GeoJSON and them from GeoJSON to KML. I'll make that software available elsewhere.
My Github Project zillow_seattle_neighborhoods contains the GeoJSON and KML files for the 78 Seattle neighborhoods defined in Zillow's dataset.
The same approach could be applied to derive GeoJSON and KML files for the entire Zillow US city dataset. I'll see if I can do that in the future... for now it just covers Seattle.

Using the Files
You can use either format of file in many GIS packages and mapping programs. Here are a few ideas for people who are new to this area of coding.
You can load KML files into your own maps using Google Maps directly. Look under 'My Places' -> 'Create Map' -> 'Import'. See Google Maps documentation for details.
You can load the KML files into Google Earth for a different experience. The command 'open .kml' on a Mac with Google Earth installed should be enough.
To use these overlays in custom Google Maps you want to use the Google Maps JavaScript API.
I'll add an example HTML page that shows how to do this in due course...
Two important things to note about testing KML overlays in Google Maps in your own HTML pages
1: Your KML files MUST be hosted on a remote server. They will be ignored if you give try and host them from a server on localhost !
2: KML files are CACHED in web pages, just like images. So if you are changing the files, such as changing the colors, these will not be visible unless you work around the caching issue. The easiest way to do this is to add a question mark followed by a random string at the end of the url for the KML file, and change this on every page reload.

Friday, April 5, 2013

Fixing incorrect timezones in Ruby

Timezones can be tricky to manage and sometimes I just get them wrong - like now...

I had been parsing strings representing local times in Seattle and then storing these without an explicit timezone on a machine which uses UTC by default - so they got stored as that time in UTC (not converted to UTC - the exact time string in UTC.

For example: Fri, 05 Apr 2013 14:24:39 (the local time in Seattle) was stored as Fri, 05 Apr 2013 14:24:39 UTC +00:00 which is 7 hours earlier than it should be.

The easiest way to fix this, that I found is to convert back to a string, strip of any record of a timezone and then parse it back into a time, having temporarily set the TZ environment variable to the correct local timezone.

NOTE that this works on UNIX/Mac OS X systems - not sure about Windows...

puts ENV['TZ']         => nil
puts t                 => 2013-04-05 14:24:39 UTC
s = t.to_s
puts s                 => "2013-04-05 14:24:39 UTC"
s.sub!(/\s+UTC/, '')
puts s                 => "2013-04-05 14:24:39"
ENV['TZ'] = 'US/Pacific'
t1 = Time.parse(s)
ENV['TZ'] = nil
puts t1                => 2013-04-05 14:24:39 -0700

Remember to reset the TZ environment variable back to nil

Be very careful about using gmtime and localtime as these methods operate in place. So ALWAYS make a copy of your time before applying those conversions

t            => 2013-04-05 15:25:41 -0700
t1 = t.dup

t.gmtime     => 2013-04-05 22:25:41 UTC
t            => 2013-04-05 22:25:41 UTC
t1           => 2013-04-05 15:25:41 -0700

Wednesday, April 3, 2013

Using Latitude and Longitude with Google and Bing Maps

When you work with maps you sometimes want to extract a Latitude/Longitude pair from the map, or you want to enter a pair and see the corresponding map.

Google Maps and Bing Maps do things slightly differently. In general I go with Google but there is not a lot in it. Here are how you work with Latitude/Longitude in both of them:

1: Enter an Address and get the Latitude/Longitude pair

Bing - Enter the address - the Lat/Lon pair will be shown below the Address in the left hand panel

Google - Enter the address - then Right Click the marker and select What's here? - the Lat/Lon pair will appear in the address entry box

2: Browse to a Location and get the Latitude/Longitude pair

Bing - Browse to the location - then Right Click and the Lat/Lon pair will be shown in the popup

Google - Browse to the location - then Right Click the marker and select 'What's Here - the Lat/Lon pair will appear in the address entry box

3: Enter a Latitude/Longitude pair and view the Location

Bing and Google - Enter the Lat/Lon pair, separated by a comma, into the address box (e.g. 47.619905,-122.320844) - Google makes a better choice of zoom level in my opinion.

Archive of Tips