Gotchas when working with HTML5 Audio

I've run into a few issues while building a demo web app that plays audio files, so I wanted to post my experiences...

1: Firefox does NOT play MP3 files (as of 2013-04-17)

This is because the MPEG format is proprietary and the Firefox/Mozilla folks insist on only open source formats. You need to use .wav, .ogg or .webm. See https://developer.mozilla.org/en-US/docs/HTML/Supported_media_formats for the allowed options.

Unfortunately they do not seem to produce a useful error message if you try and play an MP3 file.

Google Chrome plays anything - so your application may appear to work fine but may fail on Firefox.

2: Firefox is VERY strict in regard to Cross-Origin Resource Sharing (CORS)

Firefox will refuse to play an audio file on a host other than the one that served the original web page. So you can't store an audio file on, say, Amazon S3, and play it on your web site. The same goes for Ajax requests to another site. The Firefox console will, or may, display an error but this is not necessarily clear.

Google Chrome doesn't seem to care about this restriction.

See this page for more details https://developer.mozilla.org/en-US/docs/HTTP/Access_control_CORS

To work around this you can specify a header for the audio file which means that as the owner of the audio file you allow it to be referenced from any other site.
"Access-Control-Allow-Origin" = "*"

To work around the problem with Ajax requests you can use JSONP instead of JSON as I have described.

The special header ought to work with AWS S3... except....

3: AWS S3 does NOT allow the Access-Control-Allow-Origin header

You can set metadata headers with S3 files but not this one. The Amazon support forums are full of requests/complaints about this but it does not seem to have been resolved.

There does seem to be a way round this on Amazon Cloudfront, which pretty much exists to server assets for other sites, but from what I've seen this is pretty ugly.

My workaround for this was to add a proxy method to my server. The client requests a file that resides on S3 from the proxy. It fetches it from S3 and echoes it to the client, which sees it as coming from the originating server. This introduces an unnecessary delay but works file. Here is a snippet of code in Ruby for a Sinatra server. It uses the aws-sdk gem and leaves out set up for that.

  get '/proxy' do
    s3obj = bucket.objects[params['file']]
    response["Access-Control-Allow-Origin"] = "*"
    content_type 'audio/wav'
    s3obj.read
  end

Apparently Google's equivalent of S3 does not have this issue - I've not tried it.

4: You cannot play audio files from localhost

This makes testing a pain. Your audio files must be hosted on a server other than localhost. I recommend Heroku as a way to get development servers up and running quickly and at no initial cost.

If you don't realize this, then there is no error message in the browser console - it simply doesn't work and you are left scratching your head until you figure it out. I hate stuff like that...

Web Text To Speech using Bing Translate - Demonstration Sinatra App

Given the variety of sophisticated web services available today, it is surprising that a good Text To Speech API (TTS) is not readily available.

Google has one as part of its Translation API but it is not publicized or actively supported. The Bing/Microsoft Translate API also has a TTS feature and this is supported. In my experience this works very well and allows you to specify the language of the text, which changes the voice and pronunciation that is used.

Accessing this API is easy enough using a wrapper library, such as https://github.com/CodeBlock/bing_translator-gem (disclaimer: I added the speak() function to this ruby gem) but it returns binary data that represents an MP3 file. The HTML5 audio tag allows you play audio files, but not, apparently, to play the data itself.

The result is that the TTS output must be first written to a file and then played via the audio tag.

To demonstrate how these different pieces fit together, I have written a TTS demo app that consists of a Javascript in a web page that sends the query text to a Sinatra app, using ajax. The server in turn sends this to Bing and get back the audio. The server then writes this to a file on Amazon S3 and returns the URL for this back to the web page where the audio is played.

The Live Demo for this is at http://bing-translate-tts-demo.craic.com/ and the code required to implement the whole thing is freely available at https://github.com/craic/bing_translate_text_to_speech

The demo has several moving parts and setting it up for yourself requires experience with Sinatra, S3, etc.

Zillow Neighborhood Boundaries in GeoJSON and KML for Seattle

The good people at Zillow have invested a lot of time defining the boundaries of around 7,000 neighborhoods in cities around the USA.

These are in the form of ESRI ARC Shapefiles that can be used as overlays on maps.

You can find these at Zillow Neighborhood Boundaries.

Zillow have kindly made these available to the rest of us under a Creative Commons license. This allows you to share and modify the data but you need to attribute Zillow and must distribute any derivative forms of the data under the same or similar licenses. Attribution should take the form of including their logo with a link back to their site.

ESRI shapefiles can be read by many types of mapping software, but one big exception to this is Google Maps. Overlays in Google Maps are loaded from KML format files. In addition GeoJSON is a JSON-based format that is gaining in popularity.

I needed neighborhood boundaries for the City of Seattle in KML format so I can use them in Google Maps. I couldn't find a convenient way to convert from Shapefile directly to KML (although these do exist in some GIS packages) and the solution I came up with involved converting first from Shapefile to GeoJSON and them from GeoJSON to KML. I'll make that software available elsewhere.

My Github Project zillow_seattle_neighborhoods contains the GeoJSON and KML files for the 78 Seattle neighborhoods defined in Zillow's dataset.

The same approach could be applied to derive GeoJSON and KML files for the entire Zillow US city dataset. I'll see if I can do that in the future... for now it just covers Seattle.

Using the Files

You can use either format of file in many GIS packages and mapping programs. Here are a few ideas for people who are new to this area of coding.

You can load KML files into your own maps using Google Maps directly. Look under 'My Places' -> 'Create Map' -> 'Import'. See Google Maps documentation for details.

You can load the KML files into Google Earth for a different experience. The command 'open .kml' on a Mac with Google Earth installed should be enough.

To use these overlays in custom Google Maps you want to use the Google Maps JavaScript API.

I'll add an example HTML page that shows how to do this in due course...

Two important things to note about testing KML overlays in Google Maps in your own HTML pages

1: Your KML files MUST be hosted on a remote server. They will be ignored if you give try and host them from a server on localhost !

2: KML files are CACHED in web pages, just like images. So if you are changing the files, such as changing the colors, these will not be visible unless you work around the caching issue. The easiest way to do this is to add a question mark followed by a random string at the end of the url for the KML file, and change this on every page reload.

For example: http://yourserver.net/yourfile.kml?1234567

Fixing incorrect timezones in Ruby

Timezones can be tricky to manage and sometimes I just get them wrong - like now...

I had been parsing strings representing local times in Seattle and then storing these without an explicit timezone on a machine which uses UTC by default - so they got stored as that time in UTC (not converted to UTC - the exact time string in UTC.

For example: Fri, 05 Apr 2013 14:24:39 (the local time in Seattle) was stored as Fri, 05 Apr 2013 14:24:39 UTC +00:00 which is 7 hours earlier than it should be.

The easiest way to fix this, that I found is to convert back to a string, strip of any record of a timezone and then parse it back into a time, having temporarily set the TZ environment variable to the correct local timezone.

NOTE that this works on UNIX/Mac OS X systems - not sure about Windows...

puts ENV['TZ']         => nil
puts t                 => 2013-04-05 14:24:39 UTC
s = t.to_s
puts s                 => "2013-04-05 14:24:39 UTC"
s.sub!(/\s+UTC/, '')
puts s                 => "2013-04-05 14:24:39"
ENV['TZ'] = 'US/Pacific'
t1 = Time.parse(s)
ENV['TZ'] = nil
puts t1                => 2013-04-05 14:24:39 -0700

Remember to reset the TZ environment variable back to nil

Be very careful about using gmtime and localtime as these methods operate in place. So ALWAYS make a copy of your time before applying those conversions

t            => 2013-04-05 15:25:41 -0700
t1 = t.dup

t.gmtime     => 2013-04-05 22:25:41 UTC
t            => 2013-04-05 22:25:41 UTC
t1           => 2013-04-05 15:25:41 -0700

Using Latitude and Longitude with Google and Bing Maps

When you work with maps you sometimes want to extract a Latitude/Longitude pair from the map, or you want to enter a pair and see the corresponding map.

Google Maps and Bing Maps do things slightly differently. In general I go with Google but there is not a lot in it. Here are how you work with Latitude/Longitude in both of them:

1: Enter an Address and get the Latitude/Longitude pair

Bing - Enter the address - the Lat/Lon pair will be shown below the Address in the left hand panel

Google - Enter the address - then Right Click the marker and select What's here? - the Lat/Lon pair will appear in the address entry box

2: Browse to a Location and get the Latitude/Longitude pair

Bing - Browse to the location - then Right Click and the Lat/Lon pair will be shown in the popup

Google - Browse to the location - then Right Click the marker and select 'What's Here - the Lat/Lon pair will appear in the address entry box

3: Enter a Latitude/Longitude pair and view the Location

Bing and Google - Enter the Lat/Lon pair, separated by a comma, into the address box (e.g. 47.619905,-122.320844) - Google makes a better choice of zoom level in my opinion.

SODA - Socrata Open Data API

Socrata is a company that helps organizations make their data available to outside users. They work with people like the World Bank and a number of city governments, including the City of Seattle.

You can access these public data sources via SODA - the Socrata Open Data API which provides a SQL-like query language (SoQL) that you submit via HTTP request to an API endpoint for that resource. The default response is JSON but you can get CSV, XML and RDF.

They provide quite a bit of documentation but it is a bit short on examples and I had to poke around a bit to get the right syntax for the data sources that I am interested in.

A very useful resource is the 'live' query console they provide for a test dataset, which lets you try various queries.

I access their resources from Ruby. I generate the URL, use open-uri to fetch the data and then parse the JSON response. Here is an example fo the process:

The data source I want to query list the Seattle Fire Department 911 calls. That page lets you create custom views and download the data. If you click on 'Export' you can see the column names used in the database as well as the url for the API endpoint.

I want to get all records in the past hour, so I need a query along the lines of 'where datetime > my_timestamp'. Specifically I need to build a URL similar to this:

http://data.seattle.gov/resource/kzjm-xkqj.json?$where datetime > my_time_stamp

This is not a real query just yet but note a couple of things...

The API endpoint is http://data.seattle.gov/resource/kzjm-xkqj.json

This version ends in '.json' and so it will return the response in JSON. If you want CSV then replace '.json' with '.csv' - same goes for XML and RDF.

Also note the dollar sign before the 'where' - it is easy to overlook - that signifies the following word is a SoQL clause and not the name of a column.

A big problem I ran into was figuring out the format for timestamps. The web page showing the table formats them as '03/21/2013 07:35:00 AM -0700' whereas the JSON response uses "datetime" : 1363872960. But neither of these works in a query... and the documentation on data types does not describe what you should use. Trial and error with that console page led me to the correct format of '2013-03-21 07:00:00' which is similar to the ISO 8601 format.

Also note that the time needs to be quoted... again, apparently not in the documentation

So my query should now look something like this:

http://data.seattle.gov/resource/kzjm-xkqj.json?$where datetime > '2013-03-21 07:00:00'

But that URL needs to be escaped before submission. That was the other issue - URI.escape() handles the spaces, quotes, etc but for some reason it does not escape the dollar sign. So I had to do that for myself.

Here is a chunk of Ruby code that gets all the 911 calls in the past hour:

require 'open-uri'
require 'time'
require 'json'

endpoint = 'http://data.seattle.gov/resource/kzjm-xkqj.json'
timestamp = (Time.now - (60 * 60)).strftime("%F %H:%M:%S")

query = "$where=datetime > '#{timestamp}'"

url = "#{endpoint}?#{query}"
url = URI.escape(url)
url.gsub!(/\$/, '%24')

puts "query: #{query}
puts "url: #{url}

json = open(url).read
puts json

result = JSON.parse(json)
puts result.length

That will show you the query, the escaped URL, the JSON returned and the number of records in the parsed JSON - enough to get you started. Different data sources will use different column names and perhaps different formats for date/time (always be aware of time zone issues).

Having a common API to many government/public data sources is great but the Socrata-backed sources that I've looked a so far don't seem to be quite as consistent as one would hope. This is an issue for the different data providers - it's not a Socrata issue.

I would hope that one could query the endpoint in some way and get a listing of all the column names and data types. There is one for SODA v1.0 but this now deprecated... so perhaps there is a newer version but it has not been documented.

Part of the answer lies in two custom HTTP response headers (X-SODA2-Fields and X-SODA2-Types) that are returned with your data. Here they are for the data source used here:

X-SODA2-Fields: [":updated_at","address","longitude",":id","latitude","incident_number","datetime","type",":created_at","report_location"]
X-SODA2-Types: ["meta_data","text","number","meta_data","number","text","date","text","meta_data","location"]
X-SODA2-Legacy-Types: true

You can see the headers if you use 'curl -i "your_url"' on the command line with your escaped url.

HTML5 and web API code examples

Whenever I am learning a new feature of HTML5, JavaScript/Jquery or a new API, such as Google Maps, I always look for example code that I can learn from.

In many cases the examples are great, but in others they can be too clever and too heavily styled, such that it can be hard to understand the core of the feature that they are demonstrating.

So in writing my own example code I try and strip things down to the bare minimum - very little styling and code that tries to do one, and only one, thing.

I've been collecting examples that I think have some substance and that can help others learn about a feature with the minimum of confusion.

Take a look at http://html5-examples.craic.com

This site has working examples of a number of HTML5-related features, including Data Attributes, Geolocation, Web Audio and Speech Recognition.

Look at the Source for each of these pages to see annotated JavaScript, etc. which illustrates the target feature of each page.

All the code is distributed freely under the terms of the MIT license and you are encouraged to build your own applications using it.

Note that some of the examples involve new web technologies, so they may not work in some browsers and what does work now may not work in the future as the APIs mature.