Setting up the Starling Messaging Server

Starling is a simple messaging server for distributed queuing. It was written by Blaine Cook and colleagues at Twitter and is used extensively by them. It has been released as open source and is becoming widely used. It is written entirely in Ruby and uses the memcache protocol for communication.

I am looking at it as a simple way to manage distributed tasks on my local network. The basic idea is similar to other queuing systems like Amazon's SQS. It has benefits and disadvantages compared to other systems. The benefits for me are that it is simple to set up and use.

The basic idea is that you run a Starling server on one machine, listening on a specific address and port. Other machines can set or get messages from an arbitrary number of queues on the server. It should be used on a trusted network as it has no security.

Fetch the latest version as a rubygem from github at http://github.com/starling/starling/tree/master. You will also need the fiveruns-memcache-client (at least as of Jan 2009)

$ gem sources -a http://gems.github.com/
$ sudo gem install fiveruns-memcache-client
$ sudo gem install starling-starling

This should also install the eventmachine gem for you if needed.
NOTE I have not been able to install the eventmachine gem on Fedora 7 because of some problem finding -lruby. I think it has to do with having lib and lib64. Anyone got a fix for that?

The gem has two components - a starling executable that implements the server and the ruby library that you use to interact with it.

You need to copy the executable out of the gem directory. The default gem directory on Ubuntu is /var/lib/gems/1.8/gems and on (my) Mac OS X it is /usr/local/lib/ruby/gems/1.8/gems/. The current version is installed under starling-starling-0.9.9

I'm just going to cover the Ubuntu installation here:

$ sudo cp /usr/local/lib/ruby/gems/1.8/gems/starling-starling-0.9.9/bin/starling /usr/local/bin/starling
$ sudo cp /usr/local/lib/ruby/gems/1.8/gems/starling-starling-0.9.9/bin/starling_top /usr/local/bin/starling_top

I had to edit the first line of both of these Ruby scripts to point to where my Ruby interpreter was installed (#!/usr/bin/env ruby)

Test out the installation by running the server as root
# starling
This will start up a server on the default address of 127.0.0.1 and port 22122

To have the server run as system start:
1: Copy the example init script from the gem etc directory.
$ sudo cp /usr/local/lib/ruby/gems/1.8/gems/starling-starling-0.9.9/etc/starling.ubuntu /etc/init.d

2: Edit the file to change the user and group to root. This is not ideal. You should create a starling user and group but I have not been able to get this to work - probably just a permissions issue - any fixes to this are welcomed. You also need to change the IP address on which to run the service from the default of 0.0.0.0
DUSER=root
DGROUP=root
LISTEN=192.168.1.1

3: Create the log directory /var/log/starling
Look at the contents of /var/log/starling/starling.log if you have problems.

4: Create the spool directory /var/spool/starling that starling uses to back up its queues. Each queue will get a separate binary file. Poke around in the contents of these using 'od -c ' if you are so inclined. (Starling may create this automatically - but it does NOT create the log directory)

5: Start/stop the server
# /etc/init.d/starling start

6: Set up the appropriate symlinks in the /etc/rc* directories. On Fedora I would use chkconfig for this but it barfs on Ubuntu for me. Setting them manually I created K08starling in rc0.d, rc1.d and rc6.d and S92starling in rc3.d, rc4.d, rc5.d and rc6.d

To use Starling to manage messages you will use the Ruby client library. The interface is refreshingly simple. Test it out using irb:
$ irb --simple-prompt
>> require 'rubygems'
=> true
>> require 'starling'
=> true
>> star = Starling.new('192.168.1.1:22122')
=> <MemCache: 1 servers, 1 buckets, ns: nil, ro: false>
>> star.available_queues()
=> []
>> star.set('myqueue', 'Hello World')
=> nil
>> star.available_queues()
=> ["myqueue"]
>> star.sizeof('myqueue')
=> 1
>> msg = star.get('myqueue')
=> "Hello World"


Note that the Starling.new call makes a connection to the server which should already be running. It does not itself start a server. Also note that a call to get() when there are no messages in the queue will just hang waiting for the next message. So check first with sizeof(queue) if you don't want that to happen.

There are no hard and fast limits as to how large messages can be, unlike Amazon SQS. But I would imagine that very large messages are going screw up your performance. Remember this is using a memcache-like architecture and so memory can become limiting. Starling is designed to handled lots of small messages so it's probably best to use it in this way.

Starling is written entirely in Ruby. The documentation for the gem is reasonable. Look at the code in the gem if you need to really understand what is going on.

To use Starling within Rails applications you should look at the Workling plugin