Mongrel Cluster on Mac OS X

Mongrel is a HTTP server that is well suited for serving Rails web applications, often in conjunction with Apache as a 'front end' server.

Mongrel works fine as is for development applications but if you have any real number of users its performance will degrade rapidly. In this case you want to use a cluster of mongrel servers, each running on a different port and have Apache balance the load between all of them

The mongrel_cluster gem is a convenient way to set up and manage multiple mongrels. HOWEVER I do not recommend this on Mac OS X. Because of the way Mac OS X handles processes at startup and/or because I couldn't figure it out, I was not able to get mongrel_cluster to work correctly at startup on my system. Some of the guides on the web that claimed to do this did not work for me. But don't worry... mongrel_cluster is really a simple utility and you can do without it at the price of a little more configuration. That is what I will show here.

These instructions relate to Mac OS X 10.4 and they assume that you already have Apache, Mongrel and your Rails application up and running.

1. Make sure that your Rails app is using Active Record to store Session data in the database, as opposed to storing it in files, which is the default. Instructions for that can be found HERE.

2. Setup a launchd plist file for each mongrel instance that you want to set up.
The preferred way to start programs automatically under Mac OS X is through launchd instead of the traditional init process in other Unix variants. launchd takes a bit of getting used to but you should use it (and don't try and mimic init scripts using StartupItems...)

You can learn about launchd is this Apple developer note and by doing a man launchd and man launchd.plist

3. In this example I am going to setup up 4 instances of mongrel on ports 8000, 8001, 8002 and 8003

In /Library/LaunchDaemons create the file net.mongrel80000.plist with contents similar to this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/Prop
ertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>net.mongrel8000</string>
  <key>ProgramArguments</key>
  <array>
          <string>/usr/local/bin/mongrel_rails</string>
          <string>start</string>
          <string>-e</string>
          <string>development</string>
          <string>-p</string>
          <string>8000</string>
          <string>-c</string>
          <string>/Users/jones/myapp</string>
          <string>-P</string>
          <string>/Users/jones/myapp/tmp/pids/mongrel.8000.pid</string>
          <string>-l</string>
          <string>/Users/jones/myapp/log/mongrel.8000.log</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
  <key>ServiceDescription</key>
  <string>Mongrel Rails Application Server</string>
</dict>
</plist>

This ugly block of XML breaks down into the following components:
Label
This is a UNIQUE label for this launch item. In our case make this the same as the name of the file, less the '.plist' extension (net.mongrel8000)
ProgramArguments
An array of strings which, when joined together, create the command that you would run to start this instance of mongrel. Change the paths, etc to suit your application. Be VERY careful to set the port numbers to the one used in this file (8000 in this case)
RunAtLoad
This is set to true and tells launchd to run this item once when the system starts up.
ServiceDescription
An optional string that describes what this item represents.

chown/chmod this to have these permissions and ownership:

-rw-r--r--   1 root  wheel  836 Feb 19 14:53 net.mongrel8000.plist

4. Clone and modify this file for each mongrel instance
In my case I copied this into net.mongrel8001.plist, etc. and changed each instance of the port number 8000 to 8001, 8002, or 8003 as appropriate. These are marked in red in the above XML. Make absolutely sure the Label is correct and unique otherwise it won't work.

5. Test it out
Clean out any editor backup files in /Library/LaunchDaemons, check your file permissions and restart your machine. If things are setup correctly, when it restarts it will have started four instances of Mongrel that will drive your application.

Test these by using a browser on that machine and going to each port in turn, in other words these four URLs should all work:

http://localhost:8000
http://localhost:8001
http://localhost:8002
http://localhost:8003

Note that in one of the guides on the web that I saw they used a single .plist file and put the command strings for all the mongrel instances in a single ProgramArguments section. This did not work for me at all...

If my instructions don't work for you then double check the format of the XML file and your paths.

6. Configure Apache to direct requests to the Mongrel servers
Your Apache installation needs to have the mod_proxy_balancer module installed. 'httpd -l' will list the compiled in modules and hopefully you will find it there. You can find out how to compile in modules in Apache 2.2 and read my grumbles about that HERE.

Edit your Apache httpd.conf file to create a virtual host that it will respond to and a set of proxy and load balancing instructions. You can do this in various ways and get a lot more complex than this, but here is a single block that you can put at the bottom of httpd.conf.

<VirtualHost *:80>
ServerName myserver.craic.com

# Enable URL rewriting
RewriteEngine On

# Rewrite index to check for static pages
RewriteRule ^/$ /index.html [QSA]

# Rewrite to check for Rails cached page
RewriteRule ^([^.]+)$ $1.html [QSA]

# Redirect all non-static requests to cluster
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ balancer://mongrel_cluster%{REQUEST_URI} [P,QSA,L]

# You could also add a bunch of deflate rules etc here

# This is the path to static content as opposed to your Rails app
DocumentRoot  "/Users/jones/html"
<Directory "/Users/jones/html">
     Options Indexes FollowSymLinks

     AllowOverride None
     Order allow,deny
     Allow from all
 </Directory>

</VirtualHost>

# This block tells the load balancer to pass requests
# onto these four mongrels

<Proxy balancer://mongrel_cluster>
  BalancerMember http://127.0.0.1:8000
  BalancerMember http://127.0.0.1:8001
  BalancerMember http://127.0.0.1:8002
  BalancerMember http://127.0.0.1:8003
</Proxy>

All the Rewrite lines tell Apache how to handle requests for static versus dynamic (from Rails) content. The important lines are the two that rewrite requests for non-static content into ones with a prefix of balancer://mongrel_cluster. These are passed to the <Proxy> block at the end where the load balancer distributes these among the members listed here. These members are the four mongrel instances that we set up earlier. Apache doesn't care that these are mongrel servers. It just sees them as URLs and efficiently passes on the requests.

Your Apache is most likely started by a launchd plist file. Assuming it starts automatically then any reboot of your server will start it and the mongrel instances.

Giving a URL like http://myserver.craic.com/myapp will now get forwarded to one of the mongrels.


There you have it... If you read this and have a better mongrel launchd configuration please let me know. There has to be a cleaner way than what I have here...

Using Active Record for Session Storage in Rails

This setup appears in all sorts of pages but I'm including the basic steps here for my own benefit - and hopefully yours...

The default way that Rails (1.2.3) stores session data is in files in your application tmp directory. This mechanism is referred to as CGI::Session::PStore. This is fine for development but becomes a problem as you move to a real production environment.

One problem with it is that unless you actively clean out old files with a cron job and 'rm' you can end up with a massive number of old session files.

It is also a problem if you use Apache and Mongrel to serve your application and want to scale things up with mongrel_cluster. Various resources warn of bad things happening with multiple mongrels and session files.

The next step up from files is to use a database table and have Active Record store session data in that. This is easy to setup.

1: Create a migration to set up the table and run that

$ rake db:sessions:create
    exists  db/migrate
    create  db/migrate/027_add_sessions.rb
$ rake db:migrate
== AddSessions: migrating =====================================================
-- create_table(:sessions)
 -> 0.4298s
-- add_index(:sessions, :session_id)
 -> 0.2914s
-- add_index(:sessions, :updated_at)
 -> 0.0727s
== AddSessions: migrated (0.8001s) ============================================

2: Edit your app's config/environment.rb file and uncomment this line
config.action_controller.session_store = :active_record_store

3: Start your app server, interact with it and look in mysql
mysql> select * from sessions;

You will see a hexadecimal encoded session_id and a big block of encoded session data. Everything else should just work normally.

4: To avoid sessions accumulating in applications where you have users login and logout, you can call reset_session in the logout action - something like this:

  def logout
  reset_session
  flash[:notice] = "Logged out"
  redirect_to :action => "index"
end

This clears out your current session row in the table and initializes a new session object. I'm not sure why it does the latter.

Multihomed Ethernet Interfaces on Mac OS X server 10.4

I had a Mac OS X 10.4 server with two ethernet interfaces configured so that eth0 connected to my internal network and eth1 connected directly to the Internet.

I could access either interface just fine from my internal network. But unbeknownst to
me, the outside world was not able to access the external interface.

Checking the firewall or even turning the firewall off completely had no effect. The output of ifconfig looked like it should, the cables were plugged in where they ought to be. You just couldn't get to the external interface from the Internet...

It turns out that Mac OS X expects eth0 to connect to the Internet and I had it connected to my internal network. The server could see the Internet via that interface. Because it passed through my restrictive firewall, the outside world could not see it but from my subnet I could see it just fine... a nasty little gotcha.

I swapped the network configuration parameters for that eth0 was the external interface and eth1 was internal and the problem was solved.

Look at the Network control panel under 'Network Status'. Both interfaces will be active in a configuration like mine but only one will state 'You are connected to the Internet via...'. That needs to be the external interface.

There must be some way to configure which interface connects to the Internet but it is easier to follow their convention and not worry about it.

Ortho - JavaScript Graphics Library

I've written a JavaScript library called Ortho (http://www.craic.com/ortho) on top of Prototype for creating 'diagram-style' graphics in JavaScript. You can create histograms, graphs, timeline plots, 'maps' of genomic data, annotated images, tree diagrams, etc.

Unlike Canvas, it seamlessly integrates text with graphics and the output looks the same across browsers and in *print*. Unlike Flash it does not require third party software.

It uses associated CSS styles to draw rectangles (divs with a border) and horizontal or vertical lines (divs with a border on one side). A bit of a hack? You might say that, but it turns out to be very effective for the sort of graphics that I need to create on the fly.

It cannot draw curved lines or arbitrary shapes - hence the name 'ortho' for orthogonal. But for a range of applications it may offer a simple solution for creating sophisticated graphics.

It is built on top of the wonderful Prototype library. As a result it is very amenable to being extended with Prototype and Scriptaculous.

Ortho is released under an MIT-style license.

The initial release only covers 'static' graphics but functions for user interaction and Ajax are under development.

The Ortho project site (http://www.craic.com/ortho) has a number of examples that show you what you can do with the library.

Running a Rails Applications in a Subdirectory

Part 1: Running a Rails Applications in a Subdirectory

I'm running a Rails application and a separate static web site on the same host.

I want to refer to the static content with a url like http://myserver/public/index.html etc.

I want to refer to the Rails app with a url like http://myserver/db

By default Rails expects an application to reside at the top level of a site, i.e. http://myserver, so I need a way to change this default path. There are two simple steps involved:

1: In config/environment.rb add a line like this at the bottom of the file:

ActionController::AbstractRequest.relative_url_root = "/db"

...where '/db' is the subdirectory where you want your application to appear

This will add that prefix to every URL generated in your Rails app pages and will cause the Rails dispatcher to respond to that path.

2: Adding that prefix will cause your links to stylesheets and javascripts to break. Fix that by adding a symbolic link in your Rails app /public directory

% ln -s . db

... where 'db' is the prefix with no leading slash

Restart your server and try it out.

Part 2: Serving Static Content from Apache and your Rails app from Mongrel

Currently (late 2007) the preferred Web server solution for Rails applications is to use Apache2 as the front end and Mongrel as the Rails application server. All requests come into Apache. Those for a static web site are handled directly, as Apache is great at that job. Requests for the Rails app are proxied to an instance of the Mongrel web server running on a different port. You can find details of this configuration here:
http://mongrel.rubyforge.org/docs/apache.html

These are the steps needed to set up this configuration.

1: Set up the Rails app under a subdirectory as shown in Part 1.

2: Run Mongrel on a port other than 80 as described in the Apache link above. For example, you might run it on port 8000 using this command in your app directory:

% mongrel_rails start -d -p 8000 -e production -P /my/path/railsapp/log/mongrel.pid

3: Configure Apache2 to proxy Rails requests to Mongrel using a VirtualHost block similar to this:

<VirtualHost>
 ServerName yourserver.com
 ProxyPass /db http://localhost:8000/db
 ProxyPassReverse /db http://localhost:8000/db
 ProxyPreserveHost on

 DocumentRoot  /yourpath/static/html
 <Directory>
    Options Indexes FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
 </Directory>

</VirtualHost>

The VirtualHost blocks sets up a host on the standard port 80.

The ProxyPass directives tell Apache to pass any requests with URLs that start with '/db' over to the web server running on the same machine (localhost) at port 8000. It also modifies the responses from that server so that they appear to come from Apache instead of Mongrel. So from the user's perspective the Mongrel server is invisible.

Any other requests are handled by Apache directly and will fetch static content from the directory '/yourpath/static/html', which is where you would put your static web site.

For this to work, your Apache server must load the mod_proxy module, and probably some others. See the Mongrel/Apache link given above for details on that.

Look at the various Apache Proxy options if you need finer control over what gets proxied and what gets served directly. You can set up multiple different Rails apps on the same host in this way and/or multiple Mongrel instances that can serve an application under high load. Again, see the Mongrel/Apache link for details.

Installing Instiki on Mac OS X

Instiki is a Wiki clone written in Ruby and Rails. In the spectrum of wiki software, it is relatively simple with a correspondingly clean interface, unlike the default mediawiki pages. It is not perfect but if you need to get a wiki up and running quickly it may be just what you need.

The instiki pages suggest that installation is trivial, but that is a little misleading. Here are the steps needed to get it up and running on a Mac OS X 10.4.

It assumes that you have Ruby and MySQL already installed and that you have at least a passing familiarity with Rails applications.

1: Download the instiki package from: http://rubyforge.org/frs/?group_id=186&release_id=10014
The version current at the time of writing is instiki-0.11.pl1.tgz

2: Unpack the .tgz file
Use the default 'Archive Utility' or use tar on the command line. Be aware of this weird gotcha that I encountered with StuffitExpander.

3: Create the database in MySQL
The database name is not critical.
# mysqladmin -u root -p create instiki_production

4: Edit config/database.yml in the instiki directory
Replace the existing 'production' database definition with this:
production:
adapter: mysql
database: instiki_production
username: root
password: xxxxx
host: localhost
# socket: /path/to/your/mysql.sock

Add your own password, and don't worry about the socket line unless you are on Linux.

5: Install the database tables
This is a Rails command that creates all the right tables and sets up some other configuration settings:
# rake environment RAILS_ENV=production migrate

6: Start the Instiki Server

# ./instiki

This will fire up the Webrick web server built into Rails and bind the Instiki to port 2500 on the local machine. Point your browser (on that machine) to this URL:
http://localhost:2500

You should see a setup web page.

7: Configure the Wiki

In the setup page, enter the name of the wiki and the address (by which they mean the subdirectory on this site that will form the root of the wiki), and enter an administrator password.

It will return a home page with a text area into which you can enter the content of the page. Look at the hints on that page and on the Instiki project site for help on the formatting shortcuts.

You will notice the absence of any button or link that adds a new page. The way you do this in Instiki is to enter the name for that page in the parent page and surround it in double square brackets, like this:
[[Another Page]]
When you submit this page, you will see the text 'Another Page' with
a question mark next to it. Click on that to add content to the new page. That can be a little confusing until you get used to it.

With the Home Page in place, do some other setup by clicking 'Edit Web' on that page. Here you can fine tune some of the default styling, you can setup password protection for the entire site and, importantly, you can configure the site to publish a read-only version of itself in parallel to the one that you are editing. This feature can be very useful if you want to block changes to a public site or if you want to use Instiki as a simple Content Management System for a static web site.

If you opt for publishing the read-only version you can then access the two versions from similar but distinct URLs.

The URL for the regular home page looks like this (for a wiki called 'mysite')
http://localhost:2500/mysite/show/HomePage

The published (read-only) version can be found at
http://localhost:2500/craic/published/HomePage

Note you will get Rails errors if you try to access
http://localhost:2500/craic/published
or
http://localhost:2500/craic/published/show/HomePage

That had me confused for quite a while when I was testing if my installation was working.


8: Make your instiki available on the Internet

The default configuration for instiki runs on localhost at port 2500. If you want to run it as the sole public web server on your host you could change those two parameters in the installation (file script/server) or when you start instiki.
# ./instiki -b 192.168.0.0 -p 80

But a more likely scenario is that you want to add the wiki to an existing web site. Running instiki directly on a web server other than Webrick is non-trivial so your best bet is to configure your regular web server to proxy wiki requests that it receives to the instiki server.

This fairly straightforward but does involve some server configuration directives.

In this example, the main web site is called 'mysite.com' and the wiki name is 'mysite'. I want the wiki to accessed by urls like http://mysite.com/subdir/HomePage

I modified the following from this page at instiki.org
http://www.instiki.org/show/HowToUseInstikiAsWebSite

If your main web server is Apache then you want a configuration something like this, using the ProxyPass directives:
<virtualhost>
ServerName mysite.com
ServerAlias www.mysite.com
ProxyPass /subdir/ http://localhost:2500/mysite/published/
ProxyPassReverse /subdir/ http://localhost:2500/mysite/published/
</virtualhost>

If you are using lighttpd as your server, which is common in the Rails world, then things are a little cryptic and look like this:

$HTTP["host"] =~ "(^|\.)mysite.com$" {
 server.document-root = basedir + "/web/mysite.com/html/"

 # Rewrite the URL *before* entering the HTTP["url"] block
 url.rewrite_once = ("^/subdir/(.*)$" => "/mysite/published/$1" )

   # pass any /mysite/ urls to instiki on port 2500
   $HTTP["url"] =~ "^/mysite/published/\S+" {
   proxy.server = ( "" => (
                  ( "host" => "127.0.0.1", "port" => 2500),
                    ),  )
 }
}

What this does is to rewrite any input URL that include /subdir to point to /mysite/published and then to pass those on to the instiki web server.

Note that your must have mod_proxy added to your module list for this to work.
server.modules = ("mod_rewrite", "mod_fastcgi", "mod_accesslog", "mod_proxy")
Also note that you need to specify the proxy host as an IP address, not a hostname.

Restart the server and then http://mysite.com/subdir/HomePage will be passed to the instiki server as http://localhost:2500/mysite/published/HomePage

Be careful how you set up the regular expressions. Make sure that you can't mess with the URL and get the editable pages by mistake (unless you want to allow access to those). Also be aware that instiki will give cryptic Rails error dumps if you enter invalid or truncated URLs and that may confuse your users.

9: Set up instiki to run automatically
On Mac OS X you need to create a file under /Library/LaunchDaemons in Apple's plist format.
This should look something like this (here the instiki startup script is located at /Users/mysite/instiki/instiki).

<plist version="1.0">
<dict>
<key>Label</key>
<string>net.instiki</string>
 <key>OnDemand</key>
 <false>
 <key>Program</key>
 <string>/Users/mysite/instiki/instiki</string>
 <key>ProgramArguments</key>
 <array>
   <string>/Users/mysite/instiki/instiki</string>
   <string>--daemon</string>
 </array>
</dict>
</plist>>

Call your file net.instiki.plist and then reboot your machine. Assuming your web server starts up in a similar fashion, you should be able to go to your main site URL and then to the wiki link, whereupon you'll see the published version of the wiki.

10: Outstanding issues
From this point you are on your own in terms of creating your content, pages and styles. Refer to the instiki.org site for help with that.

Currently the WEBrick web server is hardwired into the instiki code. This works fine but under heavy load this would not be acceptable. Being able to replace it with lighttpd or mongrel, like you do with regular Rails applications, would solve the problem but for now you'd have to hack instiki to get this.

Thanks and acknowledgements

Instiki was created by David Heinemeier Hansson and further developed by Alexey Verkhovsky, Matthias Tarasiewicz and Michal Wlodkowski. I thank them all for their work!

Serious Gotcha with Mac OS X / Stuffit Expander / Instiki .tgz file

After a lot of screwing around I've figured out the reason why my installation of the Instiki software was failing on my installation of Mac OS X.

Hopefully this is some very esoteric combination of factors but I want to put the story out there in case it helps someone else.

Background:
I want to install the Wiki software 'instiki' on my Mac (OS X 10.4.9). I downloaded this version:
instiki-0.11.pl1.tgz
from rubyforge
http://rubyforge.org/frs/?group_id=186&release_id=10014

I downloaded it using the Camino web browser v1.5

I copied it from the downloads folder to the target folder in the Finder.
I double-clicked the .tgz file to unpack the archive. Normally the Mac OS X Archive Utility takes care of that. In this case Stuffit Expander popped up and did the job (version 8.0.2).

To cut a long story short, for some reason Stuffit Expander made two copies of certain files (not all of them). For example app/models/web.rb appeared as web.rb and web.1.rb. The real problem was that web.rb was empty (0 bytes) whereas the real content was in web.1.rb. Instiki doesn't know anything about the .1.rb files and only sees the empty versions when you fire it up. Not surprisingly it craps out with a whole slew of odd messages.

The solution for me was to either unpack the archive manually
# tar xzvf instiki-0.11.pl1.tgz
or to remove the Stuffit application. Once you've done that then the default Archive Utility should handle the unpacking and the problem will go away.

Why Stuffit should do this I don't know... very, very strange behaviour and a real pain to troubleshoot...