Ensuring a DSL Connection Stays Up

I've got DSL service through Qwest, with an Actiontec DSL modem. Every so often the connection will just hang for no apparent reason. Qwest have not been able to help. I suspect the problem is due to a flaky modem or some accumulation of 'state' in the device that causes it to hang when it reaches a certain point. Power cycling the modem fixes it every time.

Most of the time the connection is fine but I can almost guarantee that it will hang if I go away for a few days. As I run my own web site and mail server it is crucial that the connection stays up.

The fix that I've arrived at is very low tech. Simply plug the modem into a cheap programmable timer and set that to go off for one minute everyday sometime in the middle of the night. That way the modem gets reset once a day whether it needs it or not.

Most importantly it will reset when I'm not here. So even it hangs during the day it will be back within 24 hours.

Since I've been using this I've not had a problem... famous last words...

My timer is an Intermatic with an LCD display, but any reasonable timer from Home Depot, etc. will do the job. It cost less than $20.

Ruby unable to find installed Gems? You may have 2 installations of ruby

I've been bitten by this before but it has been long enough that I forgot about the issue until it tripped me up again just now.

You've installed a gem, then go to use it and your Ruby script complains it can't find it. It does so with a cryptic message along the lines of "in 'require__': no such file to load -- postgres (LoadError)" In this example it can't find the postgres gem, which I know is installed and which 'gem list --local' tells me is installed.

Why can't it see it?

It may be that you have two versions of Ruby installed on your system. Your OS may place one in /usr/bin and you may have installed a more recent version in /usr/local/bin. That's what I have on one of my machines.

The thing is that each version has its own directory of gems. If you can't see a gem that you know is installed then you might be using the wrong installation of Ruby!

This was my problem. I have /usr/local/bin at the front of my path so I automatically pick up that installation of ruby and gem on the command line.

I had mistakenly put #!/usr/bin/ruby at the top of a script. That led the script to look for 'require'd gems in the wrong gem directory, and hence it did not find it.

You'll see some folks use "#!/usr/bin/env ruby" as the first line of their scripts. This should find the right one as it uses your PATH to find the first instance of ruby.

It's a nasty gotcha and it might explain quite a few pleas for help on Ruby mailing lists.

Ruby, ActiveRecord and Postgresql

Here are some tips on getting Postgresql and Ruby working together. This has not been as simple as I had hoped but after some experimentation, here are the steps that worked for me...

My environment:
RedHat Linux (Enterprise Linux ES Release 4 - Nahant Update 3)
Ruby 1.8.6
Postgres 8.3 (?)

1: Downloaded Postgresql as source and compiled as directed, installed into /usr/local/pgsql
NOTE that I had to be root to get make to successfully compile the code - don't know why

2: Install the Ruby postgres Gem
You will see references to the postgres, ruby-postgres and postgres-rb gems. As far as I can tell, the one you want is 'postgres'
As root run:
# gem install postgres
This may very well blow up with error messages! It did with me. For whatever reason it does not know where to find the Postgresql include and lib directories.
Look at the error message and cd to the gem install directory. For me this was:
/usr/local/lib/ruby/gems/1.8/gems/postgres-0.7.9.2008.01.28/ext
Install the gem manually using these commands:
# ruby extconf.rb --with-pgsql-include=/usr/local/pgsql/include --with-pgsql-lib=/usr/local/pgsql/lib
# make
# gem install postgres -- --with-pgsql-include-dir=/usr/local/pgsql/include --with-pgsql-lib-dir=/usr/local/pgsql/lib
NOTE that the options are different for the ruby extconf.rb and gem install commands! They look very similar but the gem install options have a '-dir' suffix.

3: Create a test database with 'psql'
You will want to follow a basic tutorial on Postgresql and the client 'psql' to learn about this. They are plentiful and easy to find online.
For this example, let's assume that the database is called 'test', the table is called 'employees' (plural) and that it has two fields called 'id' (int) and 'name' (varchar(255)).
Populate that with a few rows of data.

4: Write a ruby script to interact with the database using ActiveRecord
ActiveRecord is the way that Rails interacts with databases, but you can use it outside of Rails with no problem.
Here is one that will fetch the 'employee' records from a database on the same machine as the script:

#!/usr/bin/env ruby
require 'rubygems'
require 'active_record'

# create a class for employee records (the class is singular but the table is plural)
class Employee < ActiveRecord::Base
end

# connect to the database
ActiveRecord::Base.establish_connection(:adapter => 'postgresql',
                                        :host => 'localhost',
                                        :username => 'postgres',
                                        :database => 'test');

employees = Employee.find(:all)
employees.each do |employee|
   print "#{employee.id}   #{employee.name}\n"
end

NOTE the adapter is 'postgresql', not 'postgres', you don't need to explicitly require the 'postgres' gem, and the username is 'postgres'

To connect to a remote postgresql instance you will need to change the :host value in the establish_connection call. You may need to add the port that it is listening on (default 5432) with a :port directive.

You will also need to set up Postgres to allow remote calls. In /usr/local/pgsql/data/ you will need to edit pg_hba.conf to add a line allowing connections from a defined set of remote machines. Read the documentation to make sure you don't open up your database to the entire world.
Then you need to edit postgresql.conf and uncomment the 'listen_addresses' and 'port' lines. You want to add the IP address of the database server to the listen addresses so that the default line

listen_addresses = 'localhost'

becomes something like this

listen_addresses = 'localhost,192.168.2.100'

You'll need to restart Postgresql for these to take effect.

If you have trouble making a connection from your remote ruby script then try connecting using 'psql' on the database machine, if that works, try it from the remote machine. That should help pin down where the problem lies.

Ruby, WEBrick and CGI scripts

Ruby on Rails is the obvious choice when you want to create a web application using Ruby. But this can be overkill for some projects where a regular CGI script would do the job.

Unfortunately the focus on Rails has led to CGI scripts being somewhat neglected in Ruby documentation. Here are the steps you need to get a basic CGI script up and running and the wrinkle you need to know about if you want to upload files.

1: Setting up a test web server using WEBrick
Ruby comes bundled with the WEBrick web server library and this allows you to create a basic server on your local machine for testing. Startup a server with this simple script:

#!/usr/bin/env ruby
require 'webrick'
include WEBrick
s = HTTPServer.new(
    :Port => 3000,
    :DocumentRoot => File.join(Dir.pwd, "/html")
)
trap("INT") { s.shutdown }
s.start

This will start a server at 'http://localhost:3000' that will respond to requests for files in the 'html' subdirectory of your project.
Put that into a file called, for example 'webrick.rb', 'chmod a+x webrick.rb' to make it executable and run it from your project top level directory.

WEBrick can be used in various ways, such as to serve up Java like servlets. BUt this simple configuration allows you to serve static content and CGI scripts from the given directory with no other configuration required. Great for testing, but use Apache or something else substantial for hosting real sites.

2: Create a CGI script
For WEBrick to recognize your scripts, with this minimal configuration, you MUST give them the suffix '.cgi'
Here is a minimal script that will return the value for parameter 'foo' as supplied in a corresponding form.

#!/usr/bin/env ruby
require 'cgi'
cgi = CGI.new
foo = cgi['foo']
print "Content-type: text/plain\n\n" 
print "foo: #{foo}\n"

The CGI object contains a hash of the parameters passed to it and with either a GET or POST request you can access these as shown above.

Things get more complicated when you want to upload a file from your form. In this case you specify enctype="multipart/form-data" on your html form, telling the CGI script that you are including a file among the parameters. BUT, with Ruby CGI, that affects the way you access other regular parameters that are included in the form.

You can no longer access their values by a direct hash lookup. cgi['foo'] no longer returns a string, instead it returns a StringIO object. StringIO wraps the IO class around the string so that you can use IO methods on it. This is great for handling the contents of the file that you want to upload but it makes handling regular arguments totally confusing until you realize the trick. Instead of accessing cgi['foo'] you now need to 'read' that StringIO stream and use cgi['foo'].read. The equivalent script as above for handling multipart data, and echoing the contents of an uploaded file is:

#!/usr/bin/env ruby
require 'cgi'
cgi = CGI.new
foo = cgi['foo']
print "Content-type: text/plain\n\n" 
print "foo: #{foo}\n"
print "foo.read: #{foo.read}\n"
print cgi['filename'].read

The incorrect direct use of cgi['foo'] will print a reference to a StringIO object.

You can test these out with this static HTML form:

<html><head></head><body>
<p>Two example HTML forms for testing Ruby CGI scripts</p>
<hr>
Form with (method="post")
<form action="ruby_cgi_post.cgi" method="post">
Enter Value: <input type="text" name="foo"><br/>
<input type="submit">
</form>
<br/>
<hr>
Form with (method="post" enctype="multipart/form-data") 
<form action="ruby_cgi_post_multipart.cgi" method="post" enctype="multipart/form-data">
Enter Value: <input type="text" name="foo"><br/>
Choose File: <input type="file" name="filename"><br/>
<input type="submit">
</form>
</body></html>

Mongrel Cluster on Mac OS X

Mongrel is a HTTP server that is well suited for serving Rails web applications, often in conjunction with Apache as a 'front end' server.

Mongrel works fine as is for development applications but if you have any real number of users its performance will degrade rapidly. In this case you want to use a cluster of mongrel servers, each running on a different port and have Apache balance the load between all of them

The mongrel_cluster gem is a convenient way to set up and manage multiple mongrels. HOWEVER I do not recommend this on Mac OS X. Because of the way Mac OS X handles processes at startup and/or because I couldn't figure it out, I was not able to get mongrel_cluster to work correctly at startup on my system. Some of the guides on the web that claimed to do this did not work for me. But don't worry... mongrel_cluster is really a simple utility and you can do without it at the price of a little more configuration. That is what I will show here.

These instructions relate to Mac OS X 10.4 and they assume that you already have Apache, Mongrel and your Rails application up and running.

1. Make sure that your Rails app is using Active Record to store Session data in the database, as opposed to storing it in files, which is the default. Instructions for that can be found HERE.

2. Setup a launchd plist file for each mongrel instance that you want to set up.
The preferred way to start programs automatically under Mac OS X is through launchd instead of the traditional init process in other Unix variants. launchd takes a bit of getting used to but you should use it (and don't try and mimic init scripts using StartupItems...)

You can learn about launchd is this Apple developer note and by doing a man launchd and man launchd.plist

3. In this example I am going to setup up 4 instances of mongrel on ports 8000, 8001, 8002 and 8003

In /Library/LaunchDaemons create the file net.mongrel80000.plist with contents similar to this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/Prop
ertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>net.mongrel8000</string>
  <key>ProgramArguments</key>
  <array>
          <string>/usr/local/bin/mongrel_rails</string>
          <string>start</string>
          <string>-e</string>
          <string>development</string>
          <string>-p</string>
          <string>8000</string>
          <string>-c</string>
          <string>/Users/jones/myapp</string>
          <string>-P</string>
          <string>/Users/jones/myapp/tmp/pids/mongrel.8000.pid</string>
          <string>-l</string>
          <string>/Users/jones/myapp/log/mongrel.8000.log</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
  <key>ServiceDescription</key>
  <string>Mongrel Rails Application Server</string>
</dict>
</plist>

This ugly block of XML breaks down into the following components:
Label
This is a UNIQUE label for this launch item. In our case make this the same as the name of the file, less the '.plist' extension (net.mongrel8000)
ProgramArguments
An array of strings which, when joined together, create the command that you would run to start this instance of mongrel. Change the paths, etc to suit your application. Be VERY careful to set the port numbers to the one used in this file (8000 in this case)
RunAtLoad
This is set to true and tells launchd to run this item once when the system starts up.
ServiceDescription
An optional string that describes what this item represents.

chown/chmod this to have these permissions and ownership:

-rw-r--r--   1 root  wheel  836 Feb 19 14:53 net.mongrel8000.plist

4. Clone and modify this file for each mongrel instance
In my case I copied this into net.mongrel8001.plist, etc. and changed each instance of the port number 8000 to 8001, 8002, or 8003 as appropriate. These are marked in red in the above XML. Make absolutely sure the Label is correct and unique otherwise it won't work.

5. Test it out
Clean out any editor backup files in /Library/LaunchDaemons, check your file permissions and restart your machine. If things are setup correctly, when it restarts it will have started four instances of Mongrel that will drive your application.

Test these by using a browser on that machine and going to each port in turn, in other words these four URLs should all work:

http://localhost:8000
http://localhost:8001
http://localhost:8002
http://localhost:8003

Note that in one of the guides on the web that I saw they used a single .plist file and put the command strings for all the mongrel instances in a single ProgramArguments section. This did not work for me at all...

If my instructions don't work for you then double check the format of the XML file and your paths.

6. Configure Apache to direct requests to the Mongrel servers
Your Apache installation needs to have the mod_proxy_balancer module installed. 'httpd -l' will list the compiled in modules and hopefully you will find it there. You can find out how to compile in modules in Apache 2.2 and read my grumbles about that HERE.

Edit your Apache httpd.conf file to create a virtual host that it will respond to and a set of proxy and load balancing instructions. You can do this in various ways and get a lot more complex than this, but here is a single block that you can put at the bottom of httpd.conf.

<VirtualHost *:80>
ServerName myserver.craic.com

# Enable URL rewriting
RewriteEngine On

# Rewrite index to check for static pages
RewriteRule ^/$ /index.html [QSA]

# Rewrite to check for Rails cached page
RewriteRule ^([^.]+)$ $1.html [QSA]

# Redirect all non-static requests to cluster
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ balancer://mongrel_cluster%{REQUEST_URI} [P,QSA,L]

# You could also add a bunch of deflate rules etc here

# This is the path to static content as opposed to your Rails app
DocumentRoot  "/Users/jones/html"
<Directory "/Users/jones/html">
     Options Indexes FollowSymLinks

     AllowOverride None
     Order allow,deny
     Allow from all
 </Directory>

</VirtualHost>

# This block tells the load balancer to pass requests
# onto these four mongrels

<Proxy balancer://mongrel_cluster>
  BalancerMember http://127.0.0.1:8000
  BalancerMember http://127.0.0.1:8001
  BalancerMember http://127.0.0.1:8002
  BalancerMember http://127.0.0.1:8003
</Proxy>

All the Rewrite lines tell Apache how to handle requests for static versus dynamic (from Rails) content. The important lines are the two that rewrite requests for non-static content into ones with a prefix of balancer://mongrel_cluster. These are passed to the <Proxy> block at the end where the load balancer distributes these among the members listed here. These members are the four mongrel instances that we set up earlier. Apache doesn't care that these are mongrel servers. It just sees them as URLs and efficiently passes on the requests.

Your Apache is most likely started by a launchd plist file. Assuming it starts automatically then any reboot of your server will start it and the mongrel instances.

Giving a URL like http://myserver.craic.com/myapp will now get forwarded to one of the mongrels.


There you have it... If you read this and have a better mongrel launchd configuration please let me know. There has to be a cleaner way than what I have here...

Using Active Record for Session Storage in Rails

This setup appears in all sorts of pages but I'm including the basic steps here for my own benefit - and hopefully yours...

The default way that Rails (1.2.3) stores session data is in files in your application tmp directory. This mechanism is referred to as CGI::Session::PStore. This is fine for development but becomes a problem as you move to a real production environment.

One problem with it is that unless you actively clean out old files with a cron job and 'rm' you can end up with a massive number of old session files.

It is also a problem if you use Apache and Mongrel to serve your application and want to scale things up with mongrel_cluster. Various resources warn of bad things happening with multiple mongrels and session files.

The next step up from files is to use a database table and have Active Record store session data in that. This is easy to setup.

1: Create a migration to set up the table and run that

$ rake db:sessions:create
    exists  db/migrate
    create  db/migrate/027_add_sessions.rb
$ rake db:migrate
== AddSessions: migrating =====================================================
-- create_table(:sessions)
 -> 0.4298s
-- add_index(:sessions, :session_id)
 -> 0.2914s
-- add_index(:sessions, :updated_at)
 -> 0.0727s
== AddSessions: migrated (0.8001s) ============================================

2: Edit your app's config/environment.rb file and uncomment this line
config.action_controller.session_store = :active_record_store

3: Start your app server, interact with it and look in mysql
mysql> select * from sessions;

You will see a hexadecimal encoded session_id and a big block of encoded session data. Everything else should just work normally.

4: To avoid sessions accumulating in applications where you have users login and logout, you can call reset_session in the logout action - something like this:

  def logout
  reset_session
  flash[:notice] = "Logged out"
  redirect_to :action => "index"
end

This clears out your current session row in the table and initializes a new session object. I'm not sure why it does the latter.

Multihomed Ethernet Interfaces on Mac OS X server 10.4

I had a Mac OS X 10.4 server with two ethernet interfaces configured so that eth0 connected to my internal network and eth1 connected directly to the Internet.

I could access either interface just fine from my internal network. But unbeknownst to
me, the outside world was not able to access the external interface.

Checking the firewall or even turning the firewall off completely had no effect. The output of ifconfig looked like it should, the cables were plugged in where they ought to be. You just couldn't get to the external interface from the Internet...

It turns out that Mac OS X expects eth0 to connect to the Internet and I had it connected to my internal network. The server could see the Internet via that interface. Because it passed through my restrictive firewall, the outside world could not see it but from my subnet I could see it just fine... a nasty little gotcha.

I swapped the network configuration parameters for that eth0 was the external interface and eth1 was internal and the problem was solved.

Look at the Network control panel under 'Network Status'. Both interfaces will be active in a configuration like mine but only one will state 'You are connected to the Internet via...'. That needs to be the external interface.

There must be some way to configure which interface connects to the Internet but it is easier to follow their convention and not worry about it.