Private Wi-Fi Networks, Raspberry Pi and Mac OS X Mavericks

I ran into two problems connecting to my Wi-Fi network recently - both turned out to be the result of the network being private and not broadcasting its SSID (Service Set Identifier) - the 'name' of the network.

My network runs on an Apple Time Capsule using WPA2 encryption. It has worked fine with other machines - MacBook, Win PC, Chromebook, iPhones...


#1 Getting W-Fi running on a Raspberry Pi

I was not able to get Wi-Fi running on a Raspberry Pi linux machine, despite trying a range of configurations that I found on the web.

When I changed the network to make its SSID public then it connected just fine with the basic configuration.

#2 Setting up a new MacBook Pro

When you start up a brand new Mac it walks you through several setup steps, including connecting to a network. The new MacBook Pro only has Wi-Fi so connecting to the network is essential. But in my case it did not see the private network and so I was unable to get through the regular setup.

The work around was to skip those steps until I got to the regular desktop. Then going into Preferences and Network and configuring the network manually I was able to get in.

I think that if the SSID were broadcast then I would not have had any problems.


The reason to keep an SSID private is to make it more difficult for someone to break into your network. If they don't know the network exists then they won't try and break in. In reality, however, it not that simple. A serious hacker will still be able to detect packets on all networks in the area including the 'hidden' ones and then attempt to break in. So a private network is a good idea but it does not offer great protection.

Installing the OpenCV library on Mac OS X Lion using Homebrew

OpenCV (Open Source Computer Vision) is a remarkable library of functions for real time computer vision.

I'm interested in using it on to recognize buildings in satellite/aerial photographs - I'll explain more if I get the idea to work.

The first step is to get OpenCV up and running on my Mac (OS X Lion 10.7.4). The library has a lot of dependencies that need to be installed but thankfully there is a Homebrew recipe for doing all of the heavy lifting... almost... I ran into a few problems but here is what I needed to do.

In principle all you need is:

$ brew update
$ brew install opencv

But I got two errors:
1: brew wanted me to update my version of Xcode from 4.2 to 4.3
On Lion you update Xcode via the App store and it turns out that 4.4 is available. So I installed that and reran 'brew install opencv'
2: brew then complained that the Python package 'numby' (Numerical Python) was not installed. Brew does not install Python dependencies but suggested that I do this with:

$ easy_install numby

I did that but I still got the error.

I've got two versions of python installed (via homebrew)

$ python --version
Python 2.7.3
$ python3 --version
Python 3.2.3

For our purposes you only want to deal with Python 2 - version 3 should work but the homebrew package seems to want version 2

Given the problem with easy_install, I went with a manual installation
1: Download the tar file for the latest stable version of numby
http://sourceforge.net/projects/numpy/files/NumPy/1.6.2/
2: Unpack the file into a temporary directory and cd into it
3: Run the python setup script

$ python setup.py install

Sit back and relax as this takes a while. Everything went smoothly for me and hopefully for you too.


All being well you can now run

$ brew install opencv

Again this takes a while - about 9 minutes on my box. You will get a message at the end of the output telling you to add the following to your .bash_profile

export PYTHONPATH="/usr/local/lib/python2.7/site-packages:$PYTHONPATH"

OpenCV has an excellent interface to Python, but I work primarily in Ruby so the next step for me is to get an OpenCV ruby gem set up and start to do some image processing. The next post on this blog will cover that.

Setting up Apache Solr on Mac OS X

Apache Solr is an incredibly flexible and capable text search engine that you can hook into other applications.

For a beginner it can appear to be very daunting due to the number of configuration options and when you start to read the documentation this feeling tends not to improve. But if you are using Mac OS X (a recent version) and have the homebrew package manager installed then the process is not too bad, at least to get you up and running.

These steps will help you get Solr installed and then load in data extracted from HTML, PDF and Word files.

In my configuration I have Mac OS X Lion and current versions of Java and homebrew.

1: Install Solr with homebrew - (brew update is always a good idea before a complex install)
$ brew update
$ brew install solr

This installs the software in /usr/local/Cellar/solr/3.6.0 (or whatever your version is)

2: Go to the example directory and start up Solr
$ cd /usr/local/Cellar/solr/3.6.0/libexec/example
$ java -jar start.jar
This will start up the Solr server using Jetty as the servlet container, which is just fine for our testing. Do not worry about using Tomcat etc until you are comfortable with a working Solr set up.
It is usually not good form to work inside a distribution directory - but for initial testing you should do this.
You will see a load of verbose output from Java in your terminal window, aside from any real errors, just ignore this.

3: Verify that the server is running
Browse to http://localhost:8982/solr/admin
You should the Administration interface with a gray background. There is not a lot you can do as you have not yet indexed any data, but this shows that you are running.

4: Load some data into Solr
$ cd exampledocs
$ java -jar post.jar *.xml
Note that the Solr server MUST be running before you try and load the documents.

5: Run your first query from the admin page
Enter 'video' in the Query form and hit 'Search'. You should see the contents of an XML file returned to you with 3 documents.
This demonstrates that the server is working, that you can index XML documents and query them. Solr does not provide you with a nice search interface. The intent is that your Rails, etc application sends queries and then parses out the XML results for display back to the user.

6: Extract text from other document types
What I want to use Solr for is to search text extracted from web pages, Word documents, etc. But this where the Solr documents started to really let me down.
This parsing is done by calling Apache Tika which is a complex software package with the sole aim of text extraction. The interface between Solr and Tika used to be called SolrCell and is now called the ExtractingRequestHandler. Don't worry about any of that for now! The Solr distribution has everything you need already in place - you just need to know how to use it...


Do not set up a custom Solr home directory for now. That is what the documentation suggests, it makes perfect sense but nothing will work if you do.


Save a few HTML files that contain a good amount of text into a temporary directory somewhere.

To load the contents of a file into the server and have it parse the text you need to use 'curl' to POST the data to a specific URL on the server. In this example my file is called index.html and I have cd'ed to the directory containing the file
$ curl "http://localhost:8983/solr/update/extract?literal.id=doc1&commit=true&uprefix=attr_&fmap.content=attr_content" -F "myfile=@index.html"

There are several things to note with this URL...
The server URI is http://localhost:8983/solr/update/extract
The parameters are
literal.id=doc1
commit=true
uprefix=attr_
fmap.content=attr_content

Literal.id provides a unique identifier for this this document in the index (doc1 in this case)
uprefix=attr_   adds the prefix 'attr_' to the name of each tag in the source document
fmap.content=attr_content  specifies that the main text content of the page should be given the tag attr_content
commit=true actually commits the parsed data to the index

And then note how the file to be parsed is specified... as a quoted string passed with the -F flag to curl. The quoted string consists of a name for the uploaded file (myfile) followed by the path to the file preceded by the Ampersand character (@).

You can ignore what these mean for now with the exception that each document you load must have a unique document ID.

7: Verify that the text has been indexed
Go to the Admin interface and click 'Full Interface'. Set the number of rows returned to large number (50) and then search with the query *:*   - this will return all the records in the index.

You should get back an XML page with the example data plus the text derived from your HTML file at the bottom.

8: Try loading other data types
Tika knows how to parse Word, Excel, PDF and other files.
Be aware that there may not be much text in certain files. This is especially true of PDF files which although they appear to have text when displayed, may have that text stored as an image.

9: Explore the example directory

Everything I've shown here should 'just work' as long as you worked in the example directory. In there you will see a 'solr' subdirectory which contains conf and data directories. The data directory is where the index files reside. In the conf directory you will see a solrconfig.xml file. This is the main location for specifying the various components that your Solr installation uses. The default values happen to work fine until you create your own 'Solr Home' directory elsewhere.

The problem is that the solrconfig.xml file contains various relative paths (like ../../lib) which will not work if you create a solr directory in an arbitrary location. Knowing this you can update those paths pretty easily, but if you created a custom solr home then you are in for a lot of frustration trying to figure out why nothing works. I went through that process - you shouldn't have to...

That should be enough to getting you started with Solr. The next steps are to link it to your web application, to load in a lot more data and to move it to a production servlet container like Tomcat.

Installing Apache tomcat on Mac OS X Lion using homebrew

It is hard to overestimate how much I am grateful for the homebrew Mac OS X package manager.

Today I needed to setup Apache Tomcat on a OS X Lion machine. I already have homebrew installed and have used it to install a bunch of stuff.

All that is needed is:
$ brew install tomcat

I'll admit that this failed the first time that I tried it but running 'brew update' and then 'brew install tomcat' sorted that out - I guess if you've not used it for a while you should run 'brew update' as it is still evolving quite rapildly.

That installs the code into /usr/local/Cellar/tomcat/7.0.28

To start/stop the server from the command line you use the catalina shell script
$ /usr/local/Cellar/tomcat/7.0.28/bin/catalina run
[...]
$ /usr/local/Cellar/tomcat/7.0.28/bin/catalina stop


When it is running you can go to http://localhost:8080 and see a default tomcat information page with lots more info.


For me that default port is a problem as it is also the default for my nginx installation. But changing it is easy enough. Edit /usr/local/Cellar/tomcat/7.0.28/libexec/conf/server.xml and replace the instances of 808 with your preferred port. Rerun catalina and then go to your preferred URL.


For my purposes I want to start and stop the server manually but check the documents for how to link it to an instance of Apache.

Setting a Title for a Tab in iTerm2 on Mac OS X

 iTerm2 (http://www.iterm2.com) by George Nachman is an excellent terminal emulator for Mac OS X that is highly configurable. I it prefer over the system Terminal application.

I tend to have a number of terminal sessions open at the same time and for a while I have wanted a way to give each Tab in the window an informative name. You can do that through the iTerm2 preferences but I wanted a way to set it directly from the shell command line.

iTerm2 can be scripted via AppleScript and so I wrote a simple script to do the job. It uses a system application called osascript which will execute a AppleScript script from the Unix command line.

The code is freely available at https://github.com/craic/iterm_title - see the README for more details.

The code should be a useful starting point if you want to write something in AppleScript that you can call from UNIX.

Disabling Spotlight (mds) on Mac OS X (Snow Leopard)

I run a lot of command line scripts on my laptop - some of which can run for hours. I want to continue using the machine for reading mail, etc., but I don't want any other intensive task sucking up the cpu cycles. So I shut down iTunes, don't watch any videos, etc.

But sometimes I see some other process taking all my cycles. The odds are that it is either something to do with Flash or it is a process called mds.

mds is the indexing software that powers Spotlight - the built in Mac search facility.

I suspect that when I'm generating gigabytes of data and hundreds of files in one of my compute jobs, mds is responding by trying to index them at the same time.

I don't use spotlight at all, so let's turn it off and see if that helps.

This turns it off:
$ sudo mdutil -a -i off 

This turns it back on:
$ sudo mdutil -a -i on

Turning it back on will presumably trigger a big mds run as it plays catch up, so run this command only when you can afford the cycles.

Nokogiri and Snow Leopard

I'm not alone in having problems installing the nokogiri ruby gem on a Mac that has been upgraded to Snow Leopard. The problem lies in the gem not being able to find a suitable version of libxml2, despite Snow Leopard having a recent version of that library installed. (Not sure if this is by default or only if you have the developer tools installed...).

People have tried various things but the place to look first is /opt/local/lib/xml2.*

If you have the libxml2 installed there then remove (or move) those files, along with /opt/local/libz.*

Try installing the gem again (sudo gem install nokogiri).

If it succeeds, you're good - if not then try removing the libxml2 include files in /opt/local/include - try again - and if still no luck then try trawling through other directories under /opt/local.

You should not have to specify the explicit libxml2 file locations with options to the gem install.