I use YAML as a convenient way to serialize data in a number of projects, most of which are written in Ruby and Rails
You generate the YAML representation with the .to_yaml method - really simple.
p = Paper.find(359)
puts p.to_yaml
But if the input is a ruby or rails object the output is prefixed with !ruby/object and that causes problems when you try and load that document in another script that does not know about this object
For example - here is an example of a Paper object from a Rails application.
--- !ruby/object:Paper
attributes:
id: 359
pmid: 7945531
title: 'Pharmacokinetics of a new human monoclonal antibody against cytomegalovirus.
Third communication: correspondence of the idiotype activity and virus neutralization
activity of the new monoclonal antibody, regavirumab in rat serum and its pharmacokinetics'
[...]
If I try and read this file in a separate script I get this error because that script has no concept of a Paper object.
y = YAML.load_file('test.yml')
ArgumentError: undefined class/module Paper
from /Users/jones/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/psych-2.0.5/lib/psych/class_loader.rb:53:in `path2class'
from /Users/jones/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/psych-2.0.5/lib/psych/class_loader.rb:53:in `resolve'
from /Users/jones/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/psych-2.0.5/lib/psych/class_loader.rb:45:in `find'
[...]
The way to strip off the ruby/object 'header' is to use serializable_hash before to_yaml.
---
abstract: TI-23 consists of lyophilized regavirumab (monoclonal antibody C23, MCA
[...]
id: 359
pmid: 7945531
publication_date: 1994-07-01
title: 'Pharmacokinetics of a new human monoclonal antibody against cytomegalovirus.
Third communication: correspondence of the idiotype activity and virus neutralization
activity of the new monoclonal antibody, regavirumab in rat serum and its pharmacokinetics'
It looks like the keys in the yaml block are output in alphabetical order.
It's a simple fix but I had to hunt around to find it.
Showing posts with label rails. Show all posts
Showing posts with label rails. Show all posts
Thursday, April 25, 2019
Monday, November 11, 2013
Setting up a new Application on Heroku
You can set up a new web application on Heroku in several ways. Here is my preferred way:
1: Build and test your application locally
2: Set it up as a git repository and commit all changes
3: Go to your account on the Heroku web site and cerate your new app there with your preferred name.
4: In your local git repo run:
$ heroku git:remote -a <your-app-name>
The response should be 'Git remote heroku added'
5: Upload your code with:
$ git push heroku master
6: Finish up your Heroku configuration on the we b site
7: Import any existing database as directed by this article.
1: Build and test your application locally
2: Set it up as a git repository and commit all changes
3: Go to your account on the Heroku web site and cerate your new app there with your preferred name.
4: In your local git repo run:
$ heroku git:remote -a <your-app-name>
The response should be 'Git remote heroku added'
5: Upload your code with:
$ git push heroku master
6: Finish up your Heroku configuration on the we b site
7: Import any existing database as directed by this article.
Thursday, September 5, 2013
Adding extensions to Postgresql database with Rails on Heroku
This is a follow up to my previous post Adding extensions to Postgresql database with Rails
If your database is running on Heroku you need to access it slightly differently.
Using the 'rails db' command does not work:
Instead access psql like this and then add the extension. \dx lists the current installed extensions
Note that this works just fine with a pg:basic database plan. There is no need to upgrade just to use postgres text search.
If your database is running on Heroku you need to access it slightly differently.
Using the 'rails db' command does not work:
$ heroku run rails db Running `rails db` attached to terminal... up, run.6499 Couldn't find database client: psql. Check your $PATH and try again.
Instead access psql like this and then add the extension. \dx lists the current installed extensions
$ heroku pg:psql psql (9.2.1, server 9.1.9) [...] SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256) Type "help" for help. dftmhe7vpg4dhs=> create extension unaccent; CREATE EXTENSION dftmhe7vpg4dhs=> \dx List of installed extensions Name | Version | Schema | Description ----------+---------+------------+--------------------------------------------- plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language unaccent | 1.0 | public | text search dictionary that removes accents
Note that this works just fine with a pg:basic database plan. There is no need to upgrade just to use postgres text search.
Friday, August 16, 2013
Testing for the presence of a loaded Gem in a Rails application
I'm migrating the search function in an application from meta_search to ransack (both from Ernie Miller)
The syntax has changed quite a bit between the two gems and I use the search functionality in more than 30 controllers / index pages. So I want a simple way to update each instance while still being able to switch back to meta_search if I need to.
So I need to test which of the two gems has been loaded in the running application.
The best way that I have found is this:
You can see the full list of loaded gems with Gem.loaded_specs.keys
The syntax has changed quite a bit between the two gems and I use the search functionality in more than 30 controllers / index pages. So I want a simple way to update each instance while still being able to switch back to meta_search if I need to.
So I need to test which of the two gems has been loaded in the running application.
The best way that I have found is this:
if Gem.loaded_specs['ransack'] '... do this...' else '... do that ...' end
You can see the full list of loaded gems with Gem.loaded_specs.keys
Thursday, June 13, 2013
Migrating from Rails HABTM to has_many :through
I had a has_and_belongs_to_many (HABTM) association that I needed to convert to has_many through with an explicit model that contains additional columns.
In my example a User knows about many Words and each Word is known by many Users.
You can find many pages on the web about the differences. The consensus is that has_many :through is the way to go from the start - and after this process I agree.
Making the change in itself is not a big deal - drop the existing joining table 'users_words', create the new one, update the User and Word models and you're good to go.
Problem is that I already had data in the joining table...
And because the standard way of setting up a HABTM joining table does not include an id column, you can't just use that table or directly copy each record from it. Dang it...
Here were my steps - hopefully I got them all - don't skip any !
1: Backup your database and prevent users from accessing it
2: Do not touch any of the old associations, table, etc
3: Create the new table and model with a different name from the old joining table.
My HABTM table was users_words and my new table is user_work_links
4: Update the two models
My original association was this - do not change it yet !
The new association is this - NOTE the second line is commented out for now - VERY important !
5: Copy over the data from the old joining table with a rake task
You need to go through the existing associations one by one to get the ids for records in the two tables.
Here is my script:
6: Update the two models
Now you can comment out the old associations and uncomment the new ones
In the attr_accessible lists in the two models be sure to add :user_ids in the Word model and :word_ids in the User model. If your forget this it will silently fail to create the records
7: Test Test Test
You should be back up and running with the new model
8: Remove the old table
Finally create a migration that drops the old table and run it
Not too bad as long as you think it through before you start and don't rush the steps
In my example a User knows about many Words and each Word is known by many Users.
You can find many pages on the web about the differences. The consensus is that has_many :through is the way to go from the start - and after this process I agree.
Making the change in itself is not a big deal - drop the existing joining table 'users_words', create the new one, update the User and Word models and you're good to go.
Problem is that I already had data in the joining table...
And because the standard way of setting up a HABTM joining table does not include an id column, you can't just use that table or directly copy each record from it. Dang it...
Here were my steps - hopefully I got them all - don't skip any !
1: Backup your database and prevent users from accessing it
2: Do not touch any of the old associations, table, etc
3: Create the new table and model with a different name from the old joining table.
My HABTM table was users_words and my new table is user_work_links
4: Update the two models
My original association was this - do not change it yet !
has_and_belongs_to_many :words, :uniq => true
The new association is this - NOTE the second line is commented out for now - VERY important !
has_many :user_word_links, :dependent => :destroy
# has_many :words, :through => :user_word_links, :uniq => true
5: Copy over the data from the old joining table with a rake task
You need to go through the existing associations one by one to get the ids for records in the two tables.
Here is my script:
namespace :craic do
desc "move user word data"
task :move_user_word_data => :environment do
users = User.all
users.each do |user|
user.words.each do |word|
record = UserWordLink.new({ :user_id => user.id, :word_id => word.id })
record.save
end
end
end
end
6: Update the two models
Now you can comment out the old associations and uncomment the new ones
# has_and_belongs_to_many :words, :uniq => true
has_many :user_word_links, :dependent => :destroy
has_many :words, :through => :user_word_links, :uniq => true
In the attr_accessible lists in the two models be sure to add :user_ids in the Word model and :word_ids in the User model. If your forget this it will silently fail to create the records
7: Test Test Test
You should be back up and running with the new model
8: Remove the old table
Finally create a migration that drops the old table and run it
Not too bad as long as you think it through before you start and don't rush the steps
Friday, May 31, 2013
Listing all the models and numbers of records in a Rails application
Here is a rake task that will list all the ActiveRecord models in a Rails application along with the number of database records in each and the last time that database table was updated. Note the call to eager_load which ensure that all models are loaded.
This is a simple but very useful way to assess the contents of your database.
This is a simple but very useful way to assess the contents of your database.
desc "List model in application"
task :list_models => :environment do
Rails.application.eager_load!
ActiveRecord::Base.descendants.each do |model|
name = model.name
count = model.count
last_date = count == 0 ? 'no data' : model.order('updated_at desc').first.updated_at
printf "%-20s %9d %s\n", name, count, last_date
end
end
Tuesday, February 26, 2013
General approach to building Rails applications with Devise
I have built a number of complex Rails applications that include user authentication. I want to describe my general approach to getting these up and running.
In most of them I chose the Devise gem to handle authentication and added a number of custom fields to the User model (first name, last name, etc). The gem documentation and this Railscast can help you get started with all that.
One approach is to build your core application without authentication and then add it in at a later stage but I don't like that.
Authentication can quickly get very complicated once you start adding user confirmation via email, an admin role and other custom fields. I prefer to get all that machinery up and running in the context of a very simple application and only then add my real application code into it.
Similarly, when I'm starting with a new application I don't usually hit on the right data architecture right away. That leads to me adding new columns to my database tables and models over time and it always involves quite a lot of 'rake db:rollback / rake db:migrate' operations. I prefer to go through this phase without the extra baggage of authentication, and often, without worrying a lot about CSS etc. I just need to try several alternate views of the problem before I settle on a final design.
So may approach is to build two applications side by side. The first is a minimal application with, say, a single table (usually called 'posts') to which I add Devise and configure all the options that I want.
This is a very useful piece of code for other applications, so I always create a good README and store a copy of the code for future use.
Separately I work through my core application and, once that is working, I condense all the migrations that add/remove/rename columns into a few 'clean', simple migrations.
Finally I add the core application code to the minimal authenticated application. That way it is easier for me to update the controllers, etc. to use authentication.
The bottom line is that you want to minimize the number of moving parts in the early stages of a new application design. This is a simple approach that works very well for me.
In most of them I chose the Devise gem to handle authentication and added a number of custom fields to the User model (first name, last name, etc). The gem documentation and this Railscast can help you get started with all that.
One approach is to build your core application without authentication and then add it in at a later stage but I don't like that.
Authentication can quickly get very complicated once you start adding user confirmation via email, an admin role and other custom fields. I prefer to get all that machinery up and running in the context of a very simple application and only then add my real application code into it.
Similarly, when I'm starting with a new application I don't usually hit on the right data architecture right away. That leads to me adding new columns to my database tables and models over time and it always involves quite a lot of 'rake db:rollback / rake db:migrate' operations. I prefer to go through this phase without the extra baggage of authentication, and often, without worrying a lot about CSS etc. I just need to try several alternate views of the problem before I settle on a final design.
So may approach is to build two applications side by side. The first is a minimal application with, say, a single table (usually called 'posts') to which I add Devise and configure all the options that I want.
This is a very useful piece of code for other applications, so I always create a good README and store a copy of the code for future use.
Separately I work through my core application and, once that is working, I condense all the migrations that add/remove/rename columns into a few 'clean', simple migrations.
Finally I add the core application code to the minimal authenticated application. That way it is easier for me to update the controllers, etc. to use authentication.
The bottom line is that you want to minimize the number of moving parts in the early stages of a new application design. This is a simple approach that works very well for me.
Thursday, October 25, 2012
Upgrading from Rails 3.0 to 3.1 - Issues with Javascripts
I have been upgrading a Rails app from 3.0.10 to 3.1.5.
The BIG difference with 3.1 is the use of the Assets pipeline which requires that you move images, stylesheets and javascripts from public/* to app/assets/*. In the case of Stylesheets and Javascripts you also put these under the control of a Sprockets Manifest file: application.js or application.css.scss
That Manifest file caused me problems with my javascripts...
I followed the advice in the Railscasts Episode 282 (which I recommend you look at) and put these lines into my application.js
Jquery is supposed to 'just be there' if you have the jquery-rails gem installed.
I think there were a two, maybe three, issues.
1: Sprockets requires that there are NO BLANK LINES in the Manifest file BEFORE the end of the lines that should be processed. In Rails 3.2 the default file says that explicitly but if you are creating your own that is an easy thing to overlook.
2: Look at the 4 lines shown above. The first two are 'require', followed by a SPACE and then the name of the javascript file.
The Third is 'require_self' - underscore, no space - that means process any JS code in this file.
The Fourth is 'require_tree' - underscore, no space - and that means process ALL JS files in this directory.
If you type these lines in and happen to use a space instead of the underscore then Sprockets will look for JS file called 'self' and 'tree' - and won't find them. So be very careful tying those in.
3: The main problem was the require_tree line
I have a number of JS files that are included as needed in certain views. At the top of each view page I specify those with a content_for block like this:
That has worked just fine in the past and I kept all the custom JS files in the javascripts directory. But require_tree will load ALL of them on EVERY page. I assume that there are some variable/function names that are duplicated and/or conflicting and so with all that loaded, nothing worked.
The solution is to get rid of the require_tree line completely.
If I had JS files that should be included in each page then I would 'require' each of them explicitly. This is the same approach that Ryan Bates suggests for CSS files. It just gives you more flexibility.
The BIG difference with 3.1 is the use of the Assets pipeline which requires that you move images, stylesheets and javascripts from public/* to app/assets/*. In the case of Stylesheets and Javascripts you also put these under the control of a Sprockets Manifest file: application.js or application.css.scss
That Manifest file caused me problems with my javascripts...
I followed the advice in the Railscasts Episode 282 (which I recommend you look at) and put these lines into my application.js
//= require jquery //= require jquery_ujs //= require_self //= require_tree .But when I ran my app none of my javascript functions were working. Looking at the source for one of the pages I saw this link
<script src="/assets/application.js" type="text/javascript"></script>Clicking on brought up a message, instead of a JS file, saying that Sprockets was unable to find the file 'jquery'
Jquery is supposed to 'just be there' if you have the jquery-rails gem installed.
I think there were a two, maybe three, issues.
1: Sprockets requires that there are NO BLANK LINES in the Manifest file BEFORE the end of the lines that should be processed. In Rails 3.2 the default file says that explicitly but if you are creating your own that is an easy thing to overlook.
2: Look at the 4 lines shown above. The first two are 'require', followed by a SPACE and then the name of the javascript file.
The Third is 'require_self' - underscore, no space - that means process any JS code in this file.
The Fourth is 'require_tree' - underscore, no space - and that means process ALL JS files in this directory.
If you type these lines in and happen to use a space instead of the underscore then Sprockets will look for JS file called 'self' and 'tree' - and won't find them. So be very careful tying those in.
3: The main problem was the require_tree line
I have a number of JS files that are included as needed in certain views. At the top of each view page I specify those with a content_for block like this:
<% content_for :custom_javascript_includes do %> <%= javascript_include_tag "raphael.js" %> <%= javascript_include_tag "tabs_timeline_simple.js" %> <% end %>and in my application.html file I have
<%= javascript_include_tag "application" %> <%= yield :custom_javascript_includes %>
That has worked just fine in the past and I kept all the custom JS files in the javascripts directory. But require_tree will load ALL of them on EVERY page. I assume that there are some variable/function names that are duplicated and/or conflicting and so with all that loaded, nothing worked.
The solution is to get rid of the require_tree line completely.
If I had JS files that should be included in each page then I would 'require' each of them explicitly. This is the same approach that Ryan Bates suggests for CSS files. It just gives you more flexibility.
Wednesday, October 24, 2012
Uploading contents of a PostgreSQL database to Heroku without db:push
I just created a new Rails app on my local machine and want to deploy it to Heroku. I've done this before and the uploading of the code typically works great - and it did today.
Then I wanted to upload the contents of a small PostgreSQL database to Heroku - and that's where I ran into issues.
Heroku has recently switched from a Heroku gem to Heroku Toolbelt which I installed onto my Mac OS X machine.
I should be able to run 'heroku db:push' from my Rails app and the data would get moved over using the taps gem. But it failed with this error:
I dug around in /usr/local/heroku and hacked /usr/local/heroku/lib/heroku/command/db.rb to tell me what it thought the load path for ruby gems was, using this line:
And then I found mention of a page on Heroku's site that deals with the pgbackups addon. Part of that describes how to import from a PostgreSQL backup directly, without having to use the heroku clients or taps.
I followed that and it worked great - here are the steps involved. Your starting data must be in PostgreSQL and you need to have an account on Amazon S3 (and know how to use it)
1: On your machine create a backup of your PostgreSQL database - 'mydb' is your database name and 'myuser' is your database user.
3: Check that you can get the file back out of S3 using curl
Then I wanted to upload the contents of a small PostgreSQL database to Heroku - and that's where I ran into issues.
Heroku has recently switched from a Heroku gem to Heroku Toolbelt which I installed onto my Mac OS X machine.
I should be able to run 'heroku db:push' from my Rails app and the data would get moved over using the taps gem. But it failed with this error:
$ heroku db:push ! Taps Load Error: cannot load such file -- taps/operation ! You may need to install or update the taps gem to use db commands. ! On most systems this will be: ! ! sudo gem install tapsI have taps installed, so what is going on? Well, I'm not sure but I run my rubies under rbenv which allows you to manage multiple versions - and Heroku's client doesn't seem able to figure out where the gems are located with rbenv.
I dug around in /usr/local/heroku and hacked /usr/local/heroku/lib/heroku/command/db.rb to tell me what it thought the load path for ruby gems was, using this line:
STDERR.puts "DEBUG: load path #{$:}"
The paths given in that were all in /usr/local/heroku/vendor/gems and my gem paths were nowhere to be found. I tried installing the taps, sqlite3 and sequel gems to that directory and I was able to get a bit further but I still couldn't get all the way - so I gave up on that path.And then I found mention of a page on Heroku's site that deals with the pgbackups addon. Part of that describes how to import from a PostgreSQL backup directly, without having to use the heroku clients or taps.
I followed that and it worked great - here are the steps involved. Your starting data must be in PostgreSQL and you need to have an account on Amazon S3 (and know how to use it)
1: On your machine create a backup of your PostgreSQL database - 'mydb' is your database name and 'myuser' is your database user.
$ pg_dump -Fc --no-acl --no-owner -h localhost -U myuser mydatabase > mydb.dump2: Transfer that to an Amazon S3 bucket and either make it world readable or set a temporary access url.
3: Check that you can get the file back out of S3 using curl
$ curl 'https://s3.amazonaws.com/mybucket/mydb.dump' > tmp $ ls -l tmp4: Enable the pgbackups addon in Heroku
$ heroku addons:add pgbackups5: Get the name of the Heroku database that you will restore to, by capturing a backup from the empty heroku db:
$ heroku pgbackups:capture HEROKU_POSTGRESQL_CRIMSON_URL (DATABASE_URL) ----backup---> b001 Capturing... done Storing... done
6: Restore to that database from the S3 URL
$ heroku pgbackups:restore HEROKU_POSTGRESQL_CRIMSON_URL 'https://s3.amazonaws.com/mybucket/mydb.dump'
7: Test it out - my app was already up so I just had to browse to the appropriate page and there were all my records.
8: Remove your S3 file
DONE !
Friday, July 6, 2012
Downloading CSV files in Rails
Providing a way to download data as a CSV file is a common feature in Rails applications.
There is a nice Railscasts episode on the topic here: Exporting CSV and Excel
I prefer to use a view template to generate my CSV as it gives me a lot of control on the fields that go into the file. But the standard way of invoking this from the controller does not provide a way to specify the filename for the downloaded file
respond_to do |format|
format.html
format.csv
end
With a Show action, for example '/posts/25.csv', the downloaded file would be called '25.csv' which is not useful.
In a response to this Stackoverflow question, Clinton R. Nixon offers up a nice solution.
He has a method called render_csv in his application controller that takes an optional filename. Before calling regular render on your template, it sets several HTTP headers - most importantly a Content-Disposition header with the desired filename. It adds the '.csv' suffix for you and uses the action name as the default if no name if supplied.
With this in place you modify your controller like this
respond_to do |format|
format.html
format.csv { render_csv('myfile') }
end
Very nice and very useful...
There is a nice Railscasts episode on the topic here: Exporting CSV and Excel
I prefer to use a view template to generate my CSV as it gives me a lot of control on the fields that go into the file. But the standard way of invoking this from the controller does not provide a way to specify the filename for the downloaded file
respond_to do |format|
format.html
format.csv
end
With a Show action, for example '/posts/25.csv', the downloaded file would be called '25.csv' which is not useful.
In a response to this Stackoverflow question, Clinton R. Nixon offers up a nice solution.
He has a method called render_csv in his application controller that takes an optional filename. Before calling regular render on your template, it sets several HTTP headers - most importantly a Content-Disposition header with the desired filename. It adds the '.csv' suffix for you and uses the action name as the default if no name if supplied.
With this in place you modify your controller like this
respond_to do |format|
format.html
format.csv { render_csv('myfile') }
end
Very nice and very useful...
Using Sass in Rails 3.0
Sass is the default CSS preprocessor in Rails3.1 and it is tied into the asset pipeline machinery that first appears in 3.1.
I am migrating a complex app from 3.0 to 3.1 and I want to use Sass in the 3.0 production version before I make the big jump to 3.1. So far I have been using Less and the more plugin.
I recently wrote up my experience moving from Less to Sass syntax but here I want to cover the Rails side of things.
Here are the steps involved in my Less to Sass transition in Rails 3.0.5
Part 1: Remove all traces of Less
1: Remove the less gem from your Gemfile
2: Rename the app/stylesheets folder to something else
3: Move or delete the Less plugin in vendor/plugins/more so that the app doesn't see it on start up
Part 2: Set up Sass
4: Add the sass gem to your Gemfile ( gem 'sass' )
5: Run 'bundle install'
6: Create a sass directory under public/stylesheets
7: Leave any straight css file in public/stylesheets
8: Place any SCSS files in the sass directory - these could be new files or converted Less files - give them the .scss suffix
9: Remove or rename any matching css files in the parent directory
10: Restart your server and browse to your site
All being well Sass has worked in the background, created new versions of the CSS files and put them in public/stylesheets. Look for those files and test out the operation by, say, changing a background color in the SCSS file, saving the file and reloading the page. You should see the change implemented.
There are various options that you can set in your environment.rb file or similar. But you don't need any of these. Likewise you don't need to run any rake tasks or explicitly setup sass to watch certain files. It just works.
With this set up you can now get comfortable with Sass such that moving to 3.1 and the Asset Pipeline should be straightforward.
I am migrating a complex app from 3.0 to 3.1 and I want to use Sass in the 3.0 production version before I make the big jump to 3.1. So far I have been using Less and the more plugin.
I recently wrote up my experience moving from Less to Sass syntax but here I want to cover the Rails side of things.
Here are the steps involved in my Less to Sass transition in Rails 3.0.5
Part 1: Remove all traces of Less
1: Remove the less gem from your Gemfile
2: Rename the app/stylesheets folder to something else
3: Move or delete the Less plugin in vendor/plugins/more so that the app doesn't see it on start up
Part 2: Set up Sass
4: Add the sass gem to your Gemfile ( gem 'sass' )
5: Run 'bundle install'
6: Create a sass directory under public/stylesheets
7: Leave any straight css file in public/stylesheets
8: Place any SCSS files in the sass directory - these could be new files or converted Less files - give them the .scss suffix
9: Remove or rename any matching css files in the parent directory
10: Restart your server and browse to your site
All being well Sass has worked in the background, created new versions of the CSS files and put them in public/stylesheets. Look for those files and test out the operation by, say, changing a background color in the SCSS file, saving the file and reloading the page. You should see the change implemented.
There are various options that you can set in your environment.rb file or similar. But you don't need any of these. Likewise you don't need to run any rake tasks or explicitly setup sass to watch certain files. It just works.
With this set up you can now get comfortable with Sass such that moving to 3.1 and the Asset Pipeline should be straightforward.
Wednesday, July 4, 2012
Rails 3 YAML parse error and RedCloth
I moved a Rails 3.0.5 application to a new server with ruby 1.9.2, under rbenv. It had been working fine before but now I got this:
$ rails server
/Users/jones/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/psych.rb:148:in `parse': couldn't parse YAML at line 183 column 9 (Psych::SyntaxError)
from /Users/jones/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/psych.rb:148:in `parse_stream'
from /Users/jones/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/psych.rb:119:in `parse'
from /Users/jones/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/psych.rb:106:in `load'
from /Users/jones/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/RedCloth-4.2.2/lib/redcloth/formatters/latex.rb:6:in `<module:LATEX>'
from /Users/jones/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/RedCloth-4.2.2/lib/redcloth/formatters/latex.rb:3:in `<top (required)>'
[...]
There are a lot of posts about this error on the web. Some recommend specifying the 'syck' Yaml engine in the boot.rb file or messing with your libyaml setup. This did not work for me.
The solution turned out to be simple. You can see from the error message that the error is coming from the RedCloth gem. Previously I had been using version 4.2.2 (look in Gemfile.lock) even though I had not explicitly set that version.
On the new system I had version 4.2.9 installed. When I set this explicitly in the Gemfile I could start up the server just fine, after running 'bundle install'.
gem 'RedCloth', '>= 4.2.9'
Tuesday, June 12, 2012
CSS and Rails - Migrating from Less to Sass
In my Rails projects over the past couple of years I have been using Less to make my CSS files easier to work with. Less is a CSS pre-processor that lets you use variables for color, etc.
Less is pretty similar to Sass and I didn't see any real reason to move away from it. But then Rails 3.1 made the decision to uses Sass and CoffeeScript as CSS and JavaScript preprocessors. So now I want to migrate my Less files over to Scss (the v2 format of Sass) so I can go with the flow in current Rails versions.
Sass comes with a conversion tool called sass-convert that you can find in the bin directory of the gem.
You can find where that is located with 'gem which sass'. But this gave me this error:
NameError: undefined method `build' for module `Less::StyleSheet::Mixin4'
From the sass github site it seems like sass -convert has not kept up with changes in Less and there seems little motivation to fix the problem.
Never mind - the Less and Scss syntaxes are pretty similar and my files do not include anything too complex, so manually converting them in an editor is quite feasible. In fact the main difference is that Less prefixes variables with @ and Sass uses $. The important exception here is that @import should not be changed as it is exactly the same in Sass.
Less has an equivalent to Sass Mixins. You create a custom class and then insert that into other CSS blocks. Here is an example followed by the Sass equivalent.
.my-mixin {
color: red;
}
.my-class {
width: 200px;
.my-mixin;
}
In Sass you would write:
@mixin my-mixin {
color: red;
}
.my-class {
width: 200px;
@include my-mixin;
}
Making this conversion is easy enough to do with find and replace in your editor.
The easiest way to validate the modified files is to run sass on them and check the css.
$ sass my.scss my.css
You can try validating the CSS using a service like http://jigsaw.w3.org/css-validator/validator
Less is pretty similar to Sass and I didn't see any real reason to move away from it. But then Rails 3.1 made the decision to uses Sass and CoffeeScript as CSS and JavaScript preprocessors. So now I want to migrate my Less files over to Scss (the v2 format of Sass) so I can go with the flow in current Rails versions.
Sass comes with a conversion tool called sass-convert that you can find in the bin directory of the gem.
You can find where that is located with 'gem which sass'. But this gave me this error:
NameError: undefined method `build' for module `Less::StyleSheet::Mixin4'
From the sass github site it seems like sass -convert has not kept up with changes in Less and there seems little motivation to fix the problem.
Never mind - the Less and Scss syntaxes are pretty similar and my files do not include anything too complex, so manually converting them in an editor is quite feasible. In fact the main difference is that Less prefixes variables with @ and Sass uses $. The important exception here is that @import should not be changed as it is exactly the same in Sass.
Less has an equivalent to Sass Mixins. You create a custom class and then insert that into other CSS blocks. Here is an example followed by the Sass equivalent.
.my-mixin {
color: red;
}
.my-class {
width: 200px;
.my-mixin;
}
In Sass you would write:
@mixin my-mixin {
color: red;
}
.my-class {
width: 200px;
@include my-mixin;
}
Making this conversion is easy enough to do with find and replace in your editor.
The easiest way to validate the modified files is to run sass on them and check the css.
$ sass my.scss my.css
You can try validating the CSS using a service like http://jigsaw.w3.org/css-validator/validator
Friday, June 10, 2011
Rails meta_search - sorting on custom columns
I use Ernie Miller's excellent meta_search gem for searching the contents of tables in my Rails 3 Apps.
I have a lot of index views that are tables of data and each has a search form at the top of the page that uses meta search and the column headers use the sort links provided by meta_search to reorder the rows.
Typically I also have custom columns, such as the number of objects that are associated with a given row via some relationship. Some of those relationships can be quite complex and in some cases there is no simple way to express them in a way that works with meta_search.
I handle the sorting in the controller. It's outside the database but with my current application the amount of data makes this perfectly manageable.
I've come up with a way to let my custom column sorting coexist with meta_search and I reckon it might be useful to others.
All the code can be found at this gist on Github :https://gist.github.com/1019358
It works by adding another parameter called 'custom_sort' to the existing search[] parameters. The index action in the controller sees this, performs the custom sort and passes a custom_sort value to the view.
The view calls the custom_sort_helper with a set of arguments and that creates the column header link with the correct parameters to pass back to the controller.
So far it works well and allows me to sort on my custom columns with a set of rows selected by meta_search searches.
I'm sure the code could be more elegant - leave me a column and tell my how.
I have a lot of index views that are tables of data and each has a search form at the top of the page that uses meta search and the column headers use the sort links provided by meta_search to reorder the rows.
Typically I also have custom columns, such as the number of objects that are associated with a given row via some relationship. Some of those relationships can be quite complex and in some cases there is no simple way to express them in a way that works with meta_search.
I handle the sorting in the controller. It's outside the database but with my current application the amount of data makes this perfectly manageable.
I've come up with a way to let my custom column sorting coexist with meta_search and I reckon it might be useful to others.
All the code can be found at this gist on Github :https://gist.github.com/1019358
It works by adding another parameter called 'custom_sort' to the existing search[] parameters. The index action in the controller sees this, performs the custom sort and passes a custom_sort value to the view.
The view calls the custom_sort_helper with a set of arguments and that creates the column header link with the correct parameters to pass back to the controller.
So far it works well and allows me to sort on my custom columns with a set of rows selected by meta_search searches.
I'm sure the code could be more elegant - leave me a column and tell my how.
Tuesday, June 7, 2011
Rails send_data and UTF-8
I often use send_data to output plain text from a Rails action, typically as a simple way to download, say, a CSV format file.
I'll have something like this to output the text to the browser:
send_data(@output, :type => 'text/plain', :disposition => 'inline')
I'll have something like this to output the text to the browser:
send_data(@output, :type => 'text/plain', :disposition => 'inline')
But if the text is UTF-8 and contains non-ASCII characters then you need to specify the character set.
Confusingly, to me, you don't do this with a separate argument at the Rails level. Instead you add it as a modifier on the :type argument like this:
send_data(@output, :type => 'text/plain; charset=utf-8', :disposition => 'inline')
Friday, June 3, 2011
Rails migration add_column and Postgresql
Just ran into a difference between MySQL and Postgresql when running a migration that added a new column to an existing table.
The intent of the original line was to add a new boolean column called 'active' and set the default value to true:
add_column :people, :active, :boolean, :default => true
This works fine on MySQL but with Postgresql the default value was not used on existing records. It appears that Postgresql sees that NULL is an acceptable value for the column and uses that as the value.
The way to fix this is to add a 'null' attribute to the call like this:
add_column :people, :active, :boolean, :default => true, :null => false
The intent of the original line was to add a new boolean column called 'active' and set the default value to true:
add_column :people, :active, :boolean, :default => true
This works fine on MySQL but with Postgresql the default value was not used on existing records. It appears that Postgresql sees that NULL is an acceptable value for the column and uses that as the value.
The way to fix this is to add a 'null' attribute to the call like this:
add_column :people, :active, :boolean, :default => true, :null => false
Now, running rake db:migrate sets all existing records to true
Friday, March 4, 2011
simple-tooltip Ruby Gem for tooltip help
I'm pleased to announce the release of my first Ruby Gem: simple-tooltip
Simple Tooltip helps you make tooltip help available in your Rails 3 applications.
This is often done using a JavaScript plugin which produces a small popup window containing a short text string. That works great in many applications but often the help content that you want to provide is more extensive and you many want to include richer formatting, such as links to other web pages.
Simple Tooltip acts as a Rails Generator which will create a Tooltip model in your application along with a database table, a controller and views. You create help records using a web interface using plain text, html, Textile or Markdown formatting. The help records are stored in the database and retrieved using the unique, informative titles that you give them.
In your web pages you include a Tooltip helper that refers to the title of the relevant tooltip. Depending on how you set it up, when you hover over a tooltip icon or click on it, a popup window will appear displaying your help.
This is much richer and more flexible than the basic solutions available from JavaScript alone.
More than that, if the audience for your application is not just English-speaking, you can create multiple instances of each tooltip with content in each target language, identified by a two letter locale string. If a user has a non-English locale defined in your application, Simple Tooltip will look for content in the matching locale and display that. For content longer than single sentences, this is a more practical internationalization solution than I18n functionality built into Rails 3, which is more focused on shorter strings.
The gem is available at https://rubygems.org/gems/simple-tooltip.
The code is available at https://github.com/craic/simple_tooltip and is made freely available under the MIT license.
Simple Tooltip helps you make tooltip help available in your Rails 3 applications.
This is often done using a JavaScript plugin which produces a small popup window containing a short text string. That works great in many applications but often the help content that you want to provide is more extensive and you many want to include richer formatting, such as links to other web pages.
Simple Tooltip acts as a Rails Generator which will create a Tooltip model in your application along with a database table, a controller and views. You create help records using a web interface using plain text, html, Textile or Markdown formatting. The help records are stored in the database and retrieved using the unique, informative titles that you give them.
In your web pages you include a Tooltip helper that refers to the title of the relevant tooltip. Depending on how you set it up, when you hover over a tooltip icon or click on it, a popup window will appear displaying your help.
This is much richer and more flexible than the basic solutions available from JavaScript alone.
More than that, if the audience for your application is not just English-speaking, you can create multiple instances of each tooltip with content in each target language, identified by a two letter locale string. If a user has a non-English locale defined in your application, Simple Tooltip will look for content in the matching locale and display that. For content longer than single sentences, this is a more practical internationalization solution than I18n functionality built into Rails 3, which is more focused on shorter strings.
The gem is available at https://rubygems.org/gems/simple-tooltip.
The code is available at https://github.com/craic/simple_tooltip and is made freely available under the MIT license.
Tuesday, March 1, 2011
Passing Environment Variables to the nginx web server
I need to write this down for future reference...
With the Apache web server you can pass environment variables to Rails applications, etc by specifying them using SetEnv in your configuration file. But with nginx you don't do this - or at least you don't need to.
When you fire up the server, simply tell sudo to preserve your environment. Without that, sudo will reset all your variables. With it, they are available to nginx and your applications - nice and simple...
With the Apache web server you can pass environment variables to Rails applications, etc by specifying them using SetEnv in your configuration file. But with nginx you don't do this - or at least you don't need to.
When you fire up the server, simply tell sudo to preserve your environment. Without that, sudo will reset all your variables. With it, they are available to nginx and your applications - nice and simple...
$ sudo -E /usr/local/sbin/nginx
Thursday, February 24, 2011
Rails, UTF-8 and Heroku
I've had problems with Ruby character encodings over the years, especially when pulling text with non-ASCII characters in from remote sites. I thought I had it mostly sorted but the past few days showed me that was not the case.
I have a MySQL database on my local machine and a Rails 3 app that pulls in text from remote sources, stores it in the database and does stuff with it. I am deploying the application on Heroku prior to public release. Heroku uses PostgreSQL exclusively.
I was under the belief that all the components in my system were set up to use UTF-8 encoding and therefore moving text with non-ASCII characters around should be fine. But in practice that was not the case - characters like 'α' that looked fine on my local machine showed up as 'α' on Heroku, etc. So clearly I was doing something wrong. Rather than go into all the gory details, this is the way to do it right...
Bottom-line: Make everything use UTF-8 explicitly ... EVERYTHING
0: Backup your current database!
1: MySQL
Here are the contents of my /etc/my.cnf file:
Even if your tables are held in utf8, you should add these lines. You want the following mysql command to look as shown:
2: Rails
I'm using Rails 3 - can't tell you how this works in Rails 2.x
a: In config/application.rb make sure this line is uncommented:
b: In your database.yml, specify the encoding for the databases - for example:
c: mysql2
Notice that I am using the mysql2 adapter instead of mysql. At this point (Feb 2011) the mysql gem is NOT encoding aware. Replace mysql with mysql2 in your Gemfile and run bundle.
d: In each Model that uses text add this line at the very top of the file:
With all those components in place, you should be all set. Try entering non-ASCII characters into a form - such as accented characters or greek/math symbols. These should be displayed correctly in the browser and in the mysql command line client.
With regards to Heroku, assuming you have your app already set up, you should be able to do a 'heroku db:push' to copy the database into PostgreSQL on Heroku and the characters should display correctly on the remote pages. You will see reference to using 'heroku db:push' with explicit database URLs that include an encoding option, such as '?encoding=utf8'. If your MySQL is set up correctly then this should be unnecessary.
A critical part of running apps on Heroku is the ability to pull the database back to your local database using 'heroku db:pull'. Before getting all my components set up with utf-8, this step failed for me. With everything using utf-8, and after adding the 'max_allowed_packet' lines to my my.cnf file, this process works fine.
But because I was working with data in before everything was truly utf-8, I had some instances of text in the database that had been incorrectly encoded - and thereby effectively corrupted. I could see what the 'corrupt' characters looked like and I knew what the correct versions should be. Because everything is now using utf-8 I could simply do a substitution on the text. For example:
You run this in a rails console like this:
Character encodings are HARD - Yehuda Katz wrote a nice article on the issues. For most purposes (unless you work with Japanese text) UTF-8 is your best choice for encoding and so I'm using it exclusively. Java and Python both made the same choice and things are probably easier to set up in those worlds. Ruby has it's roots in Japan and so it is not surprising that it could not go down that path.
From now on, I'm going to make sure everything I touch is configured for UTF-8. There are fews reasons not to at this stage and it allows you to handle most languages.
I have a MySQL database on my local machine and a Rails 3 app that pulls in text from remote sources, stores it in the database and does stuff with it. I am deploying the application on Heroku prior to public release. Heroku uses PostgreSQL exclusively.
I was under the belief that all the components in my system were set up to use UTF-8 encoding and therefore moving text with non-ASCII characters around should be fine. But in practice that was not the case - characters like 'α' that looked fine on my local machine showed up as 'α' on Heroku, etc. So clearly I was doing something wrong. Rather than go into all the gory details, this is the way to do it right...
Bottom-line: Make everything use UTF-8 explicitly ... EVERYTHING
0: Backup your current database!
1: MySQL
Here are the contents of my /etc/my.cnf file:
[mysqld]
datadir=/usr/local/mysql/data
bind-address = 127.0.0.1
character-set-server = utf8
max_allowed_packet = 32M
[client]
default-character-set = utf8
[mysql]
max_allowed_packet = 32M
Even if your tables are held in utf8, you should add these lines. You want the following mysql command to look as shown:
mysql> SHOW VARIABLES LIKE 'character\_set\_%';
+--------------------------+--------+
| Variable_name | Value |
+--------------------------+--------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
+--------------------------+--------+
2: Rails
I'm using Rails 3 - can't tell you how this works in Rails 2.x
a: In config/application.rb make sure this line is uncommented:
# Configure the default encoding used in templates for Ruby 1.9.MySQL and Rails use different variants of utf8/utf-8 - make sure you are using the right one. And note the comment above this line - this sets up utf-8 encoding for templates ONLY.
config.encoding = "utf-8"
b: In your database.yml, specify the encoding for the databases - for example:
development:Here you are telling the database adapter that the database uses utf-8.
adapter: mysql2
host: localhost
encoding: utf8
[...]
c: mysql2
Notice that I am using the mysql2 adapter instead of mysql. At this point (Feb 2011) the mysql gem is NOT encoding aware. Replace mysql with mysql2 in your Gemfile and run bundle.
d: In each Model that uses text add this line at the very top of the file:
# encoding: UTF-8This tells Ruby that we're using utf-8 in this model. I don't see a way to set this at the application level so you have to have to add it in all relevant model .rb file. I also don't like defining something with a comment line. I can't see how to define this in, say, an irb interactive session.
With all those components in place, you should be all set. Try entering non-ASCII characters into a form - such as accented characters or greek/math symbols. These should be displayed correctly in the browser and in the mysql command line client.
With regards to Heroku, assuming you have your app already set up, you should be able to do a 'heroku db:push' to copy the database into PostgreSQL on Heroku and the characters should display correctly on the remote pages. You will see reference to using 'heroku db:push' with explicit database URLs that include an encoding option, such as '?encoding=utf8'. If your MySQL is set up correctly then this should be unnecessary.
A critical part of running apps on Heroku is the ability to pull the database back to your local database using 'heroku db:pull'. Before getting all my components set up with utf-8, this step failed for me. With everything using utf-8, and after adding the 'max_allowed_packet' lines to my my.cnf file, this process works fine.
But because I was working with data in before everything was truly utf-8, I had some instances of text in the database that had been incorrectly encoded - and thereby effectively corrupted. I could see what the 'corrupt' characters looked like and I knew what the correct versions should be. Because everything is now using utf-8 I could simply do a substitution on the text. For example:
str.sub!(/ß/, 'β')I gathered up the character mappings that I needed (which were not may in my case) and wrote up a class method that I cloned in each model with the issue. I then ran those in the Rails console to correct the bad characters. The method is:
def self.make_utf8_cleanThis looks at your model and figures out which columns are of type String. It goes through all records and all character mappings, replacing text and updating the database as needed. Your mappings array could be much larger. There may be a better source of these, but this is a start.
mappings = [ ['α', 'α'],
['ß', 'β'],
['β', 'β'],
['’', '’'],
['“', '“'],
['â€\u009D;', '”'],
['â€', '”'],
['ö', 'ö'],
['®', '®']
]
# Get the list of String columns
columns = Array.new
self.columns.each do |column|
if column.type.to_s == 'string'
columns << column
end
end
# Go through each object -> column against all mappings
self.all.each do |obj|
columns.each do |column|
mappings.each do |mapping|
value = obj.attributes[column.name]
if value =~ /#{mapping[0]}/
s = value.gsub(/#{mapping[0]}/, "#{mapping[1]}")
obj.update_attribute(column.name.to_sym, s)
end
end
end
end
end
You run this in a rails console like this:
Loading development environment (Rails 3.0.4)It's a hack but it helped my 'fix' quite a few records that would have been a pain to recreate.
ruby-1.9.2-p0 > YourModel.make_utf8_clean
Character encodings are HARD - Yehuda Katz wrote a nice article on the issues. For most purposes (unless you work with Japanese text) UTF-8 is your best choice for encoding and so I'm using it exclusively. Java and Python both made the same choice and things are probably easier to set up in those worlds. Ruby has it's roots in Japan and so it is not surprising that it could not go down that path.
From now on, I'm going to make sure everything I touch is configured for UTF-8. There are fews reasons not to at this stage and it allows you to handle most languages.
Tuesday, February 22, 2011
Setting up PostgreSQL on Mac OS X 10.6
I needed to set up PostgreSQL on my Mac in order to troubleshoot a problem with a Rails application.
Here are the steps that I followed:
1: Using Homebrew, install and build PostgreSQL
Homebrew will give you commands to create a Launch Agent that starts the server on a reboot
2: Setup PostgreSQL
You should look into setting administration user - I skipped that for my purposes
3: Create a specific database and verify that it is running
4: Install the PostgreSQL ruby gem
You'll see different names for this gem - just use 'pg'. Using the ARCHFLAGS env variable is important. I did not need to specify the explicit path to the Homebrew installation of the PostgreSQL software.
Here are the steps that I followed:
1: Using Homebrew, install and build PostgreSQL
Homebrew will give you commands to create a Launch Agent that starts the server on a reboot
$ brew install postgresql
$ cp /usr/local/Cellar/postgresql/9.0.3/org.postgresql.postgres.plist ~/Library/LaunchAgents
$ launchctl load -w ~/Library/LaunchAgents/org.postgresql.postgres.plist
2: Setup PostgreSQL
You should look into setting administration user - I skipped that for my purposes
$ initdb /usr/local/var/postgres
The files belonging to this database system will be owned by user "jones".
This user must also own the server process.
[...]
3: Create a specific database and verify that it is running
$ createdb mydb
$ psql mydb
mydb-# select version();
mydb-# \h
4: Install the PostgreSQL ruby gem
You'll see different names for this gem - just use 'pg'. Using the ARCHFLAGS env variable is important. I did not need to specify the explicit path to the Homebrew installation of the PostgreSQL software.
$ env ARCHFLAGS="-arch x86_64" gem install pg
Subscribe to:
Posts (Atom)
Subscribe To This Site
Posts
All Comments