Setting up test:benchmark

I’m in the process of trying to establish some baseline performance metrics for increasing the performance of a website.  I thought I’d share what I learned.

The first place to start is Performance Testing Rails Applications from Rails Guides.  Everything else in this post assumes you started there (which is what I did).

Changing the host for the test

In the application I’m testing (and in a lot of other ones that I’ve worked on) the routing is conditionally different based on the host of the request (that’s a whole ‘nother blog post for directions on how to do this).  If you want to change the host for the request, its:

def setup
  host! 'yourproductionhost.com'
end

Setting up the test environment

The guide told me that tests were run in development mode.  In my experience this isn’t the case.  Instead, my tests were being run in the test environment.  For performance tests its even more important then normal to have a baseline of data you wish to test against.  Otherwise your tests are not apples-to-apples comparisons as you improve the performance of your app.

Unfortunately, in the case of a large, complex datasets necessary to do performance testing this can be a PITA to setup.  I’m using Factory Girl to make it easier.

That still leaves the problem of how to setup a test environment before performance testing.  In my case, which won’t be the same as every other case, I want to load up a whole bunch of data to run a number of tests against.  It take a couple of minutes to load all of this data and none of my tests change the data.  Test::Unit doesn’t support setting up fixture data once per-TestCase, only once per test.  But what I want is to be able to setup and teardown my data once for all the tests.

This little module did the trick.  Remember to require it at the top of test/test_helper.rb and include it in the actual TestCase you want to use it in.  Rspec, BTW, does this already with “setup :all,” but I’m unfortunately not using Rspec on this project.

module Test::SetupOnce
  module ClassMethods
    def setup_once; end
    def teardown_once; end

    def suite
      returning super do |suite|
        suite.send(:instance_variable_set, :@_test_klazz, self)
        suite.instance_eval do
          def run(*args)
            @_test_klazz.setup_once
            super
            @_test_klazz.teardown_once
          end
        end
      end
    end
  end

  def self.included(test)
    test.extend(ClassMethods)
  end
end

One last note on this, if you load data in the setup_once class method, you need to turn off fixture loading from Rails or your data will get destroyed.  Make sure and cleanup after yourself in teardown_once.  To kill the fixtures for your TestCase, redefine the setup_fixtures and teardown_fixtures methods to be no-ops.

Pulling it all together

So here’s the test:benchmark TestCase with all the features mentioned.

require 'test_helper'
require 'performance_test_help'

class BenchmarkTest < ActionController::PerformanceTest
  include Test::SetupOnce

  # don't do anything with fixtures so the data sticks around
  def setup_fixtures; end
  def teardown_fixtures; end

  def self.setup_once
    # here's where you put your data to share across tests
  end

  def self.teardown_once
    # here's where you cleanup after yourself
  end

  def setup
    host! 'yourproductionhost.com'
  end

  def test_benchmark
    get '/'
  end

  def test_benchmark2
    get '/foo'
  end
end

One last note, the GC patch that is mentioned in the benchmarking guide gives some great info and is well worth it. I was only able to patch Ruby 1.8.6 p111, though. I would love to be able to patch the same version of ruby I’m running in production. Anyone know of other patches that reveal the same information that are portable to newer Ruby versions?

Sugary new rspec syntax

A new version of rspec was just released and I love this bit of syntactic sugar (taken right out of the blog post):

describe Person do
  describe "born 19 years ago" do
    subject { Person.new(:birthdate => 19.years.ago }
    it { should be_eligible_to_vote }
    it { should be_eligible_to_enlist }
    it { should_not be_eligible_to_drink }
  end
end

The best part is that its a serious improvement to a very common use-case.  Kudos for the positive changes.  This is great.

Protoload is great

Just wanted to give a shout-out to a little javascript tool that I include in pretty much every project.  Protoload lets you mark an element as “waiting” while your AJAX is doing its bit.  It works with prototype and extends Element to add startWaiting and stopWaiting methods.

Its as simple as:

$('foo').startWaiting();

or in a slightly bigger example:

var Updater = Class.create({
    initialize: function() {
        $('button').observe('click', this.click.bindAsEventListener(this));
    },

    click: function(event) {
        $('form').startWaiting();

        new Ajax.Request('/props', {
            paramaters: { protoload: 'is cool!' },
            onSuccess: function() {
                $('form').stopWaiting();
            }
        });
    }
});

Your element will be covered in an animated gif prompting the user to wait until its ok to continue.  Works in all browsers.

It also works great to stop double-clicks on form submits in non-AJAX scenarios.

Thanks Andreas.

Working with large (thus slow) assets

You have a website with fantastic images that are big, plentiful, and full of color.  Like a fashion website, for example.  Or maybe you’ve decided that all the text on your website needs to be in a non-web font–as images–and now you have hundreds or thousands of image files to be deployed.

What to do so as to avoid heinous deploy times robbing your project of momentum and preventing following the credo of good development, “release early, release often?”

Make a separate repo just for your assets.

To accomplish this, you need the repo, you need a cap recipe and callback-hook for doing these deployments on demand (you wouldn’t want to do them every time, or you’re defeating the whole point), and you need a couple symbolic links.

STEP ONE: MAKE A REPO

Just follow Github’s directions for starting a new repo.  Once you’re all done, you’ll need to put your assets into the directory and make your first push.  Since there’s lots of these assets (that’s the whole reason you’re doing this project), it might take a while.

STEP TWO: CAP RECIPE AND CALLBACK-HOOK

For my project, and probably for yours, its important to keep separate asset repos for each environment, in the same way that you’d keep separate code repos for each environment. Are you already using capistrano multistage? I am, so these directions assume that piece. But you should be able to figure it out if you’re not.

config/deploy/staging.rb:

set :statics_deploy_to, Proc.new { "#{deploy_to}/statics-staging" }

config/deploy/production.rb:

set :statics_deploy_to, Proc.new { "#{deploy_to}/statics-production" }

Some thought went into the placement of the directory. Should it mimic the releases structure of a normal rails deployment? I decided that for my purposes it made the most sense to have a directory which didn’t change with releases such that updates could literally be git pulls instead of whole new clones. Since the file sizes are so large, I do want to avoid doing a complete deployment as much as possible.

And now your special recipe … of note here, the command variable inspiration was from the update_repository_cache method in capistrano’s recipes/deploy/strategy/remote_cache.rb and the run block was stolen from scm_run in capistrano’s recipes/deploy/strategy/remote.rb. I tried to use a slightly more abstracted API, Strategy, instead of using the Git scm API directly, but couldn’t get things to work (and since this already did, I didn’t try too hard).

lib/recipes/statics.rb:

set :statics_symlink_target, Proc.new { "#{current_path}/public/images" }
set :statics_repository, Proc.new { "git@github.com:USERNAMEHERE/PROJECTNAMEHERE-statics.git" }
set :statics_branch, 'master'

after 'deploy:symlink', 'statics:symlink'

namespace :statics do
  desc "Deploy the statics"
  task :deploy do
    gitter =
      Capistrano::Deploy::SCM::Git.new(:repository => statics_repository,
                                       :branch => statics_branch)

    # figure out the latest version of the code
    revision =
      gitter.query_revision(statics_branch) { |cmd| run_locally(cmd) }

    # clone or fetch depending on what we're up to
    command = "if [ -d #{statics_deploy_to} ]; then " +
      "#{gitter.sync(revision, statics_deploy_to)}; " +
      "else #{gitter.checkout(revision, statics_deploy_to)}; fi"

    run(command) do |ch,stream,text|
      ch[:state] ||= { :channel => ch }
      output = gitter.handle_data(ch[:state], stream, text)
      ch.send_data(output) if output
    end
  end

  desc "Dynamically link the statics into place"
  task :symlink do
    run "ln -nfs #{statics_deploy_to} #{statics_symlink_target}"
  end
end

STEP THREE: SYMBOLIC LINKS

There’s already a symbolic link that automatically gets made on the server when you deploy to connect your static assets to your code. But you have to do one locally, too. I’ll leave that as an exercise for the reader (hint, its pretty much the exact same ln -nfs line that’s in the recipe).

AND FINALLY

future-book:~/Sites/clientservice matt$ cap production statics:deploy deploy

GOTCHA: Github only allows one deploy key per-repo and now the same account on your hosting server needs to access two repos.  My approach was to setup the deploy key connected to the account of the repo owner, not the actual repo itself.  Multiple Github Accounts is another approach.