Setting up test:benchmark

I’m in the process of trying to establish some baseline performance metrics for increasing the performance of a website.  I thought I’d share what I learned.

The first place to start is Performance Testing Rails Applications from Rails Guides.  Everything else in this post assumes you started there (which is what I did).

Changing the host for the test

In the application I’m testing (and in a lot of other ones that I’ve worked on) the routing is conditionally different based on the host of the request (that’s a whole ‘nother blog post for directions on how to do this).  If you want to change the host for the request, its:

def setup
  host! 'yourproductionhost.com'
end

Setting up the test environment

The guide told me that tests were run in development mode.  In my experience this isn’t the case.  Instead, my tests were being run in the test environment.  For performance tests its even more important then normal to have a baseline of data you wish to test against.  Otherwise your tests are not apples-to-apples comparisons as you improve the performance of your app.

Unfortunately, in the case of a large, complex datasets necessary to do performance testing this can be a PITA to setup.  I’m using Factory Girl to make it easier.

That still leaves the problem of how to setup a test environment before performance testing.  In my case, which won’t be the same as every other case, I want to load up a whole bunch of data to run a number of tests against.  It take a couple of minutes to load all of this data and none of my tests change the data.  Test::Unit doesn’t support setting up fixture data once per-TestCase, only once per test.  But what I want is to be able to setup and teardown my data once for all the tests.

This little module did the trick.  Remember to require it at the top of test/test_helper.rb and include it in the actual TestCase you want to use it in.  Rspec, BTW, does this already with “setup :all,” but I’m unfortunately not using Rspec on this project.

module Test::SetupOnce
  module ClassMethods
    def setup_once; end
    def teardown_once; end

    def suite
      returning super do |suite|
        suite.send(:instance_variable_set, :@_test_klazz, self)
        suite.instance_eval do
          def run(*args)
            @_test_klazz.setup_once
            super
            @_test_klazz.teardown_once
          end
        end
      end
    end
  end

  def self.included(test)
    test.extend(ClassMethods)
  end
end

One last note on this, if you load data in the setup_once class method, you need to turn off fixture loading from Rails or your data will get destroyed.  Make sure and cleanup after yourself in teardown_once.  To kill the fixtures for your TestCase, redefine the setup_fixtures and teardown_fixtures methods to be no-ops.

Pulling it all together

So here’s the test:benchmark TestCase with all the features mentioned.

require 'test_helper'
require 'performance_test_help'

class BenchmarkTest < ActionController::PerformanceTest
  include Test::SetupOnce

  # don't do anything with fixtures so the data sticks around
  def setup_fixtures; end
  def teardown_fixtures; end

  def self.setup_once
    # here's where you put your data to share across tests
  end

  def self.teardown_once
    # here's where you cleanup after yourself
  end

  def setup
    host! 'yourproductionhost.com'
  end

  def test_benchmark
    get '/'
  end

  def test_benchmark2
    get '/foo'
  end
end

One last note, the GC patch that is mentioned in the benchmarking guide gives some great info and is well worth it. I was only able to patch Ruby 1.8.6 p111, though. I would love to be able to patch the same version of ruby I’m running in production. Anyone know of other patches that reveal the same information that are portable to newer Ruby versions?