DAVID RICE

Deployments on Heroku have gotten slower since the cedar stack, Rails 3.2 and the asset pipeline. Don’t get me wrong, I love all of these things like I love Street Fighter II, we’ve got over the novelty of new, just give me the turbo edition.

Benchmark

For the test, some things were kept controlled each time

• Emptied the S3 bucket (for asset_sync to fully run)
• No new bundled gems
• No migrations

So, using a stupidly basic Rails application (we use for testing releases of asset_sync) https://github.com/rumblelabs/asset_sync_test_3_2

git push heroku  0.02s user 0.01s system 0% cpu 1:27.61 total
git push heroku  0.02s user 0.01s system 0% cpu 1:28.91 total
git push heroku  0.02s user 0.02s system 0% cpu 1:34.35 total
git push heroku  0.02s user 0.02s system 0% cpu 1:28.45 total
git push heroku  0.02s user 0.01s system 0% cpu 1:26.65 total

=> [87.61, 88.91, 94.35, 88.45, 86.45]

AVERAGE: 89.154

Note: yes this app I am testing with is only “just” more complicated than rails new, so doing some timing on a completely fresh Heroku app would give a more accurate estimate.

Custom asset precompile task

My first idea was to only run rake assets:precompile:primary not for the faint hearted, but if you have built your application correctly to use digested assets throughout, it’s a no-brainer.

This ruby buildpack allows configuration of the task to execute for precompiling assets ENV['RAILS_ASSETS_PRECOMPILE_TASK'].

heroku config:add BUILDPACK_URL='https://github.com/rumblelabs/heroku-buildpack-ruby.git#custom_asset_precompile'
heroku config:add RAILS_ASSETS_PRECOMPILE_TASK=assets:precompile:primary

git push heroku  0.02s user 0.01s system 0% cpu 1:13.54 total
git push heroku  0.02s user 0.01s system 0% cpu 1:18.94 total
git push heroku  0.02s user 0.01s system 0% cpu 1:13.07 total
git push heroku  0.02s user 0.01s system 0% cpu 1:21.01 total
git push heroku  0.02s user 0.01s system 0% cpu 1:13.80 total

=> 73.54, 78.94, 73.07, 81.01, 73.80

AVGERAGE: 76.072

Woot! That’s an average saving of 13.082 seconds per deploy. Nice.

Note: I had to add an extra hack to get asset_sync working on deploy as we changed the precompile task.

Debug Buildpack

I was wondering if I could save even more time… so here’s a debug buildpack with a quick hack to output timings with every logged message.

heroku config:add BUILDPACK_URL='https://github.com/rumblelabs/heroku-buildpack-ruby.git#buildpack_debug'

The output looks like the following:

➜  myapp git:(master) git push heroku
Counting objects: 5, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 280 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)

-----> Heroku receiving push
-----> Fetching custom buildpack... done
-----> Ruby/Rails app detected
======> 2012-05-14 06:17:13 +0000
-----> 0.000107585 : Installing dependencies using Bundler version 1.2.0.pre
       0.073559081 : Running: bundle install --without development:test --path vendor/bundle --binstubs bin/ --deployment
       0.734531922 : Using rake (0.9.2.2)
       0.735997602 : Using i18n (0.6.0)
       0.736449923 : Using multi_json (1.3.4)
       0.737163615 : Using activesupport (3.2.0)
       0.737576858 : Using builder (3.0.0)
       0.738049721 : Using activemodel (3.2.0)
       0.738518515 : Using erubis (2.7.0)
       0.739293731 : Using journey (1.0.3)
       0.739688269 : Using rack (1.4.1)
       0.740488658 : Using rack-cache (1.1)
       0.740923127 : Using rack-test (0.6.1)
       0.741390205 : Using hike (1.2.1)
       0.742001929 : Using tilt (1.3.3)
       0.742994436 : Using sprockets (2.1.2)
       0.743355851 : Using actionpack (3.2.0)
       0.743878451 : Using mime-types (1.18)
       0.744305491 : Using polyglot (0.3.3)
       0.744781819 : Using treetop (1.4.10)
       0.74568785 : Using mail (2.4.1)
       0.747632709 : Using actionmailer (3.2.0)
       0.747760727 : Using arel (3.0.2)
       0.748128902 : Using tzinfo (0.3.32)
       0.748552448 : Using activerecord (3.2.0)
       0.74907975 : Using activeresource (3.2.0)
       0.749463748 : Using excon (0.13.4)
       0.749979571 : Using formatador (0.2.1)
       0.750377764 : Using net-ssh (2.3.0)
       0.750844938 : Using net-scp (1.0.4)
       0.751279192 : Using nokogiri (1.5.2)
       0.760296186 : Using ruby-hmac (0.4.0)
       0.76162912 : Using fog (1.3.1)
       0.778984194 : Using asset_sync (0.4.0) from git://github.com/rumblelabs/asset_sync.git (at master)
       0.779744754 : Using coffee-script-source (1.2.0)
       0.780276683 : Using execjs (1.3.0)
       0.78084057 : Using coffee-script (2.2.0)
       0.781371467 : Using rack-ssl (1.3.2)
       0.781935092 : Using json (1.6.6)
       0.782476196 : Using rdoc (3.12)
       0.783709004 : Using thor (0.14.6)
       0.784896558 : Using railties (3.2.0)
       0.785776908 : Using coffee-rails (3.2.2)
       0.786299563 : Using jquery-rails (2.0.1)
       0.787018031 : Using pg (0.13.2)
       0.787546051 : Using bundler (1.2.0.pre)
       0.788116001 : Using rails (3.2.0)
       0.788627757 : Using sass (3.1.15)
       0.790148688 : Using sass-rails (3.2.4)
       0.790633587 : Using uglifier (1.2.3)
       0.793843146 : Your bundle is complete! It was installed into ./vendor/bundle
       0.818492178 : Cleaning up the bundler cache.
-----> 2.714365124 : Writing config/database.yml to read from DATABASE_URL
-----> 4.776435204 : Preparing app for Rails asset pipeline
       4.77724766 : Running: rake assets:precompile
       22.609565246 : AssetSync: using default configuration from built-in initializer
       22.609565246 : AssetSync: using default configuration from built-in initializer
       22.609565246 : AssetSync: Syncing.
       22.609565246 : Using: Directory Search of /tmp/build_1w8jlj9e4aoaz/public/assets
       22.609565246 : AssetSync: Done.
-----> 22.63974359 : Rails plugin injection
       22.639908428 : Injecting rails_log_stdout
       22.875385302 : Injecting rails3_serve_static_assets
-----> Discovering process types
       Procfile declares types      -> (none)
       Default types for Ruby/Rails -> console, rake, web, worker
-----> Compiled slug size is 18.8MB
-----> Launching... done, v48
       http://myapp.herokuapp.com deployed to Heroku

To git@heroku.com:myapp.git
   d2b28d4..d0fc25e  master -> master

Looking at the output we can see that my custom buildpack timings only kick in within =====> CUSTOM BUILDPACK

➜  myapp git:(master) git push heroku
Counting objects: 5, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 280 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)

-----> Heroku receiving push
-----> Fetching custom buildpack... done
-----> Ruby/Rails app detected

=====> CUSTOM BUILDPACK

-----> Discovering process types
       Procfile declares types      -> (none)
       Default types for Ruby/Rails -> console, rake, web, worker
-----> Compiled slug size is 18.8MB
-----> Launching... done, v48
       http://myapp.herokuapp.com deployed to Heroku

To git@heroku.com:myapp.git
   d2b28d4..d0fc25e  master -> master

Conclusion

Taking my baseline measurements from earlier, we have “control” of about 23 Seconds of heroku deploy time out of the average of 89 Seconds measured. Therefore a saving of 13 Seconds is then pretty good.

That’s then putting an estimate that Heroku’s baseline deployment time is around 66 seconds.

So we may be able to cut our asset compilation time in half with that custom asset task buildpack. However with a baseline of 66 seconds spent “in Heroku” we can’t really get much faster than that without some help from Heroku, c’mon lads, lets get back to sub-minute deploys! As this is just the basics of an app, anything more complicated and you could easily be talking several minutes for a deploy.

Am going to implement some better statistics gathering of all our continuous deployments at Rumble. Would be very interested in hearing what other people’s average deploy times are.

Rails 4

As an aside. I recently started some work on a pull request to make rake assets:precompile faster by default in Rails 4 by making digested assets (rake assets:precompile:primary) the default and killing off the public directory entirely, moving all of those static files to app/assets.

Last year was a long year of upgrades for a lot of our clients at Rumble Labs. Going into this year, most of our Ruby work is all running on Ruby 1.9.2. A lot of those projects are Rails applications and we managed to have most of those running on Rails 3.0.x. From an OCD point of view, this pleases me.

Some of the applications we’re working on for clients are now pretty big, Rails is getting bigger and there are more libraries out there. The combined effects of which are starting to feel a little slow.

Ruby 1.9.3

Ruby 1.9.3 is the latest stable release of the 1.9.x series. It’s been great to start taking advantage of a few of the niceties in Ruby now since the move from 1.8.x.

With Ruby 1.9.3 it’s all about the speed. I tested running time rake -T before and after the upgrade in a number of our projects. Roughly speaking I saw a reduction of 50% in the time it took. This is great! If you use RVM (and I would recommend you do, or something like it) you can install with the following

rvm install 1.9.3

Bundler

A quick aside in case you haven’t heard. The latest beta version of bundler is super fast compared to the current stable release. You can install the latest version of bundler (in your global gemset) with the following.

rvm gemset use global
gem install bundler --pre

Rails 3.2.0.rc2

The latest version of Rails 3.2.0.rc2 has a few changes, the biggest of which is some performance improvements to boot time, development mode and asset compilation. For our applications running on 3.1 it was quite a painless upgrade (unlike going from 3.0.9 -> 3.1).

# add to your Gemfile
gem 'rails', '3.2.0.rc2'

# I like to create a new gemset for each specific Rails version
rvm gemset create rails-3-2-0-rc2
bundle update

Heroku

What would be the use of upgrading all these apps if we can’t try them out, in production! Thankfully it is possible to run both Ruby 1.9.3 and Rails 3.2.0.rc2 in production on Heroku thanks to their new Heroku Labs features, specifically their user_env_compile which allows us to set configuration that will be respected by the heroku build packs.

Note: not for the fainthearted Heroku mention they may remove any feature of Heroku Labs without warning, so be careful!

# Enable Heroku Labs
heroku plugins:install http://github.com/heroku/heroku-labs.git

# Enable the user_env_compile add-on (for each app)
heroku labs:enable user_env_compile -a myapp

# Enable 1.9.3
heroku config:add RUBY_VERSION=ruby-1.9.3-p0

That should be your Heroku app enabled to run on Ruby 1.9.3. Next time you push to heroku you should see something like the following in the compile output.

-----> Heroku receiving push
-----> Ruby/Rails app detected
-----> Using RUBY_VERSION: ruby-1.9.3-p0

Conclusion

I’ve upgraded about 10 apps so far and have yet to come across any showstopper bugs. It doesn’t take too long to do either. So, definitely worth the performance improvements you will gain! Happy upgrading.

So, I’ve recently joined up with the guys at Rumble Labs as their new Technical Director. Over the past two years I had been working self employed as a consultant for several local and international companies (Which was an enjoyable change from being a company director before that). However late last year as work was coming in thick and fast and I needed to collaborate with a few local freelancers more and more. I decided it was probably about time to start a another company. However at around about the same time Simon (from Rumble) asked if I’d be interested in joining the team.

It was pretty good timing as coincidentally our lovely old office building in Belfast’s Linen Quarter was being condemned (sad face). So we decided to share a new office together and knuckle down on a few collaborative projects. After a good few months of that, things were working well so we decided to start the long process of merging companies.

The paperwork was the easy part! Now we have to start all the technical side of things. For one, I’m very happy about being able to cross off a life goal of starting a software development hub in my home town!

The Rails 3.1.0 Release Candidate dropped a few days ago and thought I’d give it a try on an application I was busy upgrading already. Before starting you’ll probably want to check out the Release Notes or the railscasts overview video first.

1. Upgrade the Gemfile

source 'http://rubygems.org'

gem "rails", "3.1.0.rc1"
gem "mysql2", "0.3.2"
# Rails 3.1 - Asset Pipeline
gem 'json'
gem 'sass'
gem 'coffee-script'
gem 'uglifier'
# Rails 3.1 - JavaScript
gem 'jquery-rails'

# Rails 3.1 - Heroku
group :production do
  gem 'therubyracer-heroku', '0.8.1.pre3'
  gem 'pg'
end

Needed specifically the latest version of mysql2 as the earlier ones are incompatible. To get the asset pipeline working on heroku you’ll need therubyracer-heroku. Another issue with heroku, it appears you now need to be explicit about using postgres in production.

2. Config File Changes

Update config/boot.rb

require 'rubygems'

# Set up gems listed in the Gemfile.
ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../Gemfile', __FILE__)

require 'bundler/setup' if File.exists?(ENV['BUNDLE_GEMFILE'])

Update application.rb

In config/application.rb add the following line.

# Enable the asset pipeline
config.assets.enabled = true

Update development.rb

I seemed to have to remove the following line from config/environments/development.rb

config.action_view.debug_rjs             = true

Update production.rb

Add the following lines to enable asset compression in config/environments/production.rb.

# Compress both stylesheets and JavaScripts
config.assets.js_compressor  = :uglifier
config.assets.css_compressor = :scss

3. Move Assets

Move the Folders

mkdir app/assets
git mv public/images app/assets/images
git mv public/javascripts app/assets/javascripts
git mv public/stylesheets app/assets/stylesheets

Fix Image References

Use your favourite find and replace, check judiciously.

find: /images/
replace: /assets/

Make CSS and JavaScript Manifest Files

I already had an app/assets/application.js file so I added this to the top of mine and moved the code in there out into separate files with app/assets/javascripts.

// This is a manifest file that'll be compiled into including all the files listed below.
// Add new JavaScript/Coffee code in separate files in this directory and they'll automatically
// be included in the compiled file accessible from http://example.com/assets/application.js
// It's not advisable to add code directly here, but if you do, it'll appear at the bottom of the
// the compiled file.
//
//= require jquery
//= require jquery_ujs
//= require_tree .

I did the same for app/assets/application.css

/*
 * This is a manifest file that'll automatically include all the stylesheets available in this directory
 * and any sub-directories. You're free to add application-wide styles to this file and they'll appear at
 * the top of the compiled file, but it's generally better to create a new file per style scope.
 *= require_self
 *= require_tree . 
*/

Fix the CSS and JavaScript references

Instead of requiring multiple files you’ll want to require the manifest files created above.

<%= stylesheet_link_tag    "application" %>
<%= javascript_include_tag "application" %>

Success!

On an app with good front end conventions established, already using a method of asset bundling, etc. this isn’t a big task.

However I do find that because Rails has set such good conventions it’s hard for people to go too wrong. Of all the applications where I’ve taken over from another developer over the years. Rescue missions, etc. It’s the prospect of looking in the public/ directory that scares the bajaysus out of me.

Thankfully with all this great new front end stuff I’m hoping that in the future, rescue missions into interface land will suffer less casualties.