This post could also be entitled “Oh Espresso machine, I love thee!”
StillAlive is a reinteractive application that monitors websites are operating as expected - not just making sure the front page comes up, but by using customer written test scripts that navigate around and log into their web applications every minute and alerting on downtime and errors.
Over the weekend, we upgraded our service to use Ruby 1.9.3 to take advantage of some new features and fixes in the Ruby Net library. We did the upgrade, fixed up three gems that needed to be changed to work with 1.9.3 and ran our test suite. Happily, all 3000+ specs and steps passed. We then did a dummy upgrade with a recent restore of the production database, again all looking good, and finally deployed to staging, again, everything 100% OK. This was looking to be a very simple upgrade!
However, after deploying to production, a couple of our clients reported that some of their recipes were failing due to an encoding related problem. Happily, after a couple of hours of hunting it down, I found that the Mechanize library had already patched this, but was not released as a gem version yet. Bundler usually makes this trivial to solve, first you fork the Mechanize library to your own repository (see Bundler and Public Git Sources) and then you just put something like this in your Gemfile:
# Fixing gemfile against fork until commit 1561fae6bd458e42a10d41e5a6a728b253aafe96 is
# rolled into Mechanize, probably at the 2.1.1 release, once we are at 2.1.1 revert back to the Gem
gem "mechanize", :git => 'git://github.com/reInteractive/mechanize.git',
:ref => 'f10d647e1ff653a11433a24e965b855b2ee49fed'
Note, when I am depending on a git reference as opposed to a published gem, I always comment my Gemfile. In 5 months it is good to know exactly why you were depending on a git source instead of the gem and know for sure if it is OK to revert.
However, in the case of Mechanize, there is no mechanize.gemspec checked into the git repository as it is autogenerated by their Rakefile. As my application depends on a version of Mechanize that is at least version 2.1.0, Bundler then complains that it can’t find a version of Mechanize at the git repository:
Could not find gem ‘mechanize (>= 0) ruby’ in git://github.com/reInteractive/mechanize.git (at f10d647).
Source does not contain any versions of ‘mechanize (>= 0) ruby’
The excellent Bundler docs explain how to solve this:
If a git repository does have a .gemspec for the gem you attached it to, a version specifier, if provided, means that the git repository is only valid if the .gemspec specifies a version matching the version specifier. If not, bundler will print a warning. If a git repository does not have a .gemspec for the gem you attached it to, a version specifier MUST be provided. Bundler will use this version in the simple .gemspec it creates.
So then the Gemfile line becomes:
# Fixing gemfile against fork until commit 1561fae6bd458e42a10d41e5a6a728b253aafe96 is
# rolled into Mechanize, probably at the 2.1.1 release, once we are at 2.1.1 revert back to the Gem
gem "mechanize", "2.1.0", :git => 'git://github.com/reInteractive/mechanize.git',
:ref => 'f10d647e1ff653a11433a24e965b855b2ee49fed'
But this won’t work by itself, because gemspecs also list dependencies which if present, Bundler will automatically require. Opening up the Rakefile in the Mechanize repository, I see:
self.extra_deps << ['net-http-digest_auth', '~> 1.1', '>= 1.1.1']
self.extra_deps << ['net-http-persistent', '~> 2.3', '>= 2.3.2']
self.extra_deps << ['nokogiri', '~> 1.4']
self.extra_deps << ['ntlm-http', '~> 0.1', '>= 0.1.1']
self.extra_deps << ['webrobots', '~> 0.0', '>= 0.0.9']
self.extra_deps << ['domain_name', '~> 0.5', '>= 0.5.1']
So we need to add these to the Gemfile as well:
# These next 6 gems (nokogiri, net-http-digest_auth, net-http-persistent, ntlm-http, webrobots, domain_name)
# are needed to be explicitly required because we are creating a dummy gemspec with mechanzie
# off the github repository.
gem 'nokogiri', '~> 1.5.0'
gem 'net-http-digest_auth', '~> 1.1.1'
gem 'net-http-persistent', '~> 2.3.2'
gem 'ntlm-http', '~> 0.1.1'
gem 'webrobots', '~> 0.0.9'
gem 'domain_name', '~> 0.5.1'
# Fixing gemfile against fork until commit 1561fae6bd458e42a10d41e5a6a728b253aafe96 is
# rolled into Mechanize, probably at the 2.1.1 release, once we are at 2.1.1 revert back to the Gem.
# Also remove the 5 dependent gems above
gem "mechanize", "2.1.0", :git => 'git://github.com/reInteractive/mechanize.git',
:ref => 'f10d647e1ff653a11433a24e965b855b2ee49fed'
Finally! Now doing a bundle install actually worked with no complaints.
Running the specs though had a bunch of errors coming up. Tracing this down, I found that Mechanize was being required explicitly in my code, and now that Mechanize was also in the Gemfile, bundler was also requiring it. In my case this was causing a problem. Bundler provides a simple way to solve this by adding :require => false making the final Gemfile entry look like this:
# These next 6 gems (nokogiri, net-http-digest_auth, net-http-persistent, ntlm-http, webrobots, domain_name)
# are needed to be explicitly required because we are creating a dummy gemspec with mechanzie
# off the github repository.
gem 'nokogiri', '~> 1.5.0'
gem 'net-http-digest_auth', '~> 1.1.1'
gem 'net-http-persistent', '~> 2.3.2'
gem 'ntlm-http', '~> 0.1.1'
gem 'webrobots', '~> 0.0.9'
gem 'domain_name', '~> 0.5.1'
# Fixing gemfile against fork until commit 1561fae6bd458e42a10d41e5a6a728b253aafe96 is
# rolled into Mechanize, probably at the 2.1.1 release, once we are at 2.1.1 revert back to the Gem.
# Also remove the 5 dependent gems above
gem "mechanize", "2.1.0", :require => false,
:git => 'git://github.com/reInteractive/mechanize.git',
:ref => 'f10d647e1ff653a11433a24e965b855b2ee49fed'
So there you have it. Pushed into production, this solved the issues for our clients. Everyone is happy… even my espresso machine.
