Test coverage is a wonderful thing! It gives us confidence when developing new features and alerts us when we’ve accidentally affected a seemingly unrelated area of an application.
Simplecov is just one of the many tools we use on top of our standard tests/specs in order to track and maintain high test coverage. However, it doesn’t inspire confidence when you’re receiving inconsistent results between your local environment and the Continuous Integration (CI) service.
For more information on how to set up Simplecov for your app, please take a look at: Code Coverage for Ruby on Rails Projects
There was a curious situation occuring with one of our applications. The CI output for Simplecov was quite different to what a developer would see when running their test suite locally. CI was reporting 10-20% above what the local run was reporting with discrepancies in the lines-of-code counts.
Finished in 7.22 seconds (files took 3.47 seconds to load) 152 examples, 0 failures Coverage report generated for RSpec to myapp/coverage. 819 / 891 LOC (91.91%) covered
Finished in 6.13 seconds (files took 3.18 seconds to load) 152 examples, 0 failures Coverage report generated for RSpec to myapp/coverage. 764 / 953 LOC (80.17%) covered.
Take note of the LOC for each of the above.
Finding the cause:
The first step was to work out how the coverage report differed between local and CI, and for that, we needed to fetch the generated coverage files from the CI provider. This involved some creative commands to compress and transfer the files to an external storage space after the test run had completed.
Upon inspecting the coverage output, it seemed as though there were files that were reporting 0% coverage locally, but anywhere from 10% to 100% on the CI copy of the coverage report. This also meant that SOMETHING was loading all the files in the
app subfolders somewhere.
config/environments/test.rb, this configuration was spotted:
config.eager_load = ENV['CI'].present?
Hold on … we don’t set
ENV['CI'] locally! However, it does contain a value within the CI instances. So locally this would result in,
config.eager_load = false - we have found a likely culprit for the coverage discrepancy.
A note about
config.eager_load is set to
true (which is the default in the production environment), Rails will load all the classes and modules in the eager load paths specified in the application. This is done to ensure that all the code is loaded and available when the application starts, reducing the time it takes to handle requests. In a
development environment this is usually not necessary and can lead to undesired side effects while you’re working on your app.
After changing the
eager_load config in
true locally, and re-running the test suite:
819 / 891 LOC (91.91%) covered
We now have the exact same LOC output as CI! Culprit identified!
We don’t actually want to
eager_load the classes during the test run, as this will lead to a false coverage count, but we still want to
eager_load the application to ensure we haven’t got any broken code, so the following changes were applied:
# config/environments/test.rb config.eager_load = false
Within the CI configuration:
- bundle exec rails zeitwerk:check - bundle exec rspec - OTHER_CHECKS HERE
rails zeitwerk:check task will initialize the application with
config.eager_load = true and report any errors.
Following this change the coverage output from Local and CI both matched! We can now enjoy accurate code coverage output both locally and within CI.
YMMV: In this case, the boot time and the test suite for the application is quite fast. For larger applications, this may have a noticeable effect on your build times. If that’s the case, you should explore other optimisations to speed up your CI builds.