Test coverage is a wonderful thing! It gives us confidence when developing new features and alerts us when we’ve accidentally affected a seemingly unrelated area of an application.
Simplecov is just one of the many tools we use on top of our standard tests/specs in order to track and maintain high test coverage. However, it doesn’t inspire confidence when you’re receiving inconsistent results between your local environment and the Continuous Integration (CI) service.
For more information on how to set up Simplecov for your app, please take a look at: Code Coverage for Ruby on Rails Projects
The problem:
There was a curious situation occuring with one of our applications. The CI output for Simplecov was quite different to what a developer would see when running their test suite locally. CI was reporting 10-20% above what the local run was reporting with discrepancies in the lines-of-code counts.
CI:
Finished in 7.22 seconds (files took 3.47 seconds to load)
152 examples, 0 failures
Coverage report generated for RSpec to myapp/coverage.
819 / 891 LOC (91.91%) covered
Local:
Finished in 6.13 seconds (files took 3.18 seconds to load)
152 examples, 0 failures
Coverage report generated for RSpec to myapp/coverage.
764 / 953 LOC (80.17%) covered.
Take note of the LOC for each of the above.
Finding the cause:
The first step was to work out how the coverage report differed between local and CI, and for that, we needed to fetch the generated coverage files from the CI provider. This involved some creative commands to compress and transfer the files to an external storage space after the test run had completed.
Upon inspecting the coverage output, it seemed as though there were files that were reporting 0% coverage locally, but anywhere from 10% to 100% on the CI copy of the coverage report. This also meant that SOMETHING was loading all the files in the app subfolders somewhere.
Upon inspecting config/environments/test.rb, this configuration was spotted:
config.eager_load = ENV['CI'].present?
Hold on … we don’t set ENV['CI'] locally! However, it does contain a value within the CI instances. So locally this would result in, config.eager_load = false - we have found a likely culprit for the coverage discrepancy.
A note about config.eager_load
When config.eager_load is set to true (which is the default in the production environment), Rails will load all the classes and modules in the eager load paths specified in the application. This is done to ensure that all the code is loaded and available when the application starts, reducing the time it takes to handle requests. In a test or development environment this is usually not necessary and can lead to undesired side effects while you’re working on your app.
Verification:
After changing the eager_load config in config/environments/test.rb to true locally, and re-running the test suite:
819 / 891 LOC (91.91%) covered
We now have the exact same LOC output as CI! Culprit identified!
Solution:
We don’t actually want to eager_load the classes during the test run, as this will lead to a false coverage count, but we still want to eager_load the application to ensure we haven’t got any broken code, so the following changes were applied:
# config/environments/test.rb
config.eager_load = false
Within the CI configuration:
- bundle exec rails zeitwerk:check
- bundle exec rspec
- OTHER_CHECKS HERE
The rails zeitwerk:check task will initialize the application with config.eager_load = true and report any errors.
Following this change the coverage output from Local and CI both matched! We can now enjoy accurate code coverage output both locally and within CI.
YMMV: In this case, the boot time and the test suite for the application is quite fast. For larger applications, this may have a noticeable effect on your build times. If that’s the case, you should explore other optimisations to speed up your CI builds.
