Skip to content
By James McGrath

Background Job Processing 101

If you have a web application of modest complexity, sometimes you will want your app to perform a potentially long running or unreliable task initiated by your users. At the same time, you want your web application to always respond quickly and reliably to requests from your user's web browser. What is the solution?

Asynchronous Processing

A good technique to ensure quick and reliable responses from your web application is to identify any potentially long running or unreliable actions and to queue them up for processing by a background worker like SideKiq. This is also known as processing asynchronously.

This is a good technique for dealing with potentially unreliable processes (such as communicating with a third party). If the job is processed into a background queue, if it fails to complete it can remain stored in the queue until it can be rerun successfully.

A real life example

Let's look at a real life example of pushing a collection of images to Facebook as a photo album. This is an ideal candidate for background processing because it can take a long time and can be unreliable (imagine a network issue, or a Facebook outage).

In this example, a user selects 20 x 10MB images that currently reside in a private s3 bucket. They click "publish to Facebook. Now, our application needs to send 200MB of data to Facebook. Even with the high bandwidth that S3 and Facebook bring, transferring 200MB can take a while, and when every millisecond counts we should look for a better way.

So, instead of processing the photo transfer job in the request (synchronously), we create a set of background jobs, one for each image, and place them on a queue. The application can then continue fulfilling the browser request and respond to the user telling them "the photos will be published to Facebook soon". Now the background worker swings into action. It is constantly monitoring the queue for new jobs and when one arrives it picks up the job and starts pushing the photos out to the Facebook API. If something goes wrong (e.g. Facebook is down, network issues) the job can be retried in the future until it is successful. Because we created a separate job for each image, if any particular image upload fails (perhaps a single image is too large and is rejected by the Facebook API), it won't stop the publishing of the other images. In addition to this, we won't waste time retrying the upload of images that have already been successfully transmitted.

Application Simplification

One of the benefits from this approach is that it also simplifies our code as we can take advantage of features of the more advanced background job systems. Handling Facebook image upload failures in the request would be quite complex as we would need to inform the user about which images failed to upload and which were successful. We would have to ask the user to retry the upload, or perhaps we would create our own hand-rolled job solution for tracking past failures and kicking off retries. And this is just for one example, if we introduce another long running task, would we be able to easily abstract our solution for use in new situations? Much better to use someone else's well designed and battle hardened code.

We use this architecture constantly

So, having a background worker in the system can improve perceived performance and reliability when communicating to outside services. At reinteractive, we use this architecture in almost every application that we work on and it is considered best practice in the industry. i.e. we really believe in it and so do many others.

This is a very powerful tool for improving your applications performance and reliability and most systems can benefit from it. Once you have introduced this concept into your system, you'll find yourself identifying all sorts of places it becomes useful.

Popular Articles by Our Team

Our expert team of designers and developers love what the do and enjoy sharing their knowledge with the world.

We Hire Only the Best

reinteractive is Australia’s largest dedicated Ruby on Rails development company. We don’t cut corners and we know what we are doing.

We are an organisation made up of amazing individuals and we take pride in our team. We are 100% remote work enabling us to choose the best talent no matter which part of the country they live in. reinteractive is dedicated to making it a great place for any developer to work.

Free Community Workshops

We created the Ruby on Rails InstallFest and Ruby on Rails Development Hub to help introduce new people to software development and to help existing developers hone their skills. These workshops provide invaluable mentorship to train developers, addressing key skills shortages in the industry. Software development is a great career choice for all ages and these events help you get started and skilled up.

  • Webinars


    Webinars are our online portal for tips, tricks and lessons learned in everything we do. Make the most of this free resource to help you become a better developer.

    Learn more about webinars

  • Installfest


    The Ruby on Rails Installfest includes a full setup of your development environment and step-by-step instructions on how to build your first app hosted on Heroku. Over 1,800 attendees to date and counting.

    Learn more about Installfest

  • Development Hub

    Development Hub

    The Ruby on Rails Development Hub is a monthly event where you will get the chance to spend time with our team and others in the community to improve and hone your Ruby on Rails skills.

    Learn more about Development Hub

Get the “reinteractive Review” Monthly Email