Docker FastPath: Only Build Your Images Once

Use your preferred Git workflow to achieve fast and error free environment promotion with Docker

TL;DR Deploy Docker FastPath in your build system.

One of the most useful piece of information that I learned from the classic Continuous Delivery by Humble and Farley (got get your copy now!) is to Only Build Your Binaries Once.

Many build systems use the source code held in the version control system as the canonical source for many steps. The code will be compiled repeatedly in different contexts […]. Every time you compile the code, you run the risk of introducing some difference[J. Humble, D. Farley, Continuous Delivery, p. 133]

You can notice that the book has been written in 2011 by the use of the word “compiled”. Fast forward in 2017, and Javascript-based technologies reach a staggering 85% preference in the last StackOverlow Developer Survey and dynamic languages are dominating web development. Unsurprisingly that advice is still valuable today: you just need to change “compile” with your preferred build method (npm install, bundle install, webpack, etc.).

In a study researchers found out that:

[…] typical circumstances under which a build breaks are missing referenced files, mistakenly checking in work-in-progress, and transitive dependencies.

Also, building again and again the same codebase is time expensive and it makes harder for developers to get timely feedback.

So, nowadays it’s even more necessary to “Only Build Your Binaries Once”. In a Docker development pipeline this translates into:

Only Build Your Images Once

Your typical Git + Docker Continuous Integration pipeline

The typical continuous integration process using Docker is:

• A code push to the git repository triggers the build process(Jenkins, TravisCI, GitLab, etc.)
• A Docker image is built, (tagged) and pushed to a Docker Registry
• The image is automatically tested
• The image is deployed on your favourite Docker orchestrator (Kubernetes, ECS, Swarm, etc.) in a testing environment
• The process is repeated for the following environments in the pipeline (staging, production, etc.)

Usually, you have more than one server environment. Environment labelling varies a lot; popular choices being development, qa, staging. When the goals for a given environment are fulfilled, there’s the need to promote the change to a higher environment.

The goal here is to push our artifact (the Docker image) from one environment to he next one. One possibility is to decouple this process from the code repository: the image promotion happens on an external system. This approach is not easy to integrate with common git workflows (Gitflow,GitHub Pull Requests, GitLab Merge Requests, something else), where the control stays in the hand of the developers. But, as Humble and Farley taught us, promoting at the code level is bad (and I agree).

Introducing Docker FastPath

I advocate here to keep control in the hand of developers by letting them use their preferred Git workflow and enjoy the power of only building their images once. The method is independent of the particular Git workflow in use and it is based on two simple rules:

1. Tag the image with the Git commit ID from which the image has been built.
2. Do not rebuild on code push if the codebase is identical to an existing image: deploy that image instead.

To apply rule number 2 it is necessary to examine the Git history and cross check the result with the Docker registry tag list for the image. Note that in most cases you can have two identical codebases with different Git commit IDs: so comparing Git IDs is not enough, you need to diff the two codebases. Docker FastPath does that.

Let’s see some examples.

Example #1: Merging a feature-branch

This is the default behaviour for GitHub Pull Requests. All commits from the feature branch are added to the base branch master using a merge commit (using the --no-ff option). In this case, commit 1044aee is identical to commit e51acc3 and it does not need to be rebuilt. Image tagged 1044aee can be deployed directly.

Example #2: Squash and merge

This is the behaviour for GitHub Pull Request when you select “Squash and merge”. All feature branch commits are squashed into a single commit and added to the base branch using Git fast forward. In this case, commit af49f3d is identical to commit e51acc3 and it does not need to be rebuilt. Image tagged e51acc3 can be deployed directly.

Example #3: Rebase and merge

This is the behaviour for GitHub Pull Request when you select “Rebase and merge”: all commits from the feature branch are added to the base branch, without a merge commit. In this case, commit 1eb55b3 is identical to commit e51acc3 and it does not need to be rebuilt. Image tagged e51acc3 can be deployed directly.

Try it

Docker FastPath code is available on GitHub as a command line tool. You can download an executable for MacOS or Linux which can be easily deployed on most build systems, but you can build it from source code if you wish.

Two deployment examples are provided: one using Jenkins, and another one using Travis CI. It should be easy to use those examples to adapt FastPath to your existing software project based on Docker.

If you enjoyed this piece click the “♥︎” button below. For more pieces on DevOps and Docker, join my mailing list.