Jason Wilder's Blog

gravatar

Software developer and architect interested in scalability, performance and distributed systems.

Copyright © 2017

Squashing Docker Images

A common problem when building docker images is that they can get big quickly. A base image can be a tens to hundreds of MB in size. Installing a few packages and running a build can easily create a image that is a 1GB or larger. If you build an application in your container, build artifacts can stick around and end up getting deployed.

Large images are problematic when you start publishing images to a registry. More layers creates more requests and larger layers take longer to transfer. Unfortunately, deleting things in later layers does not actually remove them from the image due to the way AUFS layers work.

There are a few options to address this problem but this post will show you how you can squash your images to make them smaller without requiring big changes to your development and deployment workflow.

Other Solutions

Other people have written about this problem as well and have tried different solutions to the problem.

Using Small Base Images

A few strategies rely on starting with very small base images. For example, you could use buildroot and craft a barebones image. There is also the very small scratch base image that could be a starting point. Another strategy is to install a binary package into your container. If you’re using Go, building static binaries might be an option.

These options are pretty sophisticated and might work for you but they may not fit your development workflow easily. While deploying static binaries would work well for Go projects, it can be complicated if you have Python, Node or Ruby projects that may just wrap C libraries.

Publishing Tools

Another set of options out there are separate tools for modifying and creating new images from existing images. There is a python script, docker-flatten, docker-compile.pl and docker-rebase.rb as well as just runing docker export <id> | docker import -.

Unfortunately, I wasn’t able to get any of these tools to work and the docker export trick loses Dockerfile attributes such as PORT, VOLUMES, etc. which causes other problems.

Official Support

Fortunately, docker appears to be aware of the problem with large images so hopefully this problem won’t require custom tools to solve. There is already a docker squash pull request (4232), a Dockerfile syntax change proposal (7115), a squash build dependencies proposal (6906) as well as a flatten images proposal (332).

Hopefully one or more of these will address the problem in the future.

docker-squash

Since this is a problem that affects me currently, I created a tool to squash images before pushing them to a registry. docker-squashis a standalone Go application that works similarly to the idea described in 332. It’s intended to be used as a publishing tool in your workflow and would be run before pushing to a registry.

The way it works is that you save, squash and load an image with something like docker save <ID> | docker-squash -t <TAG> [-from <ID>] | docker load.

The resulting image has all of the layers beneath the initial FROM layer squashed into a single layer. The other layers defining PORT, etc.. are retained as well.

The default options retains the base image layer so that it does not need to be repeatedly transferred when pushing and pulling updates to the image.

Example

I have a simple Go test image called jwilder/whoami that I’ll use as an example. When you run it, it listens on a port 8080 and returns the hostname of the container over HTTP.

Starting Image

Viewing it’s history shows that it’s pretty big (423.7MB) for just simple 20 line Go app.

$ docker images jwilder/whoami
REPOSITORY          TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
jwilder/whoami      latest              63e174c2ca3d        29 minutes ago       423.7 MB

Viewing the history shows how the size is broken down between the layers:

$ docker history jwilder/whoami:latest
IMAGE               CREATED             CREATED BY                                      SIZE
63e174c2ca3d        14 seconds ago      /bin/sh -c #(nop) CMD [/app/http]               0 B
e4eea4411c00        14 seconds ago      /bin/sh -c #(nop) EXPOSE map[8000/tcp:{}]       0 B
c50f2b65cab3        14 seconds ago      /bin/sh -c #(nop) ENV PORT=8000                 0 B
589338fba5eb        15 seconds ago      /bin/sh -c go build -o http                     7.031 MB
651626d6e364        15 seconds ago      /bin/sh -c #(nop) WORKDIR /app                  0 B
8dfc0bb00563        16 seconds ago      /bin/sh -c #(nop) ADD dir:78239d85b32dd28e4cb   21.8 kB
fc294d2b22cb        17 seconds ago      /bin/sh -c apt-get update && apt-get install    191.3 MB
c4ff7513909d        3 days ago          /bin/sh -c #(nop) CMD [/bin/bash]               0 B
cc58e55aa5a5        3 days ago          /bin/sh -c apt-get update && apt-get dist-upg   32.67 MB
0ea0d582fd90        3 days ago          /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/   1.895 kB
d92c3c92fa73        3 days ago          /bin/sh -c rm -rf /var/lib/apt/lists/*          0 B
9942dd43ff21        3 days ago          /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic   194.5 kB
1c9383292a8f        3 days ago          /bin/sh -c #(nop) ADD file:c1472c26527df28498   192.5 MB
511136ea3c5a        14 months ago

Breaking this down:

  • 225.4MB - 511136ea3c5a..c4ff7513909d is the ubuntu:14.04 base image.
  • 191.3MB - fc294d2b22cb is installing the Go SDK (golang-go)
  • 7MB - 589338fba5eb builds my app

Clean Up

I don’t want to ship the Go SDK with my image. There is also a bunch of left-over apt-get cache data, as well as some extra packages I don’t need. I’ll remove them in a new container.

$ docker run -it jwilder/whoami:latest /bin/bash
root@5fe8a50718c3:/app# apt-get purge -y man  perl-modules vim-common vim-tiny \
> libpython3.4-stdlib:amd64 python3.4-minimal xkb-data \
> libx11-data eject python3 locales golang-go
...
$ root@5fe8a50718c3:/app# apt-get clean autoclean
$ root@5fe8a50718c3:/app# apt-get autoremove -y
$ root@5fe8a50718c3:/app# rm -rf /var/lib/{apt,dpkg,cache,log}/
$ root@5fe8a50718c3:/app# exit

Squash The Image

Next I need to create a image from that container:

$ docker commit 5fe8a50718c3
49b5a7a88d5353fe77204ad5591a3ef100fc2807a9d6dce979fd1b17a73a68d6

Then I’ll save, squash and load it. I’m tagging the new image with -t jwilder/whoami:squash:

$ docker save 49b5a7a88d5 | sudo docker-squash -t jwilder/whoami:squash | docker load

If you run docker-squash with the -verbose option, you can see what it’s actually doing to the image.

$ docker save 49b5a7a88d5 | sudo docker-squash -t jwilder/whoami:squash -verbose | docker load
Loading export from STDIN using /tmp/docker-squash683466637 for tempdir
Loaded image w/ 15 layers
Extracting layers...
  -  /tmp/docker-squash683466637/49b5a7a88d5353fe77204ad5591a3ef100fc2807a9d6dce979fd1b17a73a68d6/layer.tar
  -  /tmp/docker-squash683466637/651626d6e364ccc22ac990ba95cd0aab9256c56055087cc9a5a1790cea5250b9/layer.tar
  -  /tmp/docker-squash683466637/c50f2b65cab3b74f9bdb6f616b36f132b9a182ed883d03f11173e32fa39ab599/layer.tar
  -  /tmp/docker-squash683466637/d92c3c92fa73ba974eb409217bb86d8317b0727f42b73ef5a05153b729aaf96b/layer.tar
  -  /tmp/docker-squash683466637/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer.tar
  -  /tmp/docker-squash683466637/1c9383292a8ff4c4196ff4ffa36e5ff24cb217606a8d1f471f4ad27c4690e290/layer.tar
  -  /tmp/docker-squash683466637/589338fba5eb5cc32a25a036975a5e0938f12eff0dc70b661363c13ef1a192a5/layer.tar
  -  /tmp/docker-squash683466637/63e174c2ca3d53e2b7639a440940e16e15c1970e6ad16f740ffdcc60e59e0324/layer.tar
  -  /tmp/docker-squash683466637/9942dd43ff211ba917d03637006a83934e847c003bef900e4808be8021dca7bd/layer.tar
  -  /tmp/docker-squash683466637/0ea0d582fd9027540c1f50c7f0149b237ed483d2b95ac8d107f9db5a912b4240/layer.tar
  -  /tmp/docker-squash683466637/8dfc0bb00563dab615dfcc28ab3e338089f5b1d71d82d731c18cbe9f7667435f/layer.tar
  -  /tmp/docker-squash683466637/c4ff7513909dedf4ddf3a450aea68cd817c42e698ebccf54755973576525c416/layer.tar
  -  /tmp/docker-squash683466637/cc58e55aa5a53b572f3b9009eb07e50989553b95a1545a27dcec830939892dba/layer.tar
  -  /tmp/docker-squash683466637/e4eea4411c0065f8b0c7cf6be31dd58daa5ac04d8c64d54537cbfce2eb8e3413/layer.tar
  -  /tmp/docker-squash683466637/fc294d2b22cb53cb2440ff6fece18813ee7363f5198f5e20346abfcf7cce54fe/layer.tar
Inserted new layer 27935276f797 after 1c9383292a8f
  -  511136ea3c5a
  -  1c9383292a8f /bin/sh -c #(nop) ADD file:c1472c26527df28498744f9e9e8a8304c
  -> 27935276f797 /bin/sh -c #(squash) from 1c9383292a8f
  -  9942dd43ff21 /bin/sh -c echo '#!/bin/sh' > /usr/sbin/policy-rc.d  && echo
  -  d92c3c92fa73 /bin/sh -c rm -rf /var/lib/apt/lists/*
  -  0ea0d582fd90 /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/\1/g' /etc/apt/
  -  cc58e55aa5a5 /bin/sh -c apt-get update && apt-get dist-upgrade -y && rm -
  -  c4ff7513909d /bin/sh -c #(nop) CMD [/bin/bash]
  -  fc294d2b22cb /bin/sh -c apt-get update && apt-get install -y golang-go
  -  8dfc0bb00563 /bin/sh -c #(nop) ADD dir:78239d85b32dd28e4cb1d81ace7ffd32b8
  -  651626d6e364 /bin/sh -c #(nop) WORKDIR /app
  -  589338fba5eb /bin/sh -c go build -o http
  -  c50f2b65cab3 /bin/sh -c #(nop) ENV PORT=8000
  -  e4eea4411c00 /bin/sh -c #(nop) EXPOSE map[8000/tcp:{}]
  -  63e174c2ca3d /bin/sh -c #(nop) CMD [/app/http]
  -  49b5a7a88d53 /bin/bash
Squashing from 27935276f797 into 27935276f797
  -  Deleting whiteouts
  -  Rewriting child history
  -  Removing 9942dd43ff21. Squashed. (/bin/sh -c echo '#!/bin/sh' > /usr/sbin/policy-...)
  -  Removing d92c3c92fa73. Squashed. (/bin/sh -c rm -rf /var/lib/apt/lists/*)
  -  Removing 0ea0d582fd90. Squashed. (/bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/\1...)
  -  Removing cc58e55aa5a5. Squashed. (/bin/sh -c apt-get update && apt-get dist-upgra...)
  -  Replacing c4ff7513909d w/ new layer 72391e640b52 (/bin/sh -c #(nop) CMD [/bin/bash])
  -  Removing fc294d2b22cb. Squashed. (/bin/sh -c apt-get update && apt-get install -y...)
  -  Removing 8dfc0bb00563. Squashed. (/bin/sh -c #(nop) ADD dir:78239d85b32dd28e4cb1d...)
  -  Replacing 651626d6e364 w/ new layer bd7b4b11874a (/bin/sh -c #(nop) WORKDIR /app)
  -  Removing 589338fba5eb. Squashed. (/bin/sh -c go build -o http)
  -  Replacing c50f2b65cab3 w/ new layer e4af8871b961 (/bin/sh -c #(nop) ENV PORT=8000)
  -  Replacing e4eea4411c00 w/ new layer 6803497b6a61 (/bin/sh -c #(nop) EXPOSE map[8000/tcp:{}])
  -  Replacing 63e174c2ca3d w/ new layer 40b8c7c33bba (/bin/sh -c #(nop) CMD [/app/http])
  -  Removing 49b5a7a88d53. Squashed. (/bin/bash)
Tarring up squashed layer 27935276f797
Removing extracted layers
Tagging 40b8c7c33bba as jwilder/whoami:squash
Tarring new image to STDOUT
Done. New image created.
  -  40b8c7c33bba Less than a second /bin/sh -c #(nop) CMD [/app/http] 3.072 kB
  -  6803497b6a61 Less than a second /bin/sh -c #(nop) EXPOSE map[8000/tcp:{}] 3.072 kB
  -  e4af8871b961 Less than a second /bin/sh -c #(nop) ENV PORT=8000 3.072 kB
  -  bd7b4b11874a Less than a second /bin/sh -c #(nop) WORKDIR /app 3.072 kB
  -  72391e640b52 Less than a second /bin/sh -c #(nop) CMD [/bin/bash] 3.072 kB
  -  27935276f797 1 seconds /bin/sh -c #(squash) from 1c9383292a8f 39.49 MB
  -  1c9383292a8f 3 days /bin/sh -c #(nop) ADD file:c1472c26527df28498744f9e9e8a83... 201.6 MB
  -  511136ea3c5a 14 months  1.536 kB
Removing tempdir /tmp/docker-squash683466637

My squashed layer is down from ~198MB to 39.5MB. Roughly 80% smaller. I should be able to get it down to ~7MB if I squash some of the apt-get updates my build pulled in with the upstream ubuntu:14.04 base image and use a custom base image.

If I was to create a custom base image, I would squash that entire image down to a single layer using -from root and update my Dockerfile to use it as the FROM image.

This is what -from root looks like with my example images:

$ docker save 49b5a7a88d5 | sudo docker-squash -t jwilder/whoami:squash -verbose -from root | docker load
Loading export from STDIN using /tmp/docker-squash627981871 for tempdir
Loaded image w/ 15 layers
Extracting layers...
  -  /tmp/docker-squash627981871/d92c3c92fa73ba974eb409217bb86d8317b0727f42b73ef5a05153b729aaf96b/layer.tar
  -  /tmp/docker-squash627981871/cc58e55aa5a53b572f3b9009eb07e50989553b95a1545a27dcec830939892dba/layer.tar
  -  /tmp/docker-squash627981871/1c9383292a8ff4c4196ff4ffa36e5ff24cb217606a8d1f471f4ad27c4690e290/layer.tar
  -  /tmp/docker-squash627981871/63e174c2ca3d53e2b7639a440940e16e15c1970e6ad16f740ffdcc60e59e0324/layer.tar
  -  /tmp/docker-squash627981871/8dfc0bb00563dab615dfcc28ab3e338089f5b1d71d82d731c18cbe9f7667435f/layer.tar
  -  /tmp/docker-squash627981871/c4ff7513909dedf4ddf3a450aea68cd817c42e698ebccf54755973576525c416/layer.tar
  -  /tmp/docker-squash627981871/0ea0d582fd9027540c1f50c7f0149b237ed483d2b95ac8d107f9db5a912b4240/layer.tar
  -  /tmp/docker-squash627981871/9942dd43ff211ba917d03637006a83934e847c003bef900e4808be8021dca7bd/layer.tar
  -  /tmp/docker-squash627981871/c50f2b65cab3b74f9bdb6f616b36f132b9a182ed883d03f11173e32fa39ab599/layer.tar
  -  /tmp/docker-squash627981871/49b5a7a88d5353fe77204ad5591a3ef100fc2807a9d6dce979fd1b17a73a68d6/layer.tar
  -  /tmp/docker-squash627981871/589338fba5eb5cc32a25a036975a5e0938f12eff0dc70b661363c13ef1a192a5/layer.tar
  -  /tmp/docker-squash627981871/651626d6e364ccc22ac990ba95cd0aab9256c56055087cc9a5a1790cea5250b9/layer.tar
  -  /tmp/docker-squash627981871/e4eea4411c0065f8b0c7cf6be31dd58daa5ac04d8c64d54537cbfce2eb8e3413/layer.tar
  -  /tmp/docker-squash627981871/fc294d2b22cb53cb2440ff6fece18813ee7363f5198f5e20346abfcf7cce54fe/layer.tar
  -  /tmp/docker-squash627981871/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer.tar
Inserted new layer 6996f41c1688 after 511136ea3c5a
  -  511136ea3c5a
  -> 6996f41c1688 /bin/sh -c #(squash) from 511136ea3c5a
  -  1c9383292a8f /bin/sh -c #(nop) ADD file:c1472c26527df28498744f9e9e8a8304c
  -  9942dd43ff21 /bin/sh -c echo '#!/bin/sh' > /usr/sbin/policy-rc.d  && echo
  -  d92c3c92fa73 /bin/sh -c rm -rf /var/lib/apt/lists/*
  -  0ea0d582fd90 /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/\1/g' /etc/apt/
  -  cc58e55aa5a5 /bin/sh -c apt-get update && apt-get dist-upgrade -y && rm -
  -  c4ff7513909d /bin/sh -c #(nop) CMD [/bin/bash]
  -  fc294d2b22cb /bin/sh -c apt-get update && apt-get install -y golang-go
  -  8dfc0bb00563 /bin/sh -c #(nop) ADD dir:78239d85b32dd28e4cb1d81ace7ffd32b8
  -  651626d6e364 /bin/sh -c #(nop) WORKDIR /app
  -  589338fba5eb /bin/sh -c go build -o http
  -  c50f2b65cab3 /bin/sh -c #(nop) ENV PORT=8000
  -  e4eea4411c00 /bin/sh -c #(nop) EXPOSE map[8000/tcp:{}]
  -  63e174c2ca3d /bin/sh -c #(nop) CMD [/app/http]
  -  49b5a7a88d53 /bin/bash
Squashing from 6996f41c1688 into 6996f41c1688
  -  Deleting whiteouts
  -  Rewriting child history
  -  Removing 1c9383292a8f. Squashed. (/bin/sh -c #(nop) ADD file:c1472c26527df2849874...)
  -  Removing 9942dd43ff21. Squashed. (/bin/sh -c echo '#!/bin/sh' > /usr/sbin/policy-...)
  -  Removing d92c3c92fa73. Squashed. (/bin/sh -c rm -rf /var/lib/apt/lists/*)
  -  Removing 0ea0d582fd90. Squashed. (/bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/\1...)
  -  Removing cc58e55aa5a5. Squashed. (/bin/sh -c apt-get update && apt-get dist-upgra...)
  -  Replacing c4ff7513909d w/ new layer 09a62007c3f3 (/bin/sh -c #(nop) CMD [/bin/bash])
  -  Removing fc294d2b22cb. Squashed. (/bin/sh -c apt-get update && apt-get install -y...)
  -  Removing 8dfc0bb00563. Squashed. (/bin/sh -c #(nop) ADD dir:78239d85b32dd28e4cb1d...)
  -  Replacing 651626d6e364 w/ new layer b4f0dec85412 (/bin/sh -c #(nop) WORKDIR /app)
  -  Removing 589338fba5eb. Squashed. (/bin/sh -c go build -o http)
  -  Replacing c50f2b65cab3 w/ new layer cd499c2d09ef (/bin/sh -c #(nop) ENV PORT=8000)
  -  Replacing e4eea4411c00 w/ new layer 653dfab45562 (/bin/sh -c #(nop) EXPOSE map[8000/tcp:{}])
  -  Replacing 63e174c2ca3d w/ new layer f7f7eb6aae54 (/bin/sh -c #(nop) CMD [/app/http])
  -  Removing 49b5a7a88d53. Squashed. (/bin/bash)
Tarring up squashed layer 6996f41c1688
Removing extracted layers
Tagging f7f7eb6aae54 as jwilder/whoami:squash
Tarring new image to STDOUT
Done. New image created.
  -  f7f7eb6aae54 Less than a second /bin/sh -c #(nop) CMD [/app/http] 3.072 kB
  -  653dfab45562 Less than a second /bin/sh -c #(nop) EXPOSE map[8000/tcp:{}] 3.072 kB
  -  cd499c2d09ef Less than a second /bin/sh -c #(nop) ENV PORT=8000 3.072 kB
  -  b4f0dec85412 Less than a second /bin/sh -c #(nop) WORKDIR /app 3.072 kB
  -  09a62007c3f3 Less than a second /bin/sh -c #(nop) CMD [/bin/bash] 3.072 kB
  -  6996f41c1688 2 seconds /bin/sh -c #(squash) from 511136ea3c5a 111.9 MB
  -  511136ea3c5a 14 months  1.536 kB
Removing tempdir /tmp/docker-squash627981871

That gets my full build down to a single layer of 106.2MB:

REPOSITORY          TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
jwilder/whoami      squash              f7f7eb6aae54        About a minute ago   106.2 MB
jwilder/whoami      latest              63e174c2ca3d        29 minutes ago       423.7 MB

Since I typically have base images already loaded onto my docker hosts, I would continue to use the default squash settings and retain my parent base image so that I’m only tranferring the changes for each image.

Conclusion

Squashing images with docker-squash can reduce image sizes significantly. Since it only needs to be run before publishing to a registry, the regular docker build caching is not changed and you don’t lose any of the benefits of using Docker. Similarly, it does not require complex Dockerfile setups to get good results so you can start using it on existing projects with little effort.

If you want to try it out or learn more about it works, you can get it from github.
comments powered by Disqus