Speedup Docker image pushing and pulling
As a developer I am impatient by nature. Technology serves me, not the other way around ;) Here are two things I do immediately when setting up a new development environment for Docker.
More upload/download concurrency
First make sure to increase the docker upload/download concurrency. This will make Docker upload/download more layers at the same time and speed this up considerably. As explained in the dockerd commandline documentation:
–max-concurrent-downloads int Set the max concurrent downloads (default 3) –max-concurrent-uploads int Set the max concurrent uploads (default 5) – Daemon CLI (dockerd)
For Linux edit
/etc/docker/daemon.json and add these two parameters in the root of the JSON:
Or just use the Docker Desktop settings ‘Docker Engine’ menu option:
This is one of the lesser known features of Docker and I stumbled upon it by accident when I was searching for
something else in the moby/moby pull requests.
Docker can use
pigz instead of the slow Golang native gzip implementation when decompressing images when pulling.
Pigz performs better due:
The Golang built-in gzip library is serialized, and fairly slow at decompressing. It also only decompresses on demand, versus pipelining decompression. This fixes those problems.
I ran some manual benchmarks, and I found I was able to get about 50% better performance as opposed to the build in library. These performance gains were primarily seen on images with layers about 10MB.
So that is a big performance gain! Getting Docker to use
pigz see is simple,
you just need to make sure
pigz is available for Docker to use, it could be already installed as Docker marks it as
a recommended package on Debian. You can check it to be sure:
# apt list --installed | grep pigz
pigz/focal,now 2.4-1 amd64 [installed]
If it is missing you can install it easily with:
sudo apt install pigz
For Fedora/RHEL/CentOS/Rocky the
pigz package isn’t marked as a dependency of Docker for some reason.
But you can install it easily and enjoy the speedup:
dnf install pigz