The problem

We build a container image on a feature branch with a commit SHA tag, something like this lansible/application:c1ee81ad5d18fcecfa4af5c97b853a4121875664

After the feature branch is merged we don’t want to rebuild the image, that could result in a different image going to production since there is no guarantee that the rebuild is 100% identical. Also skipping the rebuild saves a lot of CI time and therefore waiting. So currently after a fast-forward merge to main we retag SHA hash image to a production release:

docker tag \
    lansible/application:c1ee81ad5d18fcecfa4af5c97b853a4121875664 \
    lansible/application:1.0.0

This works perfect when you just build for one architecture. Since Apple is switching to arm64 for their new CPUs there is a need to build multi-architecture images since the QEMU emulation of Docker on MacOS is slow and has segfault issues once in a while.

The solution

The easy part: Docker builds with buildx

This step was the easiest since docker/buildx makes building for multiple architectures a breeze.

First you need to setup QEMU emulation for the architectures you are missing. I use tonistiigi/binfmt for this. tonistiigi is not a random Github user, he is part of Docker and a big contributor on the docker/buildx repository.

docker run --rm --privileged tonistiigi/binfmt --install arm64

Then you need to register a new context for buildx to take advantages of these new architectures:

docker context create multiarch
docker buildx create --use multiarch

Now you are ready to build multi-archicture images with buildx and the build command (docs):

docker buildx build \
    --platform=linux/arm64,linux/amd64 \
    --push \
    --tag lansible/application:c1ee81ad5d18fcecfa4af5c97b853a4121875664 .

Even an existing docker-compose.yml with multiple images is easy to convert to multi-architecture build with the bake command (docs): The following command will build all the containers in the compose file for amd64 and arm64 without adding the platforms explicitly in the compose files.

# docker buildx bake --push --set *.platform="linux/amd64,linux/arm64"

The hard part: Retagging the multi-architecture image

Since there is now a manifest under the SHA tag and not just one image the docker tag command does not work to retag the image. I tried using buildx imagetools create (docs) but I couldn’t get it to work:

# docker buildx imagetools inspect --raw lansible/application:c1ee81ad5d18fcecfa4af5c97b853a4121875664 | jq '.manifests[].digest'
"sha256:3359c957089718e8f448a68c9c1be361b4abc6bac4c483a0c6fb33468bf7dede"
"sha256:f83d4008e2ef1209b58d6b477a0783d0a53a1d983707bd1f97d93da04b43d415"

# docker buildx imagetools create \
    -t lansible/application:1.0.0 \
    lansible/application:c1ee81ad5d18fcecfa4af5c97b853a4121875664@sha256:3359c957089718e8f448a68c9c1be361b4abc6bac4c483a0c6fb33468bf7dede \
    lansible/application:c1ee81ad5d18fcecfa4af5c97b853a4121875664@sha256:f83d4008e2ef1209b58d6b477a0783d0a53a1d983707bd1f97d93da04b43d415

error: failed commit on ref "index-sha256:fd2440420e20b8c9fc3fb0e1c1295d3bd2767eabfff99ec183ec87ad536491ff": cannot reuse body, request must be retried

The error doesn’t get much hits on Google and whatever I tried I couldn’t get it to work. But in the search I stumbled upon regclient/regclient/ and the regctl command which has an image copy that seems to do exactly what I wanted (docs). So I rewrote the docker tag from above to this:

# regctl image copy --verbosity info \
    lansible/application:c1ee81ad5d18fcecfa4af5c97b853a4121875664 \
    lansible/application:1.0.0

And we have achieved the retagging like before with docker tag except now with the multi-architecture manifest still intact and pointing to the correct images. regctl also is smart and doesn’t need to pull the image most of the times!