I'm quite thankful for the Qemu and Buildx Github Actions. I'll build and test my docker image locally as I want, and then use Github Actions to build for multiple platforms and publish. So it saves me a bunch of hassle. I usually go for:
It depends on what the build process looks like, but I find buildx to be really, really slow–and that can be costly if you're paying for the compute time to do that. (The slow part is if you have to do any compiling for the non-native platform on your host. I have tested this doing x86 compilation on an M1 system, and Arm complication on an x86 system.)
AWS has a guide for using CodeBuild where you split the work to build each Docker image using the platform of its target, and then create a manifest at the end.
I also took a similar approach using CircleCI. A multi-arch build can take 5 minutes using parallel, native builds instead of 45+ minutes (it actually timed out).
Pros: way faster and cheaper builds if you have to pay for the compute time
Cons: a much more complicated setup
I agree that buildx is really slow for multiplatform images, the advantage is provides is that if you want don't have a machines for that architecture you cannot build nativly, and with buildx you can build images for multiple platforms on your own self hosted runner.
I agree with the cons and pros you mentioned, but I am hoping that the qemu and buildx will continue to improve.
True – I'm definitely glad it's there, and I would totally use it for personal/casual projects where I likely won't have a fancy AWS/CircleCI/etc setup. At work though, its use would actually add to our bill and hold things up for teams.
I don't like the way docker buildx works at the moment. Hopefully it improves over time.
My biggest complaint is that everything is baked into a single command instead of being discreet commands. So instead of:
docker build...
docker tag...
docker push...
it's:
docker buildx build --tag... --push...
Most CI systems do something similar and you end up with an awful, monolithic build step that's essentially 'build-and-test-and-tag-and-push'. It's the same on every platform too. GitHub Actions, GitLab CI, Drone CI, etc. all seem to do the same thing.
What if I want to build in one environment that pushes and one environment that doesn't? In my experience it's much easier if 'push' can be a build step on it's own because the CI system will usually have conditional support for things like different environments.
From what I've seen it's not possible to use buildx and then conditionally tag the resulting images in a second command or build step. The 'docker' driver doesn't support multiarch and the 'docker-container' driver doesn't support '--load' [1], so you're pretty much stuck with a single command that does everything, at least for now, because the built images won't be added to the daemon used for building (ie: you won't see them with 'docker image ls'). Am I wrong?
I've always liked the way the Docker daemon uses simple commands for everything; docker pull, docker build, docker tag, etc.. It feels very unix like. It would be a shame to lose that IMO.
Nope, I think you got this right. I've been hit by this annoying buildx design when trying to get the manifest digest (sha256) of the resulting image for automation. Lo and behold since these steps are not discreet one can't just directly ask for the digest and would have to parse the command output.
I hope this is just an intermediate step towards a real solution as the idea is nice but the execution is severely lacking.
Another great tool for cross arch development is qemu-static-user. It allows you to run your nonnative architecture docker images on host using a qemu container for translation and only requires about one command to get working. It’s life saving for me when I need to test but setting up a full VM or native server is too time consuming.
--platform linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x
Which seems to cover most cases including Raspberry Pis.