How’s is this different than running a Docker daemon on remote with build kit support?

It's basically the same, just without you having to do any of the work to set up or maintain the remote machines.

When you start a container build with Depot, it will launch an Intel or an Arm VM running BuildKit (or both for a multi-platform image). It creates persistent SSD cache volumes for each project and attaches those to the build machines at launch. And it automates connecting your CLI to those machines over TLS. Then when the build is done, it shuts down those machines to save cost.

Depot does all that either in our cloud account, or you also have the option of connecting your own AWS account and Depot will act just as the orchestration layer.

So basically you get the remote build machines, but without needing to set them up, manage costs, configure access control, etc. But at its core, it's running the open-source version of BuildKit (the Depot CLI embeds the same Go library as `docker buildx build`).

congrats on launching Kyle and Jacob (hi Kyle!)

wondering what kinds of projects are/are not suitable for this. the only context i have is from working at open source devtool companies that provide docker builds for people to pull down. might speed up the release process slightly. i suspect my company https://github.com/airbytehq/airbyte/ could benefit. but is it also useful for internal usage?

Our biggest strength is building images, internal or not, in generic CI providers like GitHub Actions, CircleCI, etc. Depot excels at building images in these environments because it provides solutions to limitations like persistent SSD caching, multi-architecture support, and faster compute resources.

Things where Depot may not be a good fit are when you only ever need Docker images for local development. In those situations, the network becomes the bottleneck because you have to pull down the built image each time. That said, if you need to build multi-architecture images locally, Depot becomes useful again as it can build native images for each architecture, no emulation needed.

If you're entirely on-prem, we don't currently support that so Depot isn't a fit there.

Yup the caches for each architecture are available in parallel and multiple builds for a single architecture can simultaneously use the same build machine for a single project. So we don't limit the concurrency.

I believe Cloud Build has no persistent caching so you are forced to use remote cache saving and loading. Which can incur a network latency that can slow the build to some extent. Cloud Build with Kaniko also expires the layer cache after 6 hours by default.

GitHub Actions is similar except that there is the ability to store Docker cache using GitHub's Cache API via the `cache-to=gha` and `cache-from=gha` directives. However, this has limitations like only being able to store a total cache size of 10GB per repository. You also have network latency for loading/saving that cache as well.

With Depot, the cache is kept on a persistent disk. So no need to save/load it or incur network latency doing so. It's there ready to be used by any builds that come in for the given projects.

Sounds great! Thanks.

Congrats on the launch. Apple Silicon really threw a wrench in Docker local development workflows, and this feels like a natural solution.

Does docker compose work with this?

Thanks! We have a few users using docker-compose with a Makefile, where they run `depot build -t name:tag --load` to tag and pull the image locally, and then they reference the tag in their docker-compose.yml.

That works, but we've considered adding a `depot bake` command that does the same thing as `docker buildx bake`, to make the integration nicer. Ideally docker-compose would have the ability to communicate with buildx drivers (that's what Depot registers as), but to my knowledge it doesn't currently have that option.

imho having docker compose _build_ container images is an anti-pattern. Always do what the parent says -- build/pull separately (not using compose), then arrange to fix up the tag/version specified in the compose file to match.

Apple Silicon can still build amd64 though right? Using BuildX and arch flags?

I used it, didn’t have any issues but i realize there’s lots of reasons that might not work.

You're right, you can, it's just one more thing to think about.

I also had in mind Docker Desktop's licensing (paid-only for certain companies), but I now realize I was conflating that with Apple Silicon when it applies to Intel Macs too IIRC.

So it seems over 5x more expensive than GitHub actions per minute, not including the $25 per project cost. Is that right?

It is! At least in terms of raw minute pricing, though Depot builds are often faster than Actions, around 2-5x for Intel builds and closer to 10x faster for Arm builds, which affects the comparison a bit.

Our biggest cost at the moment is persistent disks. Each project gets its own SSD(s) that live for the lifetime of the project, and that's a major part of why Depot builds are so much faster and support BuildKit features that require persistent disks (e.g. RUN --mount=type=cache). We have some ideas for how we might improve this in the future, keeping the same or better performance profile but avoiding the continuously-provisioned disk, we'll see how that goes.

One note on "per project", each Depot project represents a pair of Intel/Arm VMs, each with a 50GB SSD. But there's otherwise no restriction on what you build using those projects, you can use a single project with multiple repos/Dockerfiles/whatever. Projects are the access control boundary, and building multiple disparate images using the same project would cycle the cache faster, but otherwise there are no limits on concurrency or what's building inside those VMs.

When people use a service like this do they bake test runs into the build? Typically in CI I might build and then run tests, but with this service my resulting docker image is in the depot environment.

Do I build and then download the image to run it? Or do I have this a bit backwards and this is for build-only use-cases?

Yep, so Depot runs BuildKit, so the same workflows work there, just with a remote builder. You can do one of three things:

1. Just build the image — in CI that would test that the image built successfully, and the build cache would be ready for future builds. So for instance if you ran a build on a branch, but only pushed the image on the main branch workflow, the second run could just re-use the cache.

2. Build and push the image to a registry — from there you could do anything with the image in the registry (pull it from any workflow, deploy it, etc)

3. Build and pull the image to the local Docker daemon. In CI, that might be ideal for integration testing the container, like you mention.

You can also use option (2) for integration testing the container, which is especially useful with multi-platform images. Docker doesn't (yet) support storing multi-platform images locally, but it will pull the correct platform from the registry.

tl;dr — Depot supports the same options as `docker buildx build`, where you can push, pull, or just build the image depending on your needs

Another unusual alternative is to model your test running _as_ a Docker build. It's what I ended up doing in Trellis, which supports Depot as a runner (https://github.com/charliermarsh/trellis): running the linter check is modeled as "build the image, then add a `cargo clippy` RUN command as the last layer".

you could use multi stage builds here too so you don't have to carry around the linter, not a bad idea

I've been using Depot since the early days (I don't know Kyle or Jacob, just stumbled across it)

It's superb. It's a drop-in replacement for `docker build` on my M1 Macbook and on GitHub Actions. I really like that they use OIDC for auth in the latter, so I don't have to provision credentials. Of course I also like that builds are 3-4x faster.

I have to admit though that the pricing model doesn't make a whole lot of sense to me. It's $25/mo/project, but what's to stop me from using one project for multiple, unrelated Docker images? It's a relatively small cost for saving me time, but it's still big enough that I have slight hesitation to spin up more projects.

Thank you for sharing your experience and trying Depot out!

We currently charge on a per project basis as behind the scenes we are provisioning and maintaining a VM with a persistent SSD cache for each architecture you are building for (i.e. Intel & Arm). We switched to usage based pricing before rolling Depot out to the public so you are only charged for your active projects and used build minutes in a month. So far it has seemed to be working well, but we are always open to feedback and ideas on how we can make pricing better.

We don't currently document this, and we probably should, but it's entirely possible to use a single Depot project for multiple Docker images. There is no limitation that prevents this and the only downside is you may fill up your cache space at a faster rate and get fewer hits if you have different image builds clobbering the cache.

> behind the scenes we are provisioning and maintaining a VM with a persistent SSD cache

How does this work if I have 5 builds for a single project at the same time?

In our current setup it takes a bit longer to boot, but concurrency is effectively unlimited. I don’t imagine that is true for Depot?

Thank you for the question!

We don't limit concurrency on the builder VMs. When you create a project you are effectively getting a dedicated build VM with a persistent SSD cache for each architecture you want to build for. We run native BuildKit on the VM and connect your `depot build` directly to the builder instance via mTLS. So your concurrency is unlimited with Depot as well (or at least as unlimited as BuildKit will allow for).

This sounds cool. At my current job we do multi-arch builds in GCP Cloud Build using `docker buildx`. The native `amd64` builds take like 15 minutes and the `arm64` component takes well over an hour with occasional time outs. We use the `--cache-to/--cache-from` to speed up repeated builds. If we can't wait the hour+ for the emulation build to run, we have a `seed-mac-arm64.sh` script that a developer (i.e., me) can run on their M1 mac to do the `arm64` half of the build, which is then picked up by GCP Cloud Build. It's a small pain, but helps us in a pinch.

This solution sounds far easier!

GCP cloud build was abnormally slow in my limited testing honestly.

Things that took seconds to build on my local machine were taking double-digit minutes to build in GCP. I ended up building everything with gitlab-ci using kaniko and pushing to GCR, insane speed up.

Nice, yeah we had one customer using Cloud Build, and in their case their native amd64 builds dropped from around 12 minutes to 1-2 minutes on Depot builders. Surprisingly to us, it was even faster to push from Depot to Artifact Registry than from Cloud Build directly, definitely an odd networking quirk.

arm64 builds should even be a more dramatic change, they should drop to at least as fast as your current amd64 builds.

Any comments on how this would be better than Google Clouds build service? I can do 'gcloud builds submit .` right now and build containers pretty easily in the cloud with this service currently.

The difference to me is that Depot is giving you what is effectively a dedicated build machine. That machine can take advantage of local BuildKit caching.

With Cloud Build, you need to use —-cache-from and other amendments to leverage remote caching. (And I believe that caching multi-stage builds require you to explicitly bush and push every stage.)

With Depot, you should in theory get all of that caching “for free”. You can also take advantage of things like BuildKit cache mounts (local directories on the host machine to cache, e.g., pip install artifacts) which AFAIK can’t be done with Cloud Build.

(No affiliation with Depot but I have played around with it.)

Thank you for the comment and sharing your thoughts!

The first thing that comes to mind is multi-platform support. Currently, it's very slow to build multi-platform images in things like Cloud Build because you have to emulate the Arm portion of the build. We don't have timings in Cloud Build for this, but in GitHub Actions a multi-platform image build can take over an hour when building for both Intel & Arm. With Depot, those builds can take less than 5 minutes because we build the image for both architectures in parallel on native CPU architectures.

Other things where Depot might be more streamlined than Cloud Build is that persistent caching is automatic. The dedicated build machines for each architecture have persistent SSD caches attached to them, so you don't need to save/load cache in between runs, it's just there ready to be used.

Cool. Thanks for the info!

Considering that GitHub actions provide no arm64 runners at this stage, this could be extremely useful for some of my projects.

So when I build a project on GitHub actions, I simply add a `platforms: linux/amd64,linux/arm64` line to the Docker build & push action. This takes care of pushing the multi-arch build to GHCR for me. Would it be as straightforward in Depot as well? Or would modifying the image manifest be my only option?

It should be just as as straightforward with Depot as well - we have a depot/build-push-action (https://github.com/depot/build-push-action) that accepts the same inputs as docker/build-push-action to make it easy (under the hood, both use BuildKit).

    steps:
      - uses: depot/setup-action@v1
      - uses: depot/build-push-action@v1
        with:
          platforms: linux/amd64,linux/arm64

> We believe that today we are the only hosted CI or build service offering the ability to natively build multi-platform Docker images

Dagger.io, by the originators of docker, can do this too, though it is not well documented. They have not yet enable public signups either by the looks of it.

Pretty sure I've seen some other BuildKit frontends that do as well, the space blew up this year with several startups

I haven't had a chance to experiment with it yet, but it should be possible to run Dagger pipelines on Depot builders. Depot is effectively a BuildKit-API-as-a-service currently, and Dagger can connect to an external BuildKit host (https://docs.dagger.io/1223/custom-buildkit/).

It would be cool to add an integration between Depot's CLI and Dagger's CLI such that Dagger could communicate with the remote builder machine. You could use Dagger with remote compute without needing to manage that compute, effectively. Depot even supports launching builders within your own AWS account, so you'd have that same ability with Dagger as well.

Dagger is Apache 2, no signup required: https://github.com/dagger/dagger#readme

I struggle to think of what I would need a SaaS service to run BuildKit for me unless they're trying to take on GitHub Actions's compute side or whatever

I think we are trying to take on GitHub Actions' compute side yeah — you'd want to either self-host your own BuildKit or use something like Depot for two major reasons:

1. Several of BuildKit's caching features either require or work best with a local disk. For instance things like `RUN --mount=type=cache` need a persistent disk, which Actions doesn't provide. You can also hit bottlenecks with saving and loading tarball cache from Actions (e.g. cache-from / cache-to) where it takes longer to download and unpack the tarballs compared to omitting the cache. Having a persistent disk makes all this "instant".

2. If you need to build images for ARM CPUs, QEMU in GitHub Actions is really slow. And if you want to build multi-platform images natively (e.g. --platform linux/amd64,linux/arm64), BuildKit needs a simultaneous connection to two native builders. Depot automates all that (it will launch two VMs at once and connect them to the build).

GitHub Actions today also only offers 2 CPUs for the build - they're launching a beta of larger machine sizes currently.

Dagger is helping companies scale BuildKit in a self-hosted setup (that I am aware of). Not sure about the Dagger + GHA self-hosted runner story, one would assume it would be a natural extension of their general self-hosted BuildKit, just point Dagger at whatever BuildKit domain/ip you are using.

Dagger is open-source, but also has paid things if you talk to them. They are very focused on a few key users in their paid plans, rather than opening the floodgates, as I understand it.

Love using Depot. The speed improvements are insane

Hey Kyle and Jacob. Congratulations on launching this and I was really excited because we were looking for something just like this. But as it stands this would be too expensive for our organization. We have a bunch of small (micro?) services and having to pay 25$ for each seems too excessive.

But maybe I'm misunderstanding the pricing?

Hey, so a Depot "project" represents the two VMs / SSDs for building each architecture + access control in the form of project-scoped API tokens. But there's nothing preventing you from building all your small services using just a single project.

You very well could build everything on just the single $25 project, there aren't any restrictions on concurrency or what images you build on what projects.

Nice looking site, grats on the launch!

container builds for services, compiled/packaged components you want to spin up somewhere, often the Dockerfile is very simple, use some base image, add config and your binary or packaged artifact (jars, minified JS etc) and done.

Often the complexity is (or should be) in the shared base not the individual images you build per commit or PR merge.. for those simple images using something like Jib (bazel can do it too..) where it just constructs the image in user land by adding files to the tarball. I imagine this works for arm binaries from x64 too as long as you can cross compile.. how do you feel about these libraries that make simple images waaay more accessible? No docker required.

I would love to see more development in this space. I'm aware of tools like bazel, nix, and jib (I feel like I've seen an equivalent to jib for Go) that can build Docker images "outside" Docker/BuildKit. I believe the .NET SDK can do something similar for .NET applications. And then there's tools like buildpacks, nixpacks, devbox, etc that still use Docker to produce the image, but don't require thinking in Dockerfiles.

I have some ideas I want to explore with Depot, but it really it feels like it should be possible to make cross-compiling and dependency encapsulation easy to use, so "building a container image" is reduced to creating a tarball in the correct OCI format, and you don't have to interact with Dockerfiles as much if you'd prefer not.

I’ve used jib and bazel and that’s exactly what they do, both are language agnostic and push files around. Jib has a java bent but not a required part of it.

I can totally see people using Depot bc it’s way easier and more obvious, works in the obvious way, and that is how things work today. It doesn’t challenge you to think, why the hell do I need a super privileged process with magic Dockerfile incantations just to take a tar of files and add another file to it? With some metadata? Docker has created some weird defaults.

How does this compare to nixpacks? I've heard good things but admittedly haven't dug in too far yet.

https://nixpacks.com/docs/how-it-works

Yeah Nixpacks is super cool! It uses nix and BuildKit to produce images for various different languages, and since it uses BuildKit under the hood, it might be able to use Depot's BuildKit to perform the actual build. I'd love to see what that integration would look like, Depot would provide the compute and persistent caching, and nixpacks would simplify the build process itself.

I am curious if it will help our builds; they are pretty slow, even to the point we have all kinds of tricks to speed them up. Thanks for trying to fix this upfront anyway.

In theory, any of the BuildKit frontends like Depot or Dagger, should be able to pass through cross-build caching from BuildKit.

This is how they all scale builds, and hopefully things become more standardized with a reduction in vendor lock in. It is surprising to me how much of the world relies on moby/buildkit, and scary if you've ever looked at the code. Primarily a single developer, not straightforward to understand, comments & docs are basically non-existent. Looks like that might be starting to change with so many companies now building and SaaS'n on top of BuildKit.

Have you tried self-hosting the runners so that you can use a beefier machine to run them on? I switched our iOS builds from Azure Pipeline's hosted runner to a rented M1 Mac Mini (from Scaleway), and our iOS build time went from 20-60 minutes (depending on how Azure was feeling that day) to a consistent 3.5 minutes.

Thank you for the comment! We have dealt with this problem throughout our own careers at multiple companies. We should support all of the tricks you may be doing to speed up your builds, but we hope you have to do fewer of them using Depot.

I see you currently have CI integrations for CircleCI, GitHub Actions, Google Cloud Build, and Travis CI. Could I use this in my Bitbucket Pipelines? If so, how?

Yes, you can definitely use this in your Bitbucket Pipelines. Our CLI is a drop in replacement for `docker build`. So in your pipeline you would still login to your registry with `docker login` but then you can use `depot build` with all of the same options you were using before and the build can be routed to our remote builders.

So your pipeline would effectively look like this.

    image: node:10.15.0
    pipelines:
      default:
        - step:
            script:
              - curl https://depot.dev/install-cli.sh | DEPOT_INSTALL_DIR=/usr/local/bin sh
              - npm install
              - npm test
              - docker login -u $DOCKER_HUB_USER -p $DOCKER_HUB_PASSWORD
              - depot build -t <my.container.registry.io>/<my_app>:<my_tag> . --push --project <DEPOT_PROJECT_ID> --token <DEPOT_PROJECT_TOKEN>
            services:
              - docker

Here I have used `--token` for passing in a project token to authenticate the build with your project in Depot. This can also be passed as an environment variable, `DEPOT_TOKEN`.

Edit: Formatting. Also thank you for pointing out this is an integration we are missing in our docs. We will get that added!

Great, thanks for the thorough reply. Hope to test this out soon and see if we can't speed up our pipelines. Congrats on the launch.

Since arm native support, any chance it also support building IOS application by a wrapped docker container on aws?

Possibly? I'm less familiar with building iOS applications, but Depot is providing a native Linux amd64 builder VM, so if you're able to build an iOS app in that environment, it should just work.

Can you build windows containers?

Not currently - Depot runs BuildKit (https://github.com/moby/buildkit) on our builders, and it does not support Windows containers to my knowledge.

This looks really cool, would love a OSS tier!