The big problem is how docker was designed. It is essentially a jail that is supposed to contain a Linux binary.
Things are straight forward on Linux. You build your binary, place in a docker container and you are done. The nix code will also be straight forward. If you can build your code, then creating a container is just one more operation away.
Unfortunately docker requires Linux binary and you are on Mac. So the docker desktop actually runs a Linux VM and performs all operations on it, abstracting this away from you.
Nix doesn't do that and you have two options:
1. Do cross compilation, the problem is that for this to work you need to be able to cross compile down to glibc, the problem is that while this will work for most community used dependencies you might get some package where the author didn't put effort making sure it cross compile. To make things worse the Hydra that populates standard caches that nix uses, doesn't do cross compile builds, so you will run into lengthy processes that might potentially end with a failure.
2. You can have a Linux builder, that you add to your Mac and configure to send build jobs for x86_64-linux to that builder. Now you could have a physical box, create a VM, or even have a NixOS docker container (after all docker ion Mac runs inside of the VM).
The #1 seems like the proper way, while #2 is more of a practical way.
I think you are running into issues, because you're likely trying #1, and that requires a lot of experience not only with Nix, but also with cross compiling. I wish Nix's Hydra would also build Darwin to Linux cross compilation as that would not only provide caches, but also help making sure the cross compilation doesn't break, but that would also increase costs for them.
Hydra not populating with cross compile builds is the bane of my existence.
I'm using `clang` from `pkgs.pkgsCross.musl64.llvmPackages_latest.stdenv` to cross-compile Rust binaries from ARM macos to `x86_64-unknown-linux-musl`. It _works_, but every time I update my `flake.nix` it rebuilds *the entire LLVM toolchain*. On an M2 air, that takes something like 4 hours. It's incredibly frustrating and makes me wary of updating my dependencies or my flake file.
The alternative is to switch to dockerized builds but:
1) That adds a fairly heavyweight requirement to the build process
2) All the headache of writing dockerfiles with careful cache layering
Not sure if this applies to your situation, but I believe you can avoid a full rebuild by modularizing the flake.nix derivations into stages (calling a separate *.nix for each stage in my case). That is how it appears to be working for me on a project (I am building a cc toolchain without pkgscross).
I pass the output of each stage of a toolchain as a dependency to the next stage. By chaining the stages, changes made to a single stage only require a rebuild of each succeeding stage. The final stage is the default of the flake, so you can easy get the complete package.
In addition, I can debug along the toolchain by entering a single stage env with nix develop <stage>
Not sure if this is the most optimal way, but it appears to work in modularizing the rebuild.(using 23.11)
I use Orbstack and it works flawlessly to do this. Really good tool. I use Docker to cross-compile for {aarch64,amd64} x {linux,darwin} since not all the cross-compiling is super robust across our stacks (I'm using a specific glibc for one Linux part, etc.). Just a bunch of docker on my Darwin aarch64 and it compiles everything. Good experience.
I installed Orbstack and found that I didn't really need it, so I removed the directory in /Applications. Wow, for weeks and weeks I found remnants of it in a lot of places. Very disappointing that it left so much cruft around. They should have an uninstaller. It left a really bad taste and I'm unlikely to try it again.
Before someone asks. I've been using macOS for a long time. I've never seen remnants like this from a program. Sure, there are often directories left in ~/Library/Application Support/, but this was more than that. Unfortunately, I didn't write down the details, but I ran across the bits in at least 3-4 places.
Dev here — I've been meaning to update the Homebrew cask to be more complete on zap, but there's a good reason that all of these are needed:
- ~/.orbstack
- Docker context that points to OrbStack (for CLI)
- "source ~/.orbstack/shell/init.zsh" in .zprofile/bash_profile (to add CLI tools to PATH)
- ~/.ssh/config (for convenient SSH to OrbStack's Linux machines)
- Symlinks to CLI tools in ~/.local/bin, ~/bin, or /usr/local/bin depending on what's available (to add CLI tools to existing shells on first install — only one of these is used, not all)
- ~/OrbStack (empty dir for mounting shared files)
- /Library/PrivilegedHelperTools (to create symlinks for compatibility)
Not sure what the best solution is for people who don't use Homebrew to uninstall it. I've never liked separate uninstaller apps, and it's not possible to detect removal from /Applications when the app isn't running.
IMO documenting this (and uninstall section in GUI with link) would be enough for me. Used that and never felt neglected by devs.
And cough since we’re at with - did you consider Nixpkgs distribution?
I’m slowly moving deeper and deeper into ecosystem and use Home Manager for utilities that I use often (and use nix shell/nix run for one offs). Some packages are strictly GUI and while they aren’t handled flawlessly (self-updaters) it’s nice to have them on a single list.
Yet based on your list it’s a definitely a nixventure…
And yet, you are the only(?) one with that knowledge, so the alternative seems to be replying to HN threads with a curated list of things that a user must now open iTerm2 and handle by themselves. Something, unless I'm mistaken, that computers are really good at doing (Gatekeeper and privilege elevation nonsense aside)
Even just linking to the zap portion of your brew cask could go a long way since it would be the most succinct manifest if I correctly understand what it does
I agree this should be documented, but I still appreciate uninstallers.
Also, I'm a little confused about your statement:
> Not sure what the best solution is for people who don't use Homebrew to uninstall it.
You said at the start you've "been meaning to update the Homebrew cask to be more complete on zap" ... does that mean Homebrew uninstall will not do a complete job?
But unfortunately cross compiling quickly broke when I started doing mild customization (and one reasons I’m doing this is a complex setup that’s very sensitive to version changes).
In the end solution was to “simply” get darwin.linux-builder up but that pulled a lot of weight behind it.
It works, but it’s not the first time I spent my time on nix-ventures.
On the flip side, once you have fixed the problem it has a very strong tendency to stay fixed. More importantly, the fix does not typically require me to remember that fix months later.
If something does break, rollbacks are free and an integral part of Nix.
macOS is a definitely rougher. I use colima there, and it does alright. There are one or two bugs with it, but I think those are primarily around volumes. But it does alright with building Docker images.
The rougher part is the speed of it; it's a one-two punch between the hardware & the fact that Docker has to emulate a Linux VM.
The way I've set this up for our macos devs at work was a script that runs nix builds inside docker-for-desktop using the official nixos upstream docker image (and some tricks to get ssh forwarding, filesystem mounts, ...) working. Works quite alright. Benefit is you don't need some weird Linux remote builder vm with ssh running.
Nix is absolutely the best tool for what I want to achieve but it has those dark forsaken corners that just suck your soul out dry.
I love it but sometimes it feels like being a Morty on Rick’s adventure to the compilerland.