Good data for VM folks, but I really have accepted at this point that the VM experience is almost completely distinct from the bare metal experience.
I'm really looking for a dependable pseudo-desktop/server OS for the monolithic legacy box at home. In terms of immutable, NixOS is it right now, but tbh it's had enough random uptime issues as of late, I'm starting to doubt my decision. Though I still haven't narrowed it down to this machine as opposed to some network/routing/storage problem
Have you tried Fedora Silverblue or Kinoite (the KDE version)? They also offer a fully immutable file system. The only hard part is adapting to life with an immutable filesystem, getting used to running things in containers/toolboxes, using flatpaks/appimage/snaps, etc.
I've been on it exclusively on my desktop and laptop for the past couple of months and have grown to like it a lot.
It is great to see immutable distributions becoming more popular. One thing that Ubuntu does very well however is the 5-year standard support for LTS releases (up to 10 years available via extended support).
I wasn't able to find any information on VanillaOS's support roadmap. Since the project's goal is to have stability of the underlying OS, it would be great if VanillaOS had an LTS-like support plan in mind.
It boots from a USB stick, loads system to a RAM disk, mounts configuration from a directory, and then hosts VM zones from ZFS datasets. The “root” system remains immutable.
To patch or upgrade, you just write a new system image to the USB stick and reboot. It’s great.
To skip the USB stick, you can do the whole thing over PXE.
After running a cluster on SmartOS for many years, moving back to Linux and installing the OS feels fragile, dirty, and weird.
The A/B-root trick exists purely to avoid having to physically visit a server to change the USB stick. (Can't overwrite in-place, as it might crash in the middle.)
PXE is its own hell, having local storage for the OS avoids some horror stories like "my whole cluster booted after a power outage and now PXE fails for everything".
I only learned about smartos recently, after using openindiana and learning about illumos based systems. SmartOS is a really neat idea; I'm surprised it's not more popular
I'm surprised immutable systems aren't more popular.
Back in the Knoppix days it was first a novelty, and then a blessing
that you had to boot from a CD-ROM, because it led to one amazing
outcome:
Less tinkering.
Or rather - it split use from tinkering.
Systems today are designed around the principles of deferral and
volatility. You can add or change anything at any time. The user has
absolute freedom to tinker, but also the vendor of always-connected
products has endless possibilities to update. The result is a mess of
dissatisfaction and half-bakery. Nothing is ever finished or fully
right. It also, maybe paradoxically, leads to systems that feel less
under your control.
Systems like TinyCore and Live CD distros take a different approach
that the OS is finished. You have two choices, take it or leave it.
Unless you are prepared to cross a non-trivial barrier to remix and
update the non-volatile image, you are forced to just use what you
have. That leads to more productivity because you adapt to the tool
rather than constantly adapting the tool to you.
I like TinyCore because it's looking to a middle ground of baking
immutable systems at key stage points and keeping changes separate
from the immutable core. I can change the core if I want to, but
rarely.
I see that as a separate prospect than "appliance platforms" like
Android and a PhoneOS onto which you can only load "apps".
What ideas and favourite solutions do other's have for using
immutability, or not liking it?
I helped write parallelknoppix when I was an undergrad - our university's 2nd cluster ended up being a bunch of laptops with broken displays running it. Took me a whole summer.
Then the next semester I am denied the ability to take a parallel computing class because it was for graduate students only and the prof. would not accept a waiver even though the class was being taught on the cluster me and a buddy built.
That I still had root on.
So I added a script that would renice the prof.'s jobs to be as slow as possible.
This is so typical German (I say that as German). It has changed quite a bit fortunately, but German professors are so often sticklers to the rules "aus Prinzip" (for a matter of principle).
> I'm surprised immutable systems aren't more popular.
They once were, before the internet made software updates "easy". There was a time when software was released (and often embedded without any option for upgrade/modification).
But immutable (mostly correct and reliable) software is more expensive and takes longer to develop, so it is out of fashion. Instead we now push out alpha quality MVPs and depend on updates to resolve known and unknown unaddressed issues as well as promised by not delivered features.
To be a little fair to developers, the user expectations these days are 10x or more greater than they were pre-internet-update days. After all, we now carry "supercomputer" devices in our pockets which also make phone calls, and we expect features and integrations which previously were not even imagined.
Even cars are now becoming update dependent for features and bug fixes. This is approaching a breaking point, I think. It will likely happen; there will be regulation and/or revolt; there will be a return to write-once approaches.
I idly wonder how long it will be before an OS could be "done". OS is proven to be free of security exploits, applications run entirely isolated permissions (network, disk, etc), and any remaining performance gains are incremental and optional.
I'd like to think it's possible, but sadly we don't even have the tooling to begin to correctly understand the problem in software let alone the unsafe hardware interactions.
Does macOS count? There are large chunks of the system that are impenetrable in real-world use. For example, in order to install the `yabai` window manager, here's what you need to do on your system:
If part of your program's installation process is "reboot into recovery mode", I feel like you're working with an OS that isn't quite as malleable as we sometimes are led to believe it is. :D
(btw I use `yabai` and it is totally worth doing all these things...I wouldn't set it to auto-update, and I review every release carefully, but so far so good. It's a great WM for macOS!)
Fedora Silverblue certainly seems popular here. I use it regularly and it’s been solid. I’m not sure how many people use Silverblue, but I recommend evaluating it!
I have been tempted recently to install Fedora Silverblue for trying it a daily driver in a spare computer I have at home, however, I read in many forums people having to implement some nasty workarounds for common things I use (e.g Docker/Podman). Perhaps in a near future when I have more time.
Technically you can install whatever you want the traditional way by "layering" it on your base image. I.e. just do a `rpm-otree install <package>` instead of `dnf install <package>` then reboot.
The hard/fun part is just figuring out a way to install and use the software you want without doing that, since that kind of defeats the purpose of using silverblue in the first place. In most cases, you can use a flatpak, a toolbox, or a container.
However, if you can't figure it out, there's no real harm in giving up and layering whatever you want.
> I'm surprised immutable systems aren't more popular.
Well technically near every router and embedded device is "immutable", with usually only config partition being changed, and firmware update being just re-loading whole image.
Core of macos itself is immutable and won't let you even add kernel modules or enable/disable system services, and IIRC it lives on a separate partition that is R/O.
> I'm surprised immutable systems aren't more popular.
Sadly, because they inherently make it so hard to keep up to date with the latest version of proprietary software like nVidia. Or they don't provide support over a wide range of (desktop and embedded) architectures.
Often the best choice you can make is the distro that has most users, so you are sure that it will get the attention and support from third parties.
That's exactly how I run my daily driver, my system is read-only, and I use overlayfs to put a read-write layer on top stored in RAM. This way all changes get removed every time I close the computer.
For my own files I have a partition which I use exclusively for data.
When I want to make changes to the system I have a boot option to disable overlayfs which I use once every few months for updates.
It depends, for most local apps you just have to log-in once while in upgrade mode (read-write on base system and overlayfs disabled), then the cookies will persist indefinitely and you'll be logged-in whenever you boot your computer in read-only mode.
For websites the cookies tend to expire once every few months so I'll boot back to upgrade mode and refresh my log-ins everywhere at once every few months at the same time as I install my system updates.
You can also selectively copy some software's local data to your persistent read-write disk partition and create a symbolic link in the original directory, that way this particular piece of software will be white-listed in a sense and always writable. But I haven't needed to do that.
My only issue has been google which has a weird cookie implementation, it throws cookie-mismatch errors all the time so I have to delete them and relog-in every day. If it were a big issue I'd make my firefox profile always writable as described above but I don't use google much.
My take on why they aren't more popular is that they are difficult to produce. Packer works, but it's slow and the artifacts it produces are bloated. I've been dreaming up a better tool for a while now that would make it possible to build images much faster regardless of whether you're on macOS/Windows/Linux. The hope is that it would be good enough to replace mutable systems managed with configuration management. Not really ready for use yet though.
Fedora Silverblue is probably one of our favorite Linux distributions nowadays. It was also our first experience with GNOME Wayland and we don't regret it.
I have found IODD 2531 (there are likely others) to be super-useful for that sort of a thing.
It presents an ISO (using the rocker on the side as a selector) from its HDD as a bootable CDROM target, and still exposes the HDD/SSD as a separate target.
So, it is possible to boot what is an RO OS image, and use the HDD/SSD target as an overlay, but because it is not flash memory, the wear issue is far less of a problem.
And, since it is presenting ISO images as boot targets, it is possible to (relatively) easily test upgrades while keeping the overlays separate.
Immutable doesn’t mean no updates. It just means that the user can’t change the system, and also doesn’t have to. An immutable System can have updates every night, as long as they are compatible.
It’s a bit like iOS and android, the system is the same on all devices of the same model, there are just different configurations and different apps.
It's as immutable as burning a new bootable LiveCD every time you update except that the images are kept on the HD for convenience and speed. Once booted, files stay pristine until the next reboot.
Everybody got scared from Windows XP having to reboot twice per day and wait 10 minutes just to change your display resolution or install some Windows Update security patches. Updating display drivers without restart, was back then announced as a major feature of Windows Vista.
Funny because now, that's how both Android and iOS behave, with the difference that OS updates only come every other month and AB partitions making the installation process perceived as fast as only rebooting.
> That leads to more productivity because you adapt to the tool rather than constantly adapting the tool to you.
This is just vastly untrue.
A bespoke tool will always be more productive than a one-size-fits-most tool, and thankfully with development tools the work/time of fitting your tool is marginal to the hours you work with it, and the effort is heavily front-loaded. After a few years you have a solid setup that only requires tiny amounts of polishing every few hundred hours.
The paradox of choice is real. A well configured bespoke tool may be more efficient, but if it’s hard to reach a configuration that you like, then a non configurable tool that has well selected choices may be more efficient simply because it eliminates the time, energy and mental effort spent on configuring and thinking of configuring the tool, and trying to get comfortable with the various configurations you may tinker the tool into.
So the real correct answer to whether bespoke or pre-configured tools are more efficient is “it depends”.
> A bespoke tool will always be more productive than a one-size-fits-most tool
Others have noted the FOMO/paradox of choice issue, so I’ll list another: multi-context environments. I work in a highly collaborative environment where I need to share tools and sometimes whole machines. I rarely get to use the same configuration for any particular task. So bespoke, to me and many of my colleagues, means more stuff to learn. Standard options and configurations mean much higher levels of productivity.
Short version is that MacOS made it so root cannot modify certain files without rebooting, disabling security and then booting to change those files. It can also stage changes to reload on boot, so for example, when you get a new version of the Zoom client, you actually have to reboot to make the camera permission go into effect. It's probably safer, but as a user, having to reboot just to turn on the camera is... MS Vista like...
I've been running the latest version of macOS and I've never had to reboot to enable camera permissions for any app. I haven't had to do it on fresh install or upgrade. The only thing I've had to do is close the app and open it again after giving it camera permission on first install. Subsequent updates seem to keep that permission enabled...
"I'm surprised immutable systems aren't more popular."
In the security world, an immutable system is an unpatchable system. If a 0-day is found in the system image, you can't fix it, you have to get a new image going. This causes wasted personnel-hours, wasted money, and some times wasted hardware if the image happens to be on a ROM-type of data storage. I've seen immutable systems on WORM HDDs. The people running those types of systems quickly got tired of having to do hardware replacements for every patch.
You are misunderstanding what immutable infrastructure is. You don't buy some SSD with a preinstalled OS and use it as it is for 10 years.
The point is in not making changes on the fly on live systems. You are perfectly able (and encouraged) to deploy new nodes with updated software and roll over the old nodes.
That's popular, but that's not an immutable OS we're talking about that's just a hypervisor. Routers also work like that, but we're clearly not talking about them.
My impression from the splash page is you're more encouraged to install packages to the base system. Silverblue only has that as the last resort (after flatpak, toolbox and podman).
>you're more encouraged to install packages to the base system
From documentation, I'm not even sure how this is done. The only options for packages documented are platform-independent packages (Flatpak, etc) and apx (distrobox-based toolbox alternative).
This is the command for silverblue. For Chrome to work you'll need to enable it in "Software Repositories" (which you get to through the hamburger menu in the "Software" app).
rpm-ostree install google-chrome-stable vim mozilla-openh264 virt-manager libvirt
Maybe. I'm more comfortable signing into my accounts using Chrome packaged by Google than by a third party. I should probably familiarise myself with the linked repo to show I'm worrying about nothing.
It not only is packaged by a third party but also uses a community-written patch[1] to replace Chrome's built-in sandbox with Flatpak's own sandbox. It's unclear to me whether Chromium upstream has vetted or is even aware of this modification.
Google only packages Chrome for Ubuntu (deb) officially. If you are not using Ubuntu, you are probably not using Chrome directly from Google. However, if you use flatpaks you might as well trust this one package from flathub since the trust model is the same.
From a quick look: The A/B partitions can be modified but it doesn't look like that is carried forward ("rebased") as with rpm-ostree. For package installations independent from the base OS there is something similar to (Fedora) Toolbox, called apx, where you also manage permantent containers for GUI or CLI apps that aren't available through Flatpak/Snap/AppImage or similar.
I noticed it lets you pick a primary package format to use, I just want a centralized package manager that highlights where my package came from or what format its in. I dont care if I use 5 different approaches thats already quite typical.
What I do want to know is what package format is best for my use case: I want latest version of Python and other packages, and I am on Ubuntu, I dont want a new OS or docker. No idea which would be ideal or the pros and cons of each.
The A/B root layout originated in ChromeOS, which is the most widely-used desktop Linux in the world, with somewhere around a third of a billion deployed units now.
(It also is where CoreOS got it from, before Red Hat bought them and the product largely disappeared, AFAICS.)
As such, I'd say it's an extensively field-tested and well-proven disk structure. :-)
Sounds like the only immutable part is the non-writable filesystem of the root partition, which is updated by having a live and non-live copy (A/B partitions) with the live updating the non-live and then switching on reboot. Similar to how Android works with its read-only partitions.
From a whole filesystem perspective I think it's not accurate to call this immutable though, as you can presumably work around this with bind mounts that can be used to mutate (but not persist) any part of the read-only filesystem while the system is still running.
Just glancing at it (FYI, I run NixOS on my main dev machine), it looks like it uses ABRoot which gives you 1 install to fall back on if 1 fails. I love that NixOS gives me N installs to fall back on (where N is whatever you configure it to be) because I've often needed to go back more than 1 to identify a problem that I didn't notice until 2-3 updates later.
Actually, as a natural tinkerer, what I call the "NixOS seatbelts" (being able to pick any of the last N updates via GRUB to boot into) have saved me numerous times already. I no longer feel comfortable using any other Linux distro. The declarative config and per-project dev environments are cake as well (once you get past the Nix language, which is basically just "JSON-like, plus pure functions")
My main problem with Nix isn't the language, which is fine. My problem is that Nix shoehorns everything into it; I have to write my nginx config (for example) in Nix for some bespoke nixpkg.
I would much rather just version my nginx config written in nginx config, and let Nix pull that up, or something like that. Then you get all the benefits of Nix (I think?) but my configuration is also readable for someone who doesn't know Nix. Plus I can use my existing knowledge (as well as the wealth of online available docs), instead of something obscure.
NixOS needs to handle generating the config to make service configurations composable. If you don't want that, you might as well write your own systemd unit.
That being said, most services have an option to use verbatim configuration files[0] or fragments[1][2][3].
> If you don't want that, you might as well write your own systemd unit.
This might sound glib, but it's a great idea for a simple workaround for users who want to use some other system to template their config files. Wouldn't take much boilerplate at all.
A small "operation" note: immutable is an was an ancient kind-of dream, having something that's always in a known state even if it's run for significant amount of time. Formally it should give easy debug due to a real and substantial reproducibility.
In practice it never works well though: first immutable means far longer to update and these days updates are a continuous stream, secondly even if the system is really immutable the complete infra tend to be not, making the immutable part next to useless in reproducibility terms.
In modern terms a new concept born "idempotent" witch FORMALLY means "you can run it countless of time, it will works the same and do not even re-do already done steps, it ensure consistency of a system final state no matter the initial one". Such concept have more more practical applications, again in theory, but in practice it fail to be really idempotent beside trivial use cases. From mere Ansible Playbooks for an infra to NixOS idempotence is partially there but results tend to be not.
Long story short: IMVHO the road have a name DAMN SIMPLER DESIGN, simpler infra, as the sole way to keep anything working and easy to restore when it does not.
A bottomline: reproducibility for a server infra have some reasons, for desktops... Well... IMO it's a bit overrated in the era of "endpoint".
Just tried to install it in qemu and the installer was broken. It couldn't select the vda disk (the selection was disabled). There was no way to proceed further.
It's like booting off a CD-ROM every time. User files are overlaid over the boot image. Installing updates creates a new image from which to reboot. Old images are kept in case of failed update. Prevents some hacking too.
> Vanilla OS is an immutable operating system, core parts of the system are locked down to prevent unwanted changes and corruption from third-party applications or a faulty update.
Can't imagine the HN responses if this was Windows marketing text :)
The difference is that the user is still in control here, rather than having Microsoft foist another thing on them that makes it harder for the user to control their own machine.
depends, if the Windows text was honest it'd have to mention that the parts it locks down relate to exfiltrating your data and violating user freedoms.
* The installer lags unless you temporarily disable 3D acceleration until open-vm-tools are installed
* The Norwegian keyboard layout defaults to Dvorak, which I suspect isn't the most common one ;)
* The default installer resolution was a tiny 800x600. Click Activities and type Displays to get to the relevant settings.
Now for the more exciting stuff, like testing if Tailscale works out of the box.