To me, a big benefit of ZFS' rampant layering violation is that it shrinks the surface of weird interactions and edge cases to learn about. You get to know the FS, play with it, and then off you go. Most other solutions require a lot of thought and planning. Perhaps there is a better way to tune the storage setup to exactly what you want, but with ZFS you aren't seduced into getting creative with the architecture.

So I'm in favor of native ZFS encryption due to its usability.

That said, it is not as mature as I'd like, and there is at least one very big issue to know about: I recently sent an encrypted FS with zfs send/receive, and the received copy ended up being unreadable. I'll see if I can find the bug again. So that means you should test your setup precisely.

[edit] Here is the issue: https://github.com/openzfs/zfs/issues/12594

The technical detail from that thread: "This happens because when sending raw encrypted datasets the userspace accounting is present when it's not expected to be. This leads to the subsequent mount failure due a checksum error when verifying the local mac. I tried unsuccessfully to tackle this in #11300. See also: #10523, #11221, #11294.

Edit: If you have critical data lost due to this case I could help you recover them."

And there's this comment [https://github.com/openzfs/zfs/issues/12594#issuecomment-929...] which has a lot more pointers to recent problems.

> [I recommend](https://github.com/openzfs/zfs/issues/11679) [not using](https://github.com/openzfs/zfs/issues/12014) [native encryption](https://www.illumos.org/issues/14003) [until it gets](https://github.com/openzfs/zfs/issues/12270) [a fair bit](https://github.com/openzfs/zfs/issues/12439) [more polish](https://github.com/openzfs/zfs/issues/12418) [in the future](https://github.com/openzfs/zfs/issues/11983) ([I'm only so hopeful](https://openzfs.topicbox.com/groups/developer/T86620b1082ed9...)).

Unfortunately this knowledge isn't wide-spread, and as far as I see the zfs devs seem to NOT recommend using encryption right now without accepting the risks of dataloss

That said, very few if any of them have involved actual irrecoverable data loss versus just requiring lots of handholding to get your data back.

edit: I suppose at the point where I'm being quoted on HN about this, I might as well link my list of encryption bugs.

https://docs.google.com/spreadsheets/d/1OfRSXibZ2nIE9DGK6sww...

I still need to add a couple of new bugs, but almost all of them are dupes or related to one of two flaws - a narrow race in the ARC (...probably), where you can sometimes NULL dereference for fun and sadness, and zfs change-key and send -i resulting in the receiver incorrectly resetting the metadata for which key an encryptionroot is encrypted by.

Another reason to favour it is because if you do ZFS Send/Recieve to do incremental backups (like with rsync.net), you can backup your encrypted datasets to an off-site location without shipping the encryption keys.

That means you don't have to be afraid of your backups somehow resulting in breaking your confidentiality.

So you can still retain your incremental ZFS based backup-flow, and you don't have to change any tooling to use it. With LUKS you will either have to store your backups unencrypted, or you lose the ability to do incremental ZFS-based backups.

If you want to do incremental ZFS send/recieve backups, you have to do it on the unencrypted ZFS volume, and thus your backup wont be encrypted.

That is you can't have both encrypted and incremental backups at the same time.

With native ZFS encryption you do get that.

But given that the bugs exist, anyone using encryption should test and verify all the functionality before using it in production.

The worst thing about this bug is that it doesn't manifest on the local machine but in a situation where things seem ok.

https://news.ycombinator.com/item?id=32342900

There is an outstanding issue [0] that I've encountered on two separate machines running in this setup. The issue occurs when using ZFS Native Encryption + (NVMe?) SSDs. The issue appears to be snapshots getting corrupted (or maybe failing to create?) occasionally. It happens roughly 1/1000 snapshots in my case. I take a lot of snapshots so this occurs weekly.

I haven't lost any data yet but this issue is annoying because I get spooky alerts from ZFS warning me about "potential data corruption". To clear them I have to manually intervene by deleting the corrupt snapshots, restart the machine, then scrub.

What's most annoying is that this breaks ZFS send/recv until I intervene. send/recv was the whole reason I went with native encryption instead of LUKS in the first place.

Again, no data corruption so far (if you don't count the snapshots that get lost, which I don't, because I have so many of them). Just very annoying and tedious.

[0]: https://github.com/openzfs/zfs/issues/12014

As others have said, I like the benefits, which are easy send/receive, etc (but mind the bugs some other commenter mentioned).

I will also give a few drawbacks / points in favor of using LUKS.

In my case, Arch usually has up-to-date kernels. These are not always compatible with the latest stable OpenZFS release. Issues go from benign: ultra-high CPU usage so my 980 PRO ssd is slower than spinning rust to the module not compiling at all.

Another drawback is that booting with systemd in initramfs doesn't support zfs native encryption. There's no support for using a TPM [0]. So you have to type in your password every time you boot up the computer. If you want hibernation to work, you need a separate swap, encrypted separately, so that's two passwords to type in.

Systemd recently got support for TPM so can automatically decrypt devices. I have this set up on a separate machine (with ext4) and it's nice.

---

[0] I know not everyone likes the idea of automatically decrypting the drive, but that's another debate. If your threat model allows for this, it can be done with LUKS but not with ZFS.

As far as I know, the issue is now resolved, but I haven't followed this closely.

[1] https://github.com/openzfs/zfs/issues/8259

I actually followed this closely, since that's when I had got a new Laptop with a new SSD and when I tried to benchmark it, I was going crazy trying to find the culprit. Because an older PC with a slower ssd (and older kernel at the time of benchmarking) was something like 25 times faster.

I don't think I'd use it on cold storage though, I like using it on systems I daily drive, such as workstations and laptops, where I feel sacrificing convenience is not worth it, and where I don't want to lose the option to not be able to select what's encrypted and what's not (for instance, I don't want or need for all of my VMs ZVOLs to be encrypted). With ZFS you can pick what gets encrypted, and with what key, without having to fumble around with loopback devices and other strange machinery.

In my experience it is very reliable, I didn't really feel any performance degradation after moving my system on top of encrypted datasets on any of my systems. I also happen to have to use every once in a while a system installed on an external USB-C SSD (with Arch+ZFS), and it works very well without any performance hiccup even with encryption on.

The only time I had to fumble with the dataset was when encryption still wasn't stable and I had to rsync everything to newer format datasets - something around 2018 I guess? Still, it was a pretty straightforward experience. I never had any other issue with encryption after it got moved to stable (3 years ago I think?), daily driving several (personal and work) systems.

On my cold storage, such as drives I backup my stuff onto, I still use LUKS + XFS/ext4/ZFS because it's tested and solid though - I don't really feel that moving disks I often keep offline to ZFS native encryption would help at all. I honestly worry more about them being misplaced or lost, than convenience.

In particular, with ext4+LUKS, there is no way to know if there are silent bit errors.

I wonder what is a compelling case to create a new ext4 volume as of now.

[0] https://github.com/longhorn/longhorn/issues/3597

[0] https://www.phoronix.com/news/Linux-5.14-File-Systems

I can't speak to the metadata leakage, but from my grumpy perspective the fewer layers in my IO path the better. Disks should be boring (ish).

That being said, I do use it on a laptop where losing the data wouldn't be a big deal, but having it easily compromised would be. It was just so much easier to transition to encrypted datasets; the whole thing was done with maybe 10 minutes of downtime.

Shameless plug but i wrote a nice cli script to manage luks containers.

https://codeberg.org/emanresu/Luksman

I'm not sure if complex/stronger is the right phrase to use here. If someone wants to learn more, this is a nice summary of differences https://crypto.stackexchange.com/questions/84188/is-xts-alwa...

https://sockpuppet.org/blog/2014/04/30/you-dont-want-xts/

ZFS's encryption is also certainly not as thorough as LUKS is. Depends on your threat model if that's a problem for you.

LUKS and other FDE schemes tend not to have the ability to have fine grained encryption keys. If you have raw access to the LUKS device on which your filesystem resides, you can recover plaintext of blocks that have not been overwritten or erased via TRIM or similar.

zfs on a self encrypting drive seems like a good combination. Even if the Opal implementation is garbage (see CVEs from a couple years back) the exposure is minimal because zfs is encrypting data. My measurements of performance impact IO with and without encryption suggest there is no performance nor increased cpu consumption with self-encrypting drives.

Suppose that I have encrypted ZFS datasets. Naturally, sometimes the process of working with datasets is interrupted (due to, eg, power loss and so on). Is it known how much plaintext could potentially leak to disk outside encrypted ZFS datasets, possibly accessible by forensic analysis?

That would violate the confidentiality guarantee in the event of an accident.

Related to this, if I recall correctly, if the encryption key is changed, ZFS may not destroy the old wrapped key and may keep it stored on disk outside the dataset. This is apparently due to the copy-on-write system.

When a write to an encrypted dataset happens, no plaintext data is written to disk. Your data is every bit as secure from disclosure as if the write completed normally.

Think of ZFS encryption like you would TLS encryption. They both hide your data, but leak metadata. With ZFS encryption, someone with access to the disk can see what you name your datasets (don't name your dataset "people-i-killed" or name your snapshot @before-starting-the-fire), the order that each block was written (via birth txg in block pointer), how much data was written to which datasets, when the last write happened (via ub_timestamp in uberblock copies in vdev labels), etc. Likewise, with TLS someone with access to the network can tell who the endpoints of the connection are, when each party sent data, and how much data was sent.

If you have full disk encryption under encrypted ZFS, you get the goodness of encrypted ZFS plus after your drive has been powered off, no one can determine anything about the content of the drive (except maybe that which is leaked via SMART and similar data). This is kinda like having a VPN and using TLS. Someone with access to the network under the VPN can still see that there's activity but has a much harder time figuring out information about the data and metadata with the VPN.

More information at: https://blog.heckel.io/2017/01/08/zfs-encryption-openzfs-zfs...

With ZFS there are other things to consider as well, but I would recommend looking at an article like here https://arstechnica.com/gadgets/2021/06/a-quick-start-guide-... that goes into much detail.

Like always it mostly depends on your use case/scenarios.

It works well but be aware that SWAP specifically on encrypted ZFS will lead to deadlock: https://github.com/openzfs/zfs/issues/7734

Thus I have planned to move the swap back to LUKS. If I cannot unlock everything by entering only 1 password entering due to that, I might move everything back to LUKS for that reason.

Nixos with natively encrypted zfs mirror.

Next step is impermanence

OpenZFS has a very healthy and active community working on it.

And the code has been much improved compared to originally open sourced version by Sun. Technology has moved on since then and so has ZFS.

My perception is that the ZoL based fork had a lot more instability and bugs compared to the parallel development of ZFS by in the Open Source Solaris world. That OpenZFS is based on the more unreliable code base should be evidence that the values which made ZFS excellent under Suns steermanship are insufficiently maintained and that features integrated into ZFS after the fork who did come from the ZoL side should not be used if one wants the reliability of ZFS.

The fact that not all metadata is encrypted shows they are doing things wrong from a principled standpoint. Sun didn't halfbake important stuff like that. The ZoL community will in the long run make the reliability of OpenZFS converge against the reliability of BTRFS because that is the level of reliability open source allows for. Initially higher stability be dammed.

With Solaris zfs, you saw features after they were exercised by up to a few hundred engineers for months to years. Now you are getting bits fresh from the factory. Like when software is developed behind closed doors, bugs happen and they are progressively squashed leading to a more stable product.

That is disprove by the fact that original Sun encryption did the same thing and the one currently in Oracle ZFS does so too.

This was one of my concerns when I first encountered this trade off in Solaris zfs. That is, you claim that the Solaris zfs encryption is better in this regard but it’s not.

And when something breaks (maybe because one of those numerous bugs) there are no proper tools for repair. Your ZRAID and ZLOG probably won't matter. The to-go tool for removing corrupt superblocks is a 10 year old hacky python script, it's ridiculous.

I have seen folks get in over their heads though. Like -- "OMG I've somehow broken boot for ZFS on root and I don't know how to fix!!" or "I'm using a custom kernel and OMG things don't work!!"

Canonical has even been complicit. When you download an HWE kernel, you don't also download a newer/matching versions of the tools. This should pretty clearly be considered a bug[0], and fixed but Canonical perhaps doesn't have the bandwidth?

I just can't lay any of these problems at the feet of ZFS though. Right now I trust ZFS a hell of a lot more than I trust Linux to Do The Right Thing. When ZFS seems broken (though I have little experience with native encryption so this may not apply to it), I am far more likely to think the issue is Linux, my distribution, or me, than any problem with ZFS.

I'm a pretty serious ZFS lover though.[1]

[0]: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/193... [1]: https://github.com/kimono-koans/httm

In 15 years of using zfs, I’ve never lost a pool when it was in a mirror or raidz layout.

I've used ext4 for more than seven years and ZFS for a few months. ZFS shat itself irrecoverably - ZRAID, ZLOG (PLP), ECC RAM and UPS be damned. People's issues sit open on GitHub for years.

ZFS on Linux is empirically and anecdotally not trustworthy.

If you stream your ZFS data from one host to another then the encrypted filesystem remains encrypted at the destination.

If your encryption is done outside of ZFS then the data on the destination is in plaintext.

This might seem obvious to those experienced but to a newcomer it caught me off guard and the interaction between recursive send/receive and encryption was more complicated than I had expected.

I should add that I haven’t looked into this for a while and could be completely wrong but it’s worth researching, at least, because my spideysense is definitely tingling to the tune of something about this was odd and worth knowing about.

This post might give a better hint to the actual details: https://www.truenas.com/community/threads/cant-zfs-send-betw...

Sorry to not be more authoritative. Bottom line: if you are going to use send/receive, spend some time to build a run book for your use case sooner rather than later.

Arch has a good wiki page comparing LUKS vs ZFS native vs a few others: https://wiki.archlinux.org/title/Data-at-rest_encryption

And more discussion of differences over at /r/zfs: https://www.reddit.com/r/zfs/search?q=zfs+luks&restrict_sr=o...

I guess it is about to proceed onto zfs native encryption.