BSD and Solaris (which also has kqueue, and something else IIRC) got this really right while Linux got this really wrong.

I still prefer NT's IOCP, but kqueue isn't bad. epoll… is what happens when you get someone who's never done asynchronous I/O to design a system for doing asynchronous I/O.

Solaris has its own form of completion ports too, I believe. https://web.archive.org/web/20110719052845/http://developers...

NT's IOCP is good, sensible model overall having used it some recently. It's mildly put off by some "gotchas" (like async calls occasionally returning synchronously if there's nothing else on the queue, so you have to be careful with send, recv, etc), but the actual design is good. Thread pools being a convenient, native API in Vista+ is also a bonus.

FreeBSD also has Netmap and DTrace out of the box. They will also get TLS support in sendfile(2) as soon as the patches from Netflix land in HEAD.

I understand the rationale for this (performance), but it scares me a bit to have something as complex as a TLS stack running in kernel-space, which I assume it will do.

The sendfile/TLS work from Netflix does _not_ rely on the TLS stack running in kernel space. TLS session management, negotiation, and data framing is dealt with at the application layer, via nginx/openssl. Once the TLS handshake is completed and session keys are derived, you bind your session key to a socket. The FreeBSD kernel then sees this, and when you call `sendfile` to push static data out of the socket through the page cache, it does "bulk encryption" at that point (presumably using nothing more than an AEAD or whatever TLS negotiated for you, with the given key). Basically, instead of read()/encrypt()/write() going through userspace, `sendfile` can just directly do something like `encrypt(static_data_in_page_cache_to_send, out_socket_addr)` when you call it, right there in kernel space.

The need for these cryptographic primitives isn't too onerous on its own, either; FreeBSD already needs them for IPSec (among other things probably), so the vast majority of needed, kernel-level encryption code was already there before this.

> ... which I assume it will do

I hate to lament or anything, but: ... why assume at all - they've written about it? I really wish people would just read the paper about this feature, because this is a large misconception about their work that nobody ever seems to get right, and I say this as someone who doesn't use FreeBSD at all, I just found the paper interesting. Everyone hears "sendfile with TLS support" and immediately jumps without actually reading.

Again, sorry to lament. It's just a personal nitpick or something I guess; the paper is very approachable though, so I encourage you to read it before assuming. I encourage you to do this with most papers - even if you don't understand it, since you might still learn something :)

A google search for "freebsd tls sendfile" brought me this immediately:

https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf

The main points about this are on page 2, last paragraph on the right column, and page 4, paragraph after the bullet points on the left column.

That said, you always have to look at the code to determine if it's really something worth going upstream, for all the usual reasons.

Sometimes I let my biases get the better of me. Thanks for the reference.

I agree, but there's already so much stuff that doesn't belong in kernel space inside Linux and FreeBSD, that this doesn't make it any worse. We will hopefully see the rise of something based on seL4 (microkernel) or Barrelfish's multikernel design (one kernel per core, no shared memory), and fix those issues at the root.