It always struck me as odd that Go has select{}, but no way to select on a file descriptor such a socket. There's literally no way to poll a file descriptor: You have to read from it, and deal with the result.
That also means reads aren't interruptible (unless you count closing the descriptor as interrupting, which is a blunt hammer indeed). AFAIK the only way to do this is to set SetReadDeadline() on the connection to a smallish value, then let the interrupter block on a channel that's invoked when the deadline is reached.
Go's concurrency model is pretty friendly, but it's also unfriendly in many surprising ways. (Don't get me started on the whole nil channel thing, or the problem of channel closure/ownership, or channel performance overall.) It doesn't surprise me the least (while it did surprise the Go team) that Go fell flat among the C/C++ system programmer demographic.
Anyone want to chip in about how Rust, Swift and Nim compare here (regarding the article)?
> You have to read from it, and deal with the result.
They try to keep it simple. As with all I/O: it's blocking your current goroutine, so if you want it non-blocking you spawn a new goroutine to do this in, pass it a channel, the spawned routine proxies the result to it and your main routine does select on that.
The semantics for reading sockets are different from reading channels (compare the types; one is multi valued, the other single valued---EDIT: no they're not, exactly, see [1]), so it kind of makes sense you can't share the same construct.
I'm not a Go apologist by any means, I think it has a tremendous amount of flaws. But I/O, (a)synchronicity and concurrency are not among them. They are simple and consistent, and (most importantly) orthogonal building blocks which you can combine at your leisure to create more complex constructs.
(Besides, how do you interrupt a read, anyway? Or any syscall? You can't do that in C, either...?)
[1] edit: fair is fair, go channel reads are (of course) also multi valued. however, the types still don't exactly match, since you can't pass arguments (buffer) to a channel read. you'd need some extra hoops to jump through, as other commenters suggest, which starts becoming magic and breaking down the simplicity and orthogonality of the constructs.
Sorry for being unclear. You don't interrupt the read as such. You interrupt the select.
The usual trick on POSIX is to use a self-pipe, since select()/epoll() only work on file descriptors and not, say, pthread primitives. Windows is slightly better here: WaitForMultipleObjects supports waiting on any primitive that supports blocking, so you wait on the socket + on an event object, and signaling the event object will unblock the wait. Windows also has completion ports, and the APIs are generally more extensive (not a fan of Windows, but they got this part very right).
I do understand why Go is the way it is. But it's important to recognize that it's a compromise in design that inherently reduces your flexibility — such as in this particular case. I don't think there's anything technically preventing Go from supporting non-blocking I/O, though.
Channels and FDs/sockets are both conceptually multi-valued; what did you mean by that? One major difference is that reads can fail with an error, and channel reads can't.
Edit: Also, I didn't mean that select{} on a file descriptor ought to actually return data. It would be more appropriate for it to yield a true/false value to signal that the FD has data, so you'd do "case <-fd: fd.Read(bufSize)".
Race condition? It's no different from select() followed by recv() in C. But I'm talking about semantics here, not suggesting that this would necessarily be how you'd write it. The main idea was allowing select{} to work on non-channel sources of notifications, such as file descriptors.
You can't select on a read, though. select{} only works on channels. You can run the read in a separate goroutine, but that doesn't stop the read from happening.
There's a problem lurking behind this "simple" model: the goroutine you park on that blocking IO operation can't be collected reliably itself.
I care about collecting goroutines, and would advocate that everyone should care about this. There's been a litany [1] of blogs about this recently, if anyone doesn't consider this self-evident. It's just systematically sane. Letting goroutines spin off into the distance with no supervision is extremely likely to represent either a resource leak, a missing error handle path, or an outright logic bug.
In a dream world, we might have Erlang-style supervisor trees so this can all be done with less boilerplate. It's almost even possible to do this as a library in go! ... except for this blocking IO problem, which throws a wrench in things: any supervisor of a goroutine that may be IO-blocked may itself be indefinitely stuck, and so on up-tree, making the concept unworkable.
---
[1] Some recent articles hammering the point of "collect your goroutines":
It takes the Wait-for-children behavior path by default, and that seems much safer to me. The downside is, as we're describing: if you do get stuck in IO somewhere, you're a pretty unhappy camper. This library makes the choice of (by default; it's a handler func you can override) logging warnings to stderr if a task doesn't exit within 2 seconds of being told to quit... but this still feels like something of a hack; more of a debug measure than a concrete way to improve system stability. Better ideas would be highly welcome!
So, soon I will have some stuff to point you to with regards to how Rust is doing it, but we're not _quite_ there, so as a summary...
With Tokio, you write non-blocking IO stuff as "futures", similarly to promises in JS. These then get compiled down to what's effectively a state machine, and they run in an event loop on top of epoll/kqueue. That event loop is single threaded, and fully non-blocking.
I am not a super, super expert on these low-level details, but I believe that this should mean that it is closer to the C code than the Go code.
This sounds similar to what we have in Nim right now. There are multiple layers (from highest to lowest):
* Async await
* The async dispatcher (event loop)
* Selectors module
Each builds on top of the layers below it. The selectors module is an abstraction for the various system `select()` mechanism such as epoll or kqueue. If you wish you can use it directly, and some people have done so successfully[1]. But you must be aware that on Windows it only supports `select()`.
The async dispatcher implements the proactor pattern and also has an implementation of IOCP. It implements the proactor pattern for other OS' via the selectors module. Futures are also implemented here, and indeed all the procedures return a future.
The final layer is the async await transformation. This allows you to write asynchronous code in a synchronous style and works similar to C#'s await. The transformation works by converting any async procedures into an iterator.
This is also single threaded and fully non-blocking and does give you a lot of control. In theory you should be able to work with any of these layers interchangeably (as long as you only care about non-Windows platforms).
I like futures, though cancelation has historically been a controversial topic there, too (at least in the JavaScript world; JS promises are not cancelable, which is problematic for HTTP requests [1]).
I do like that you guys chose to not make green threads a language-level construct (as Go did with "go" and goroutines), which means you have the option of supporting multiple models, for the programmer to choose according to their particular use case.
Cancellation falls out of the model: you drop the future, and it Just Works. (drop being the regular old Rust destructor stuff, which happens when it goes out of scope, or when you call std::mem::drop.)
Yup! The danger there is that the ecosystem may fragment, however, basically everyone is rallying around tokio, including the other people who were working on similar libraries, so in practice, this should be fine.
I still think that John Reppy's Concurrent ML design is the best thought out way to do non-blocking IO and event handling. There has been a lot of progress on CML but Reppy's 1992 PhD dissertation remains a very readable introduction: http://people.cs.uchicago.edu/~jhr/papers/1992/phd-thesis.ht...
Of course it is a bit of a pain to write OS specific variants if you are building a completely crossplatform solution, but it is supported out of the box:
That said, I agree [regarding the] puzzling decision to not to treat 'select' as a general contract for all asynchronous ops as that fits perfectly with the CSP approach.
This would be solved if the "select" keyword was also working on I/O objects. It could gain special io.ReadReady and io.WriteReady channels that return a boolean.
This would also simplify most of the code that tries to pull events from different sources, including I/O. Right now each I/O needs to be extracted in it's own goroutine (adding error handling makes it worse):
fromIO := make(chan []byte)
go func() {
for {
buf := myPool.Get()
myIO.Read(buf)
fromIO <- buf
}
}()
for {
select {
case a := <-fromIO:
// do one thing
case b := <-fromOtherChan:
// something else
}
}
After the change:
for {
select {
case <-myIO.ReadReady:
buf := myPool.Get()
myIO.Read(a)
case b := <-myOtherChan:
// ...
}
I/O isn't an object, it's an interface (Go has no objects). There is nothing magical about I/O or any interface--they are just functions--so this change would require extra magic to be implemented which I strongly oppose.
One of the beautiful things about Go is the designers carefully chose to confine language magic to only a few fundamental areas (channels, goroutines, arguably maps) and resisted going beyond that. Go approximates "C plus Concurrency and Better Data Structures" and nothing more.
On the contrary, the magic is required because things like select and channels and coroutines are implented at the language level in go. If they were a library with proper extension points and, sure possibly languge sugar on top, it would be doable.
The compiler actually just transforms everything channel related into function calls within the runtime package. The majority (probably > 99%) of the implementation of channels and select is Go itself.
"There is nothing magical about I/O or any interface--they are just functions--so this change would require extra magic to be implemented which I strongly oppose."
"Extra", yes, "magic", debatable. (Which, to be clear, I mean straight, not as a rhetorically-weakened pure disagreement.) There is precedent for using the io.Reader and io.Writer interfaces but allowing structs that implement those interfaces to implement additional interfaces to gain additional capabilities already, especially in the io.Copy function: https://golang.org/pkg/io/#Copy (Click on the Copy name to pass through to the underlying implementation, which bounces through to another function which should still be on your screen, and the first few lines show the special casing.) Though I also find myself with some frequency having to accept a Reader or a Writer and examining it to see if it's also an io.Closer, because if I want to wrap a new io.Reader around an underlying io.Reader for some reason, something I do quite a lot, it's important to propagate proper closing behavior, especially when wrapping with gzip or something else that really needs to be closed and not just sort of trail off at an arbitrary point.
Making io.Reader "just work" in a switch statement would be magic, but using a similar technique to the above it might be possible to offer a new "io.ReadableChan(io.Reader) <-chan struct{}" (or the obvious WritableChan extension) function that files or sockets would implement using a polling mechanism, and everything else would get an automatic new goroutine created that used just io.Reader's existing interface. This could offer a zero-buffer-until-IO implementation without having to modify io.Reader. The resulting channel coming back from ReadableChan might have to be a new sort of magic channel internally, but that would be an internal detail that shouldn't ever rise up to your code.
Alternatively, it might be a "io.ReadableChan(func() []byte) <-chan []byte", waiting internally until it can be read then determining the []byte to use for the read by invoking the provided function. I'm just spitballing here, not preparing a change proposal. (For such a thing I'd also want to consider whether that should be "func () ([]byte, error)", for instance.)
This is one of those places where Go favors practicality over purity, and is very much not channeling Haskell. Checking whether an interface value implements another interface is probably something you shouldn't be doing all the time, but it's not inherently un-Go-like or anything. You're just taking more responsibility. Much like how you are supposed to share memory by communicating but Go will still let you communicate by sharing memory if you want.
The epoll/select syscalls and the file descriptors for TCPConns are exposed, so a lot can be done in third-party packages; I linked some relevant stuff in another comment. It's kinda hard for me to imagine explicit wait APIs being deeply integrated with std, though; maybe if things reached the point where sitting on buffers became the bottleneck for a lot of apps (in which case, dang).
The fsnotify lib at https://github.com/fsnotify/fsnotify has a similar challenge at a high level (ferry events from various syscalls into a Go chan) and may be interesting as a reference.
But: perspective. If KBs per idle conn are eating a well-provisioned modern server box whole, then back of envelope you might be at, like, hundreds of thousands to a million conns for a single box (few KB/conn * 1M = few GB), and, like, huge congrats for getting that far! Not the absolute limit of the hardware, maybe, but clearly not a toy!
Other folks also note that once you're juggling that many connections there are other things that could become big factors in scalability, like whether your kernel and network will remain happy, can you keep up if more conns than expected suddenly become active, and (I'd add) will the rest of your code (the stuff besides reads/writes) be able to keep up.
Still, if anyone wants to wrap the epoll-results-to-a-chan thing in a package, they can always post something + see if anyone uses it to do cool stuff. As you suggested, it's plausible there are places it can make code cleaner.
This would kill composition. If my reader were a tls.Conn wrapping a net.TCPConn, what would it mean to be ready? Does every io.Reader now need to be an io.ReadyReader? How can a framed Reader know if it's ready without holding a buffer?
There is a race condition, or at least an avenue for a race condition, in this design. Even if myIO.ReadReady is unbuffered, there is nothing to stop another goroutine from calling myIO.ReadReady after the select case is entered. The channel receive would have to return an exclusive handle of some type that could used to read from myIO. And then at that point, how do you signal to myIO that you are done reading? Should myIO "unlock" after a single Read call? After all pending data has been consumed? What if you're using a parsing package that calls Read multiple times? How do you keep it from blocking?
Usually IO read (or writes) should only be handled by a single goroutine at the same time. I don't think that the Read() function provides concurrent access guarantees either.
But you're right that myIO.ReadReady doesn't give the guarantee that there is enough data to fill the len(buf) on the next Read call, which would make it then blocking.
One of the features which I would most love to see in Go is a Selectable interface, which could make things like this doable (albeit as a bit of a hack).
Even if a socket's Selectable implementation just abstracted away the goroutine from your first example, it would make the developer experience a lot nicer for me.
Agreed, this would be a cool change. Part of the problem is that the only way to "kill" the goroutine doing the read is to close out the fd from under it. However, depending on what you actually have open (e.g. a unix device), it often gets stuck.
Having select{} work on io.Reader and io.Writer would be great!
Forgive me if I'm wrong, but I believe this is because the select keyword is a language syntax convenience whereas actual I/O needs a full go routine because that's the unit of parallelism.
No, depending on the compiler it is where the call that calls main() and calls global initialization lives, also where support functions for compiler intrisics might live, also the implementation support for data types not directly supported by the architecture, e.g. floating point emulation.
Right... which means that "select" is like "class". It means some shit, but it isn't supposed to mean "KERNEL ASYCIO INTERFACE" the same way that "class" doesn't mean struct.
It's true goroutines waiting on the network are going to sit on a few KB (small stack, buffer, user-mode sched bookkeeping), and if you multiply that out by a million or something you're talking gigs. But I think much of the appeal of the standard Go way, and similar approaches in other languages, is you can write more or less as if you were doing simple synch I/O, and the runtime work that other folks have done gives you something decent with AIO, small stacks, user-space thread switching, etc. without you having to think much about it.
I think the author says something similar put differently in the first paragraph.
Something like rakoo's trick is interesting, with the caveat zzzcpan added that it takes at least a tiny buf, not zero-len, but if you don't actually hit this wall I'd charge forward the regular way.
You actually can observe the same behavior in most async APIs which follow a "pull model". E.g. if you do "await socket.ReadAsync(buffer...)" in C# you would also need to have preallocated the buffer and keep it alive for the whole duration of the async operation. Same is true for C++ boost asio async reads. Even if you use the pure C WinAPI IOCP methods you have to preallocate the buffers.
The readyness model avoids this, as described in the article. But it will also lead to a more event driven programming model.
I think the preallocation might be a problem if the use case is a server that holds an enormous amounts of mostly inactive connections (e.g. websocket server). For most servers I don't see a problem with it.
This method also has a benefit for performance: You often can totally avoid dynamic memory allocation during runtime with this model. You allocate the receive buffer once for the connection and then reuse it for the whole lifetime. In the push model you either need to allocate a fresh buffer for the new data or retrieve one from a pool, which is still more expensive.
> This method also has a benefit for performance: You often can totally avoid dynamic memory allocation during runtime with this model. You allocate the receive buffer once for the connection and then reuse it for the whole lifetime.
There is a security concern here; you'd better be sure to flush the buffers correctly or you could cause memory to leak between sessions.
If I were writing a server to handle a sensitive task such as executing payment transactions I'd probably take the less memory efficient approach of allocating a new buffer for every connection to lower the risk of data spilling between connections.
allocating a new buffer doesn't give any guarantee as you might be given back a just dealocated buffer by the allocator with its original content mostly intact. You need to explicitly scrub any sensitive data yourself before deallocating. At that point you might just reuse the buffer yourself.
Read() being a blocking call, wouldn't some hack like that work ?
_, err := conn.Read([]byte{})
if err != io.EOF {
return err
}
// we know there is something to read
buf := pool.Get().([]byte)
n, err := conn.Read(buf)
// process n, err and buf as needed
// if there is more to read, you may need to loop over conn.Read
// after some inactivity timeout
buf = buf[:0]
pool.Put(buf)
No, it checks for zero-length buffers and returns immediately. You would have to read at least a byte and copy it into a new buffer.
Won't matter much though, as the memory usage of a single goroutine is quite significant and doubling it by having a buffer preallocated is not something to care about.
> Won't matter much though, as the memory usage of a single goroutine is quite significant
2K as of 1.4 (was 4K before 1.2, 8K in 1.2 and 1.3).
But yeah it's a good point, the intrinsic memory overhead of goroutines means even if buffer pooling worked the memory use of the system would still mostly follow the "naive" estimate.
You may want to message the author to remind them of that, they may not have thought of that concern.
Depends on how Read() is implemented. Unless the documentation explicitly says otherwise (and it doesn't look like it does), it would be perfectly reasonable for the method to say "this buffer has a size 0, so I can just return immediately without even touching the fd".
Have never tried it with Go and don't know that the contract is there, but the approach in general is valid. Afaik you can use it with Windows IOCP to avoid the occupied buffer problem: Start an async read with a zero byte buffer and when you get the completion notificaton issue a non-blocking read with the then allocated target buffer.
> Suppose that typically 5% of the client connections are actually active, and the other 95% are idle with no pending reads or writes.
I suppose the intent is to then use that 95% memory savings for other work.
So what's the point of making this optimization? The system is going to be hugged to death if the number of client connections approaches 100%, because the system will not have enough available memory.
If 100% client connections is causing memory problems, rather than keeping a pool of buffers so that inactive client connections have less footprint, it seems like a better solution is to decrease the number of available client connections. Then, provision more servers or more memory if more client connections are required.
Provisioning for the worst possible case is sometimes necessary (e.g. hard real-time use cases), but can also be very expensive.
Since HTTP connections are typically kept alive between requests, it's very common for them to be idle. It could be the case that 99.9% of the time, X bytes of memory is enough to avoid transfer delays, but to cover the 100% case, you'd need 10X that.
I think it's reasonable to decide that avoiding delays in the 100% case is not worth 10X the cost in memory.
The answer can always be "throw more servers/resources at it", and that's exactly why we end up with simple websites/programs reacting slower than their equivalent two decades ago. I have too many apps on my phone that don't work properly and take ages to load or perform simple tasks. I tried to report this for some apps, the answer usually is that my phone is too old or my internet too slow (I know not directly related to a server, but I guess developers wouldn't think differently there).
If you use a magnitude fewer resources, you can combine more services on one machine, ultimately allowing faster servicing and a better user experience.
And as I understand that's actually how many Go programmers think. If you don't care about resources, you can set up a simple server in Python. The beauty about Go is that it gives a good developer experience while still allowing to write efficient, optimized programs.
It uses a pool of memory. If the pool is full because too many clients become active at the same time, only a part of the clients will be served in a timely manner. The other clients will have to wait for their reads to be processed. For a proxy handling long-running connections, this seems acceptable to have a slight delay in the 1% of cases where too many clients wake up at the same time. Buying 20× the number of servers don't seem a sensible solution.
Instead of supplying a buffer to each read call itself, you can use something like a sync.Pool which will give metered and concurrency-safe access to a buffer: https://golang.org/pkg/sync/#Pool.
I think you misunderstand. It's not that you can't get a pooled buffer in Go. The problem is that you don't know if/when you need that buffer without calling Read(), and the call to Read() requires a buffer that's ready to use.
Yep, you're totally right. Reread a couple times and the crux of the matter seems to lie here: "Unlike with C, in Go there is no way to know if a connection is readable, other than to actually try to read data from it."
That wouldn't make much sense since the Reader interface requires []byte, but the specific connection could inherit WriterTo: https://golang.org/pkg/io/#WriterTo
I wasn't very clear. A hypothetical conn.Read(w Writer) would call Write on w instead of filling a byte array. Write in turn allocates as needed. WriterTo is a cleaner way of doing this.
> A hypothetical conn.Read(w Writer) would call Write on w instead of filling a byte array.
Yes I understand the purpose — hence suggesting WriterTo which is supposed to do that — I simply noted that conn.Read(w Writer) would make Conn not extend io.Reader anymore since it takes a buffer.
This is something i've wondered about for years. If you have a million sockets, and you've set SO_SNDBUF to 4 kilobytes for all of them, does that mean the kernel has 4 gigabytes of memory sitting around waiting for TCP segments?
In the classic C10k scenario, most sockets don't have a segment in flight at any point in time, so their buffers will be empty. It would be a huge waste if they were actually occupying memory.
You're right. But there will realistically still be some protocol-level buffering in user-mode for every socket.
Although that might be a good use-case for a hash table of buffers, as most of the sockets will not have outstanding partial packets most of the time in typical (frequent small message) applications.
Is this inherent in Go's concurrency model or just a limitation of the standard library? Couldn't the standard library provide a function that takes a buffer pool as an argument instead of a buffer?
One could imagine a function that takes a set of net.Conns and returns the ones that are available for reading. You'd then pass those off to a goroutine for processing with blocking I/O as usual.
Since IO Multiplexing in go is based on Edge-triggered epoll, you always need to do a write/read syscall (it happens underneath) before blocking the goroutine to arm the epoll to poll that specific FD. Also this approach relies on parking/unparking the goroutine and there is no way around it, since it is part of the runtime.
One way go can make it nonblocking is to do the read/write and return the result immediately, but this kind of call requires a call back function to be provided, so instead of parking/unparking the goroutine, the call back function is called (either directly or by "go cb_function"), It can be tricky to make it work, since the initial goroutine might not be allowed to call read on FD anymore (maybe exit from the goroutine if the read was unsuccessful). In addition, you still need to provide a buffer to read from, the 1 byte trick works though, but the buffer management can be done either if the read was successful or in the call back function. Also this approach makes the programming asynchronous and similar to event programming.
One way to solve the problem is to have read return a buffer instead of providing the buffer, so that the internal read multiplexing code can do the buffer-pool trick under the hood. As long as all your readers ask for the same maximum size for the read, the pool can be shared. Under the hood, the implementation makes a select and request a buffer from the pool of the requested size.
That also means reads aren't interruptible (unless you count closing the descriptor as interrupting, which is a blunt hammer indeed). AFAIK the only way to do this is to set SetReadDeadline() on the connection to a smallish value, then let the interrupter block on a channel that's invoked when the deadline is reached.
Go's concurrency model is pretty friendly, but it's also unfriendly in many surprising ways. (Don't get me started on the whole nil channel thing, or the problem of channel closure/ownership, or channel performance overall.) It doesn't surprise me the least (while it did surprise the Go team) that Go fell flat among the C/C++ system programmer demographic.
Anyone want to chip in about how Rust, Swift and Nim compare here (regarding the article)?