I'm also not seeing why this is necessary. Range over int is pure syntax sugar, and while range over func could be useful, iterators can be easily implemented with current Go features.
I love Go because of its simplicity, and because it strictly adheres to the "there's only one way to do it" mantra. Adding syntax sugar muddles this, and introducing another keyword and construct makes the code less readable.
I hope that this proposal is rejected, and that it's not a sign of Go becoming yet another kitchen sink language.
> iterators can be easily implemented with current Go features.
It's rare that can is relevant when it comes to language design. You typically can do anything in any language.
Adding dedicated support for iterators would promote it as the way to iterate. That would actually simplify the language - one way to express a common need as opposed to the current mess of ~5 different ways [1]:
"In the standard library alone, we have archive/tar.Reader.Next, bufio.Reader.ReadByte, bufio.Scanner.Scan, container/ring.Ring.Do, database/sql.Rows, expvar.Do, flag.Visit, go/token.FileSet.Iterate, path/filepath.Walk, go/token.FileSet.Iterate, runtime.Frames.Next, and sync.Map.Range, hardly any of which agree on the exact details of iteration. Even the functions that agree on the signature don’t always agree about the semantics."
Go would have been a far simpler language had it originally shipped with generics, iterators, and sum types.
I agree with you to an extent. But adding more features is always a tradeoff between introducing complexity, and supporting a use case that might not be useful in all programs.
For example, generics were a highly controversial topic, and many people still believe that Go didn't need them. I'm partially in that camp. I've encountered maybe one or two situations where generics would've been convenient since Go 1.18, and in both cases not jumping straight at the opportunity to use them, but being forced to refactor and approach the solution in a different way, has produced simpler and more readable code. To this day, I struggle with Go's generics syntax, and it just looks alien to me.
Iterators are a useful feature, but are they really generally useful to deserve a change in the language spec? Like generics, I've yet to encounter a situation where they're truly required, and when I do, implementing them myself is trivial.
So, sure, language designers can cram every programming construct we've invented in the last 50 years, as Rust and Raku have done, and many programmers will find this very convenient and flexible, but a more focused and conservative approach produces a simpler language that is easy to pickup and read. This is what drew me to Go in the first place, and lately it seems that this is changing.
Iterators have been around for literally decades at this point, and have proven their usefulness in every language they've been introduced in (just like generics). It's not some kind of experimental feature.
Generics very nice as a library writer but in my own application code, I usually have no uses of them.
You have language features which are not needed for the ones using directly (for the lack of better term) the language and but really nice for the people providing building blocks for them.
I am using generics in other languages than Go but not in Go as I am normally not writing libraries in Go or they are already so specific that generics are not needed.
First-class tuples would make this code kind of unpleasant:
func F() (A, B) { ... }
func G(a A, b B) { ... }
G(F())
Is the return value of F a tuple? Is the parameter list of G a tuple? Given H:
func H(ab (A, B)) { ... }
How and when is H(F()) different from G(F())?
(Before you suggest a Python-like star operator, personally I consider using it for non-homogenous arguments about as hygienic as keeping your toothbrush and dirty socks in the same drawer.)
> First-class tuples would make this code kind of unpleasant:
Which assumes this code matters much, outside of errors for which intermediate processors can just take... a tuple of a value and an error.
Also assumes the magical splatting of MRVs is a good thing in the first place.
> (Before you suggest a Python-like star operator, personally I consider using it for non-homogenous arguments about as hygienic as keeping your toothbrush and dirty socks in the same drawer.)
That is literally what Go currently does for MRVs, except opt-in instead of magical.
There is nothing about such splatting which wouldn't work with proper tuples, if anything it's the opposite. It is, essentially, the value side of `uncurry`.
I don't necessarily disagree, but it depends on your goals as a language designer. I think it's OK to sacrifice a niche feature for improved readability in the common case. Anonymous structs are not as fundamentally useful as e.g. anonymous functions.
> Also assumes the magical splatting of MRVs is a good thing in the first place.
IMHO, G(F()) reads nicer than some variant of G(*F()) or G(F()...). Lua does this too, and is (not unlike Go) considered a simple and elegant language.
> Which assumes this code matters much, outside of errors for which intermediate processors can just take... a tuple of a value and an error.
Then you'll sooner or later find yourself writing a G((a, b)), which again, I think is a net loss to readability.
I think my argument would've been much stronger if Go had had generics in 1.0, and the stdlib/ecosystem wasn't already full of MustParse's and MustCompile's. But in that case we could've had a Result type as well.
> Adding dedicated support for iterators would promote it as the way to iterate. That would actually simplify the language - one way to express a common need as opposed to the current mess of ~5 different ways [1]:
There is one. Return a channel. Just a bit iffy when you want to stop iteration in the middle and throw away the rest.
Returning a channel is not really a cheap operation. They internally have locks that slow everything down as they are intended mostly as a safe concurrency primitive
And in this case there is really no need to spawn goroutines to support iterators
because go developers are sick of being pure and like to type shorter things (and I don't mean variable names) to get their jobs done.
programming is a lazy hobby. it's why as soon as generics were released, I immediately implemented 'Keys', 'Values', 'ToSet', etc. Writing multiple lines to do the equivalent of these functions might be 'pure' but it's not easier to read. Worse: instead of standardizing these common operations, you just have everyone rolling their own implementations of the exact same functions with slightly different naming/parameters/etc.
Am I confused in that there is some merit to this approach beyond dogma? Why do we, as a community, need to accept so few batteries? If you want to only use assembly in your code (the purest!), that's always an option.
Well, Go authors decided that adding features to the language comes with cost to every user (which is correct) and decided to be careful with that.
Which is, looking at absolute hulking abomination C++ or Java has become, is the right approach in theory
They just made a massive miss with lack of generics and sum types on the start. They could've avoided nils entirely with that for example. Or have less annoying error handling.
> They could've avoided nils entirely with that for example. Or have less annoying error handling.
TBF neither generics nor sum types were necessary for either, these could have been builtins or language-level features.
To start with it's not like Go has been shy about those for things they didn't want to formalise (yet), but even in languages where they're library-ish features it's common for them to have language-level support.
It's not about purity or dogma. It's about keeping complexity out of the language as much as possible, which is much harder to do. Language designers always make a tradeoff between making programming tasks convenient, and introducing more complexity that makes the language more difficult to maintain and less approachable, and programs written in it more difficult to read and reason about.
Being productive in a language is not just about using every convenience at your disposal, and typing less. It's about producing code that is easy to read and approachable for others or yourself years down the line. The more language features one needs to be aware of when writing or reading code, the less approachable it is.
Having less features also forces the programmer to rethink their solution in a less clever way. In dynamic languages like Python and Ruby it's very convenient to use some fancy trick to write less code. In languages like Rust or Raku, you can (ab)use some obscure feature in a non-intuitive way, and feel good about the "elegance" of the solution, but a fresh pair of eyes might struggle to understand what the code actually does. Considering reading code is much more important than writing it, as we literally write code for humans, not machines, optimizing for simplicity and having less features in a language is a more productive long-term approach.
> It's not about purity or dogma. It's about keeping complexity out of the language as much as possible, which is much harder to do. Language designers always make a tradeoff between making programming tasks convenient, and introducing more complexity that makes the language more difficult to maintain and less approachable, and programs written in it more difficult to read and reason about.
Complexity is not something you can remove.
Complexity removed from language moves to the application.
The best thing you can do in the language is to make expressing complex concepts in easier, more clear way. Going too simple just produces more boilerplate and makes it harder to read.
Keeping complexity out of the language does not remove the complexity. It moves it into the programs. MIPS assembly is a very simple language with specs fitting on 10 pages. Yet you probably wouldn't want to write a web application in it. Brainfuck is even simpler as a language, but the programs are even harder to understand.
sometimes it moves the complexity up the developer's head : "are you sure you really need to do all this, and there isn't a conceptually simpler solution to your specific problem ? Think again !"
That's the real value of "simple" languages like go.
“Am I really sure I want to loop over batches instead of fetching one record at a time (or the whole set of results at once)? Ugh, it’s a complete pain in the ass though, one at a time it is.”
You literally don’t have to use iterators if you don’t need them. You can just range over a collection as you do already. But when you want something a bit more performant like looping individually over batches or generating an enormous sequence of records on the fly, they make what would either be tedious or difficult into something trivial.
Go being “simple” just makes application code unnecessarily verbose and makes dealing with even moderately complicated problem domains feel like solving novel computer science problems when it’s be four lines of boring code in better languages.
Aren't we talking about standardizing iterators? That makes the compiler slightly more complex, I guess, but for the language it makes it much easier to read and more approachable imo. That's the reason that range exists in the first place.
I don't see why the author's implimentation is so complicated. I really just want an interface that works with the range operator, and to call it a day.
type Iterable[T any] interface {
Next() (T, bool)
}
so that
for _, el := range someIterable {
}
is essentially the same as
for el, ok := someIterable.Next(); ok; el, ok := someIterable.Next() {
}
>because go developers are sick of being pure and like to type shorter things
"go developers" is not a homogenous group. I write Go and am pretty happy with the language design as it is. I'm also fearful of it becoming a kitchen sink language because of a vocal minority who want Go to be more like language X, instead of just going and writing in language X.
There's plenty of things could be added to make code more concise, but they come with a hidden cost.
I too hope this will be rejected, the proposal is terrible. One of the core advantages of Go was, that if newbie colleague comes to the team, he will be able to read and understand the code very quickly. Things like this will over time kill this Go's advantage.
> Adding syntax sugar muddles this, and introducing another keyword
There is no new syntactic sugar nor there is a new keyword here though. It just changes how range works.
That being said, I'm not a fan of range having two modes of functionality. Perhaps it could just use an interface, now that we have generics it can be implemented on the existing supported types. And compiler can do whatever with it if it would be too slow.
Exactly. Go making it's way faster than I expected to be replaced by yet another language that discovers that there's a demand for feature restriction and simplicity. The very thing that made it popular in the first place. It should be rejected but I'm losing hope. I have less and less faith in rsc as time passes.
I think the pace has been extremely slow. So many of the changes to Go have been tool chain or library enhancements. Outside of generics related stuff, what are some added core language features that you'd prefer were not?
That kind of suffers if you want to be able to cheaply cancel the iteration.
Lets say the logic is:
for item := range list {
if condition(item) {
break
}
}
That is nice and readable, but if the thing your iterating is returning items in a batch then this gets far, far uglier. Its possible to end up with one loop that fetches x items at a time, and another that iterates over those items.. etc. But if you switch that to channels you maintain the readability and everything is nice! .. Except, you need a way to cancel the iteration, and it can't be closing the channel or the writer will panic.
items, cancelFunc := apiCall()
for item := range items {
if condition(items) {
cancelFunc()
for _ := range items { }
break
}
}
close(items)
Now, this is even worse if you need some kind of finalization of the goroutine that spawns to write to the given channel. Now you have a wait group, or a second channel.. etc. If its possible to error in the middle of iteration then things get complicated as well.
This is the way how Java iterators work. And having worked with them for many years I must say - please don't go that route. Having separate hasNext/next methods is terrible for implementers of the iterator. The problem arises when you need to actually compute the next item to learn if it exists. So then your hasNext is actually effectful and needs to cache the result internally. There are so many ways to get it wrong, I've even seen some terrible implementations which required the caller to call hasNext or otherwise they would not advance the iterator.
Iterator design is another area where sum types would help simplify things a lot. Just one method returning Some/None would be so much better.
Since we're talking about implementing it in usercode, as opposed to adding it to the language, we fortunately have the option to rework it (or have a couple completing implementations to see what works in practice).
If you're looking for those particular features, I'd recommend adopting go's `, ok` idiom that appears in several places in the language. A type like:
type Iterator[T any] interface {
Next() (T, bool)
}
Lets you write code like this, if that's more to your liking:
for current, ok := i.Next(); ok; current, ok = i.Next() {
// do things here
}
Which, really seems like a more go-like way to do this in the language as well (ranging over a function then becomes syntactic sugar for the above).
The design of channels is constrained by the need to support concurrency. If you do not need concurrency, using something designed to support it will be suboptimal. And because it's specifically concurrency, the overhead will be substantial.
... that's the theory behind it. For more grounded explanations, see [1] & [2].
Channels and the goroutine scheduler need to be thread safe. They use locks and/or synchronized instructions that run much slower because they need to synchronize between the different cores of your CPU (or between CPUs, if you have more than one, which is slower still.) Running goroutines also saves and restores registers, trash part of the cache, or move to another core with a cold cache. Threads have more overhead, and processes more still, but that still is very slow.
Thats how I would do it. You can have a goroutine send on a channel (buffered or non buffered), and then another function can consume the chan as needed which lets the goroutine generate results as needed as well.
you can just return channel. range works over channels. Close channel once you're done with iterator. Technically there is side effect of that (iterator done like that would always produce extra value before exiting) but you could probably work around it without much boilerplate
Channels introduce a ton of overhead (https://syslog.ravelin.com/so-just-how-fast-are-channels-any...), plus concurrency issues, and termination concerns (they are essentially a resource leak). It's nice that you can iterate channels, but they're a terrible iteration protocol.
They probably meant that programming language designers/developers can only advance their careers by adding/implementing features, not presiding over something complete.
This approach feels quite invasive compared to adding a standard interface that `range` accepts. Some discussions have happened before, eg https://github.com/golang/go/discussions/54245
A standard interface feels a cleaner than these nested callback functions.
The function-based approach is kind of nice because you can use defer for cleanup, and you don’t have to write an entire struct and track state via props, etc. I kind of like this proposal.
The interface version is nicer for consumers in a world where you can’t range over a function, but the function version is way easier to write, so once you can range over functions, the advantages of the interface versions mostly go away.
This is a weird approach. They are essentially manually implementing the “yield return” functionality from c#.
C# defines an interface IEnumerable<T> with a “next” method, and originally you’d need to implement this interface which was a bit of boiler plate.
They added “yield return” which would create compiler generated syntax sugar to return a generator function like this go proposal, but your code looked identical to a non iterative version.
The go proposal feels like the worst of both worlds and like the team misunderstood the purpose.
How is that better than the C# solution of just making the subroutine return an IEnumerable, then yielding its elements in the calling function? (aside from maybe marginally better perf characteristics)?
It works with subroutines that don't know anything about iterators or yielding but do know about first-class functions. It's also less boilerplate, since there isn't a `yield from`, I guess.
It seems like a different kind of boilerplate: instead the functions need to take a yield so you are threading the function through you program. I don't know if it's better to expect a return type or to force a function argument. I lean towards the former, but that might be because of familiarity.
I'm guessing this is following the same / highly similar motivation as rsc's coroutines post about a month ago: https://research.swtch.com/coro
I welcome people embracing function closures more, as I feel they're a drastically under-used language tool (particularly in Go where everyone crams everything into single-func interfaces which are almost exclusively far worse). More exposure to it helps people become comfortable with it, and now that we have generics it's finally not a pile of pain and agony at all times. And just using functions makes it very flexible / doesn't repeat the "X for me but not for thee" issues of past Go versions.
I'm not entirely sure how I feel about this proposal though. It kinda feels like it's trying to put a round peg into a square hole. If you try hard enough, everything can go in the square hole! But that doesn't mean it's a good fit.
> particularly in Go where everyone crams everything into single-func interfaces which are almost exclusively far worse
Depending on scope, I've grown to prefer the single func interface.
For a smaller scope use, say filepath.Walk, absolutely, pass in a closure, good stuff.
For larger stuff, like say a chain of http middleware and handlers, I think it can be very limiting.
you get a big benefit from using the interface over a function: the underlying type _can_ have more methods and implement.
this gives room to implement a parallel interface, for example, to collect all possible routes, or to trace what handlers a request would pass through.
I don't think there's a one-size-fits all approach, use whatever gives the effort:utility tradeoff you're looking for.
Func references can have methods as well fwiw :) They just need to have a named type.
But sure, if you want to build a multi-method thing (or upcast to detect optional methods), an interface is the natural choice - it's no longer a single func.
It would be nice if the proposal defined the semantics of getting single item from the iterator (or detecting that it is out of items) and exposed an api for this, then defined the behavior of range as equivalent to some pattern using these primitives.
As is the proposal says "there should be some coro.Pull type thing" which is a bit terse and says nothing about whether the implementation of coro.Pull will be fast or whether it will be built on range + channels
I suspect getting a single item out will be that you just make a
func next[T](i iterfunc[T]) T {
var item T
i(func(it T) bool){
item = it
return true // likely
})
return item
}
and use it. Semantics should be the same as ranging would desugar to (since that's all this is), though the variable arity will be slightly annoying of course.
Aah, yes, that's an excellent point. Thanks! That does complicate things a lot more.
Quite a lot more, as it essentially means you can only consume from iterators you create... which sorta defeats the purpose of iterators as a value, and degrades them almost completely to just a `range` helper.
>To convert one of these functions to a "next-based" or "pull-based" iterator, we plan to add a function to the standard library that runs the yield-based ("push-based") iterator in a coroutine, along the lines of the coro.Pull function in my recent blog post.
that's... an option I guess. Unless the runtime special-cases this coroutine though it seems like it'll probably have non-trivial performance cost, roughly equivalent to the channel + goroutine equivalents people sometimes build now. Which seems... not great. Bad enough that anyone even slightly performance-sensitive will probably avoid it.
(his coro post also mentions a proposed runtime change to improve the performance, but afaik previous attempts to get this kind of change into the language haven't worked out reliably, and you can't trust callers to be single-threaded)
I don't believe so - the post is pretty much all about how you can do this right now in normal Go code. And e.g. a bit after that is a link to the "full code" which is runnable: https://go.dev/play/p/hniFxnbXTgH
Possibly you're interpreting yield as a new keyword? It's just an argument to Pull.
The only not-(currently-)available-in-Go stuff that I can recall from the post is the performance optimization at the very end.
Iterators are great, I really loved simplicity of `yield` in c# and python.
Go's proposed syntax is god awful. Why they are doing that to devs and themselves? Should we expect it to improve in like 15-20 versions, like it happened with any and generic mix/max functions?
This looks like a very complicated way of solving something for which there is already a solution: channels and contexts. Here's the pattern I often use:
It should be doable to express this somewhat simpler using a (genericized) helper function to take out some of the work of writing the ListItems function.
Since there is already a way to do this I don't think the language should be cluttered with more ways to essentially do the same thing. It is better to push people to develop a vocabulary of idioms that uses fewer features rather than introduce "more stuff".
This does not solve the problem.
With channels you are preparing things (even just one thing) ahead of time and you have to do it in a separate goroutine.
One could possibly (haven't tried or given more than 2s of thought) implement a channel (or at least part of it) using iterators.
Not sure how I feel about this particular approach to iterators. I'd love for there to be some generate way to `range` over a generic collection, but idk about this.
Honestly I would just prefer an interface that can be implemented (not this interface.. one like it that is better thought out).
type Iterator interface{} {
// Returns the next object and true, or nil and false if iteration has ended.
func NextIteration() (interface{}, bool)
}
Even better (in my opinion) if it also adds a finalizer function to help clean up resources used during iteration and some kind of error handling.. but that might be asking too much given my track record of getting golang improvements approved. I still want else on for statements dang it! =)
Wait, so you return `false` to terminate the iterator? Stupid!! Now I can’t have an iterator that returns Booleans.
UPDATE: I misread the article. This isn’t what happens. My bad.
Honestly, I feel like Go focuses so much on superficial simplicity that you wind up with bad/leaky abstractions all over the place as well as bad primitive obsession.
No that's not at all how this works. The iterator could "return" a boolean by passing a boolean to "yield". The return value of "yield" just indicates if the generator should continue, or stop because the actual loop has exited (due to break or return).
> Now I can't have an iterator that returns Booleans.
No, you've misunderstood it.
The iterator function gets a yield function which can be called with a value (the value to be yielded). The yield function returns a boolean indicating whether the process should continue or terminate.
func SomeIterator(yield func(bool) bool) bool {
for whatever {
if !yield(a_calculated_boolean_value) {
return false
}
}
return true
}
I have no opinion on this specific feature, but I have a general feeling that by far the hardest thing for language developers is to know when to stop.
I feel this might be the case because language developers are themselves the worst reference for a typical user. They’re experts and no matter how big a language gets, they easily wield it. Even if they recognize that complexity is going up, they can’t truly grok the cost of that.
I think Bob Ross talked about this often. That it can be tricky to know when you’re done. And it can be easy to clutter the canvas.
This design avoids the inefficiencies and most of the problems of having an?_iterator_, like the classical Java `hasNext`/`moveNext` or C# `MoveNext`/`Current` interfaces.
Those iterators require you to keep the state between elements as values, effectively CPS transforming and defunctionalizing the control flow of actually iterating the elements.
This iterator function here is really just a `forEach`/`Each` method where the callback can return a boolean. You can't easily use that to split iteration, say iterate the first five elements, then go do something else, and come back to iterate the rest, doing something completely different.
But that also means it doesn't have to worry, as much, about the iterator going stale because the underlying structure changes (`ConcurrentModificationError`), or not knowing when to clean up, because somebody might call `hasNext`at any later time.
So this is basically just a nicer `forEach` method.
I want Go to remain simple but having written a hell of a lot of code in Go that needs to iterate over a stack whose size cannot be computed ahead of the iteration, I can confirm that writing such code in Go does get very complicated very fast. There are a lot of footguns to take into account:
- how do you acknowledge the end of a stack?
- how do you cancel the operation mid iteration?
- how do you make it thread safe?
- how do you make fast?
The proposal here simplifies these concerns and thus makes it easy to write and use iterators. It makes it easier to read iterators too. And it does so without adding hacks to the standard library.
This is one of those instances where the proposal might seem to add complexity but actually it simplifies the code a lot (unlike generics, for example, which complicates the syntax of your code in its effort to dedup lines of code. To be clear, I’m not being critical against generics here. Just using that as an example to show contest).
I was thinking exactly this. Coming from Rust, but also as a fan of Go, I can't help that the building blocks just aren't there to construct a satisfying form of iterators, and like you said that would be language support for tuples and expanding the flexibility of Go's generics for methods to support the composition Rust allows, such as: iter.skip(10).take(10).map()...
I have used some iterator libraries for go and they do make for much more manageable code, especially if you start nesting iterators, like iterating over JSON which reads from an iterator over a blob from blob storage. And it also makes it simpler to apply filters at various points. Using callbacks also works but makes for nastier code.
I much preferred the iterate by function approach described by one of the blogger guys a while ago (Muratori? Bendersky? Cassettieri? I can't quite recall the name atm).
> But there are some disadvantages to this way of doing it. First, we have to wait until the whole slice has been generated before we can start to process the first element of it.
Uh, range over channel ? The author seems to be very bad at go
Channels come with significant overhead [1]. While that is fine (or unavoidable) when communicating between goroutines, in single threaded code that iterates some custom container you don't want a channel.
That is why stdlib map and vector have their own range magic, but as library author this isn't available to you. That is the point of the proposal.
Channels are also not great when you want to break off / cancel iteration.
The overhead also compounds when you compose / chain iterators.
Plus the "lag" they induce due to context switching, although I think I saw an optimisation (proposal) where the runtime would directly switch from one routine to the other in some cases of channel interactions? Or am I thinking of an other language entirely?
a) one item at a time means slow because when you are iterating maps/slice/array you have it all in memory so it is fast. with one item you have to constantly rewrite memory, move data and whatnot
b) yo can always use for loop in a traditional for way with standard arguments(init, stop, next)
c) yield is poor way of doing channels, i do not see why we should be degrading our code
I love Go because of its simplicity, and because it strictly adheres to the "there's only one way to do it" mantra. Adding syntax sugar muddles this, and introducing another keyword and construct makes the code less readable.
I hope that this proposal is rejected, and that it's not a sign of Go becoming yet another kitchen sink language.