This fork brings some up deep issues, perhaps intractable, around
about the development of syntax-y languages. Useful source
transformations require marginalized forks of the compiler and all the
development burden and risks that entails. In some sense CoffeeScript
inherits the actual problem with JavaScript - a select few determine
the introduction of language features. Perhaps these language
features are best shipped as libraries?
Maybe it doesn't matter. Perhaps most users don't need it. Or perhaps
as we continue to develop software we'll find that we'd rather defer work to the
compiler / macros and the high cost of syntax is simply not worth it.
I'm looking forward to see how 3 languages I immensely enjoy (JavaScript, CoffeeScript, ClojureScript) co-evolve :)
Regarding development burden, if ICS is implemented as additions to the language, then it can get all updates to the rest of CoffeeScript from "underneath" - the only proviso is that CoffeeScript syntax doesn't change incompatibly with it. And as ICS can be seen as a simple macro (plus libraries it uses), I would guess it is implemented this way (simple source transform on top of CoffeeScript, not hacking at the Jison grammar). This is one way around the issue you raise.
Regarding the whether it's part of the language or a library as a distribution issue, it depends on what benefits users. For syntax, there's some benefit in uniformity (other developers can read it). I'm seeing ICS as a suggestion of a new feature in CoffeeScript. If it's really good (meets a need; bugs and design mistakes are fixable and fixed; it doesn't break other things), it could be back-integrated into CoffeeScript proper. Whether to do so can be better decided once data exists. :-)
It's a great forking model for exploring new syntax, IMHO.
EDIT but it breaks the CoffeeScript philosophy of readable JavaScript, and (eg) addes debug line numbers in to the source code (see "load" on the sample code). Also, other examples and more details on ICS here https://github.com/maxtaco/coffee-script/blob/iced/iced.md
> What language exists in which this is not the case?
Languages which have very little "own shape" (syntax) and let you transform it easily (via macros) or provide high flexibility.
Mostly concatenative languages and Lisps, but e.g. Smalltalk or IO can probably be fit in that category as well, they have a minuscule syntactic core and that core is used to build abstractions and structures putting both the language's designers and the language's users on a level ground.
Of course that creates other issues, where every developer or organization has its own lingo and structures, making even going from one codebase to the next more expensive. It's a tradeoff.
Still, realizing that you have pretty much the same power as the language builders themselves when it comes to creating abstractions and datastructures and control flows... is an enjoyable feeling.
> Language by committee sounds terrible.
Can end horribly, and can end pretty well. Haskell was designed by a committee over ~10 years (FPCA '87 to the release of "The Haskell 98 Report" in 1997).
I'm not talking about language design by committee. For example, ClojureScript is far more restrictive in its design than either JavaScript or CoffeeScript. I'm simply questioning where the line is drawn concerning what constitutes language extension and what is perhaps best served as a library.
I was going to complain that I could find no mention of TameJS (from which the await and defer keywords/features clearly originated). Then I noticed that the IcedCoffeeScript author is also the author of TameJS.
I like the generality of the defer/await concepts of TameJs (and by extension, iced coffee). But I was curious if these concepts could be used without code generation, as in Async.js. So I came up with Queue.js:
There's currently an open pull request for Iced to be merged with CoffeeScript, and the merge will happen if the fork manages to:
* Provide a seamless veneer over sync patterns in async code: defer in loops, with try/catch, within the middle of an expression...
* Handle both synchronous and parallel style "maps"...
* Prove to be more pleasant / powerful to use than callbacks and promises, in practice.
* And then the hardest bit -- do it all without breaking CoffeeScript's golden rule: it has to compile into straightforward JavaScript -- i.e., the callbacks you would have written in the first place.
It's a tall order, but if Max manages to pull all that off, it'll be pretty great.
streamline.js already fits the bill, including your golden rule: the core idea was precisely to generate the callbacks that the developer would write otherwise.
Today it works with both CoffeeScript and Javascript, but in a decoupled way: the streamline transformation is applied to the JS generated by the CS compiler. It could easily be repackaged as a CS language extension.
It has been around for more than a year and the CS+streamline combination is powering at least one live site: http://www.thethingdom.com/
This is a fork of the language, not an official feature. It’s as if you wrote your own version of JavaScript that trampolines function calls and performs tail-call optimization. You still need to go through the social process to get your ideas into the main “trunk.” Selling people on adopting your fork might be one way to get everyone excited about the idea.
That being said, right up front he explains the two reasons why this is a departure from CoffeeScript’s core philosophy: `await` and `defer` are changes in the semantics of JavaScript, and not surprisingly, code written in IcedCoffeeScript no longer compiles into more-or-less recognizable JavaScript.
Modulo syntax and sugar, CoffeeScript is JavaScript with some minor changes to the AST (such as soaking up nils with the Elvis operator). IcedCoffeeScript is another language that evolved from JavaScript and transpiles to JavaScript.
Before adding such features willy-nilly to CoffeeScript, folks need to be comfortable walking away from a core value. I’m not saying that’s a bad thing, but it’s a problem related to the Innovator’s Dilemma: To embrace this new value, CoffeeScript will have to alienate some portion of its user base that value compiled JavaScript that is easily recognizable to anyone who has read the CoffeeScript source.
As a user, you don't really need to walk away from anything. There isn't any difference between Iced CoffeeScript output that doesn't use await and vanilla CoffeeScript output. You could literally switch out somebody's name-brand CoffeeScript for Iced and he'd never know the difference unless he had a function called await. It's purely an ideological concern, not a practical one.
But one objection we will both encounter repeatedly is that if a language supports feature X, and you work on a team/inherit code/use libraries, there will always be people who don’t want to use feature X but wind up dealing with it any ways because someone else used it.
So the naysayers will claim that if it’s added to the core language, it’s only a matter of time before they wind up dealing with it whether they like it or not. Given the way that most Ruby users have to live with monkey-patching in Ruby whether they do it themselves or not, I fully expect that if await and defer are added to core CS, there will be libraries or CommonJS modules that use it and presto, you may find yourself looking at some of its output in the debugger one day.
I’d rather use the feature myself and benefit from it, but I can accept what the luddites are saying even if I don’t end up coming to the same conclusion about whether the feature should be added or withheld :-)
UPDATE: I recall being told that tail-call optimization should not be added to certain languages because it encourages programmers to write recursive functions that are “hard to read.” You might argue that if you never write recursive functions, why should you care? But people do care.
Maybe. Tail recursion optimizes itero-recursive algorithms, turning them into iterative algorithms by eliminating the overhead.
Confused? Here are the iterative, recursive, and itero-recursive ways of writing the factorial function:
function fact_r(N) {
return N <= 0 ? 1 : N * fact_r(N - 1);
}
function fact_i(N) {
var acc = 1, i;
for (i = 1; i <= N; i += 1) {
acc *= i;
}
return acc;
}
function fact_ir(N, acc) {
acc = acc || 1; // default value of acc
return N <= 0 ? acc : fact_ir(N - 1, N * acc);
}
So the accumulator variable becomes an argument variable which may have to be separately initialized. The key feature which makes something optimized for "tail calls" is that when f(something) wants to recurse, it returns f(something else). In other words, certain conditional logics can happen, but don't leave an operation like the "n " in "n recurse(n - 1)" on the stack or else you can't get rid of the function call. It's not even a special optimization -- basically you can move one line in the compiler a couple steps ahead of where you might have automatically written it, and it suddenly treats iteration and tail-recursion the same.
So that's the difference. If you can understand fact_ir, which does the iterative idea in a recursive syntax, then you can understand tail recursion.
> code written in IcedCoffeeScript no longer compiles into more-or-less recognizable JavaScript
So this is a more fundamental fork than first appears... And it explains why it includes debugging information (loading up the code examples reveals artifacts like lineno: 6 in the output JavaScript).
I recall a debate about debug info, turning on the values you note. Can't find it right now.
(though, apparently extra-linguistic Source Maps are being introduced into browsers http://news.ycombinator.com/item?id=2862629)
Personally, I think "clean JavaScript" is the right adoption strategy, for now, in a JavaScript-dominated world. When CoffeeScript has native browser support, that value can be dropped (it might not happen: MS supports VB, Google supports Dart... even if Mozilla supports CS, the others mightn't).
Finally, I don't think the resulting JavaScript is that hard to understand - you can easily see the correspondences with the source. However, it's the thin-edge of the wedge. What changes will be added next?
I haven't looked at this implementation specifically, but the issue with many similar proposals/implementations is that the generated JavaScript is ugly. One of the nice things about CoffeeScript is that there's no magic--it's very easy to figure out what the generated JavaScript is doing, and there's a straightforward CS <-> JS mapping. jashkenas (CoffeeScript's author) has been reluctant to change that.
IIRC, Jeremy Ashkenas actually brought this up in the pull request discussion, but he sounded pretty impressed by the minimalism of the code ICS generates. There seems to be more ideology-based opposition among the community.
I did some simple tests, and frankly the generated code is ugly, certainly by comparison to vanilla CS. Gone is the "oh I see how it did that, it's just a Javascript version of my Coffee!" It's now more like what C++ used to look like when compiled to C (remember cfront? probably not. Be glad you don't).
As someone else said, if you avoid await/defer entirely, it seems to back off to exactly what Coffeescript does, so there is a nice progression path.
Being able to syntactically assemble several parallel requests together has an obvious advantage in making code more readable. Having a hard time wrapping my mind around the keywords. From the examples I think more wait/collect than await/defer. For instance:
It's fantastic to see so much enthusiasm and innovation happening around JavaScript.
While I'd love to see some of these features find their way back into the core language (or native support for coffeescript in node or browsers) it's fair to say that's a long time off if it's ever going to happen.
In the meantime, I'm getting to massively clean up the logic and syntax of my apps. Huge thanks to everyone working on this!
The way that this manages to transform those examples makes me think that it would make all kinds of other things more palatable than the vanilla coffee script. I've written chains of callbacks in plain JS that this would make so much nicer to work with.
Can't agree more. There is no other language in the world that needs this as badly as JS does.
- I am hoping that this would be merged back if it could be somehow concluded that there isn't a much cleaner way to achieve the same result. Even if the resulting JS isn't as clean as what CS produces now. The ICS approach looks safe; they could be assured by similar approaches in more mainstream languages, most prominently C# (and F#) which have much bigger teams attacking the same problems.
If the prospect of this getting merged back looks bleak, I might consider switching ICS once I am convinced there aren't any other breaking issues.
So was there more discussion about await and defer in the many github issues on the subject? This output seems more debuggable than previous CScript defer attempts.
I remember there was another discussion before where they first started talking about Max's implementation, but I can't seem to find it at the moment. That did seem to be the general consensus, though.
Why is defer needed? Why can't I follow C# and do:
result = await search("test")
Is this just so that it plays nicely with non-standard callback APIs? I'd prefer to introduce Promises (i.e., q) and have await interact with that. I guess this is not the NodeJS way though.
In other words, JavaScript doesn't return a Promise/Future/Task<T>, so the defer syntax is necessary. You could get rid of it by simply capturing the callback params and returning them as a value, though your code would end up having a lot of `[0]`'s in them, since most callbacks are of the form (thing, err)
This reminds me of my Celluloid::IO system in Ruby (which uses coroutines to provide a synchronous API), except Celluloid::IO lets you have as many event loops as you want:
It's a great way to write defensive code for those of us who find writing regular unit tests a little onerous even (if they are often necessary). I'm planning on using it in all my CoffeeScript, up to -- but not including -- generated production Javascript.
This is certainly a development to applaud, but for me, the most important feature about async JS programming is to be able to catch errors in the asynchronous code, and do it without code clutter. So I hope that if this gets accepted to mainline CoffeScript, it will provide for a working try/catch around the await/defer, along with caller+callee stacktraces capturing (at least in debug mode). At that point, it will be hard for me to not justify the use of one compilation layer above JavaScript, which currently certainly is, and even so with await/defer.
EDIT: To clarify - an uncaught error in async handler means your Node.js is going down (unless you use the catch-all handler, which is an even poorer practice). In browser it means your callback never returns, leaving you in void (with a short an ambiguous stacktrace) to search where the problem is.
Nice work, I love the syntax. Was a bit disappointed at the verbosity of the compiled js though. Wondering if the same thing could be achieved without the secret functions, classes & variables? (Deferrals, findDeferral, __iced*...)
I think I can imagine a more direct translation of await & defer, though of course I haven't actually tried to implement it (yet).
> The function defer returns a callback, and a callee in an await block can fulfill a deferral by simply calling the callback it was given.
I'm wanting in CoffeeScript knowledge and/or intelligence, so could someone explain the above please? (In the examples, defer seems to specify the variable that gets the result of await.)
Thank you for your thoughtful answer, it's clarified some issues. I'm afraid it also raises more questions for me (I probably need to go study CoffeeScript some more). BTW, the text I added is actually from the first example - the simplest:
await $.getJSON url, defer json
Is the following right? Syntactically, "defer json" is an argument to the function "$.getJSON". So you could write it like this:
await $.getJSON( url, defer(json) )
The code we are waiting for is the "$.getJSON" call. That code indicates that it is finished by calling the function returned by defer.
But what does the function returned by defer do? It looks like it (somehow) sets its argument to the result... so in the above example, "json" (somehow) gets the result of the call; in your example, "x" (somehow) gets a value. I'm guessing it's some convenient compiler magic that ICS does, and not an issue of CoffeeScript syntax.
An upshot of this is you can't give defer any old argument (e.g. a numerical value) - it has to be a variable that is capable of being set.
Also, who is the "callee"? A "callee" is a function that is called... Oh, does it mean (eg) "$.getJSON" is a callee, and it calls the callback function given to it (in the jQuery docs, it's notated as "success(data, textStatus, jqXHR)"; here, that success function is the function returned by defer). I guess, given the purpose of ICS, there will always be a callee in the block - but it seems to me that, syntactically, you could just directly call the function returned by defer. Ah... but (I bet) part of the compiler magic is that you are only allowed to call the function returned by defer from within another function - this is because the result of that other function is needed in order to set the argument of defer to it. Whew.
One last thing: the description talks about "deferrals" begin "fulfilled", but it doesn't say where these defarrals come from. Now, I think they are created by calling defer.
Hmmm.... it's probably better for me to think about this in terms of lisp macros, instead of C calls, as macros allow you to do weird source code transformations, like setting the value of a variable given as an "argument". i.e. ICS is a macro.
Note that I haven't tested ICS so it's only based on my understanding of the examples/docs.
"Is the following right? Syntactically, "defer json" is an argument to the function "$.getJSON". So you could write it like this:
await $.getJSON( url, defer(json) )"
Yes, exactly.
"'m guessing it's some convenient compiler magic that ICS does"
Yes, exactly. When you say "defer(x)", it basically returns a callback like this:
function(result) {
*x = result;
Code to terminate the await block*
}
"One last thing: the description talks about "deferrals" begin "fulfilled", but it doesn't say how they are created. Now, I think they are created by calling defer."
Yeah, defer returns a callback and when you call it, it "fulfills" the await, i.e. you tell await that you're done with that async block.
Thanks a second time, I'm getting there! I realized one can look at the transformed javascript output, to see just what it does. The setting of the variable in the defer becomes:
Presumably, when
the function that __iced_deferrals.defer returns is
called by the code in .getJSON that delivers the result, it will set arguments[0] to that result, and then call the assign_fn above. This will set json to that result.
Also, I realize that it doesn't set json to the returned value of $.getJSON, as $.getJSON doesn't return anything - it uses a callback to deliver its result. It seems that ICS always works this way - which I guess implies that the use-case it's for always uses callbacks...
BTW: I'm not sure why the above has a function which is then called immediately. Maybe a coffeescript thing, for separating namespaces? I assume the lineno: 6 is for debugging - I recall some debate against this. Good to see it got in.
About the function which is called automatically, it's a common pattern in javascript. It's used to create a new namespace/modules as block are defined per function instead of {}.
And, you are right that we're not assigning the return value of $.getJSON. Basically, once $.getJSON has finished with the request, it will call a function with the result. This result is set to the variable passed to defer. It's a bit of a brain teaser but that's the whole point of the await/defer thingy. Instead of using a callback to retrieve the result as an argument, you just write a variable after defer and you receive it.
If I understand correctly, everything after "await ..., defer" block will be executed only after the asynchronous operations from the await block are completed.
Does it mean that it's impossible to return any value from a function that contains await block?
module =
dataPath: "http://search.twitter.com/search.json?q=blah&callback=?"
init: () ->
# Fetch data and do some stuff with it
await
$.getJSON @dataPath, defer @data
# Return yourself (this won't work)
return @
The aim of your docs should be to show how simple and readable IcedCoffeeScript is: using multi character variable names in your examples would help to achieve that.
Do you really get lost or confused? Are you actually complaining about a problem, or just a violation of your favorite style guide? Personally, I'm looking for what makes IcedCoffeeScript unique; all else is just cruft, so keeping that terse is just fine with me.
Just on the website itself: it's very impressive presentation and usability for playing with examples. Couple of issues:
1. The floating title looks cool esp with greying out above it, but it page-down/space moves down more than a page (in effect), so it skips some text. I didn't realize I was missing some text for a while.
2. In the Animal example, the "Run" button doesn't work (though "run" works from the "load" overlay for it). I'm uising Firefox 9.0.1 on Ubuntu 10.04
Maybe it doesn't matter. Perhaps most users don't need it. Or perhaps as we continue to develop software we'll find that we'd rather defer work to the compiler / macros and the high cost of syntax is simply not worth it.
I'm looking forward to see how 3 languages I immensely enjoy (JavaScript, CoffeeScript, ClojureScript) co-evolve :)