promises | Communicating Event-Loop Concurrency and Distribution
On Thu, Jan 27, 2011 at 8:00 AM, Irakli Gozalishvili <rfobic at gmail.com>wrote:
Hi,
I was curious to know what is the state of the following proposal: strawman:concurrency
It's on the agenda for the upcoming March meeting. I was planning to do some more work on it before declaring it ready for discussion. But since you raise it, I'm happy enough with it in its current state. I can clarify my remaining issues with it as discussion proceeds. So
This page is now ready for discussion.
I do expect that this strawman is too large to be accepted into ES-next as a whole. To get some of it accepted -- syntactic sugar + some supporting semantics -- we need to find a clean boundary between what kernel and extension mechanism the platform needs to provide vs the remaining functionality that should be provided by libraries using these extension mechanisms.
For example, Tyler's web_send uses such an extension API in ref_send to stretch these operations onto the network, by mapping them onto JSON/RESTful HTTPS operations. Kris Kowal's qcomm library uses Q.makePromise (like that proposed above) to stretch these operations over WebSockets, whose connection-oriented semantics enables a better mapping at the price of more specialized usage. I hope that Kevin Reid's caja-captp can also be reformulated as a library extending the Q API.
(See links to ref_send, web_send, qcomm, and caja-captp at the bottom of the strawman page.)
I do believe that having ES native promises could provide drastically better alternative for writing async code in comparison to currently popular nested callback style. Also even though there are few implementations of Q API adoption is still low IMO that's due to non obvious and verbose syntax. Syntactic sugar described in the proposal really makes a lot of difference. Also I don't see proposal for
Q.whensyntax and would love to know what is the a plan for that.
Given a lightweight closure syntax, I don't think Q.when actually needs any further syntactic sugar. For example, here's the asyncAnd example from Figure 18.1 of erights.org/talks/thesis in JS with this strawman +
destructuring + the lightweight closure syntax from HOBD (Harmony of Brendan's Dreams):
#asyncAnd(answerPs) {
let countDown = answerPs.length;
if (countDown === 0) { return true; }
const {promise, resolver} = Q.defer();
answerPs.forEach(#(answerP) {
Q.when(answerP, #(answer) {
if (answer) {
if (--countDown === 0) { resolver(true); }
} else {
resolver(false);
}
}, #(ex) {
resolver(Q.reject(ex));
});
});
return promise;
}
The original asyncAnd in Figure 18.1 is in E, whose syntax was designed without legacy burden to make such code pleasant. Nevertheless, I don't think the code above suffers much in comparison.
Of course, if you have a suggestion of how a sugared Q.when can improve on this enough to be worth the cost of yet more sugar, please suggest it. Thanks.
This looks like a case of creating in-language for a library. This was done with json2.js because it was one of the most widely used libraries in existence and similar JSON handlers were used in numerous libraries. The ref_send library and sibling Q style implementations are not anywhere close to that level of adoption. It seems like there are a large number of library functions that have proven useful enough (by real usage) ripe for incorporation into the language before the Q api. In fact, even with the realm of JavaScript promise libraries, this implementation/API, which has been discussed and known about for a number of years has not become the dominate interface API, there are other approaches that have been found more use. While I certainly like and appreciate the Q API, I don't think it has proven itself worthy to be added to the language.
The real pain point with promises isn't that it takes too many characters to type Q.post. How many bytes does the average JavaScript program spend on promise API calls? It negligible, this proposal isn't solving any real world problem. The main challenge and overhead of promises, or any CPS-esque code flow is the complexity and overhead of continuing flow through callbacks. This is why there are proposals for shorter anonymous functions and single-frame continuations. Single frame continuations provide the tools for building cleaner, simpler code flow with promises or any other callback style mechanism without standardizing on a single library. Like Brendan mentioned in last blog post (well, I guess it came from Guy Steele), good language empowers users to build on it, not stifle them with single library approach.
Thanks, Kris
On Thu, Jan 27, 2011 at 8:23 PM, Kris Zyp <kris at sitepen.com> wrote:
This looks like a case of creating in-language for a library. This was done with json2.js because it was one of the most widely used libraries in existence and similar JSON handlers were used in numerous libraries. The ref_send library and sibling Q style implementations are not anywhere close to that level of adoption. It seems like there are a large number of library functions that have proven useful enough (by real usage) ripe for incorporation into the language before the Q api. In fact, even with the realm of JavaScript promise libraries, this implementation/API, which has been discussed and known about for a number of years has not become the dominate interface API, there are other approaches that have been found more use. While I certainly like and appreciate the Q API, I don't think it has proven itself worthy to be added to the language.
The real pain point with promises isn't that it takes too many characters to type Q.post. How many bytes does the average JavaScript program spend on promise API calls? It negligible, this proposal isn't solving any real world problem.
Hi Kris, I appreciate your other points. However, from my own experience having programmed in this style with syntactic support, and attempting to bring myself to program in this style without syntactic support, I disagree. In this particular case, the syntactic difference between
Q.post(a, "foo", [b, c])
and
a ! foo(b, c)
makes a tremendous difference in one's willingness to use these abstractions, and in one's ability to read and understand code expressed in these abstractions. I believe this difference is adequate to deter most usage -- even in my experience by people like me, who already like these abstractions.
The main challenge and overhead of promises, or any CPS-esque code flow is the complexity and overhead of continuing flow through callbacks. This is why there are proposals for shorter anonymous functions and single-frame continuations. Single frame continuations provide the tools for building cleaner, simpler code flow with promises or any other callback style mechanism without standardizing on a single library. Like Brendan mentioned in last blog post (well, I guess it came from Guy Steele), good language empowers users to build on it, not stifle them with single library approach.
I don't understand why single-frame continuations are not at least as vulnerable to the criticism you make above. They are even more unproven among JavaScript programmers than are the concurrency abstractions in the strawman. The main benefit of moving support for single-frame continuations into the language is notational, which you dismiss above. I don't get it.
And in case you missed it, I am not proposing that we bake in any one library for remote communications, whether web_send, qcomm, or captp. I am hoping for an extension API that would allow library authors to create an open ended set of remote communications libraries that plug into this extension API. As I understood your previous engagement with these issues, I thought you were also focussed on such extensibility.
On Thu, Jan 27, 2011 at 9:37 PM, Mark S. Miller <erights at google.com> wrote:
I don't understand why single-frame continuations are not at least as vulnerable to the criticism you make above. They are even more unproven among JavaScript programmers than are the concurrency abstractions in the strawman. The main benefit of moving support for single-frame continuations into the language is notational, which you dismiss above. I don't get it.
We have some experience with single-frame continuations in the Firefox code base (and elsewhere such as threads.js from Neil Mix), based on our years-old implementation. The thing I like most about it, versus working from a callback-based model, is that you can use the language's flow control constructs. Having to turn everything inside out via function chaining hurts a lot, even with helper libraries that are popular in the node.js community.
for (var i in hash) { doSomethingAsyncAndWait(); }
versus gross hackery like breaking out of the enumeration every time and deleting the just-processed property. Nothing that code to be written out of order, or shattered by function decomposition and glued back together in the programmers mind, will be as satisfying I fear.
Mike
On 1/27/2011 10:37 PM, Mark S. Miller wrote:
On Thu, Jan 27, 2011 at 8:23 PM, Kris Zyp <kris at sitepen.com <mailto:kris at sitepen.com>> wrote:
This looks like a case of creating in-language for a library. This was done with json2.js because it was one of the most widely used libraries in existence and similar JSON handlers were used in numerous libraries. The ref_send library and sibling Q style implementations are not anywhere close to that level of adoption. It seems like there are a large number of library functions that have proven useful enough (by real usage) ripe for incorporation into the language before the Q api. In fact, even with the realm of JavaScript promise libraries, this implementation/API, which has been discussed and known about for a number of years has not become the dominate interface API, there are other approaches that have been found more use. While I certainly like and appreciate the Q API, I don't think it has proven itself worthy to be added to the language. The real pain point with promises isn't that it takes too many characters to type Q.post. How many bytes does the average JavaScript program spend on promise API calls? It negligible, this proposal isn't solving any real world problem.Hi Kris, I appreciate your other points. However, from my own experience having programmed in this style with syntactic support, and attempting to bring myself to program in this style without syntactic support, I disagree. In this particular case, the syntactic difference between
Q.post(a, "foo", [b, c])and
a ! foo(b, c)
makes a tremendous difference in one's willingness to use these abstractions, and in one's ability to read and understand code expressed in these abstractions. I believe this difference is adequate to deter most usage -- even in my experience by people like me, who already like these abstractions.
Of course syntax makes a difference, but doesn't that same argument apply to the thousands of other JavaScript libraries out there? If I just had syntactical support for my cool library, I know people would start using it...
On 1/27/2011 10:47 PM, Mike Shaver wrote:
On Thu, Jan 27, 2011 at 9:37 PM, Mark S. Miller <erights at google.com> wrote:
I don't understand why single-frame continuations are not at least as vulnerable to the criticism you make above. They are even more unproven among JavaScript programmers than are the concurrency abstractions in the strawman. The main benefit of moving support for single-frame continuations into the language is notational, which you dismiss above. I don't get it. We have some experience with single-frame continuations in the Firefox code base (and elsewhere such as threads.js from Neil Mix), based on our years-old implementation. The thing I like most about it, versus working from a callback-based model, is that you can use the language's flow control constructs. Having to turn everything inside out via function chaining hurts a lot, even with helper libraries that are popular in the node.js community.
for (var i in hash) { doSomethingAsyncAndWait(); }
versus gross hackery like breaking out of the enumeration every time and deleting the just-processed property. Nothing that code to be written out of order, or shattered by function decomposition and glued back together in the programmers mind, will be as satisfying I fear.
Exactly. On the NodeJS mailing list there is constant, never-ending stream of messages from (generally new users) who are complaining about the pain of callbacks (not from typing some API, I have yet to ever get a request for shorter syntax for node-promise, which is one of the commonly used helper libraries). In fact I think someone recently created another continuation library (in the same vein as Narrative, generators, etc.), continuing to add to our collective experience in this area. I could look up the address if desired.
Thanks, Kris
On Fri, Jan 28, 2011 at 6:05 AM, Kris Zyp <kris at sitepen.com> wrote:
Exactly. On the NodeJS mailing list there is constant, never-ending stream of messages from (generally new users) who are complaining about the pain of callbacks (not from typing some API, I have yet to ever get a request for shorter syntax for node-promise, which is one of the commonly used helper libraries). In fact I think someone recently created another continuation library (in the same vein as Narrative, generators, etc.), continuing to add to our collective experience in this area. I could look up the address if desired.
Please do. More collective experience is good.
However, I think your comparisons show your missing the point of this strawman. AFAIK, all the systems you cite, including node-promise which implements CommonJS promises/A and your proposed use of shallow continuations, are all only for helping express asynchrony within a single vat. None of these are about supporting communication to objects in separate or remote vats. So none of these should be taken as competitors for this strawman. This strawman includes means for spawning new vats, talking between the spawner and the spawned, and an underspecified-at-the-moment (but see qcomm's Q.makePromise()) pluggable extension point for extending this semantics over networks.
I expect this is mostly my own fault by giving the strawman the title "concurrency", which emphasizes the wrong first point.
Vat's, are asynchronously coupled to each other by remote references that can fail. This helps vats be units of transparent distribution as well as concurrency, partial failure, partition, and preemptive termination. With appropriate support, they can also become the unit of migration, separate persistence, and resource control.
I have received much helpful private correspondence suggesting many other needed clarifications and improvements to that strawman page (thanks). Having announced "ready for discussion" before I'd intended, please everyone indulge me while I temporarily withdraw this page in order to make these suggested improvements and shifts of emphasis. I will re-declare this "ready" soon. I promise ;).
Mark
Does this have some parallels with channel messaging (www.w3.org/TR/html5/comms.html#channel-messaging)?
Sent from my iPad
On 1/28/2011 8:43 PM, Mark S. Miller wrote:
On Fri, Jan 28, 2011 at 6:05 AM, Kris Zyp <kris at sitepen.com <mailto:kris at sitepen.com>> wrote:
Exactly. On the NodeJS mailing list there is constant, never-ending stream of messages from (generally new users) who are complaining about the pain of callbacks (not from typing some API, I have yet to ever get a request for shorter syntax for node-promise, which is one of the commonly used helper libraries). In fact I think someone recently created another continuation library (in the same vein as Narrative, generators, etc.), continuing to add to our collective experience in this area. I could look up the address if desired.Please do. More collective experience is good.
Looking at the language in the docs, this may just use event-loop stacking to "wait" or yield.
However, I think your comparisons show your missing the point of this strawman. AFAIK, all the systems you cite, including node-promise which implements CommonJS promises/A and your proposed use of shallow continuations, are all only for helping express asynchrony within a single vat. None of these are about supporting communication to objects in separate or remote vats. So none of these should be taken as competitors for this strawman. This strawman includes means for spawning new vats, talking between the spawner and the spawned, and an underspecified-at-the-moment (but see qcomm's Q.makePromise()) pluggable extension point for extending this semantics over networks.
So Web Workers are the competition? You can build far references on Web Workers, right? Kris
On Fri, Jan 28, 2011 at 9:54 PM, Kam Kasravi <kamkasravi at yahoo.com> wrote:
Does this have some parallels with channel messaging ( www.w3.org/TR/html5/comms.html#channel-messaging)?
On Sat, Jan 29, 2011 at 7:17 AM, Kris Zyp <kris at sitepen.com> wrote:
So Web Workers are the competition? You can build far references on Web Workers, right?
Kind of, yes. Vats, remote references, and eventual message sends can be straightforwardly mapped (in two different ways) onto WebWorkers, message ports, and postMessage respectively.
The reason I say "kind of" is that this is intended as an object oriented abstraction layer than can be mapped onto many kinds of substrate, including postMessage, JSON/RESTful HTTPS (ref_send), WebSockets (qcomm), and connection-oriented use of TLS (caja-captp), etc. By contrast, with WebWorkers and message ports themselves, you only get "far" references among vats within the same browser.
More later...
On Fri, Jan 28, 2011 at 01:09, Mark S. Miller <erights at google.com> wrote:
On Thu, Jan 27, 2011 at 8:00 AM, Irakli Gozalishvili <rfobic at gmail.com>wrote:
Hi,
I was curious to know what is the state of the following proposal: strawman:concurrency
It's on the agenda for the upcoming March meeting. I was planning to do some more work on it before declaring it ready for discussion. But since you raise it, I'm happy enough with it in its current state. I can clarify my remaining issues with it as discussion proceeds. So
This page is now ready for discussion.I do expect that this strawman is too large to be accepted into ES-next as a whole. To get some of it accepted -- syntactic sugar + some supporting semantics -- we need to find a clean boundary between what kernel and extension mechanism the platform needs to provide vs the remaining functionality that should be provided by libraries using these extension mechanisms.
For example, Tyler's web_send uses such an extension API in ref_send to stretch these operations onto the network, by mapping them onto JSON/RESTful HTTPS operations. Kris Kowal's qcomm library uses Q.makePromise (like that proposed above) to stretch these operations over WebSockets, whose connection-oriented semantics enables a better mapping at the price of more specialized usage. I hope that Kevin Reid's caja-captp can also be reformulated as a library extending the Q API.
(See links to ref_send, web_send, qcomm, and caja-captp at the bottom of the strawman page.)
I'm familiar with those libraries and actually I have a fork of Kris's q-comm where I prototyped de-sugared implementation of this proposal for Mozilla Jetpack, where it will allow cross process async messaging. Gozala/vats
I do believe that having ES native promises could provide drastically better alternative for writing async code in comparison to currently popular nested callback style. Also even though there are few implementations of Q API adoption is still low IMO that's due to non obvious and verbose syntax. Syntactic sugar described in the proposal really makes a lot of difference. Also I don't see proposal for
Q.whensyntax and would love to know what is the a plan for that.Given a lightweight closure syntax, I don't think Q.when actually needs any further syntactic sugar. For example, here's the asyncAnd example from Figure 18.1 of erights.org/talks/thesis in JS with this strawman
destructuring + the lightweight closure syntax from HOBD (Harmony of Brendan's Dreams):
#asyncAnd(answerPs) { let countDown = answerPs.length; if (countDown === 0) { return true; } const {promise, resolver} = Q.defer();
answerPs.forEach(#(answerP) { Q.when(answerP, #(answer) { if (answer) { if (--countDown === 0) { resolver(true); } } else { resolver(false); } }, #(ex) { resolver(Q.reject(ex)); }); }); return promise;}
The original asyncAnd in Figure 18.1 is in E, whose syntax was designed without legacy burden to make such code pleasant. Nevertheless, I don't think the code above suffers much in comparison.
Of course, if you have a suggestion of how a sugared Q.when can improve on this enough to be worth the cost of yet more sugar, please suggest it. Thanks.
I see your point, I was actually thinking of an E like syntax, but I do agree that Q.when is good enough, but then there still should be some build-in APIs to allowing to build Q.when on top, which leads me to a few questions:
- Is there supposed to be a build-in type that will represent a promise ?
- I believe there should be a build-in API to test whether value is promise or not, how that will look ?
- There should be build-in API to create / fulfill / brake promises and also observe state changes, which will allow lib implementers to implement Q.when like methods.
Also I think it would be useful to split up proposal into 2 parts:
- Concentrating on promises and eventual operations
- vats
I was curious to know what is the state of the following proposal: strawman:concurrency
I do believe that having ES native promises could provide drastically better alternative for writing async code in comparison to currently popular nested callback style. Also even though there are few implementations of Q API adoption is still low IMO that's due to non obvious and verbose syntax. Syntactic sugar described in the proposal really makes a lot of difference. Also I don't see proposal for
Q.whensyntax and would love to know what is the a plan for that.Thanks
Irakli Gozalishvili Web: www.jeditoolkit.com Address: 29 Rue Saint-Georges, 75009 Paris, France goo.gl/maps/3CHu