Discuss
  • natively negotiating sync vs. async...without callbacks

natively negotiating sync vs. async...without callbacks

# Getify Solutions (13 years ago)

I am aware that I'm kicking a potentially angry beehive by bringing up this topic, but I wanted to just have some discussion around if there's any possibility that JavaScript can, in the near term, have a native mechanism added to it for better managing of sync vs. async coding patterns, without nested callbacks.

The more formalized approach to this topic is typically labeled "promises" (and/or "defers"). While what I'm going to propose is going to borrow some language and ideas from that crowd, let me just say: I am not qualified as anything remotely like proficient in all the ins and outs of promises/defers. I am just a JavaScript developer who's trying to boil the formalized stuff down into some simpler ideas, and explore ways that JavaScript could natively be enhanced with building blocks to address those issues.

I will not bog down this email thread with a detailed description of all the problems with callbacks, but I encourage you to read the first half of my most recent blog post, as I go into much detail about those issues there.

blog.getify.com/2010/12/native-javascript-sync-async

It's not just a question of syntactic sugar -- I believe there's a fundamental paradigm with callbacks that is deficient, and actually that is my primary motivator. Where my proposal will have possibly shorter/simpler syntax, i consider that a benefit, but not the primary motivator.

Moreover, I don't necessarily want to stoke the flames of syntactic perferences. I'd rather discuss some underlying basic pieces of the puzzle. Also, Brendan suggested exactly the same -- that we start with discussion of the semantics and exeuction model, rather than syntax.

My proposal (semantics/processing model first, syntax second) is located in the second half of that post, here:

blog.getify.com/2010/12/native-javascript-sync-async/#proposal

Briefly, in summary, I'm suggesting a statement like this in JavaScript:

???? X(1,2) ???? Y(3,4) ???? Z("foo") ????;

X, Y, and Z are all possibly function calls which will not complete synchronously, but may defer their fulfillment asynchronously until a later time/event.

The spirit of the proposal is that this special type of statement be a linear sequence of function executions (as opposed to nested function-reference callbacks delegating execution to other code).

The special behavior is that in between each part/expression of the statement, evaluation of the statement itself (NOT the rest of the program) may be "suspended" until the previous part/expression is fulfilled. This would conceptually be like a yield/continuation localized to ONLY the statement in question, not affecting the linear execution of the rest of the program.

The model for the suspension and resuming of such evaluation would be, under the covers, identical (as much as possible) to how async callbacks are deferred to execute until a later time (turn based, etc).

Any regular statement that appears after this special type of statement would be evaluated/executed as soon as either the statement in question completes (if all parts are in fact synchronous and not-deferred), or as soon as some part of the statement in question defers, to "suspend" evaluation of the rest of the statement.

There are other details to the proposed mechanism, including "message passing" between each step in the sequence of expressions in the statement, and also conditional execution, for allowing both a "success" and "failure" path to be defined. "Success" path would be taken if the deferred/suspended expression fulfills itself under expected conditions, and "Failure" path otherwise. The mechanism would ensure that only one of either the "success" or "failure" paths was taken for any particular deferral/suspension upon resuming.

Lastly, no special behavior with respect to the state of surrounding variables/scope is implied during the resumed-from-suspension execution of the statement. Normal scoping/closure rules would apply, meaning the current state at time of resume would be respected.

I'm not trying to solve all of the possible issues/use-cases with dealing with complex async/chained logic through promises/defers. I'm not suggesting that noone will ever have to use a promise/defer "lib" of some sort again for the more complex negotiation tasks. I'm not suggesting that my proposal will be the cure all.

What I am suggesting is that the best first step to take in helping deal with sync/async negotiation in JavaScript is for there to be a small, narrowly defined, natively implemented mechanism, which will serve as a building block for more complex usage, and will simultaneously provide very streamlined and simple syntax for a wide array of common sync/async tasks (like event binding, etc).

I welcome any thoughts/feedback on the viability of such a proposal, or pitfalls or structural shortcomings I may not have forseen yet. But again, let's not get mixed up in the weeds of syntax preference just yet.

Thanks, Kyle (aka @getify)

# Mark S. Miller (13 years ago)

[+ravi]

Hi Kyle, you may also be interested in taking a look at the strawman at < strawman:concurrency> and the Orleans

paper and video just announced in the email below. One lesson to take from these (and similar systems going back to Actors) is: once you support asynchronous interactions between objects well, you are already most of the way towards supporting distributed object computing. Since JavaScript in practice (Browsers with postMessage and WebWorkers, async xhr, NodeJS) is already de-facto using communicating event loop concurrency, this seems like a natural step. We're already using the abstractions on that strawman page via the ref_send libraries, but without the syntactic support proposed by that strawman.

The kind of local chaining you're talking about can be expressed straightforwardly using Q.when, though a bit verbosely because of the "function" and "return" keywords. If we had a shorter function syntax, e.g., "#" for "function" and "(expression)" for "{ return expression; }", then we could code your example as

const xP = X(1,2);
const yP = Q.when(xP, #(x) (Y(3,4)));
const zP = Q.when(yP, #(y) (Z("foo")));

This still seems much more verbose, but it accomplishes something else that you typically want: it brings the resolution of each expression into scope (x, y) for use within the body of the next one.

---------- Forwarded message ---------- From: Ravi Pandya <ravi at iecommerce.com>

Date: Fri, Dec 3, 2010 at 6:35 AM Subject: [e-lang] Orleans framework for cloud computing To: e-lang at mail.eros-os.org

I've been working on a project in Microsoft Research that builds on some ideas from E and related work - promises, turn-based execution, object-capability patterns, etc. It is a framework and runtime for writing distributed applications to run in the cloud (and eventually on the client as well). It also includes functionality such as optimistic lightweight transactions, transparent persistence, and object replication & synchronization. We're not yet ready to release the code externally, but we have published a technical report and an online video describing the project. I'd be interested in your comments.

Tech report: research.microsoft.com/apps/pubs/?id=141999

Video: channel9.msdn.com/Shows/Going+Deep/Project-Orleans-A-Cloud-Computing- Framework

Thanks,

Ravi

# Breton Slivka (13 years ago)

On Wed, Dec 8, 2010 at 8:48 AM, Getify Solutions <getify at gmail.com> wrote:

I am aware that I'm kicking a potentially angry beehive by bringing up this topic, but I wanted to just have some discussion around if there's any possibility that JavaScript can, in the near term, have a native mechanism added to it for better managing of sync vs. async coding patterns, without nested callbacks.

snip

blog.getify.com/2010/12/native-javascript-sync-async/#proposal

Briefly, in summary, I'm suggesting a statement like this in JavaScript:

???? X(1,2) ???? Y(3,4) ???? Z("foo") ????;

X, Y, and Z are all possibly function calls which will not complete synchronously, but may defer their fulfillment asynchronously until a later time/event.

The spirit of the proposal is that this special type of statement be a linear sequence of function executions (as opposed to nested function-reference callbacks delegating execution to other code).

The special behavior is that in between each part/expression of the statement, evaluation of the statement itself (NOT the rest of the program) may be "suspended" until the previous part/expression is fulfilled. This would conceptually be like a yield/continuation localized to ONLY the statement in question, not affecting the linear execution of the rest of the program.

es-discuss mailing list es-discuss at mozilla.org, mail.mozilla.org/listinfo/es-discuss

I haven't found anyone else that has worked this out, and I have been keeping it kind of a secret, but seeing as I haven't yet done anything with it, I will reveal that what you propose is more or less possible with plain vanilla ecmascript 3. To modify your syntax proposal, what the syntax would look like is something like this:

$(document).X(1,2).Y(3,4).Z("foo");

this call returns an object, with a call, and an apply method (possibly a function object?) that when called, starts a chain of execution that is suspended at the finishing of each operand. Most of the time this suspension would be caused by a simple call to setTimeOut(next, 0); With other functions that are obviously asyncronous, such as XHR calls, the suspension is resumed at the point when the callback would normally be called.

This would be made possible by a library, I will call MND for the purposes of this discussion. You start by passing MND an object with a set of related methods:

var MyMonad = MND.make(MyObject);

the MyMonad object has methods with all the same names as the object that you passed in, but when you call them, the names and arguments are simply placed on an internal stack, and an object with call/apply methods is returned.

When the call function is invoked, it loops through the version of the stack stored on that particular object, by executing each method in turn, and in the spot where each method requires a callback, it places an additional helper function to continue execution of the stack. Additionally, the return value of the previous method call is passed in a parameter (perhaps as a property of "this") to the next function to be executed. The initial function call (above, as $(document)), allows you to pass in some starting value for this chain of execution.

The only drawback is that the call/apply method would itself require a callback function, for when the whole chain finishes, but that is still far preferable than multiply nested callbacks, which is, I presume, the problem you are attempting to solve.

I think the syntax for this type of library is simple enough to not require new syntax. In fact it's pretty similar to jQuery, which as you probably know is quite popular specifically for its simplicity.

# Tom Van Cutsem (13 years ago)

The spirit of the proposal is that this special type of statement be a linear sequence of function executions (as opposed to nested function-reference callbacks delegating execution to other code).

The special behavior is that in between each part/expression of the statement, evaluation of the statement itself (NOT the rest of the program) may be "suspended" until the previous part/expression is fulfilled. This would conceptually be like a yield/continuation localized to ONLY the statement in question, not affecting the linear execution of the rest of the program.

This reminds me of a proposal by Kris Zyp a couple of months ago ("single frame continuations") esdiscuss/2010-March/010865

I don't think that discussion lead to a clear outcome, but it's definitely related, both in terms of goals as well as in mechanism. I also recall it prompted Dave Herman to sketch the design space of (single-frame) continuations for JS: esdiscuss/2010-April/010894

# David Herman (13 years ago)

I pretty much abandoned that line of investigation with the conclusion that generators:

http://wiki.ecmascript.org/doku.php?id=strawman:generators

are a good (and well-tested, in Python and SpiderMonkey) design for single-frame continuations. They hang together well; in particular, they don't have the issues with `finally' that some of the alternatives I talked about do. Moreover, the continuation-capture mechanisms based on call/cc or shift/reset require additional power in the VM to essentially tail-call their argument expression. When I tried prototyping this in SpiderMonkey, I found this to be one of the biggest challenges -- and that was just in the straight-up interpreter, not in the tracing JIT or method JIT.

Generators work well for lightweight concurrency. As a proof of concept, I put together a little library of "tasks" based on generators:

http://github.com/dherman/jstask

Somebody reminded me that Neil Mix had written a very similar library several years ago, called Thread.js:

http://www.neilmix.com/2007/02/07/threading-in-javascript-17/

and there's another library called Er.js that built off of that to create some Erlang-like abstractions:

http://www.beatniksoftware.com/erjs/
# David Herman (13 years ago)

PS To be concrete, here's an example code snippet using my jstask library that chains several event-generated actions together in a natural way (i.e., in direct style, i.e. not in CPS):

var task = new Task(function() {
    var request = new HttpRequest();
    try {
        var foo = yield request.send(this, "foo.json");
        var bar = yield request.send(this, "bar.json");
        var baz = yield request.send(this, "baz.json");
    } catch (errorResponse) {
        console.log("failed HTTP request: " + errorResponse.statusText);
    }
    ... foo.responseText ... bar.responseText ... baz.responseText ...
});

I should also point out that the core of jstask is 7 lines of code. :)

# Shriram Krishnamurthi (13 years ago)

It might surprise some who know me to hear this, but I agree with Dave on this. There's a huge gap between the single-frame mechanism and going the whole hog. Going all out does buy you some expressive power, but at the cost of complicating everything. If the simple mechanism gives you most of what you want (and I feel it goes pretty far), why commit to a lifetime of pain?

Shriram

# Kris Zyp (13 years ago)

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

The generators from JS 1.7 are too specific to provide much help with promises, IMO. Adding a yield operator fundamentally alters the semantics of the entire surrounding function, violating the principle of locality. Consequently you need special mechanisms (like Neil's library) to even call an async function, and you can't "return" values either (with the return operator). Solving these problems was the point behind the shallow continuation proposal that we worked on. Does the generator strawman build on that work, or does it still fail to preserve locality?

My understanding of Kyle's proposal was that he did not want to introduce real continuations, but this was more of a syntax for a specialized callback chain, reducing the verbosity of anon functions, callback registration. I'd prefer more of a generalized solution for reducing the size of anon functions (which has been discussed to great lengths here in the lambda threads). However, my belief on async is that the verbosity is only part of the pain, working with async within various control flows (loops, branches, etc) is more of a burden, hence the need for continuations (shallow/single frame, of course). Kris

# Getify Solutions (13 years ago)
 var task = new Task(function() {
    var request = new HttpRequest();
    try {
        var foo = yield request.send(this, "foo.json");
        var bar = yield request.send(this, "bar.json");
        var baz = yield request.send(this, "baz.json");
    } catch (errorResponse) {
        console.log("failed HTTP request: " +

errorResponse.statusText); } ... foo.responseText ... bar.responseText ... baz.responseText ... });

This style is similar to what I'm proposing, except for a few important things:

  1. I'm not trying to open the can-o'-worms around block level changes. The above code suggests that a 'yield' suspension of execution is local to the nearest container { } block, in this case the try { } block. While I wouldn't be strongly opposed to it, it seems a little unnatural to have to wrap a collection of like-behaving stuff in a { }... if that were the case, just use an anonymous function (without or without the shorter syntax) and not have to change the meaning of { } blocks. But I can see some merit in this concept. Just not hugely in favor of it.

  2. It breaks a fundamental thing that I happen to think is really important... it puts the control of whether a function is yielded or not into the hands of the calling code. I happen to think the function call itself should get to decide if he wants to yield asynchronously, or finish synchronously.

For instance, imagine a function that retrieves a value from some "database", but employs some memoization/local caching. In the first call case, that function will need to return asynchronously. But in the cached case, that function can and should return immediately. It'd be nice for the function to get to decide conditionally if it wants to "yield" or "defer" itself based on that state.

And the calling code, by virtue of something like the native operator I suggested, can be written just one way and not care about that hidden detail.

lookupDBValue("something") @ printValue();

For async, printValue will be held up from executing until lookupDBValue signals that it's complete by fulfilling its promise. For sync, lookupDBValue will already have its promise fulfilled at the time it returns, so the @ can move to immediately/synchronously executing printValue.

# Getify Solutions (13 years ago)
 This reminds me of a proposal by Kris Zyp a couple of months ago

("single frame continuations") esdiscuss/2010-March/010865

I don't think that discussion lead to a clear outcome, but it's definitely related, both in terms of goals as well as in mechanism. I also recall it prompted Dave Herman to sketch the design space of (single-frame) continuations for JS: esdiscuss/2010-April/010894

I have to throw up a red flag and claim some naivety/ignorance here. Even

after reading those links, I'm confused on what "single frame continuations" means. Perhaps what I have in mind is like that, but perhaps not, and I need a little help in understanding better the implications of that terminology.

what is a "single-frame"? I think I roughly understand a 20,000ft level idea of what continuation is.

It might surprise some who know me to hear this, but I agree with Dave

on this. There's a huge gap between the single-frame mechanism and going the whole hog. Going all out does buy you some expressive power, but at the cost of complicating everything. If the simple mechanism gives you most of what you want (and I feel it goes pretty far), why commit to a lifetime of pain?

As I said, I'm confused, so I'm not sure where my idea fits in the spectrum

of what you're suggesting. Is my idea closer to the simple mechanism "single-frame continuation" side or is it closer to the "whole hog" side?

I feel like my idea is pretty simple and limited compared to what I've seen from the broader scope ideas of full program continuations, true concurrency, etc. But maybe I'm in completely the wrong "frame" of reference and my idea is way far out there in complexity?

Solving these problems was the

point behind the shallow continuation proposal that we worked on. Does the generator strawman build on that work, or does it still fail to preserve locality?

OK, so is "shallow continuation" the same as "single-frame continuation" or are we now talking about different concepts? Again, on the surface, "shallow continuation" sounds like it might be akin to what my idea is, but I may be completely wrong.

My understanding of Kyle's proposal was that he did not want to

introduce real continuations, but this was more of a syntax for a specialized callback chain, reducing the verbosity of anon functions, callback registration.

In my naive understanding of the words, I think what I want to introduce is

statement-localized continuations.

I want for a statement, which can consist of two or more expressions, any or all of them being function calls which can chose to yield/defer/suspend their completion asynchronously.

Only a statement which would use the @ operator on two or more expression operands would be able to take advantage of this statement localized continuation, though. A function that called p.defer() inside itself, but which was called in a normal expression/statement without @ operator, would simply complete and program execution would continue as normal. Of course, the function would still be able to complete asynchronously at a later time, just like a callback to setTimeout can.

function foo() { console.log("foo"); } function bar() { console.log("bar"); }

foo() @ bar(); // normal sync statement that prints "foobar".

function foo() { var p = promise; setTimeout(function(){ console.log("foo"); p.fulfill(); },1000); p.defer(); } function bar() { console.log("bar"); }

foo() @ bar(); // async statement, which will print "foobar", but not until 1000ms from now.

And what I mean by "statement-localized continuation" is that yield/defer/suspend/continuation would only affect that single statement with the @ in that case, and any statements after the @ statement would continue synchronously if the @ statement indeed gets suspended at any point during its evaluation.

So:

function baz() { console.log("baz"); }

baz() @ foo() @ bar(); console.log("yes");

would result in "bazyes" immediately, and 1000ms from now, "foobar" right after it.

Does that make any more sense in terms of clarifying my idea, or does it just complicate things worse?

# Mike Shaver (13 years ago)

On Thu, Dec 9, 2010 at 12:22 PM, Getify Solutions <getify at gmail.com> wrote:

  1. I'm not trying to open the can-o'-worms around block level changes. The above code suggests that a 'yield' suspension of execution is local to the nearest container { } block, in this case the try { } block

No, as implemented in JS1.7, the suspension is of the innermost frame, not the innermost lexical block. (You think that it would pause that block, and then jump to the statement following it, to execute from there until the yield...yielded?)

developer.mozilla.org/en/Core_JavaScript_1.5_Guide/Iterators_and_Generators has more.

Mike

# David Herman (13 years ago)

I'm not trying to open the can-o'-worms around block level changes. The above code suggests that a 'yield' suspension of execution is local to the nearest container { } block, in this case the try { } block.

No, that's not the case. That code was not hypothetical: it works in SpiderMonkey using extensions that have been around for several years. You might want to take some time to explore generators by following the docs that Mike linked to, and by playing with SpiderMonkey.

I have to throw up a red flag and claim some naivety/ignorance here. Even after reading those links, I'm confused on what "single frame continuations" means.

Both "single-frame continuation" and "shallow continuation" are terms we've used informally for the same idea: being able to suspend a single function activation (aka stack frame) as a JavaScript value that you can use later to resume the suspended activation.

I feel like my idea is pretty simple and limited compared to what I've seen from the broader scope ideas of full program continuations, true concurrency, etc.

Nobody is recommending full continuations or heavyweight concurrency facilities. Especially if it involves shared-memory threads (which most straightforward extensions would) -- that will definitely not happen in ECMAScript).

But maybe I'm in completely the wrong "frame" of reference and my idea is way far out there in complexity?

I haven't fully understood your proposal yet, so FWIW it's hard for me to say just yet.

In my naive understanding of the words, I think what I want to introduce is statement-localized continuations.

I want for a statement, which can consist of two or more expressions, any or all of them being function calls which can chose to yield/defer/suspend their completion asynchronously.

I think you might find you need to work through this part in more detail. Part of the essential trickiness in nailing down the semantics of a continuation operator is specifying exactly how much of the continuation is suspended.

I don't understand the |promise| "auto-variable" you mention. What does it do? Does it affect control flow as well, or is it no difference from (new Promise()) for some built-in Promise constructor? (BTW, automatically-bound variables like |arguments| are a mis-feature IMO, and I'd urge you to reconsider that part.)

What does it mean when you say in your post that "the @ operator will wait to continue"? What does it mean in JS to "wait?" Does it mean that evaluation of the entire statement suspends and the result is a new promise? Where does that promise go? Statements discard their result, more or less (notwithstanding the completion value, which in most cases in the language really is discarded), so how does that promise not just disappear?

You mention a ternary form but I can't find it anywhere in the blog post. Do you describe it in any more detail somewhere else?

What is the specification of p.defer()? Does it cause its containing function to suspend? Or just its containing statement?

# Brendan Eich (13 years ago)

On Dec 9, 2010, at 3:27 PM, David Herman wrote:

I'm not trying to open the can-o'-worms around block level changes. The above code suggests that a 'yield' suspension of execution is local to the nearest container { } block, in this case the try { } block.

No, that's not the case. That code was not hypothetical: it works in SpiderMonkey using extensions that have been around for several years.

Works in Rhino too, thanks to Norris Boyd, Steve Yegge, et al.

# Bradley Meck (13 years ago)

I really like the proposal, but think the syntax is confusing many of us.

I think that the proposal is closer to post hooking than continuation in spirit. It appears that the syntax is misleading but reading the text of the desired effects promise.defer() is acting like a point to hook the next part of the expression rather than a yield (not sure if it would make more sense to make this construct in whatever syntax return from the current function). The reason that I believe this is the case is the fact that the stack is unwound / "normal" closure abnormalities apply it seems. The interesting part of this proposal is the fact that you can pass messages between hooks to me. Overall, I find a hooking proposal to be beneficial since soo many times I am making anonymous functions to do so. Having to declare a hooking point might be overkill, but the message passing and a generic code pathing on error would be amazing for async dev time personally.

I think given the side effects wanted these are the list of features in this proposal:

  1. code run in this "statement" does so at the first opportune moment (sync runs immediately)
  2. code in this statement does not start execution until some condition is true
  3. code that encounters and error in this statement is routed to a construct in this statement automatically.

Given these I propose a different syntax, since #2 is the main issue with why it is being considered similar to a continuation. I think that this construct should declare up front instead of inline when it defers. This syntax is still poor but should help to illustrate (using a binary ! since js doesnt have one).

foo = function(cb) { setTimeout(cb,1000) } bar = function(cb) { ... cb(1) ... } end = function(cb,x) { log(x) } // will log 1 foo @ bar @ end ! err

Would mean that foo should wait until cb is fired in defer. Then if it does not error it would go to bar and pass along some context arguments. If it errors it would go to err. If bar does not error it would go to end. If bar errors it would go to err still.

So how does this relate to making things easier?

  1. Error routing callbacks. This would be a god send if you have ever used node.js .
  2. No confusing continuations (many people like them but I find them hard to track code wise).
  3. Passing data between callbacks would not require a closure construction which is generally costly.
  4. What is being defered on does not break functions since it is an argument automatically passed (non-ideal since it could cause issues with existing code). Like Kyle proposed a keyword maybe needed etc.
# Getify Solutions (13 years ago)

No, that's not the case. That code was not hypothetical: it works in SpiderMonkey using extensions that have been around for several years. You might want to take some time to explore generators by following the docs that Mike linked to, and by playing with SpiderMonkey.

OK, clearly I have a lot to learn about SpiderMonkey and extended JavaScript stuff like that. I'm sorry I'm so behind the game. I'll try to catch up :)

Both "single-frame continuation" and "shallow continuation" are terms we've used informally for the same idea: being able to suspend a single function activation (aka stack frame) as a JavaScript value that you can use later to resume the suspended activation.

The wording "suspend ...as a JavaScript value you can use" seems to me to indicate that a defered-promise/yielded-function is a first-class JavaScript value type, that can be assigned and manipulated. Is that correct? Which also seems to possibly imply that the code that calls a function is in control of suspending the function and of resuming it?

I may obviously be mis-interpreting, but I'd say my idea is a little more reserved than that. There is no first-class "Promise" type. It's just an implicit mechanism by which the asynchronous negotiation of each subsequent expression of the @ statement occurs.

Put another way, every function (and indeed every valid expression) call has a "promise". A function (and only a function) called can choose to defer its internal promise. If a function call doesn't defer its promise, or if the expression is not a function call, then the promise for that expression is immediately flagged as fulfilled.

But the calling code doesn't interact with the promise directly. That is, a function doesn't return a first-class promise value that the calling code directly receives and manipulates. A function can only flag its own internal state as being deferred. A function that flags itself as deferred can still do a normal return statement and send back an immediate value to the calling code statement/expression. For instance:

function foo() { var p = promise; setTimeout(function(){ p.fulfill(); }, 1000); p.defer(); return 12; } function bar() { console.log(x+1); }

var x = 0; (x = foo()) @ bar(); console.log(x);

This code example would immediately output "12", and then a second later, output "13".

The calling code only indirectly knows of a function call's promise deferral, via the @ operator. The @ operator is able to inspect if the previous function call's internal promise was deferred or not, and if it was deferred, the rest of that current @ statement in the calling code is suspended. Otherwise, the statement continues its evaluation immediately.

When the deferred function signals that its now complete, the suspended @ statement receives that signal and resumes evaluation of the statement.

The function code doesn't "suspend" itself or create a continuation inside itself. It simply can flag (with defer()) that it won't be finished yet when function return occurs. And the function hands the "key" (with fulfill()) to fulfilling its promise over to some asynchronously executing code -- for instance, inside an inner function that's assigned as a callback for an event, XHR call, or timeout, etc.

If a function call doesn't flag that it will defer its completion, it is always assumed to have an immediate completion, which means that any standard function call in an @ statement will simply keep going with no suspension of the statement at that point.

I don't understand the |promise| "auto-variable" you mention. What does it do? Does it affect control flow as well, or is it no difference from (new Promise()) for some built-in Promise constructor? (BTW, automatically-bound variables like |arguments| are a mis-feature IMO, and I'd urge you to reconsider that part.)

promise as an auto-variable is a special accessor to that current function call's internal promise state. It has fulfill(), fail() and defer() as its 3 methods. It also has a messages property, which is an array of the arguments passed to the previous deferred function's fulfill(...) or fail(...) call. If there is no message (either an implied promise from an expression or normal function call, or no parameters passed, then the messages is empty.

The reason it's not just new Promise() is because its not an object that is negotiable (can't be returned, etc). It only has meaning inside the context of the function call.

I'm not wild about having to create another auto-variable either. But this isn't sufficient since it takes on a function's activation object, and arguments.callee is deprecated. I need some way to have a function refer to its own internal promise state. I'd welcome ideas for a different name for it, or some other way to access this internal state that's not so awkward.

What does it mean when you say in your post that "the @ operator will wait to continue"? What does it mean in JS to "wait?" Does it mean that evaluation of the entire statement suspends

Yes.

and the result is a new promise?

No. The "result" of a suspended statement would be the the last immediate return value from a function call (or the implied undefined if the function didn't have a return), or from the expression (like b=5 or d++, etc). Since my idea doesn't model promises as a first-class value returned from the function, but just as internal state of the function call, it wouldn't make any sense to have that promise be the statement's value.

You mention a ternary form but I can't find it anywhere in the blog post. Do you describe it in any more detail somewhere else?

Sorry, the ternary form is illustrated on the original gist thread where I first proposed this idea:

gist.github.com/727232#file_ex2:extended_@_:_usage

Essentially:

function foo() { setTimeout(function(){ if (blah == 5) { p.fulfill(); } else { p.fail(); } }, 1000); } function yay() { console.log("yay"); } function bummer() { console.log("bummer"); }

var blah = 5; foo() @ yay() : bummer(); // after 1 second, "yay"

foo() @ yay() : bummer(); // after 1 second, "bummer" blah = 10;

What is the specification of p.defer()? Does it cause its containing function to suspend? Or just its containing statement?

The p.defer() doesn't create any kind of side effect inside the function. It doesn't suspend the function or create a continuation. It simply flag's that function's internal promise state that it will be deferred, and that at a later time, fulfill() or fail() will be called to complete the promise.

Some have suggested that "fail" not be an explicit thing but only if a JavaScript error/exception is thrown. I don't suppose I'd be terribly opposed to that, but I do see some complications with it. First, if I programmatically want to trigger the "fail" path, then I have to artificially create a JS error with throw. That's a little more awkward than just calling fail, especially since I can directly call fulfill in the same context. Also, if I want to pass some sort of value other than an error object as my "message" to the "fail" path, I have to wrap that message inside the Error object I throw. Again, more awkward. fail() seems more direct to me.

# Getify Solutions (13 years ago)

I think given the side effects wanted these are the list of features in this proposal:

  1. code run in this "statement" does so at the first opportune moment (sync runs immediately)
  2. code in this statement does not start execution until some condition is true
  3. code that encounters and error in this statement is routed to a construct in this statement automatically.

There's some other really important aspects of my proposal:

  1. Control of execution (that is, the actuall calling of a function) is linear in the chain in the calling code, rather than being controlled by the code inside a function. If some function in my chain is a third-party function I don't control and don't fully trust, then I don't want to pass it any references to my code's functions (as a callback) or any of my code's variables/scope. I simply want that function to "signal" to my code that it's complete. My code then is able to "wait" for that signal (if I chose to set up the statement that way with @ usage) and then continue evaluation of the rest of the statement.

For instance:

function foo() { var p = promise; setTimeout(function(){ p.fulfill(20); }, 1000); p.defer(); // this could be p.willDefer = true or some other less confusing syntax, // for flagging the current function's internal promise as being deferred. }

Now, let's assume that function is a third party code that I don't control. All I know that it's "async/promise aware". Now, let's also assume I have a bar function in my code that I need to pass an object to and have it do something, but I only want bar to run after foo completes.

In other words, I want to kind of conceptually observe the async behavior of foo and "subscribe" to his "completion" event. But I don't know or trust foo enough to pass him any of my functions, nor do I want to pass him references to any of my scope's variables (which he could then mess with). I just want to "listen" for him to signal that he's fully complete, and then move on.

function bar(y) { console.log(y.some_value); }

var obj = { some_value: 10 };

foo() @ bar(obj); // whenever foo completes, output: "10"

  1. I don't want to conflate a function's parameters, or its return value, with the negotiation of the asynchronous chaining, suspension, resume, message passing, etc. This is intended to preserve existing code to the greatest extent possible.

For instance:

function foo(x) { var p = promise; setTimeout(function(){ p.fulfill(x+20); }, 1000); p.defer();

return x+10; // return an immediate value }

Again, let's assume that function is a third party code that I don't control. All I know that it's "async/promise aware", and that it has both an immediate return value and a deferred result.

Let's then say that in some part of my code, I just want to get the immediate value, and I don't care what that function's eventual/deferred result is:

var x = foo(4); console.log(x); // "14" ...

This code would operate exactly like normal JavaScript. Even though foo() has a deferred part to it, the calling code is choosing not to observe this behavior, and just continue as normal, effectively discarding any async behavior that may or may not be inside of foo. This is identical to existing code patterns where I can call a function, and it can create async side effects that I may or may not be aware of or care about.

Later in the code, I may decide that I do actually want to care about the foo function's async behavior, so I want to construct a statement that explicitly observes foo and "waits" for that fulfilled value. Without needing any kind of different/changed foo signature in any way, I can then choose to do this:

function bar(y) { var _x = (promise.messages.length) ? promise.messages[0] : 1; console.log(_x * y); }

bar(3); // immediately, "3" var x; (x = foo(4)) @ bar(3); // after 1 second: "42" console.log(x); // but, immediately: "14"

I recognize the syntax I'm proposing is quite different from normal code (but I'm hoping eventually that it's seen as different in a good way, not a confusing or bad way). But the syntax is explicitly and intentionally separating the concern of async promise/deferral and fulfillment/continuation from the concern of function parameter passing or function return values.

I think I should be able to choose in my calling code if I want to construct an @ statement that observes the deferral or if I just want to call the function and only observe its immediate effects.

This is conceptually the same as me saying that if there's some function I want to call that requires a callback, but for whatever reason I don't care about the eventual effects, I can just pass it a dummy no-op callback and accomplish the ignoring (the not observing) of that async effect.

Given these I propose a different syntax, since #2 is the main issue with why it is being considered similar to a continuation. I think that this construct should declare up front instead of inline when it defers. This syntax is still poor but should help to illustrate (using a binary ! since js doesnt have one).

foo = function(cb) { setTimeout(cb,1000) } bar = function(cb) { ... cb(1) ... } end = function(cb,x) { log(x) } // will log 1 foo @ bar @ end ! err

Where does cb come from or get defined? Is it an explicit thing the calling code would create, or is it an implicit thing that the @ operator is going to create on the fly and pass to all the operand function calls?

In this code example, none of your foo, bar, end or err functions are being explicitly called with a (). Is this intentional, to suggest that the @ operator will implicitly call them? I'm guessing so, because it also appears that the @ operator is taking care of passing that cb to all of them. Or perhaps I'm still misunderstanding.

So how does this relate to making things easier?

  1. Error routing callbacks. This would be a god send if you have ever used node.js .

No question, I agree that "error handling" (but not strictly in the sense of trapping JS errors, rather a higher level definition of "error" which is any state the function decides is unexpected/undesired) needs to be part of the mechanism.

  1. No confusing continuations (many people like them but I find them hard to track code wise).

The confusion is a big part of the reason I want to contain them only localized to a single statement. I'm relying on the fact that a lot of developers already do things within single statements where they "tolerate" the idea that what they're expressing in the single statement won't fully complete right away.

jQuery chaining is an example of this. With jQuery chains, I can express a chain of behavior, but in some circumstances, even though lexically the whole chain will execute, the full completion effects of the chain will not be finished when the chain is done parsing.

Similarly, LABjs does the same thing. And people don't seem to have too much trouble understanding the idea that $LAB.script("1.js").wait().script("2.js") will process the whole chain immediately, but it will create an asynchronous chain of events that will complete at a later time.

  1. Passing data between callbacks would not require a closure construction which is generally costly.

Agreed. I think the message passing is important, but I want to separate it from the function call signature or return value, and have it be instead accessible through the function call's internal "promise" state.

  1. What is being defered on does not break functions since it is an argument automatically passed (non-ideal since it could cause issues with existing code). Like Kyle proposed a keyword maybe needed etc.

On the contrary, I think by conflating the function's parameter list with the negotiation token for the async behavior, we've explicitly created a system where function signatures will have to change to work with the new system.

By contrast, I'd like to have it so that all (or most) function signatures that currently exist don't have to be changed, only the internal behavior of a function needs to be slightly adjusted to "teach" it about this new "promise" interface.

Lastly, I'd say that function signatures which currently accept a callback also don't have to change. In fact, there may be some value in allowing both styles (existing callbacks, and my new proposed style) to co-exist in the same function, giving the calling developer the freedom to choose. Imagine an event binding function:

function onclick(obj, cb) { var p = promise; obj.addEventListener("click", function(){ if (cb) cb(); else p.fulfill(); }, true); p.defer(); }

function clicked() { console.log("btn clicked!"); }

var obj = document.getElementById("btn"); onclick(obj, clicked); // or onclick(obj) @ clicked();

# Getify Solutions (13 years ago)

Lastly, I'd say that function signatures which currently accept a callback also don't have to change. In fact, there may be some value in allowing both styles (existing callbacks, and my new proposed style) to co-exist in the same function, giving the calling developer the freedom to choose.

Another example of this would be existing interfaces in JavaScript which accept callbacks, like setTimeout. Those interfaces could be extended to also be aware of this new behavior, without changing their function signature at all:

function foo() { console.log("foo rocks"); }

setTimeout(foo, 1000); // or setTimeout(null, 1000) @ foo();

# Getify Solutions (13 years ago)
  1. Control of execution (that is, the actuall calling of a function) is linear in the chain in the calling code, rather than being controlled by the code inside a function.

Another reason I feel it's important to address being able to retain control of the execution of the chained functions in my calling code is the awkwardness around function context objects and them losing their this binding.

We know that this:

function foo(cb) { setTimeout(cb,1000); } var obj = { val: 1, bar: function() { this.val = (this.val || 0) + 1; console.log(this.val); } }; foo(obj.bar);

...is subject to the this binding being lost from "obj" and becoming "window". So, we have to go through hoops to "bind" a function's this tightly, with Function.prototype.bind(), or we create a manual anonymous function wrapper, etc. While it's possible to correct for, obviously, it creates extra work for the developer and is a common trap for subtle errors.

Instead:

foo() @ obj.bar();

Since my calling code is in control of the execution of bar, I call it on the proper object and this doesn't get lost.