How would shallow generators compose with lambda?

# Mark S. Miller (16 years ago)

Two features that have been proposed for Harmony are shallow generators and lambda.

In the absence of lambda, I think I understand shallow generators. Because of the way my brain is wired, for whatever reason, my path to understanding them is as a source-to-source cps-style transform of the generator function -- the function immediately containing the 'yield'. Because generators are shallow, only this function need be translated. I do understand that other people find this route to understanding shallow generators unnatural.

In the absence of generators, I am confident I understand lambda. A return in a lambda returns from the immediately containing function activation, so long as that function activation is still active. If the containing function has already returned, then a postponed attempt to return from it again will fail[1], presumably with a thrown error. This applies as well to break, continue, labeled break, labeled continue, and labeled return if we introduce such a thing. What happens to a postponed 'yield'? Also, does it make as much sense to introduce a labeled yield as it does to introduce a labeled return?

Given both shallow generators and lambda, I don't understand how they would interact.

[1] Schemers would say "dynamic extent continuations" vs Scheme's "indefinite extent continuations". Indefinite extent continuations -- which can be returned from multiple times -- are IMO a horror. Indefinite extent continuations would enable deep generators, which only increases my sense of horror. Perhaps combining lambda with shallow generators produces something like delimited continuations? The literature on these makes my brain hurt.

Two features that have been proposed for Harmony are shallow
generators and lambda.

In the absence of lambda, I think I understand shallow generators.
Because of the way my brain is wired, for whatever reason, my path to
understanding them is as a source-to-source cps-style transform of the
generator function -- the function immediately containing the 'yield'.
Because generators are shallow, only this function need be translated.
I do understand that other people find this route to understanding
shallow generators unnatural.

In the absence of generators, I am confident I understand lambda. A
return in a lambda returns from the immediately containing function
activation, so long as that function activation is still active. If
the containing function has already returned, then a postponed attempt
to return from it again will fail[1], presumably with a thrown error.
This applies as well to break, continue, labeled break, labeled
continue, and labeled return if we introduce such a thing. What
happens to a postponed 'yield'? Also, does it make as much sense to
introduce a labeled yield as it does to introduce a labeled return?

Given both shallow generators and lambda, I don't understand how they
would interact.

[1] Schemers would say "dynamic extent continuations" vs Scheme's
"indefinite extent continuations". Indefinite extent continuations --
which can be returned from multiple times -- are IMO a horror.
Indefinite extent continuations would enable deep generators, which
only increases my sense of horror. Perhaps combining lambda with
shallow generators produces something like delimited continuations?
The literature on these makes my brain hurt.

-- 
    Cheers,
    --MarkM

# Jason Orendorff (16 years ago)

On Thu, May 14, 2009 at 12:25 PM, Mark S. Miller <erights at google.com> wrote:

Given both shallow generators and lambda, I don't understand how they would interact.

This is a good question. I don't have the answer, but thinking aloud:

You can implement shallow generators either by doing the CPS transformation you described or by storing (a reference to) the top stack frame in the generator-iterator object. (The latter is how SpiderMonkey and CPython currently do it.)

Now, suppose 'yield' and 'return' are extended in analogous ways in lambdas.

A lambda can escape and be called from other code. So both implementation approaches must be modified:

The CPS transformation would have to be applied to almost all functions, even functions not lexically inside the generator-function. This is a major change.
The stack-snapshot approach would have to store (a reference to) a range of stack frames, which may include frames for functions not lexically inside the generator-function. This might be a minor change.

So much for duality. :-|

With shallow generators, there is a check so that you can't call .next() on a generator if it's already executing. It just throws a TypeError. This enforces an invariant:

In the continuations view, each captured continuation will be called (resumed) at most once.
In the stack-snapshot view, no stack frame may ever appear twice in the stack.

The implementation of the check is unaffected by lambdas--.next() can trivially check a bit in the generator-iterator, regardless of implementation. But with lambdas, the invariants take on a new meaning. In particular, note that generators+lambdas are not fully as powerful as delimited continuations. They are more like coroutines. ...In fact, you might be able to use them to implement coroutines.

When a lambda yields, the lines of code corresponding to the topmost captured frame (or, the beginning of the delimited continuation) and the bottommost (the tail end of the delimited continuation) are both lexically inside the generator-function. But there may be other functions on the stack, in between. You can't always statically tell which ones. This means that generator semantics affect the integrity of code that isn't in a generator.

In ES5, when you call a function, you can expect it to return or throw eventually. (Unless you run out of memory, or time, and the whole script gets terminated.) With shallow generators, this is still true. A 'yield' might never return control, but function calls are ok. But with generators+lambdas, almost any function call anywhere in the program might never return or throw. This weakens 'finally', at least.

On Thu, May 14, 2009 at 12:25 PM, Mark S. Miller <erights at google.com> wrote:
> Given both shallow generators and lambda, I don't understand how they
> would interact.

This is a good question. I don't have the answer, but thinking aloud:

You can implement shallow generators either by doing the CPS
transformation you described or by storing (a reference to) the top
stack frame in the generator-iterator object. (The latter is how
SpiderMonkey and CPython currently do it.)

Now, suppose 'yield' and 'return' are extended in analogous ways in lambdas.

1. A lambda can escape and be called from other code.  So both
implementation approaches must be modified:

* The CPS transformation would have to be applied to almost all
functions, even functions not lexically inside the generator-function.
This is a major change.

* The stack-snapshot approach would have to store (a reference to) a
range of stack frames, which may include frames for functions not
lexically inside the generator-function. This might be a minor change.

So much for duality. :-|

2. With shallow generators, there is a check so that you can't call
.next() on a generator if it's already executing.  It just throws a
TypeError.  This enforces an invariant:

* In the continuations view, each captured continuation will be called
(resumed) at most once.

* In the stack-snapshot view, no stack frame may ever appear twice in the stack.

The implementation of the check is unaffected by lambdas--.next() can
trivially check a bit in the generator-iterator, regardless of
implementation.  But with lambdas, the invariants take on a new
meaning.  In particular, note that generators+lambdas are not fully as
powerful as delimited continuations. They are more like coroutines.
...In fact, you might be able to use them to implement coroutines.

3. When a lambda yields, the lines of code corresponding to the
topmost captured frame (or, the beginning of the delimited
continuation) and the bottommost (the tail end of the delimited
continuation) are both lexically inside the generator-function. But
there may be other functions on the stack, in between. You can't
always statically tell which ones.  This means that generator
semantics affect the integrity of code that isn't in a generator.

In ES5, when you call a function, you can expect it to return or throw
eventually.  (Unless you run out of memory, or time, and the whole
script gets terminated.)  With shallow generators, this is still true.
 A 'yield' might never return control, but function calls are ok.  But
with generators+lambdas, almost any function call *anywhere* in the
program might never return or throw.  This weakens 'finally', at
least.

-j

# Brendan Eich (16 years ago)

We still have open controversy over lambdas, if I recall correctly.
Separate thread, but I wanted to confirm that there is disagreement on
completion value being the implicit return value, at least. Some
strawman syntax for explicit result value was mooted.

I also seem to recall some folks (Waldemar in particular) thinking
that lambdas are redundant, and also potentially confusing to average
users precisely because of Tennant Correspondence being two-edged in
languages that have statements as well as expressions. The question
for naive readers is: "return" from what? The answer proposed is
always from the nearest enclosing function (ignoring return to label),
but some may stop at the nearest enclosing lambda, and resulting bugs
might hide in plain sight.

On May 14, 2009, at 10:25 AM, Mark S. Miller wrote:

A return in a lambda returns from the immediately containing function activation, so long as that function activation is still active. If the containing function has already returned, then a postponed attempt to return from it again will fail[1], presumably with a thrown error. This applies as well to break, continue, labeled break, labeled continue, and labeled return if we introduce such a thing. What happens to a postponed 'yield'?

Same thing: yield is very much like return, except that the activation
is not deallocated because the generator-iterator g, created by
calling a generator function f that contains yield expresssions, which
was invoked via g.next(), g.send(v), or g.throw(e), holds a strong
reference to that activation for generator function f.

So a lambda that tries to yield from a deactivated function that
encloses it should fail with a big fat exception, same as for return.

Also, does it make as much sense to introduce a labeled yield as it does to introduce a labeled return?

Return to label, aka "escape continuations", are motivated largely by
the expression-oriented lambda proposal with its final expression
evaluating to the return value, for working around the consequent lack
of "early returns".

Note that return to label as proposed works in any labeled statement,
and does not require a lambda to surround the return. It works via TC
in lambdas just as return does. The motivation via lambda having
implicit return of completion value is what connects the return to
label and lambda proposals.

In this light, yield to label is plausible, although the "early/ exceptional return from lambda" motivation is missing in the sense
that you can write yield expressions in larger expressions -- you
can't use return in the middle of an expression. But yield to label
could be just as useful as return to label.

The syntax works when the yield expression forms a complete expression
statement:

 yield : expr;

but it requires parenthesization in the middle operand of the ?:
ternary operator.

Indeed the low-precedence (same as assignment) nature of the yield
unary caused JS1.7 to follow Python in requiring parentheses around
yield expressions in lists: in comma expressions and actual parameter
lists (although we diverged from Python by allowing an unparenthesized
yield expression as the last actual argument in a call or new
expression's parameter list -- Python requires foo((yield bar))
instead of foo(yield bar)).

We still have open controversy over lambdas, if I recall correctly.  
Separate thread, but I wanted to confirm that there is disagreement on  
completion value being the implicit return value, at least. Some  
strawman syntax for explicit result value was mooted.

I also seem to recall some folks (Waldemar in particular) thinking  
that lambdas are redundant, and also potentially confusing to average  
users precisely *because* of Tennant Correspondence being two-edged in  
languages that have statements as well as expressions. The question  
for naive readers is: "return" from what? The answer proposed is  
always from the nearest enclosing function (ignoring return to label),  
but some may stop at the nearest enclosing lambda, and resulting bugs  
might hide in plain sight.

On May 14, 2009, at 10:25 AM, Mark S. Miller wrote:

> A return in a lambda returns from the immediately containing function
> activation, so long as that function activation is still active. If
> the containing function has already returned, then a postponed attempt
> to return from it again will fail[1], presumably with a thrown error.
> This applies as well to break, continue, labeled break, labeled
> continue, and labeled return if we introduce such a thing. What
> happens to a postponed 'yield'?

Same thing: yield is very much like return, except that the activation  
is not deallocated because the generator-iterator g, created by  
calling a generator function f that contains yield expresssions, which  
was invoked via g.next(), g.send(v), or g.throw(e), holds a strong  
reference to that activation for generator function f.

So a lambda that tries to yield from a deactivated function that  
encloses it should fail with a big fat exception, same as for return.

> Also, does it make as much sense to
> introduce a labeled yield as it does to introduce a labeled return?

Return to label, aka "escape continuations", are motivated largely by  
the expression-oriented lambda proposal with its final expression  
evaluating to the return value, for working around the consequent lack  
of "early returns".

Note that return to label as proposed works in any labeled statement,  
and does not require a lambda to surround the return. It works via TC  
in lambdas just as return does. The motivation via lambda having  
implicit return of completion value is what connects the return to  
label and lambda proposals.

In this light, yield to label is plausible, although the "early/ 
exceptional return from lambda" motivation is missing in the sense  
that you can write yield expressions in larger expressions -- you  
can't use return in the middle of an expression. But yield to label  
could be just as useful as return to label.

The syntax works when the yield expression forms a complete expression  
statement:

     yield : expr;

but it requires parenthesization in the middle operand of the ?:  
ternary operator.

Indeed the low-precedence (same as assignment) nature of the yield  
unary caused JS1.7 to follow Python in requiring parentheses around  
yield expressions in lists: in comma expressions and actual parameter  
lists (although we diverged from Python by allowing an unparenthesized  
yield expression as the last actual argument in a call or new  
expression's parameter list -- Python requires foo((yield bar))  
instead of foo(yield bar)).

/be

# Brendan Eich (16 years ago)

On May 14, 2009, at 12:24 PM, Jason Orendorff wrote:

A lambda can escape and be called from other code. So both implementation approaches must be modified:

The CPS transformation would have to be applied to almost all functions, even functions not lexically inside the generator-function. This is a major change.

The stack-snapshot approach would have to store (a reference to) a range of stack frames, which may include frames for functions not lexically inside the generator-function. This might be a minor change.

So much for duality. :-|

It's overrated :-P.

But this may be a problem with lambdas as proposed, not generators per
se.

When a lambda yields, the lines of code corresponding to the topmost captured frame (or, the beginning of the delimited continuation) and the bottommost (the tail end of the delimited continuation) are both lexically inside the generator-function. But there may be other functions on the stack, in between. You can't always statically tell which ones. This means that generator semantics affect the integrity of code that isn't in a generator.

Right. We crossed the finally-may-not-run Rubicon along with Python
(with a minor improvement, avoiding GeneratorExit), already, for
finally clauses in the generator function itself. But this extends the
finally integrity degradation outside of the lexical scope of the
generator function. Good point.

In ES5, when you call a function, you can expect it to return or throw eventually. (Unless you run out of memory, or time, and the whole script gets terminated.) With shallow generators, this is still true. A 'yield' might never return control, but function calls are ok. But with generators+lambdas, almost any function call anywhere in the program might never return or throw. This weakens 'finally', at least.

To make this clear with an example (thanks to Jason for some IRC
interaction):

I think finally is the only issue, since how else can you tell that
foo didn't see a return or exception from the callback?

It's true that this finally-may-not-run wrinkle is confined to
generator functions in JS1.7, i.e., without lambda.

But I think any function that invokes an arbitrary other function in a
try has to fear that its finally might not run. There could be a hard
stop due to iloop prevention, or an uncatchable security fail-stop
error.

FWIW I'm cool on lambda and return to label atm.

On May 14, 2009, at 12:24 PM, Jason Orendorff wrote:

> 1. A lambda can escape and be called from other code.  So both
> implementation approaches must be modified:
>
> * The CPS transformation would have to be applied to almost all
> functions, even functions not lexically inside the generator-function.
> This is a major change.
>
> * The stack-snapshot approach would have to store (a reference to) a
> range of stack frames, which may include frames for functions not
> lexically inside the generator-function. This might be a minor change.
>
> So much for duality. :-|

It's overrated :-P.

But this may be a problem with lambdas as proposed, not generators per  
se.

> 3. When a lambda yields, the lines of code corresponding to the
> topmost captured frame (or, the beginning of the delimited
> continuation) and the bottommost (the tail end of the delimited
> continuation) are both lexically inside the generator-function. But
> there may be other functions on the stack, in between. You can't
> always statically tell which ones.  This means that generator
> semantics affect the integrity of code that isn't in a generator.

Right. We crossed the finally-may-not-run Rubicon along with Python  
(with a minor improvement, avoiding GeneratorExit), already, for  
finally clauses in the generator function itself. But this extends the  
finally integrity degradation outside of the lexical scope of the  
generator function. Good point.

> In ES5, when you call a function, you can expect it to return or throw
> eventually.  (Unless you run out of memory, or time, and the whole
> script gets terminated.)  With shallow generators, this is still true.
> A 'yield' might never return control, but function calls are ok.  But
> with generators+lambdas, almost any function call *anywhere* in the
> program might never return or throw.  This weakens 'finally', at
> least.

To make this clear with an example (thanks to Jason for some IRC  
interaction):

function gen(arg) {
     foo((lambda (x) yield x), arg);
}
function foo(callback, arg) {
     try {
         callback(arg);
     } finally {
         alert("I'm ok!");
     }
}
g = gen(42);
print(g.next()); // tell the user the meaning of life, etc.
g = null;
gc();

I think finally is the only issue, since how else can you tell that  
foo didn't see a return or exception from the callback?

It's true that this finally-may-not-run wrinkle is confined to  
generator functions in JS1.7, i.e., without lambda.

But I think any function that invokes an arbitrary other function in a  
try has to fear that its finally might not run. There could be a hard  
stop due to iloop prevention, or an uncatchable security fail-stop  
error.

FWIW I'm cool on lambda and return to label atm.

/be

# Waldemar Horwat (16 years ago)

This whole thing is another nail in the coffin of generators. Generators are the root of the troublesome longjmp's which don't play well with others.

Waldemar

This whole thing is another nail in the coffin of generators.  Generators are the root of the troublesome longjmp's which don't play well with others.

    Waldemar


Brendan Eich wrote:
> To make this clear with an example (thanks to Jason for some IRC 
> interaction):
> 
> function gen(arg) {
>     foo((lambda (x) yield x), arg);
> }
> function foo(callback, arg) {
>     try {
>         callback(arg);
>     } finally {
>         alert("I'm ok!");
>     }
> }
> g = gen(42);
> print(g.next()); // tell the user the meaning of life, etc.
> g = null;
> gc();
> 
> I think finally is the only issue, since how else can you tell that foo 
> didn't see a return or exception from the callback?
> 
> It's true that this finally-may-not-run wrinkle is confined to generator 
> functions in JS1.7, i.e., without lambda.
> 
> But I think any function that invokes an arbitrary other function in a 
> try has to fear that its finally might not run. There could be a hard 
> stop due to iloop prevention, or an uncatchable security fail-stop error.
> 
> FWIW I'm cool on lambda and return to label atm.
> 
> /be

# Brendan Eich (16 years ago)

On May 14, 2009, at 1:47 PM, Waldemar Horwat wrote:

This whole thing is another nail in the coffin of generators.
Generators are the root of the troublesome longjmp's which don't
play well with others.

Are you talking about generators, or lambdas?

Generators have no longjmps at all. The only issue here is due to the
lambda following Tennent's Correspondence Principle.

On May 14, 2009, at 1:47 PM, Waldemar Horwat wrote:

> This whole thing is another nail in the coffin of generators.   
> Generators are the root of the troublesome longjmp's which don't  
> play well with others.

Are you talking about generators, or lambdas?

Generators have no longjmps at all. The only issue here is due to the  
lambda following Tennent's Correspondence Principle.

/be

>
>
>   Waldemar
>
>
> Brendan Eich wrote:
>> To make this clear with an example (thanks to Jason for some IRC  
>> interaction):
>> function gen(arg) {
>>    foo((lambda (x) yield x), arg);
>> }
>> function foo(callback, arg) {
>>    try {
>>        callback(arg);
>>    } finally {
>>        alert("I'm ok!");
>>    }
>> }
>> g = gen(42);
>> print(g.next()); // tell the user the meaning of life, etc.
>> g = null;
>> gc();
>> I think finally is the only issue, since how else can you tell that  
>> foo didn't see a return or exception from the callback?
>> It's true that this finally-may-not-run wrinkle is confined to  
>> generator functions in JS1.7, i.e., without lambda.
>> But I think any function that invokes an arbitrary other function  
>> in a try has to fear that its finally might not run. There could be  
>> a hard stop due to iloop prevention, or an uncatchable security  
>> fail-stop error.
>> FWIW I'm cool on lambda and return to label atm.
>> /be

# Mark S. Miller (16 years ago)

On Thu, May 14, 2009 at 1:22 PM, Brendan Eich <brendan at mozilla.com> wrote:

On May 14, 2009, at 12:24 PM, Jason Orendorff wrote:

When a lambda yields, [...] there may be other functions on the stack, in between. You can't always statically tell which ones. This means that generator semantics affect the integrity of code that isn't in a generator.

[...] this extends the finally integrity degradation outside of the lexical scope of the generator function. Good point.

[...] with generators+lambdas, almost any function call anywhere in the program might never return or throw. This weakens 'finally', at least. [...]

function gen(arg) { foo((lambda (x) yield x), arg); } function foo(callback, arg) { try { callback(arg); } finally { alert("I'm ok!"); } } g = gen(42); print(g.next()); // tell the user the meaning of life, etc. g = null; gc();

Thanks all, this has been very clarifying. You both have put your finger on what was nagging at me and explained it clearly.

It seems that either lambda or generators by themselves may be ok, but together they make a fatal combination. Whichever we might eventually decide to add to Harmony, we probably forever preclude the other. I do not yet have an opinion on which.

On Thu, May 14, 2009 at 1:22 PM, Brendan Eich <brendan at mozilla.com> wrote:
> On May 14, 2009, at 12:24 PM, Jason Orendorff wrote:
>> 3. When a lambda yields, [...]
>> there may be other functions on the stack, in between. You can't
>> always statically tell which ones.  This means that generator
>> semantics affect the integrity of code that isn't in a generator.
>
> [...] this extends the finally integrity
> degradation outside of the lexical scope of the generator function. Good
> point.
>
>> [...] with generators+lambdas, almost any function call *anywhere* in the
>> program might never return or throw.  This weakens 'finally', at
>> least.
> [...]

> function gen(arg) {
>    foo((lambda (x) yield x), arg);
> }
> function foo(callback, arg) {
>    try {
>        callback(arg);
>    } finally {
>        alert("I'm ok!");
>    }
> }
> g = gen(42);
> print(g.next()); // tell the user the meaning of life, etc.
> g = null;
> gc();


Thanks all, this has been very clarifying. You both have put your
finger on what was nagging at me and explained it clearly.

It seems that either lambda or generators by themselves may be ok, but
together they make a fatal combination. Whichever we might eventually
decide to add to Harmony, we probably forever preclude the other. I do
not yet have an opinion on which.

-- 

    Cheers,
    --MarkM

# Brendan Eich (16 years ago)

On May 14, 2009, at 2:10 PM, Mark S. Miller wrote:

It seems that either lambda or generators by themselves may be ok, but together they make a fatal combination. Whichever we might eventually decide to add to Harmony, we probably forever preclude the other. I do not yet have an opinion on which.

Just to be extra clear (for my benefit, at least ;-), is the problem
the finally-may-not-run issue?

If so, did you have a different way of reasoning about the reasons
today why finally might not run that I mentioned (iloop detection or
other hard stop)?

On May 14, 2009, at 2:10 PM, Mark S. Miller wrote:

> It seems that either lambda or generators by themselves may be ok, but
> together they make a fatal combination. Whichever we might eventually
> decide to add to Harmony, we probably forever preclude the other. I do
> not yet have an opinion on which.

Just to be extra clear (for my benefit, at least ;-), is the problem  
the finally-may-not-run issue?

If so, did you have a different way of reasoning about the reasons  
today why finally might not run that I mentioned (iloop detection or  
other hard stop)?

/be

# Mark S. Miller (16 years ago)

On Thu, May 14, 2009 at 2:38 PM, Brendan Eich <brendan at mozilla.com> wrote:

On May 14, 2009, at 2:10 PM, Mark S. Miller wrote:

It seems that either lambda or generators by themselves may be ok, but together they make a fatal combination. Whichever we might eventually decide to add to Harmony, we probably forever preclude the other. I do not yet have an opinion on which.

Just to be extra clear (for my benefit, at least ;-), is the problem the finally-may-not-run issue?

Yes.

If so, did you have a different way of reasoning about the reasons today why finally might not run that I mentioned (iloop detection or other hard stop)?

Those hard stops kill all further activity within that event loop. Once a universe has been destroyed, no further bad things can happen in that universe.

Infinite loops don't kill their universe, so this case is similar but different. As the halting problem teaches us, an infinite loop is generally indistinguishable from a loop that hasn't terminated yet. Since control flow has not yet escaped the loop, it hasn't yet bypassed the finally, and so no invariants have yet been violated.

On Thu, May 14, 2009 at 2:38 PM, Brendan Eich <brendan at mozilla.com> wrote:
> On May 14, 2009, at 2:10 PM, Mark S. Miller wrote:
>
>> It seems that either lambda or generators by themselves may be ok, but
>> together they make a fatal combination. Whichever we might eventually
>> decide to add to Harmony, we probably forever preclude the other. I do
>> not yet have an opinion on which.
>
> Just to be extra clear (for my benefit, at least ;-), is the problem the
> finally-may-not-run issue?

Yes.

> If so, did you have a different way of reasoning about the reasons today why
> finally might not run that I mentioned (iloop detection or other hard stop)?

Those hard stops kill all further activity within that event loop.
Once a universe has been destroyed, no further bad things can happen
in that universe.

Infinite loops don't kill their universe, so this case is similar but
different. As the halting problem teaches us, an infinite loop is
generally indistinguishable from a loop that hasn't terminated yet.
Since control flow has not yet escaped the loop, it hasn't yet
bypassed the finally, and so no invariants have yet been violated.

-- 
    Cheers,
    --MarkM

# Brendan Eich (16 years ago)

On May 14, 2009, at 2:50 PM, Mark S. Miller wrote:

If so, did you have a different way of reasoning about the reasons
today why finally might not run that I mentioned (iloop detection or other
hard stop)?

Those hard stops kill all further activity within that event loop. Once a universe has been destroyed, no further bad things can happen in that universe.

There's always the next universe (new event starts another control
flow). Life goes on, in the JS serial multiverse, and those finally
clauses failed to run even though control abruptly left the lambda
under the hypothesis.

Infinite loops don't kill their universe, so this case is similar but different. As the halting problem teaches us, an infinite loop is generally indistinguishable from a loop that hasn't terminated yet. Since control flow has not yet escaped the loop, it hasn't yet bypassed the finally, and so no invariants have yet been violated.

True enough, but see above.

This isn't entirely academic, since information leaks include
termination channels.

On May 14, 2009, at 2:50 PM, Mark S. Miller wrote:

>> If so, did you have a different way of reasoning about the reasons  
>> today why
>> finally might not run that I mentioned (iloop detection or other  
>> hard stop)?
>
> Those hard stops kill all further activity within that event loop.
> Once a universe has been destroyed, no further bad things can happen
> in that universe.

There's always the next universe (new event starts another control  
flow). Life goes on, in the JS serial multiverse, and those finally  
clauses failed to run even though control abruptly left the lambda  
under the hypothesis.

> Infinite loops don't kill their universe, so this case is similar but
> different. As the halting problem teaches us, an infinite loop is
> generally indistinguishable from a loop that hasn't terminated yet.
> Since control flow has not yet escaped the loop, it hasn't yet
> bypassed the finally, and so no invariants have yet been violated.

True enough, but see above.

This isn't entirely academic, since information leaks include  
termination channels.

/be

# Brendan Eich (16 years ago)

On May 14, 2009, at 3:42 PM, Brendan Eich wrote:

Those hard stops kill all further activity within that event loop. Once a universe has been destroyed, no further bad things can happen in that universe.

There's always the next universe (new event starts another control
flow). Life goes on, in the JS serial multiverse, and those finally
clauses failed to run even though control abruptly left the lambda
under the hypothesis.

Infinite loops don't kill their universe, so this case is similar but different. As the halting problem teaches us, an infinite loop is generally indistinguishable from a loop that hasn't terminated yet. Since control flow has not yet escaped the loop, it hasn't yet bypassed the finally, and so no invariants have yet been violated.

True enough, but see above.

This isn't entirely academic, since information leaks include
termination channels.

And that previous universe ended, detectably. It's not as if the loop
is still running in the lambda when the next event-flow little bang
blooms into a universe, since there is a big fat shared heap (no
quantum uncertainty obscuring inter-verse sharing!) accessible by
which to detect mutations. If the iloop were incrementing a counter,
it is guaranteed that the counter stopped before the new event-flow
started.

No racing scripts; run-to-completion is the observable execution model
(ignoring web workers, which are shared nothing in JS heap terms but
may share effects in DOM storage, etc).

On May 14, 2009, at 3:42 PM, Brendan Eich wrote:

>> Those hard stops kill all further activity within that event loop.
>> Once a universe has been destroyed, no further bad things can happen
>> in that universe.
>
> There's always the next universe (new event starts another control  
> flow). Life goes on, in the JS serial multiverse, and those finally  
> clauses failed to run even though control abruptly left the lambda  
> under the hypothesis.
>
>
>> Infinite loops don't kill their universe, so this case is similar but
>> different. As the halting problem teaches us, an infinite loop is
>> generally indistinguishable from a loop that hasn't terminated yet.
>> Since control flow has not yet escaped the loop, it hasn't yet
>> bypassed the finally, and so no invariants have yet been violated.
>
> True enough, but see above.
>
> This isn't entirely academic, since information leaks include  
> termination channels.

And that previous universe ended, detectably. It's not as if the loop  
is still running in the lambda when the next event-flow little bang  
blooms into a universe, since there is a big fat shared heap (no  
quantum uncertainty obscuring inter-verse sharing!) accessible by  
which to detect mutations. If the iloop were incrementing a counter,  
it is guaranteed that the counter stopped before the new event-flow  
started.

No racing scripts; run-to-completion is the observable execution model  
(ignoring web workers, which are shared nothing in JS heap terms but  
may share effects in DOM storage, etc).

/be

# Igor Bukanov (16 years ago)

2009/5/14 Brendan Eich <brendan at mozilla.com>:

function gen(arg) { foo((lambda (x) yield x), arg); } function foo(callback, arg) { try { callback(arg); } finally { alert("I'm ok!"); } } g = gen(42); print(g.next()); // tell the user the meaning of life, etc. g = null; gc();

I think finally is the only issue, since how else can you tell that foo didn't see a return or exception from the callback?

It's true that this finally-may-not-run wrinkle is confined to generator functions in JS1.7, i.e., without lambda.

lambda can be combined with the generators if the yield semantic would be altered to execute any finally block it resides in in the same way the return does. Then in the above example the alert would be executed when the lambda does yield.

This is how continuations in Rhino works with any continuation long jump respecting lexical finally blocks.

Igor

2009/5/14 Brendan Eich <brendan at mozilla.com>:
> function gen(arg) {
>    foo((lambda (x) yield x), arg);
> }
> function foo(callback, arg) {
>    try {
>        callback(arg);
>    } finally {
>        alert("I'm ok!");
>    }
> }
> g = gen(42);
> print(g.next()); // tell the user the meaning of life, etc.
> g = null;
> gc();
>
> I think finally is the only issue, since how else can you tell that foo
> didn't see a return or exception from the callback?
>
> It's true that this finally-may-not-run wrinkle is confined to generator
> functions in JS1.7, i.e., without lambda.

lambda can be combined with the generators if the yield semantic would
be altered to execute any finally block it resides in in the same way
the return does. Then in the above example the alert would be executed
when the lambda does yield.

This is how continuations in Rhino works with any continuation long
jump respecting lexical finally blocks.

Igor

# Waldemar Horwat (16 years ago)

Brendan Eich wrote:

On May 14, 2009, at 1:47 PM, Waldemar Horwat wrote:

This whole thing is another nail in the coffin of generators.
Generators are the root of the troublesome longjmp's which don't play well with others.

Are you talking about generators, or lambdas?

Generators. They cause control flow to jump around between two independent functions.

THe same problem occurs without lambdas:

function gen(arg) { try { yield 1; yield 2; } finally { alert("Finally called up to three times?!"); } }

That finally will be called either 1, 2, or 3 times depending on what the caller does, which violates the principle that finally is called exactly once, no matter how you leave a scope.

Waldemar

Brendan Eich wrote:
> On May 14, 2009, at 1:47 PM, Waldemar Horwat wrote:
> 
>> This whole thing is another nail in the coffin of generators.  
>> Generators are the root of the troublesome longjmp's which don't play 
>> well with others.
> 
> Are you talking about generators, or lambdas?

Generators.  They cause control flow to jump around between two independent functions.

THe same problem occurs without lambdas:

function gen(arg) {
  try {
    yield 1;
    yield 2;
  } finally {
    alert("Finally called up to three times?!");
  }
}

That finally will be called either 1, 2, or 3 times depending on what the caller does, which violates the principle that finally is called exactly once, no matter how you leave a scope.

    Waldemar

# Brendan Eich (16 years ago)

On May 14, 2009, at 6:04 PM, Waldemar Horwat wrote:

Brendan Eich wrote:

On May 14, 2009, at 1:47 PM, Waldemar Horwat wrote:

This whole thing is another nail in the coffin of generators.
Generators are the root of the troublesome longjmp's which don't
play well with others. Are you talking about generators, or lambdas?

Generators. They cause control flow to jump around between two
independent functions.

THe same problem occurs without lambdas:

function gen(arg) { try { yield 1; yield 2; } finally { alert("Finally called up to three times?!"); } }

That finally will be called either 1, 2, or 3 times depending on
what the caller does, which violates the principle that finally is
called exactly once, no matter how you leave a scope.

Did you actually test this? Firefox 2 and up work the same.

js> g = gen() [object Generator] js> g.next()

1 js> g.next()

2 js> g.next()

typein:6: ReferenceError: alert is not defined js> alert=print

function print() { [native code] } js> g = gen(42) [object Generator] js> g.next()

1 js> g.next()

2 js> g.next()

Finally called up to three times?! uncaught exception: [object StopIteration]

Exactly one finally.

On May 14, 2009, at 6:04 PM, Waldemar Horwat wrote:

> Brendan Eich wrote:
>> On May 14, 2009, at 1:47 PM, Waldemar Horwat wrote:
>>> This whole thing is another nail in the coffin of generators.   
>>> Generators are the root of the troublesome longjmp's which don't  
>>> play well with others.
>> Are you talking about generators, or lambdas?
>
> Generators.  They cause control flow to jump around between two  
> independent functions.
>
> THe same problem occurs without lambdas:
>
> function gen(arg) {
> try {
>   yield 1;
>   yield 2;
> } finally {
>   alert("Finally called up to three times?!");
> }
> }
>
> That finally will be called either 1, 2, or 3 times depending on  
> what the caller does, which violates the principle that finally is  
> called exactly once, no matter how you leave a scope.

Did you actually test this? Firefox 2 and up work the same.

js> g = gen()
[object Generator]
js> g.next()
1
js> g.next()
2
js> g.next()
typein:6: ReferenceError: alert is not defined
js> alert=print
function print() {
     [native code]
}
js> g = gen(42)
[object Generator]
js> g.next()
1
js> g.next()
2
js> g.next()
Finally called up to three times?!
uncaught exception: [object StopIteration]

Exactly one finally.

/be

# kevin curtis (16 years ago)

If a function is used instead of a lambda, do the the same - syntactic or semantic - problems arise?

foo((lambda (x) yield x), arg);

to:

foo((function (x) yield x), arg); // js1.7 expression closure syntax OR foo(function (x) { yield x}, arg);

If a function is used instead of a lambda, do the the same - syntactic
or semantic - problems arise?

 foo((lambda (x) yield x), arg);

to:

 foo((function (x) yield x), arg);  // js1.7 expression closure syntax
OR
 foo(function (x) { yield x}, arg);

On Fri, May 15, 2009 at 6:11 AM, Brendan Eich <brendan at mozilla.com> wrote:
> On May 14, 2009, at 6:04 PM, Waldemar Horwat wrote:
>
>> Brendan Eich wrote:
>>>
>>> On May 14, 2009, at 1:47 PM, Waldemar Horwat wrote:
>>>>
>>>> This whole thing is another nail in the coffin of generators.
>>>>  Generators are the root of the troublesome longjmp's which don't play well
>>>> with others.
>>>
>>> Are you talking about generators, or lambdas?
>>
>> Generators.  They cause control flow to jump around between two
>> independent functions.
>>
>> THe same problem occurs without lambdas:
>>
>> function gen(arg) {
>> try {
>>  yield 1;
>>  yield 2;
>> } finally {
>>  alert("Finally called up to three times?!");
>> }
>> }
>>
>> That finally will be called either 1, 2, or 3 times depending on what the
>> caller does, which violates the principle that finally is called exactly
>> once, no matter how you leave a scope.
>
> Did you actually test this? Firefox 2 and up work the same.
>
> js> g = gen()
> [object Generator]
> js> g.next()
> 1
> js> g.next()
> 2
> js> g.next()
> typein:6: ReferenceError: alert is not defined
> js> alert=print
> function print() {
>    [native code]
> }
> js> g = gen(42)
> [object Generator]
> js> g.next()
> 1
> js> g.next()
> 2
> js> g.next()
> Finally called up to three times?!
> uncaught exception: [object StopIteration]
>
> Exactly one finally.
>
> /be
>
> _______________________________________________
> es-discuss mailing list
> es-discuss at mozilla.org
> https://mail.mozilla.org/listinfo/es-discuss
>

# Brendan Eich (16 years ago)

On May 15, 2009, at 2:34 AM, kevin curtis wrote:

If a function is used instead of a lambda, do the the same - syntactic or semantic - problems arise?

No, because yield in a function makes it a generator, whereas in the
lambda proposal extended to treat yield as return/break/continue are
treated (per Tennent's Correspondence Principle), yield does not make
the lamda some kind of generator-lambda -- instead it yields the
nearest enclosing function (if active; else throw).

foo((lambda (x) yield x), arg);

to:

foo((function (x) yield x), arg); // js1.7 expression closure syntax OR foo(function (x) { yield x}, arg);

That passes a generator function to foo. If foo calls it, foo gets a
generator-iterator, which if used in a for-in construct, or explicitly
iterated via .next/send/throw, yields x once then throws StopIteration.

On May 15, 2009, at 2:34 AM, kevin curtis wrote:

> If a function is used instead of a lambda, do the the same - syntactic
> or semantic - problems arise?

No, because yield in a function makes it a generator, whereas in the  
lambda proposal extended to treat yield as return/break/continue are  
treated (per Tennent's Correspondence Principle), yield does not make  
the lamda some kind of generator-lambda -- instead it yields the  
nearest enclosing function (if active; else throw).

> foo((lambda (x) yield x), arg);
>
> to:
>
> foo((function (x) yield x), arg);  // js1.7 expression closure syntax
> OR
> foo(function (x) { yield x}, arg);

That passes a generator function to foo. If foo calls it, foo gets a  
generator-iterator, which if used in a for-in construct, or explicitly  
iterated via .next/send/throw, yields x once then throws StopIteration.

/be

# Waldemar Horwat (16 years ago)

Brendan Eich wrote:

On May 14, 2009, at 6:04 PM, Waldemar Horwat wrote:

Brendan Eich wrote:

On May 14, 2009, at 1:47 PM, Waldemar Horwat wrote:

This whole thing is another nail in the coffin of generators.
Generators are the root of the troublesome longjmp's which don't play well with others. Are you talking about generators, or lambdas?

Generators. They cause control flow to jump around between two independent functions.

THe same problem occurs without lambdas:

function gen(arg) { try { yield 1; yield 2; } finally { alert("Finally called up to three times?!"); } }

That finally will be called either 1, 2, or 3 times depending on what the caller does, which violates the principle that finally is called exactly once, no matter how you leave a scope.

Did you actually test this? Firefox 2 and up work the same.

js> g = gen() [object Generator] js> g.next() 1 js> g.next() 2 js> g.next() typein:6: ReferenceError: alert is not defined js> alert=print function print() { [native code] } js> g = gen(42) [object Generator] js> g.next() 1 js> g.next() 2 js> g.next() Finally called up to three times?! uncaught exception: [object StopIteration]

Exactly one finally.

That behavior is wrong in the case that g gets forgotten after calling next just once. You're silently exiting through a finally clause without calling it.

Waldemar

Brendan Eich wrote:
> On May 14, 2009, at 6:04 PM, Waldemar Horwat wrote:
> 
>> Brendan Eich wrote:
>>> On May 14, 2009, at 1:47 PM, Waldemar Horwat wrote:
>>>> This whole thing is another nail in the coffin of generators.  
>>>> Generators are the root of the troublesome longjmp's which don't 
>>>> play well with others.
>>> Are you talking about generators, or lambdas?
>>
>> Generators.  They cause control flow to jump around between two 
>> independent functions.
>>
>> THe same problem occurs without lambdas:
>>
>> function gen(arg) {
>> try {
>>   yield 1;
>>   yield 2;
>> } finally {
>>   alert("Finally called up to three times?!");
>> }
>> }
>>
>> That finally will be called either 1, 2, or 3 times depending on what 
>> the caller does, which violates the principle that finally is called 
>> exactly once, no matter how you leave a scope.
> 
> Did you actually test this? Firefox 2 and up work the same.
> 
> js> g = gen()
> [object Generator]
> js> g.next()
> 1
> js> g.next()
> 2
> js> g.next()
> typein:6: ReferenceError: alert is not defined
> js> alert=print
> function print() {
>     [native code]
> }
> js> g = gen(42)
> [object Generator]
> js> g.next()
> 1
> js> g.next()
> 2
> js> g.next()
> Finally called up to three times?!
> uncaught exception: [object StopIteration]
> 
> Exactly one finally.

That behavior is wrong in the case that g gets forgotten after calling next just once.  You're silently exiting through a finally clause without calling it.

    Waldemar

# Brendan Eich (16 years ago)

On May 15, 2009, at 1:47 PM, Waldemar Horwat wrote:

Did you actually test this? Firefox 2 and up work the same. js> g = gen() [object Generator] js> g.next() 1 js> g.next() 2 js> g.next() typein:6: ReferenceError: alert is not defined js> alert=print function print() { [native code] } js> g = gen(42) [object Generator] js> g.next() 1 js> g.next() 2 js> g.next() Finally called up to three times?! uncaught exception: [object StopIteration] Exactly one finally.

That behavior is wrong in the case that g gets forgotten after
calling next just once.

Progress! You're not alleging longjmps or multiple finallys ;-).

You're silently exiting through a finally clause without calling it.

You must mean where I hit that ReferenceError due to alert not being
defined, then reassigned to g without closing it or calling g.next()
and hitting the StopIteration.

(I //totally// so meant to do that in order to provoke this
discussion. :-P)

The alternative, which we prototyped and rejected, but which CPython
implements, is to automate calling the generator's close method from
the garbage collector (close "throws a return" at the generator,
causing finallys to run). But this is a burden on either or both of
content authors and implementators, since it makes finally execution
unpredictably tardy, requires certain kinds of GC, exposes two-phase
finalization, etc.

In browsers, automating close from the GC opens up trivial denial of
service attacks that could prevent pages from being unloaded, which
browsers can and must defeat in ways that again mean "finally does not
always run."

The fact is, in browsers at least, finally does not always run --
ignoring generators. There are good DOS-prevention reasons for
skipping it.

If you want automated close calling in the absence of DOS suppression,
use for-in. This was suggested by Chris Hansen here:

esdiscuss/2007-January/003743

That whole "Immediate closing of iterators" thread is worth a read if
you missed it.

On May 15, 2009, at 1:47 PM, Waldemar Horwat wrote:

>> Did you actually test this? Firefox 2 and up work the same.
>> js> g = gen()
>> [object Generator]
>> js> g.next()
>> 1
>> js> g.next()
>> 2
>> js> g.next()
>> typein:6: ReferenceError: alert is not defined
>> js> alert=print
>> function print() {
>>    [native code]
>> }
>> js> g = gen(42)
>> [object Generator]
>> js> g.next()
>> 1
>> js> g.next()
>> 2
>> js> g.next()
>> Finally called up to three times?!
>> uncaught exception: [object StopIteration]
>> Exactly one finally.
>
> That behavior is wrong in the case that g gets forgotten after  
> calling next just once.

Progress! You're not alleging longjmps or multiple finallys ;-).

> You're silently exiting through a finally clause without calling it.

You must mean where I hit that ReferenceError due to alert not being  
defined, then reassigned to g without closing it or calling g.next()  
and hitting the StopIteration.

(I //totally// so meant to do that in order to provoke this  
discussion. :-P)

The alternative, which we prototyped and rejected, but which CPython  
implements, is to automate calling the generator's close method from  
the garbage collector (close "throws a return" at the generator,  
causing finallys to run). But this is a burden on either or both of  
content authors and implementators, since it makes finally execution  
unpredictably tardy, requires certain kinds of GC, exposes two-phase  
finalization, etc.

In browsers, automating close from the GC opens up trivial denial of  
service attacks that could prevent pages from being unloaded, which  
browsers can and must defeat in ways that again mean "finally does not  
always run."

The fact is, in browsers at least, finally does not always run --  
ignoring generators. There are good DOS-prevention reasons for  
skipping it.

If you want automated close calling in the absence of DOS suppression,  
use for-in. This was suggested by Chris Hansen here:

https://mail.mozilla.org/pipermail/es-discuss/2007-January/003743.html

That whole "Immediate closing of iterators" thread is worth a read if  
you missed it.

/be

# Brendan Eich (16 years ago)

On May 14, 2009, at 2:10 PM, Mark S. Miller wrote:

On Thu, May 14, 2009 at 1:22 PM, Brendan Eich <brendan at mozilla.com>
wrote:

On May 14, 2009, at 12:24 PM, Jason Orendorff wrote:

When a lambda yields, [...] there may be other functions on the stack, in between. You can't always statically tell which ones. This means that generator semantics affect the integrity of code that isn't in a generator.

[...] this extends the finally integrity degradation outside of the lexical scope of the generator function.
Good point.

[...] with generators+lambdas, almost any function call anywhere
in the program might never return or throw. This weakens 'finally', at least. [...]

function gen(arg) { foo((lambda (x) yield x), arg); } function foo(callback, arg) { try { callback(arg); } finally { alert("I'm ok!"); } } g = gen(42); print(g.next()); // tell the user the meaning of life, etc. g = null; gc();

Thanks all, this has been very clarifying. You both have put your finger on what was nagging at me and explained it clearly.

I think we missed an alternative that comports with Tennent's Oversold
Correspondence Principle, and composes. Thanks to Dave Herman for
pointing it out.

function gen(x) { foo( lambda (x) (yield x*x) ); }

need not yield from gen if the lambda is called from foo or another
function -- it can throw the same error it would throw if the lambda
escaped upward/heapward and was called after gen had returned. There's
no requirement that yield not throw in any case where the lambda is
not applied in the context of gen.

I've seen magical thinking applied to both TCP and LSP, to the
detriment of the concrete application of these principles. TCP does
not mean yield plus lambdas must require delimited continuation.

The big advantage of yield other than the utility (good not great,
less than some folks want, and hard to compose), and the Python
brainprint re-use (also good IMHO, but this could degrade over time),
is the ease of implementation. Ecma TC39 is going to have a hard time
standardizing deeper continuations. Never mind the challenges facing
lambda.

On May 14, 2009, at 2:10 PM, Mark S. Miller wrote:

> On Thu, May 14, 2009 at 1:22 PM, Brendan Eich <brendan at mozilla.com>  
> wrote:
>> On May 14, 2009, at 12:24 PM, Jason Orendorff wrote:
>>> 3. When a lambda yields, [...]
>>> there may be other functions on the stack, in between. You can't
>>> always statically tell which ones.  This means that generator
>>> semantics affect the integrity of code that isn't in a generator.
>>
>> [...] this extends the finally integrity
>> degradation outside of the lexical scope of the generator function.  
>> Good
>> point.
>>
>>> [...] with generators+lambdas, almost any function call *anywhere*  
>>> in the
>>> program might never return or throw.  This weakens 'finally', at
>>> least.
>> [...]
>
>> function gen(arg) {
>>   foo((lambda (x) yield x), arg);
>> }
>> function foo(callback, arg) {
>>   try {
>>       callback(arg);
>>   } finally {
>>       alert("I'm ok!");
>>   }
>> }
>> g = gen(42);
>> print(g.next()); // tell the user the meaning of life, etc.
>> g = null;
>> gc();
>
>
> Thanks all, this has been very clarifying. You both have put your
> finger on what was nagging at me and explained it clearly.

I think we missed an alternative that comports with Tennent's Oversold  
Correspondence Principle, *and* composes. Thanks to Dave Herman for  
pointing it out.

function gen(x) {
   foo( lambda (x) (yield x*x) );
}

need not yield from gen if the lambda is called from foo or another  
function -- it can throw the same error it would throw if the lambda  
escaped upward/heapward and was called after gen had returned. There's  
no requirement that yield not throw in any case where the lambda is  
not applied in the context of gen.

I've seen magical thinking applied to both TCP and LSP, to the  
detriment of the concrete application of these principles. TCP does  
not mean yield plus lambdas must require delimited continuation.

The big advantage of yield other than the utility (good not great,  
less than some folks want, and hard to compose), and the Python  
brainprint re-use (also good IMHO, but this could degrade over time),  
is the ease of implementation. Ecma TC39 is going to have a hard time  
standardizing deeper continuations. Never mind the challenges facing  
lambda.

/be

# Jim Blandy (16 years ago)

On 05/28/2009 11:08 AM, Brendan Eich wrote:

On May 14, 2009, at 2:10 PM, Mark S. Miller wrote:

On Thu, May 14, 2009 at 1:22 PM, Brendan Eich <brendan at mozilla.com> wrote:

On May 14, 2009, at 12:24 PM, Jason Orendorff wrote:

When a lambda yields, [...] there may be other functions on the stack, in between. You can't always statically tell which ones. This means that generator semantics affect the integrity of code that isn't in a generator.

[...] this extends the finally integrity degradation outside of the lexical scope of the generator function. Good point.

[...] with generators+lambdas, almost any function call anywhere in the program might never return or throw. This weakens 'finally', at least. [...]

function gen(arg) { foo((lambda (x) yield x), arg); } function foo(callback, arg) { try { callback(arg); } finally { alert("I'm ok!"); } } g = gen(42); print(g.next()); // tell the user the meaning of life, etc. g = null; gc();

Thanks all, this has been very clarifying. You both have put your finger on what was nagging at me and explained it clearly.

I think we missed an alternative that comports with Tennent's Oversold Correspondence Principle, and composes. Thanks to Dave Herman for pointing it out.

function gen(x) { foo( lambda (x) (yield x*x) ); }

need not yield from gen if the lambda is called from foo or another function -- it can throw the same error it would throw if the lambda escaped upward/heapward and was called after gen had returned. There's no requirement that yield not throw in any case where the lambda is not applied in the context of gen. "Not applied in the context of gen" means what, exactly? Called directly from gen? Called only by lambdas enclosed by gen? Called in some gen's dynamic scope?

Would yield need to work normally in this case?

function f(x) { ((lambda (g, h, x) { return g(g, h, x); }) (lambda (g, h, x) { if (x > 0) return g(g, h, x-1); else return h(x); }, lambda (x) { yield 42; }, x)); } f(10);

Nothing there but local lambdas. If lambda isn't transparent in such cases, then its value as something to safely desugar to is pretty weak.
If yield does work, then we're capturing deep stacks. A more plausible example:

function j(x) { (lambda (y) { (lambda (z) { yield z; } (y + " and a dyne")); } (x + ", a poundal")); } print(j("I love you").next());

This is just a desugaring of some nested lets, but we still have yield capturing many frames. If this doesn't work, lambda is really useless.

For what it's worth, speaking as a long-time Scheme fan, I wouldn't add lambda to ES. It seems too similar to function; there will be endless blog posts explaining the differences and motivation, mostly slightly wrong. The best ones will say, "Don't use it; just use function." And lambda introduces an awful lot of subtlety for something whose main claim to utility would be in allowing precise and clear definitions of new control constructs through desugaring. Natural language is bad, but not this bad.

On 05/28/2009 11:08 AM, Brendan Eich wrote:
> On May 14, 2009, at 2:10 PM, Mark S. Miller wrote:
>
>> On Thu, May 14, 2009 at 1:22 PM, Brendan Eich <brendan at mozilla.com> 
>> wrote:
>>> On May 14, 2009, at 12:24 PM, Jason Orendorff wrote:
>>>> 3. When a lambda yields, [...]
>>>> there may be other functions on the stack, in between. You can't
>>>> always statically tell which ones.  This means that generator
>>>> semantics affect the integrity of code that isn't in a generator.
>>>
>>> [...] this extends the finally integrity
>>> degradation outside of the lexical scope of the generator function. 
>>> Good
>>> point.
>>>
>>>> [...] with generators+lambdas, almost any function call *anywhere* 
>>>> in the
>>>> program might never return or throw.  This weakens 'finally', at
>>>> least.
>>> [...]
>>
>>> function gen(arg) {
>>>   foo((lambda (x) yield x), arg);
>>> }
>>> function foo(callback, arg) {
>>>   try {
>>>       callback(arg);
>>>   } finally {
>>>       alert("I'm ok!");
>>>   }
>>> }
>>> g = gen(42);
>>> print(g.next()); // tell the user the meaning of life, etc.
>>> g = null;
>>> gc();
>>
>>
>> Thanks all, this has been very clarifying. You both have put your
>> finger on what was nagging at me and explained it clearly.
>
> I think we missed an alternative that comports with Tennent's Oversold 
> Correspondence Principle, *and* composes. Thanks to Dave Herman for 
> pointing it out.
>
> function gen(x) {
>   foo( lambda (x) (yield x*x) );
> }
>
> need not yield from gen if the lambda is called from foo or another 
> function -- it can throw the same error it would throw if the lambda 
> escaped upward/heapward and was called after gen had returned. There's 
> no requirement that yield not throw in any case where the lambda is 
> not applied in the context of gen.
"Not applied in the context of gen" means what, exactly?  Called 
directly from gen?  Called only by lambdas enclosed by gen?  Called in 
some gen's dynamic scope?

Would yield need to work normally in this case?

function f(x) {
   ((lambda (g, h, x) { return g(g, h, x); })
    (lambda (g, h, x) {
       if (x > 0)
         return g(g, h, x-1);
       else
         return h(x);
     },
     lambda (x) { yield 42; },
     x));
}
f(10);

Nothing there but local lambdas.  If lambda isn't transparent in such 
cases, then its value as something to safely desugar to is pretty weak.  
If yield does work, then we're capturing deep stacks.  A more plausible 
example:

function j(x) {
   (lambda (y) {
      (lambda (z) {
         yield z;
       }
       (y + " and a dyne"));
    }
    (x + ", a poundal"));
}
print(j("I love you").next());

This is just a desugaring of some nested lets, but we still have yield 
capturing many frames.  If this doesn't work, lambda is really useless.

For what it's worth, speaking as a long-time Scheme fan, I wouldn't add 
lambda to ES.  It seems too similar to function; there will be endless 
blog posts explaining the differences and motivation, mostly slightly 
wrong.  The best ones will say, "Don't use it; just use function."  And 
lambda introduces an awful lot of subtlety for something whose main 
claim to utility would be in allowing precise and clear definitions of 
new control constructs through desugaring.  Natural language is bad, but 
not this bad.

# David-Sarah Hopwood (16 years ago)

Brendan Eich wrote:

I think we missed an alternative that comports with Tennent's Oversold Correspondence Principle, and composes. Thanks to Dave Herman for pointing it out.

function gen(x) { foo( lambda (x) (yield x*x) ); }

need not yield from gen if the lambda is called from foo or another function -- it can throw the same error it would throw if the lambda escaped upward/heapward and was called after gen had returned. There's no requirement that yield not throw in any case where the lambda is not applied in the context of gen.

Well, that depends on what lambda is expected to be used for.

If it is expected to be used to implement general user-defined control structures, then this restriction would prevent a yield from appearing in the body of any such structure.

For the use of lambda in built-in expansions, OTOH, this would probably be adequate, assuming the check that the lambda is called from the body of the generator function is applied after expansion.

Brendan Eich wrote:
> I think we missed an alternative that comports with Tennent's Oversold
> Correspondence Principle, *and* composes. Thanks to Dave Herman for
> pointing it out.
> 
> function gen(x) {
>   foo( lambda (x) (yield x*x) );
> }
> 
> need not yield from gen if the lambda is called from foo or another
> function -- it can throw the same error it would throw if the lambda
> escaped upward/heapward and was called after gen had returned. There's
> no requirement that yield not throw in any case where the lambda is not
> applied in the context of gen.

Well, that depends on what lambda is expected to be used for.

If it is expected to be used to implement general user-defined control
structures, then this restriction would prevent a yield from appearing
in the body of any such structure.

For the use of lambda in built-in expansions, OTOH, this would probably
be adequate, assuming the check that the lambda is called from the body
of the generator function is applied *after* expansion.

-- 
David-Sarah Hopwood  ⚥  http://davidsarah.livejournal.com

# Dave Herman (16 years ago)

[Sorry for my absence lately.]

If yield does work, then we're capturing deep stacks. A more
plausible example:

function j(x) { (lambda (y) { (lambda (z) { yield z; } (y + " and a dyne")); } (x + ", a poundal")); } print(j("I love you").next());

This is just a desugaring of some nested lets, but we still have
yield capturing many frames. If this doesn't work, lambda is really
useless.

This issue really comes down to whether lambda' is properly tail calling. If so, then theyield' in this example occurs as if it were
directly within the body of j', because there's no stack beyond the activation frame ofj'.

To be a little more precise: when people refer to "Tennent," the main
property they're talking about is that (modulo syntax) an expression E
is equivalent to the expression (lambda () E)(). Namely, in any
expression context (roughly, a position in a program where you can
place an expression), either expression will result in the same
program behavior.

If we don't want generators to capture deep stacks, then we can make
it a dynamic error for `yield' to occur in a non-empty sub-stack. Then
in order to preserve the property that (lambda () E)() for all E, we'd
have to mandate that (lambda () E)() is really a tail call and
guaranteed not to push any stack.

Alternatively, we could allow generators to capture deep lambda- stacks, but not deep function-stacks. However...

For what it's worth, speaking as a long-time Scheme fan, I wouldn't
add lambda to ES. It seems too similar to function; there will be
endless blog posts explaining the differences and motivation, mostly
slightly wrong. The best ones will say, "Don't use it; just use
function." And lambda introduces an awful lot of subtlety for
something whose main claim to utility would be in allowing precise
and clear definitions of new control constructs through desugaring.
Natural language is bad, but not this bad.

I hear you. I love lambda, I love the syntactic simplicity of
Smalltalk blocks (and we could probably approximate its syntactic
simplicity with a literal syntax), and I love the idea of adding a
properly tail-calling function form. Nevertheless, I recognize that
the similarity to `function' is a usability hazard.

[Sorry for my absence lately.]

> If yield does work, then we're capturing deep stacks.  A more  
> plausible example:
>
> function j(x) {
>  (lambda (y) {
>     (lambda (z) {
>        yield z;
>      }
>      (y + " and a dyne"));
>   }
>   (x + ", a poundal"));
> }
> print(j("I love you").next());
>
> This is just a desugaring of some nested lets, but we still have  
> yield capturing many frames.  If this doesn't work, lambda is really  
> useless.

This issue really comes down to whether `lambda' is properly tail  
calling. If so, then the `yield' in this example occurs as if it were  
directly within the body of `j', because there's no stack beyond the  
activation frame of `j'.

To be a little more precise: when people refer to "Tennent," the main  
property they're talking about is that (modulo syntax) an expression E  
is equivalent to the expression (lambda () E)(). Namely, in any  
expression context (roughly, a position in a program where you can  
place an expression), either expression will result in the same  
program behavior.

If we don't want generators to capture deep stacks, then we can make  
it a dynamic error for `yield' to occur in a non-empty sub-stack. Then  
in order to preserve the property that (lambda () E)() for all E, we'd  
have to mandate that (lambda () E)() is really a tail call and  
guaranteed not to push any stack.

Alternatively, we could allow generators to capture deep lambda- 
stacks, but not deep function-stacks.  However...

> For what it's worth, speaking as a long-time Scheme fan, I wouldn't  
> add lambda to ES.  It seems too similar to function; there will be  
> endless blog posts explaining the differences and motivation, mostly  
> slightly wrong.  The best ones will say, "Don't use it; just use  
> function."  And lambda introduces an awful lot of subtlety for  
> something whose main claim to utility would be in allowing precise  
> and clear definitions of new control constructs through desugaring.   
> Natural language is bad, but not this bad.

I hear you. I love lambda, I love the syntactic simplicity of  
Smalltalk blocks (and we could probably approximate its syntactic  
simplicity with a literal syntax), and I love the idea of adding a  
properly tail-calling function form. Nevertheless, I recognize that  
the similarity to `function' is a usability hazard.

Dave

# Brendan Eich (16 years ago)

On May 28, 2009, at 11:48 AM, Jim Blandy wrote:

This is just a desugaring of some nested lets, but we still have
yield capturing many frames. If this doesn't work, lambda is really
useless.

Dave replied, but I wanted to join in agreeing on the following:

For what it's worth, speaking as a long-time Scheme fan, I wouldn't
add lambda to ES. It seems too similar to function; there will be
endless blog posts explaining the differences and motivation, mostly
slightly wrong. The best ones will say, "Don't use it; just use
function." And lambda introduces an awful lot of subtlety for
something whose main claim to utility would be in allowing precise
and clear definitions of new control constructs through desugaring.
Natural language is bad, but not this bad.

I'm "cool" toward lambda, meaning not in favor. I'd rather reform
function to the extent that doing so helps programmers. I'm not
convinced that TCP matters a lot to those humans. To some programmers,
and of course for macros and other things involving code generation
and analysis, lambdas are great. But you can't please every(one|thing).

On May 28, 2009, at 11:48 AM, Jim Blandy wrote:

> This is just a desugaring of some nested lets, but we still have  
> yield capturing many frames.  If this doesn't work, lambda is really  
> useless.

Dave replied, but I wanted to join in agreeing on the following:

> For what it's worth, speaking as a long-time Scheme fan, I wouldn't  
> add lambda to ES.  It seems too similar to function; there will be  
> endless blog posts explaining the differences and motivation, mostly  
> slightly wrong.  The best ones will say, "Don't use it; just use  
> function."  And lambda introduces an awful lot of subtlety for  
> something whose main claim to utility would be in allowing precise  
> and clear definitions of new control constructs through desugaring.   
> Natural language is bad, but not this bad.

I'm "cool" toward lambda, meaning not in favor. I'd rather reform  
function to the extent that doing so helps programmers. I'm not  
convinced that TCP matters a lot to those humans. To some programmers,  
and of course for macros and other things involving code generation  
and analysis, lambdas are great. But you can't please every(one|thing).

/be