Discuss
  • yield syntax

yield syntax

# David-Sarah Hopwood (16 years ago)

Brendan Eich wrote:

This may be breaking a butterfly on a wheel, but I am game if it improves the state of the strawman proposals.

On May 15, 2009, at 7:16 PM, David-Sarah Hopwood wrote:

My point was that the example of 'yield (foo)' (that is, yield as a prefix operator applied to the expression '(foo)') shows that the prefix operator syntax cannot possibly be easier to specify than the function call syntax -- contrary to what you appeared to be arguing above.

Analogous to direct vs. indirect eval in ES5 (15.1.2.1.1), there is no purely syntactic specification for what Neil proposes. A runtime check is required. So I don't see why you are focusing only on syntax here.

Maybe I misunderstood what Neil was proposing, but I took him to mean something equivalent to:

  • parse using essentially the current grammar.
  • if the text of the CallExpression in 'CallExpression Arguments', or of the MemberExpression in 'MemberExpression Arguments', is "yield" (excluding comments and whitespace), then the call is a yield-expression.

This is a purely syntactic specification.

It does have some quirks: if you define a function called 'yield' and try to call it, then the call will still be treated as a yield-expression rather than a function call. For that reason I'm going cold on the idea. I would prefer something entirely unambiguous, like

@yield expr

or an opt-in that makes 'yield' (and 'let') reserved everywhere. Either would make parentheses around a yield expression unnecessary, or at least not necessary in as many cases. ('yield' would behave syntactically like 'typeof', perhaps with lower precedence.)

[...]

Show me an unambiguous grammar for the prefix yield operator, and then I'll concede the point :-)

Python 2.5 has essentially the same issues,

Python does not attempt to contextually reserve 'yield'. It uses an opt-in that makes 'yield' a keyword that is reserved everywhere. From PEP 255:

A new statement is introduced:

yield_stmt: "yield" expression_list

"yield" is a new keyword, so a future statement[8] is needed to phase

this in: in the initial release, a module desiring to use generators

must include the line

from future import generators

near the top (see PEP 236[8]) for details). Modules using the

identifier "yield" without a future statement will trigger warnings.

In the following release, yield will be a language keyword and the

future statement will no longer be needed.

and it has a formal grammar (several, the one CPython uses and at least one other at antlr.org). From CPython's Grammar/Grammar EBNF:

$ grep yield Grammar expr_stmt: testlist (augassign (yield_expr|testlist) | ('=' (yield_expr|testlist))*) flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt | yield_stmt yield_stmt: yield_expr atom: ('(' [yield_expr|testlist_gexp] ')' | yield_expr: 'yield' [testlist]

That would be ambiguous, if (hypothetically) 'yield' were not a keyword in Python. For example

(yield (foo))

would match both a parenthesised <yield_expr>, and a parenthesised <NAME> 'yield' followed by a trailer giving function arguments. (This involves the <power>, <trailer>, <arglist> and <argument>

productions that weren't found by your grep.)

Similarly for (yield - 1) or (yield + 1) or (yield /re/g) in Harmony. If these are treated as being yield-expressions, then that is not compatible with any ES3/5 code using 'yield' as a variable name, but if they are treated as having their ES5 meaning, then there are a bunch of ugly special cases for what token can validly begin the expression after 'yield'.

What we are discussing is almost trivial in terms of usability, but the difference is big enough to care about, IMHO.

Unambiguous grammar first, please.

# David-Sarah Hopwood (16 years ago)

Brendan Eich wrote:

On May 17, 2009, at 12:43 PM, Mark S. Miller wrote:

On Sun, May 17, 2009 at 11:00 AM, Brendan Eich <brendan at mozilla.com> wrote:

Analogous to direct vs. indirect eval in ES5 (15.1.2.1.1), there is no purely syntactic specification for what Neil proposes. A runtime check is required. So I don't see why you are focusing only on syntax here.

I don't follow. What runtime check? For the eval operator, the runtime check is whether the value of the eval variable is the original global eval function. It makes no sense to have a corresponding global yield function value.

If we reserve yield then you're right. One of the appealing (at least to me) aspects of Neil's suggestion was that it would avoid opt-in versioning required by reserving yield (which is used in extant web content, or was when we tried reserving it without opt-in versioning -- the particular use was as a formal parameter name, used as a flag not a function).

Oh, right. We've been talking at cross-purposes. I assumed that you were suggesting that 'yield' should be contextually reserved. That is what I've been saying couldn't work due to ambiguities.

# Brendan Eich (16 years ago)

On May 17, 2009, at 5:05 PM, David-Sarah Hopwood wrote:

Brendan Eich wrote:

On May 17, 2009, at 12:43 PM, Mark S. Miller wrote:

On Sun, May 17, 2009 at 11:00 AM, Brendan Eich <brendan at mozilla.com> wrote:

Analogous to direct vs. indirect eval in ES5 (15.1.2.1.1), there
is no purely syntactic specification for what Neil proposes. A runtime check is required. So I don't see why you are focusing only on
syntax here.

I don't follow. What runtime check? For the eval operator, the
runtime check is whether the value of the eval variable is the original
global eval function. It makes no sense to have a corresponding global
yield function value.

If we reserve yield then you're right. One of the appealing (at
least to me) aspects of Neil's suggestion was that it would avoid opt-in versioning required by reserving yield (which is used in extant web content, or was when we tried reserving it without opt-in
versioning -- the particular use was as a formal parameter name, used as a flag
not a function).

Oh, right. We've been talking at cross-purposes. I assumed that you
were suggesting that 'yield' should be contextually reserved.

Oh, I should have seen that. Thanks for clarifying.

That is what I've been saying couldn't work due to ambiguities.

RIght, no sane way to reserve yield or another keyword contextually
(e.g., only in callee position; "contextually" in the sense of get and
set in object initialisers).

In JS1.7+, we reserve yield with opt-in version selection via <script
type="...;version=1.7"> and the like. In such a script, yield and let

are reserved the same way the other identifiers are, which is how ES5
does it (with an extension we've discussed -- the extension where a
reserved identifier can be used after 'function' -- but that's not
interesting for this discussion).

# David-Sarah Hopwood (16 years ago)

Brendan Eich wrote:

On May 17, 2009, at 6:39 PM, Neil Mix wrote:

  • we could always allow parenthesis to be dropped when the yield is the entire expression of an expression statement or the right-hand side of an assignment.

Right-hand side of assignment is ok without parens in Python because assignment is a statement.

In JS if you allow assignment expressions ending in unparenthesized yields, then you can have unparenthesized yields in argument and initialiser lists, comma expressions, and in the middle and final operand positions in ternary (?:) expressions.

  • in my experience with JS 1.7 I almost always had to parenthesize the yield expression when it was in some other kind of expression. An in the cases where parenthesis weren't required, I parenthesized anyway to avoid ambiguity and maintain coding style consistency. (And because I got tired of predicting incorrectly whether or not parens would be required in a particular context.)

The only contexts we allow you not to parenthesize in JS1.7 are assignment expressions and final argument in list. But see above -- the assignment expression loophole is big enough to allow

foo(a = yield b, c);

One argument, or two?

If it is allowed, then it should be two. It would be very surprising if

foo(a = b, c);

had two arguments (as it does), but the above expression with yield had one.

But I agree that it may be better not to allow it.

So I would argue that there are two syntactical forms of yield, yield E and (yield E), and that the rules regarding the requirement for parenthesis are hard to predict (from personal experience). Therefore, I argue that it would make sense to simplify a bit:

  • the yield E form may be used when it is the entire expression of an expression statement
  • all other times it must be parenthesized

Agreed; this closes the assignment expression loophole.

This would disallow

foo(yield x);

which seems unnecessary.

# David-Sarah Hopwood (16 years ago)

Igor Bukanov wrote:

2009/5/18 Brendan Eich <brendan at mozilla.com>:

On May 18, 2009, at 2:25 AM, Igor Bukanov wrote:

The plus side of this is that an empty generator can be created with a straightforward:

Generator(function() {})

and not with a rather unnatural

(function() { if (false) yield; })() No one makes empty generators.

For me the problem with the way the generators are defined is that a dead code like that "if (0) yield;" affects the semantic by mere presence of it. Surely, this is not the first feature in ES that has that property - "if (0) var a;" is another example. But "if (0) yield;" sets a new record affecting the nature of the whole function.

A more explicit alternative is to require some kind of decoration on the function definition, e.g. (just a straw man):

function generator foo() { ... }

# Brendan Eich (16 years ago)

On May 19, 2009, at 2:00 PM, David-Sarah Hopwood wrote:

If it is allowed, then it should be two. It would be very
surprising if

foo(a = b, c);

had two arguments (as it does), but the above expression with yield had one.

But I agree that it may be better not to allow it.

Good; it was a loophole in JS1.7 and up, which I think should be closed.

Agreed; this closes the assignment expression loophole.

This would disallow

foo(yield x);

which seems unnecessary.

I thought so too, but Python keeps its grammar simple this way, and
simpler is better, ceteris paribus. JS1.7 allows this but also suffers
the assignment loophole. Getting rid of the latter but not the former
is grammar-hacking busy work. I haven't done it in the WebKit/ JavaScriptCore/parser/Grammar.y.

# Brendan Eich (16 years ago)

On May 19, 2009, at 2:08 PM, David-Sarah Hopwood wrote:

A more explicit alternative is to require some kind of decoration on
the function definition, e.g. (just a straw man):

function generator foo() { ... }

Or just (we discussed this briefly last summer in Oslo):

generator foo() { ... }

In JS1.7, we followed Python, which uses def, same as for defining a
function, and distinguishes generators by presence of yield (which
makes value returns illegal). It seems unwise to diverge from Python
without more evidence of an actual usability problem. We have no such
evidence from shipping JS1.7 and up in Firefox 2 and up.