Discuss
  • yield and new : SpiderMonkey and the draft Spec

yield and new : SpiderMonkey and the draft Spec

# Dmitry A. Soshnikov (15 years ago)

Hello,

I hope you can help with explaining of what is going on with this value inside the body of a generator?

Consider e.g. the following case:

// infinite objects generator let g = new function () { this.x = 10; while (true) { yield; } };

// generate an array of 3 objects let objects = [1, 2, 3].map(function(i) g.next());

console.dir(objects);

Results:

[ [[Class]]: "Array", length: 3, 0: { [[Class]]: "Object", x: 10 }, 1: { [[Class]]: "Object" }, 2: { [[Class]]: "Object" } ]

Only first object has x property. Also:

console.log(objects[0] == objects[1]); // false console.log(objects[1] == objects[2]); // false

As I understand, [[Construct]] activated by the new calls [[Call]] of the function, which produces the g generator. I look here: strawman:generators and see:

Calling

Let |f| be a generator function. The semantics of a function call |f(x1, ..., xn)| is:

Let E = a new VariableEnvironment record with mappings for |x1| ... |xn| Let S = the current scope chain extended with E Let V = a new generator object with [[Scope]] = S [[Code]] = f.[[Code]] [[ExecutionContext]] = null [[State]] = "newborn" [[Handler]] = the standard generator handler Return V

So, g will be the generator with the needed [[Code]] and empty [[ExecutionContext]]. Notice, there nothing is said about this value.

In calling next (i.e. send(undefined)) we get into:

Internal method: send

G.[[Send]]

Let State = G.[[State]] If State = "executing" Throw Error If State = "closed" Throw Error Let X be the first argument If State = "newborn" If X != undefined Throw TypeError Let K = a new execution context as for a function call K.currentGenerator := G K.scopeChain := G.[[Scope]] Push K onto the stack Return /Execute/(G.[[Code]]) G.[[State]] := "executing" Let Result = /Resume/(G.[[ExecutionContext]], normal, X) Return Result

We see that a new context is created but again, nothing is said about its this value.

When evaluating /Execute/(G.[[Code]]) we with yield get into:

Yielding

The semantics of evaluating an expression of the form |yield e| is:

Let V ?= Evaluate(e) Let K = the current execution context Let O = K.currentGenerator O.[[ExecutionContext]] := K O.[[State]] := "suspended" Pop the current execution context Return (normal, V, null)

Btw, what does "?=" mean?

Here the K is the context created on in the send method (still we haven't any info about this value).

The following call to next will again enter send method with undefined but we already get into:

Resuming generators

Operation /Resume/(K, completionType, V)

Push K onto the execution context stack Let G = K.currentGenerator Set the current scope chain to G.[[Scope]] Continue executing K as if its last expression produced (completionType, V, null)

where we proceed with evaluating previously saved continuation. And again, nothing about this is said.

As I see, during all these steps always the same K is passed around and evaluated. This value is a property of the context and K has it, but which?

As was shown in the example above, this is set to the newly created object, but it's a current SpiderMonkey's behavior; don't know how it correlate with this draft spec.

Consider e.g. the following example (tested in SpiderMonkey):

g = new function() { yield new Boolean((yield) == this) };

console.log(''+ g.send(g.next())); // false

g = new function() { yield new Boolean(this == (yield)) };

console.log(''+ g.send(g.next())); // true

Why in first call yield wasn't newly created object and in the second one -- it was? What actually should yield without an argument yields? undefined I guess.

So don't know which behavior is correct and whether it's a strange behavior in current SpiderMonkey.

Thanks, Dmitry.

# Brendan Eich (15 years ago)

It looks to me like you've just found a bug in SpiderMonkey's generator implementation, which pre-dates the strawman:generators proposal by four+ years. Please file it in the right place: bugzilla.mozilla.org. Thanks.

The bug is a regression. Here's what my Firefox 3-era JS shell does:

js> // infinite objects generator

js> let g = new function () { this.x = 10; while (true) { yield; } }; js>

js> // generate an array of 3 objects

js> let objects = [1, 2, 3].map(function(i) g.next());

js> objects ,, js> uneval(objects) [(void 0), (void 0), (void 0)]

Clearly, yield; should not yield a mysterious object as the return value of g.next().

# Dmitry Soshnikov (15 years ago)

Yep, thanks Brendan,

I filed the bug bugzilla.mozilla.org/show_bug.cgi?id=641436

But the thing with this is still interesting for me. So in this particular case this should be set to undefined. Should it be always set to undefined (strict) / global object and regardless -- calling from new or via simple call? There nothing is said about this value of the [[ExecutionContext]] created for the "newbord" generator. What should it be set to?

Thanks, Dmitry.

# Brendan Eich (15 years ago)

On Mar 14, 2011, at 3:02 AM, Dmitry Soshnikov wrote:

Yep, thanks Brendan,

I filed the bug bugzilla.mozilla.org/show_bug.cgi?id=641436

But the thing with this is still interesting for me. So in this particular case this should be set to undefined.

No, |this| is not set to undefined. The generator function yields undefined each time (via |yield;| in the loop body).

That undefined is what the anonymous expression closure you pass to map should (absent the bug) return in turn (i.e., undefined is the return value of g.next() and thus the return value of function(i) g.next()).

Notice how you evaluate a |new| operator exactly once, in the top-level assignment expression statement:

// infinite objects generator let g = new function () { this.x = 10; while (true) { yield; } };

// generate an array of 3 objects let objects = [1, 2, 3].map(function(i) g.next());

A generator function called as a constructor via new receives a fresh Object instance as |this|, same as for any function.

However, there's no way for a generator function to return that instance, because a generator function always implicitly returns a fresh generator iterator when invoked. It could store |this| in the heap, and whatever value |this| receives is saved as part of the shallow continuation capture by the generator.

If you write |return this| anywhere in the body of the generator function you'll get an early error. So really all |new| is doing for you here is giving a fresh object to |this|, which seems fine if that's what you want.

# Dmitry A. Soshnikov (15 years ago)

On 14.03.2011 16:51, Brendan Eich wrote:

On Mar 14, 2011, at 3:02 AM, Dmitry Soshnikov wrote:

Yep, thanks Brendan,

I filed the bug bugzilla.mozilla.org/show_bug.cgi?id=641436

But the thing with this is still interesting for me. So in this particular case this should be set to undefined. No, |this| is not set to undefined. The generator function yields undefined each time (via |yield;| in the loop body).

That undefined is what the anonymous expression closure you pass to map should (absent the bug) return in turn (i.e., undefined is the return value of g.next() and thus the return value of function(i) g.next()).

Yes, in this particular case I see that yield should return undefined and that's the bug is that it instead returns a newly created object (every time different btw, as there would different calls to new, however by logic there is only one call to new, i.e. [[Construct]]).

What I asked is the this value inside a generator's body in general, not in this exact case with the new. I'll described the question in some details below.

Notice how you evaluate a |new| operator exactly once, in the top-level assignment expression statement:

Yes, it's true and that exactly I wrote in the first letter.

// infinite objects generator let g = new function () { this.x = 10; while (true) { yield; } };

// generate an array of 3 objects let objects = [1, 2, 3].map(function(i) g.next()); A generator function called as a constructor via new receives a fresh Object instance as |this|, same as for any function.

Yep. But it's from the point of view how [[Construct]] works. This is exactly about what I was asking. However, according to the draft spec, [[Call]] applied to the generator function (no matter from [[Construct]] or simply) does not enter any new context (which means, a caller doesn't/shouldn't provide any this value for the new context, since there is no entering the context). Only a new generator instance is created with the needed [[Code]], [[ExecutionContext]] set to null and [[State]] as "newborn".

Only when next() method (or send(undefined) -- which is the same) is called, there is a creation of new execution context for a function call. This is again how it's written in that draft spec. What does it mean? -- it means that entering the execution context as for a function code, a caller, i.e. the next (or send) method should provide this value for the callee -- a newly created context. And exactly about this case I was asking -- which this value should be provided in this case? Because on the draft page there nothing is said about this value of the context (saved continuation) passing around.

It's clearly seen in current JS 1.8.5 that this value is set to global object (as for every normal function called with the base either an activation object or undefined => global).

let global = this;

let isGlobal = (function () { yield this == global; // true })().next();

console.log(isGlobal); // true

Moreover, in strict mode it's (again in current JS 1.8.5) undefined:

let global = this;

let isUndefined = (function () { "use strict"; yield this == undefined; // true })().next();

console.log(isUndefined); // true

So the answer is in the next() / send() method which is a caller and which is provides this value for the context. However, next/send just resumes the same context K which was created in first send(undefined)

# Brendan Eich (15 years ago)

I.e. again, what I notice (and asked), is that there's no any info in the draft spec about this value of the generator's context. I think it would be good to add this information.

A generator function is still a function, and |this| binding is orthogonal to yield making the surrounding function a generator:

js> function gen() { yield [this, 1]; yield [this, 2]; yield [this, 3]; }

js> var obj = {foo: gen, toString: function(){return "[obj]"}}

js> var a = []

js> for (v in obj.foo()) a.push(v)

3 js> a[0][0] === obj

true js> a[0][0] === obj

true js> a[1][0] === obj

true js> a[2][0] === obj

true js> a.toString() "[obj],1,[obj],2,[obj],3"

We'll clarify this in the strawman.

# Dmitry A. Soshnikov (15 years ago)

On 14.03.2011 23:02, Brendan Eich wrote:

I.e. again, what I notice (and asked), is that there's no any info in the draft spec about this value of the generator's context. I think it would be good to add this information. A generator function is still a function, and |this| binding is orthogonal to yield making the surrounding function a generator:

js> function gen() { yield [this, 1]; yield [this, 2]; yield [this, 3]; } js> var obj = {foo: gen, toString: function(){return "[obj]"}} js> var a = [] js> for (v in obj.foo()) a.push(v) 3 js> a[0][0] === obj true js> a[0][0] === obj true js> a[1][0] === obj true js> a[2][0] === obj true js> a.toString() "[obj],1,[obj],2,[obj],3"

Oh, I see.

A possible place to store |this| according to the current strawman is:

Calling

Let |f| be a generator function. The semantics of a function call |f(x1, ..., xn)| is:

Let E = a new VariableEnvironment record with mappings for |x1| ... |xn| Let S = the current scope chain extended with E Let V = a new generator object with [[Scope]] = S [[Code]] = f.[[Code]] [[ExecutionContext]] = null

**[[ThisBinding]] = GetBase(Reference(base, "f"))

[[State]] = "newborn" [[Handler]] = the standard generator handler Return V

I.e. since here we don't create the context, at least can save |this| of the reference. For the example of the strawman it will be global, since f(...). And for your example with obj.foo, it's the obj object. OK, now it's clear.

And then in:

Internal method: send

G.[[Send]]

Let State = G.[[State]] If State = "executing" Throw Error If State = "closed" Throw Error Let X be the first argument If State = "newborn" If X != undefined Throw TypeError Let K = a new execution context as for a function call

Let K.[[ThisBinding]] = G.[[ThisBinding]]

K.currentGenerator := G K.scopeChain := G.[[Scope]] Push K onto the stack Return /Execute/(G.[[Code]]) G.[[State]] := "executing" Let Result = /Resume/(G.[[ExecutionContext]], normal, X) Return Result

And now everything's becoming clear.

We'll clarify this in the strawman.

Yep, would be good.

Thanks again Brendan,

Dmitry.

# Dmitry A. Soshnikov (15 years ago)

Moreover, forgot to mention. Passing the generator function ("g-function" and "g-object" for shortness) as an argument for the Generator constructor is not good for dynamically bound this value (notice, that in Python's self is just a casual variable/argument, which should be passed manually anyway).

I.e. we should have this as foo in the following example:

let foo = { bar: function () { yield (this == foo); } };

let g = new Generator(foo.bar); g.next(); // false

However,

let g = foo.bar(); g.next(); // true

You may of course pass the context object as the second argument (thus, providing explicit self as in Python):

function Generator(fn, self) { return fn.apply(self); }

let g = new Generator(foo.bar, foo); g.next(); // true

But I don't see a big need in such a wrapper. If the rules are specified, there should be no any confusion with using call expression for getting g-object, and then use next to executed the continuation.

Dmitry.

# Dmitry A. Soshnikov (15 years ago)

On 15.03.2011 20:21, David Herman wrote:

P.S.:

A small change, e.g. can be to make next as a getter since it doesn't accept arguments.

g.next; // 1 g.next; // 2

But, it's a cosmetic and actually not so needed change. -1

The purpose of the next interface is to change the state of the iterator. A getter interface obscures the statefulness.

Yeah, true.

Dmitry.

# John J. Barton (15 years ago)

On 11:59 AM, Brendan Eich wrote:

However, there's no way for a generator function to return that instance, because a generator function always implicitly returns a fresh generator iterator when invoked. It could store |this| in the heap, and whatever value |this| receives is saved as part of the shallow continuation capture by the generator.

The implicit return of a fresh generator iterator makes the example confusing. It also makes simple examples difficult. For example from developer.mozilla.org/en/Core_JavaScript_1.5_Guide/Iterators_and_Generators

function simpleGenerator(){ yield "first"; yield "second"; yield "third"; for (var i = 0; i< 3; i++) yield i; }

var g = simpleGenerator(); print(g.next()); // prints "first"

The developer has to read the code for simpleGenerator and scan an arbitrary number of lines for 'yield' to realize that the statement var g = simpleGenerator(); is an implicit constructor of a special object and only when g.next() is run will any statement from simpleGenerator-the-real-function-we-wrote be called. A debugger stopped on the line that creates |g| will want to step-into simpleGenerator(); if it does the result is puzzling, it has to step back immediately; if does not step-into simpleGenerator() the result is puzzling, why did the debugger fail? So we are stuck with some warning-message special-case solution, "oh this function is a generator!".

Suppose we had a built-in function Generator: var g = new Generator(simpleGenerator); Now I read once the documentation for Generator. I see it takes a function argument which must contain the keyword 'yield' and it creates a special object with two methods, next() and send(). I never need to read the body of simpleGenerator. (Generator.create(simpleGenerator) would also work).

Furthermore if I stop a debugger at the line above and single step into, it would be perfectly reasonable for the debugger to simply step to the next line and show |g| is now a Generator object. That's how built-in things work, it's ok.

Now consider Dmitry's example:

// infinite objects generator

let g = new function () { this.x = 10; while (true) { yield; } };

It becomes: // infinite objects generator let g = new new Generator (function () { this.x = 10; while (true) { yield; } };

Brendan's explanation of this case is now not necessary: by making the implicit constructor explicit we see immediately that the |new| is not doing what one expected from Dmitry's original code.

jjb

# Brendan Eich (15 years ago)

On Mar 14, 2011, at 11:50 PM, John J. Barton wrote:

On 11:59 AM, Brendan Eich wrote:

However, there's no way for a generator function to return that instance, because a generator function always implicitly returns a fresh generator iterator when invoked. It could store |this| in the heap, and whatever value |this| receives is saved as part of the shallow continuation capture by the generator.

The implicit return of a fresh generator iterator makes the example confusing.

It's just like Python.

# John J. Barton (15 years ago)

On 3/14/2011 10:08 PM, Brendan Eich wrote:

On Mar 14, 2011, at 11:50 PM, John J. Barton wrote:

On 11:59 AM, Brendan Eich wrote:

However, there's no way for a generator function to return that instance, because a generator function always implicitly returns a fresh generator iterator when invoked. It could store |this| in the heap, and whatever value |this| receives is saved as part of the shallow continuation capture by the generator.

The implicit return of a fresh generator iterator makes the example confusing. It's just like Python.

and Python is perfect? Even successful features have room for improvement.

jjb

# Dmitry A. Soshnikov (15 years ago)

On 15.03.2011 17:58, John J. Barton wrote:

On 3/14/2011 10:08 PM, Brendan Eich wrote:

On Mar 14, 2011, at 11:50 PM, John J. Barton wrote:

On 11:59 AM, Brendan Eich wrote:

However, there's no way for a generator function to return that instance, because a generator function always implicitly returns a fresh generator iterator when invoked. It could store |this| in the heap, and whatever value |this| receives is saved as part of the shallow continuation capture by the generator.

The implicit return of a fresh generator iterator makes the example confusing. It's just like Python.

and Python is perfect? Even successful features have room for improvement.

Pyhton's and JS's implementation is just a one of. But for those who work with such an implementation, there should be no a big confusion. Lua in contrast e.g. uses a special coroutine.create(generatorFn) (lua-users.org/wiki/CoroutinesTutorial) -- similarly to your proposal. But you may do it yourself if it's confusing

function Generator(fn) { return fn(); }

let g = new Generator(function() { yield 1; yield 2; });

g.next(); 1 g.next(); 2

From JS/Python's viewpoint, this may be considered as superficial thing -- if you know that the function is a generator function, be prepared that the call creates the generator object. The other call to the same generator function, creates a fresh g-object. And so on. There is no confusion as it would be that the first call create the g-object, and the other calls already yield, no. For yielding next method is.

P.S.:

A small change, e.g. can be to make next as a getter since it doesn't accept arguments.

g.next; // 1 g.next; // 2

But, it's a cosmetic and actually not so needed change.

Dmitry.

# David Herman (15 years ago)

P.S.:

A small change, e.g. can be to make next as a getter since it doesn't accept arguments.

g.next; // 1 g.next; // 2

But, it's a cosmetic and actually not so needed change.

-1

The purpose of the next interface is to change the state of the iterator. A getter interface obscures the statefulness.

# David Herman (15 years ago)

I agree with Dmitry on these points. In short: the new Generator(...) syntax still requires its argument to be a generator function, so you've just added pointless extra make-work to create one. And as Dmitry points out, you can always implement that constructor yourself in JS.

What I think is more important is to have an API that lets you actually test whether a function (which may have been created somewhere else entirely) is a generator function. That's why I added an API to the spec:

Function.isGenerator(f)

which returns a boolean indicating whether f is a generator function. See:

http://wiki.ecmascript.org/doku.php?id=strawman:generators#api

JJB: you could use this to make your implementation of a Generator constructor more robust, e.g.:

function Generator(fn, self, args) {
    if (!Function.isGenerator(fn))
        throw new TypeError("not a generator function");
    this.generatorFunction = fn.apply(self, args);
}

Generator.prototype = { ... }
# John J. Barton (15 years ago)

On 11:59 AM, Dmitry A. Soshnikov wrote:

Moreover, forgot to mention. Passing the generator function ("g-function" and "g-object" for shortness) as an argument for the Generator constructor is not good for dynamically bound this value (notice, that in Python's self is just a casual variable/argument, which should be passed manually anyway).

I.e. we should have this as foo in the following example:

let foo = { bar: function () { yield (this == foo); } };

let g = new Generator(foo.bar); g.next(); // false

g.next.apply(foo,[]); // true

However,

let g = foo.bar(); g.next(); // true

Could you please explain the reasoning here? I can't figure it out. To me, g.next() has to mean that in "next", this===g. I can't see a path to conclude that foo is involved. It's only role is to hold the property 'bar' which is a function constructing 'g'.

But I don't see a big need in such a wrapper. If the rules are specified, there should be no any confusion with using call expression for getting g-object, and then use next to executed the continuation.

Perhaps we need a little user testing to see whose claims hold up.

jjb

# John J. Barton (15 years ago)

On 11:59 AM, David Herman wrote:

P.S.:

A small change, e.g. can be to make next as a getter since it doesn't accept arguments.

g.next; // 1 g.next; // 2

But, it's a cosmetic and actually not so needed change. -1

The purpose of the next interface is to change the state of the iterator. A getter interface obscures the statefulness.

Yes! An API inconsistent with the rest of the language makes programming more difficult. The implicit constructor for generators obscures the nature of the generator object: same principle.

jjb

# Dmitry A. Soshnikov (15 years ago)

On 15.03.2011 21:13, John J. Barton wrote:

On 11:59 AM, Dmitry A. Soshnikov wrote:

Moreover, forgot to mention. Passing the generator function ("g-function" and "g-object" for shortness) as an argument for the Generator constructor is not good for dynamically bound this value (notice, that in Python's self is just a casual variable/argument, which should be passed manually anyway).

I.e. we should have this as foo in the following example:

let foo = { bar: function () { yield (this == foo); } };

let g = new Generator(foo.bar); g.next(); // false g.next.apply(foo,[]); // true

No, here you set this value of the next activation, not for the context of the continuation. Notice, the state of the CC's (current continuation) context is set only once -- at it's creation. The creation is done (by the strawman spec) at first call to next method. In this creation of the CC's context, this value is taken from the g-object itself (and g-object got it when was created by the call to the g-function).

Currently (in non-standardized version in SpiderMonkey), next method in every single/next call doesn't change this value of the CC's context, since as said, uses the same context passing it around.

I don't see currently the case when the this value should/could be changed in every next entering for yielding (in other case it reminds me Python again when you can change self in runtime).

So all you've done with g.next.apply(foo,[]); is just provided this value as foo for the next method. And which is caused an exception since next method is applied not for a generator object, but for the foo:

g.next.apply(foo, []); // not true, but TypeError

Side note: Notice though, that current SpiderMonkey 1.8.5 shows the following exception message: "TypeError: Generator.prototype.next called on incompatible Object". However, Generator binding is not defined. So it's good either to make Generator binding available, or to change the error message. Note: it's not Generator function used in the previous example, it's a built-in generators constructor:

console.log(typeof Generator); // "undefined"

let g = (function () {yield}()).next.call(null); // TypeError: Generator.prototype.next called on incompatible global

However,

let g = foo.bar(); g.next(); // true Could you please explain the reasoning here? I can't figure it out. To me, g.next() has to mean that in "next", this===g.

Yes, absolutely correct. And it does mean exactly this.

However, I was talking about this value of the CC's context (not of the nexts context). Perhaps it can be convenient to have a g-function as a method of an object and use this as that object during all reenters to that continuation.

I can't see a path to conclude that foo is involved.

Well, I can assume this (especially if a programmer isn't so familiar with such a style of creating of these coroutines-generators). However, if the programmer is familiar with that, then the information how foo is involved here is taken from the code of the creation of the g -- at the moment of g-function activation, i.e. when this value is provided by the caller.

It's only role is to hold the property 'bar' which is a function constructing 'g'.

As is said, we can't say exactly whether it's the only its purpose, repeat, it's likely that this value normally can be used inside the code of the g-function and refer to the needed object.

The other thing is mutable over the reentering the CC this value. E.g. when we pass it to next and next then pass it throw to the CC's context. Thus, we can't have different this value in every next call to CC. But it breaks some existing scheme, e.g. with the same checking of the this to be exactly the generator object in the next method. And I'm not sure whether different this value is needed in different reenters to the CC. Moreover, if to consider such coroutines as threads, then I guess this value should be the same during the whole code of the g-function body.

But I don't see a big need in such a wrapper. If the rules are specified, there should be no any confusion with using call expression for getting g-object, and then use next to executed the continuation. Perhaps we need a little user testing to see whose claims hold up.

Maybe, I'm not against it and opened to different styles and variants, though, as I mentioned, this thing that such a generators way seems to you unfamiliar can be just a matter of a habit.

Dmitry.

# Brendan Eich (15 years ago)

Sorry, this is both foolish consistency and a boilerplate tax on users. Worse, it removes a special form (syntax dedicated to a new primitive) that implementations need to parse, in order to compile and optimize the function using the primitive. We are not going to add a Generator constructor.