API to get stack frame info from generator
Le 14/06/2013 16:56, Bruno Jouhier a ?crit :
I'm using ES6 generators to implement a little async/await library and I'm quite pleased with the result so far but I'm lacking one API: a function to get stack information from a generator object. Ideally it would return the name of the current generator function, the filename and the line number where it last yielded.
If I had this API I 'd be able to provide a complete trace of the stack of await calls when an exception is caught.
ES5 strict mode poisoned .caller and .callee. The reason is that it isn't necessarily a good idea (security, maybe performance reasons as well) to give authority to the runtime to inspect stack frames. It's more of a debugger use case.
The Debugger API in Firefox (only available to "Chrome-level" privileged code) has a way to know the function being called, and to navigate across the different frames like the caller frame ("older" property) 1 and some infos that help finding the function name, filename and line number.
Thanks David. The problem is that I need it first in node.js, and if possible with a standardized API that works in all ES6 JS engines.
I see this as being similar to asking for a portable "stack" property in Error objects. I don't know if it is actually mandated by ES6 but it looks like all major JS engines support it now.
Note that caller and callee would not work for my use case. I need the frame info for a generator object which has been suspended, not from the generator function itself.
If I understand this right, essentially the problem is that an exception could occur when some of the involved code is not alive on the stack (because the generator(s) are suspended), and without the ability to capture this information, the end user has no actual knowledge of why the exception occurred. Yeah?
That makes it hard to leave up to debuggers, since the debugger isn't aware of your generator library and it's pretty hard to extend debuggers. This is a common problem with custom task schedulers based on generators - we had the same problem with imvu.task (debugging tasks in python) and I had the same problem personally with my task scheduler in C#. The simplest solutions (like aggressively capturing stack traces at some appropriate point), even when they work, are pretty expensive.
Is it possible to 'throw into' an ES6 generator? I can't remember if that made it in - in Python, at least, that allows you to basically terminate a generator wherever it is, and through doing so potentially capture a stack trace for the generator after it terminates, from the error bubbling up through the generator... that is, if I remember the semantics for when python sets/overwrites tracebacks correctly. Using such a mechanism, you could reconstruct an 'accurate' stack trace for any number of suspended generators.
On 6/14/2013 7:02 PM, Kevin Gadd wrote:
Is it possible to 'throw into' an ES6 generator?
This is what generator.throw(exception) does. Or are you referring to
something else?
Yes, that's the problem. My library is keeping a stack of suspended generators that corresponds to the stack of "await" calls. When an exception occurs I rethrow it into the suspended generators until it gets caught (see the run function in bjouhier/galaxy/blob/master/lib/galaxy.js). This works fine and the exception propagates through the stack of "await" calls exactly as you would expect it to (I have a series of unit tests that check all sorts of try/catch/finally combinations and they work). So the propagation of exceptions through a stack of await calls works fine.
What I am lacking is a way to reconstruct the stack trace of await calls when an exception is caught. As the exception goes through my little run function when it bubbles up I could easily collect the stack frames of the suspended generators into the exception object. The only thing that's missing is an API to get stack frame information from a suspended generator.
My objective is not too much to get this await stack trace in the debugger, it is to be able to log it when the application is run without a debugger.
A generator object represents a computation which has been suspended. We have an API to resume this computation (next/throw). What's missing is an API to get information about this suspended computation (which function, where in the source).
Le 15/06/2013 11:18, Bruno Jouhier a écrit :
A generator object represents a computation which has been suspended. We have an API to resume this computation (next/throw). What's missing is an API to get information about this suspended computation (which function, where in the source).
As an aside, note that we already have this sort of problem today with event loop turns.
function schedule(){
if(Math.random() < 0.5)
throw new Error('whatever');
else
setTimeout(schedule, Math.random()*100);
}
setTimeout(schedule, (Math.random()*100) |0)
setTimeout(schedule, (Math.random()*100) |0)
There is no way to know how many scheduling happened before the first thrown error, nor whether the error comes from the first or second scheduling originally. And that's a dummy 5-line example. It gets worse when you have a sequence of different events (or promise resolutions) called at different times, adding other listeners, etc. Q solves that with long stack traces [1] (only for promises I believe). It might be worth looking into it.
Back to your problem, I worry that this kind of information (a call to get the stack trace of where the generator last yielding with line number) may partially break encapsulation which wouldn't be good for security. I'm thinking of something like:
(exports => {
var someCrucialInfo = // boolean
exports.bla = function*(){
if(someCrucialInfo){
doX();
yield 27;
}
else{
doY();
yield 27;
}
}
})(this);
var v = this.bla();
var l = getLastBlaYieldLine(bla);
// given l, the encapsulated value of someCrucialInfo can be inferred
cc'ing MarkM to get his eyes on it as I don't feel qualified to assess the gravity. oh... and we have the source code of functions by default with Function#toString. This sort of inference can happen in regular error stack traces too, but requires for a function to throw which happens only if it's supposed to, not at the caller's will. Though I realize now that anyone holding a reference to a generator can force an error being thrown and, if uncaught, it generates a stack trace leaking the line number even without the API you're asking for (but that would work only once)
I see this as being similar to asking for a portable "stack" property in Error objects. I don't know if it is actually mandated by ES6 but it looks like all major JS engines support it now.
I believe the state of TC39 on Error#stack can be found at strawman:error_stack
David
[1] kriskowal/q#long
On Sat, Jun 15, 2013 at 10:17 AM, David Bruant <bruant.d at gmail.com> wrote:
cc'ing MarkM to get his eyes on it as I don't feel qualified to assess the gravity.
Thanks for calling my attention to this thread. Generators and multiple turns aside (see below), we've been over the security issue before but a recap is useful. The stack trace information cannot be publicly accessible from the Error object because it violates encapsulation. (Historical note: When E entered its first security review, it had this vulnerability. Thanks to David Wagner for catching it.) Instead, some sort of rights amplification is required. One possible interface is a privileged function, like the getStack function at code.google.com/p/google-caja/source/browse/trunk/src/com/google/caja/ses/debug.js#221, where getStack(err) returns the stacktrace. If you don't have the getStack function, you can't get the stacktrace. getStack is implemented using a WeakMap associating Errors with stacktraces.
Another approach, if strawman:relationships are adopted in ES7, is to have a relationship associating Error objects with stacktraces. The full read/write relationship would be internal to the implementation, but a relationship representing a readonly facet of that relationship, say @stacktrace, could be made available in the same privileged manner as getStack above. Then err at stacktrace would return the same stacktrace that getStack(err) would return above.
If relationships don't happen in ES7 (it's too late for ES6) and private symbols do, then @stacktrace could be a private symbol.
Multiple turn distributed debugging is indeed the next frontier, and one that will become ever more urgent as promises continue to catch on. I participated in a cool project, Causeway, that explored some of this territory well www.hpl.hp.com/techreports/2009/HPL-2009-78.html, code.google.com/p/causeway
Although the project has since been abandoned, its trace log format is a good place to start, in order to be able to support something like this in the future: wiki.erights.org/wiki/Causeway_Platform_Developer, wiki.erights.org/wiki/Causeway_Platform_Developer:_Ajax, wiki.erights.org/wiki/Causeway_Platform_Developer:_Promises
code.google.com/p/google-caja/source/browse/trunk/src/com/google/caja/ses/debug.js also provides some support for the Causeway log format. See getCWStack.
See also crpit.com/confpapers/CRPITV135Murray.pdf and infoscience.epfl.ch/record/181543/files/EPFL_TH5533.pdf for other interesting approaches to extended debugging
As for generators specifically, it seems to me that whatever rights amplification operation one applies to an Error object to get a stack trace, one should also be able to apply to a generator instance to get a 1-level stacktrace showing where it is currently suspended.
On Sat, Jun 15, 2013 at 6:56 PM, Mark S. Miller <erights at google.com> wrote:
As for generators specifically, it seems to me that whatever rights amplification operation one applies to an Error object to get a stack trace, one should also be able to apply to a generator instance to get a 1-level stacktrace showing where it is currently suspended.
Although these 1-level generator stacks aren't part of a larger synchronous stack, they are part of the overall causality graph that Causeway tries to trace. It would be interesting to think about where these would go in a Causeway log, or if the Causeway log format would need to be extended to accommodate this.
On Sat 15 Jun 2013 19:17, David Bruant <bruant.d at gmail.com> writes:
Le 15/06/2013 11:18, Bruno Jouhier a écrit :
A generator object represents a computation which has been suspended. We have an API to resume this computation (next/throw). What's missing is an API to get information about this suspended computation (which function, where in the source).
Back to your problem, I worry that this kind of information (a call to get the stack trace of where the generator last yielding with line number) may partially break encapsulation which wouldn't be good for security.
Given the plans to have a Reflect module, I would put it there, in that case. Presumably people that can get at Reflect can get at anything.
Andy
No, actually we've carefully designed the Reflect module so that it grants no special privilege.
I'm the author of IMVU's task system, which is very similar to what Bruno describes except with Python Futures and Python generators. Python provides stack traces as a linked list of activation records from the thrower to the catcher. On the other hand, Error().stack in most JavaScript implementations returns a complete stack trace all the way up to the main event loop. Python's linked list stack traces allows stitching them together so that if async generator A yields to async generator B, and B throws, both A and B can be made to appear in the stack together. Finally, and this is key, Python attaches stack traces to the act of throwing and not to the exception object itself.
I'm not familiar with the security or encapsulation issues with Error().stack (is it that bad if Chrome and Firefox appear to support the .stack property?), but I would like to express my support for being able to stitch together stack traces on the open web so that if something goes wrong developers have a shot at diagnosing what happened. Native application developers have dozens of fantastic options for phone-home crash reporting, but on the web the best you can hope for is that your customer's browser supports Error#stack and window.onerror provides the thrown exception.
Thanks, Chad
p.s. E was hugely influential in my thinking about concurrency and led me to build the task system at IMVU in the first place, so
On Mon, Jun 17, 2013 at 11:33 AM, Chad Austin <chad at imvu.com> wrote:
I'm the author of IMVU's task system, which is very similar to what Bruno describes except with Python Futures and Python generators. Python provides stack traces as a linked list of activation records from the thrower to the catcher. On the other hand, Error().stack in most JavaScript implementations returns a complete stack trace all the way up to the main event loop. Python's linked list stack traces allows stitching them together so that if async generator A yields to async generator B, and B throws, both A and B can be made to appear in the stack together. Finally, and this is key, Python attaches stack traces to the act of throwing and not to the exception object itself.
I hope we can stitch together stack traces in JS as well. This is desperately needed to make asynchronous and distributed code practical, more on a par with local synchronous code. Please follow the Causeway links in the email you're responding to.
The Causeway work was done in the context of event-loops and promises, but not generators. We should explore how to extend its stack stitching approach to include generators.
I'm not familiar with the security or encapsulation issues with Error().stack (is it that bad if Chrome and Firefox appear to support the .stack property?)
Yes. Fortunately on Chrome we can effectively remove it.
, but I would like to express my support for being able to stitch together stack traces on the open web so that if something goes wrong developers have a shot at diagnosing what happened. Native application developers have dozens of fantastic options for phone-home crash reporting, but on the web the best you can hope for is that your customer's browser supports Error#stack and window.onerror provides the thrown exception.
The built-in window.onerror can have/use the rights amplification powers of the getStack or @stacktrace in my previous message. This would not violate any security constraint, since untrusted scripts within a frame are denied direct access to their true window and global scope anyway.
p.s. E was hugely influential in my thinking about concurrency and led me to build the task system at IMVU in the first place, so thanks!
You are very welcome! Always nice to hear.
For info, I've created a small node.js add-on that exposes a getStackFrame(generator) function: bjouhier/galaxy-stack.
With this I could implement long stacktrace in my "galaxy" module;
Temporary hack but it's there for experiments until the API gets sorted out.
Excellent. This looks like the right kind of low level enabler.
Bruno: cool, maybe we can build on this in SpiderMonkey too. Cc'ing Jason and Dave.
Cc'ing Arv in case he is reading es-discuss only sporadically.
Bruno, looks very useful. We should make sure that whatever stack API we propose also allows you to get the stack frame for generator.
Thanks all.
A few words for the API: I introduced two calls because I wanted to have minimal overhead to capture the stack. So I'm only capturing the continuation value (an integer). I'm using the other call (GetStackFrame) to format the stack trace. The disadvantage of this approach is that I have to keep a handle on the stack of generators in the error object. So the generators won't be garbage collected until the error object goes away.
An alternative would be to capture the stack frame directly with GetStackFrame(gen). The capture operation would be a little heavier but this would lead to a simpler API.
I let you sort this out because I'm not familiar with the best practices in this area.
Bruno
2013/6/22 Erik Arvidsson <erik.arvidsson at gmail.com>
I'm using ES6 generators to implement a little async/await library and I'm quite pleased with the result so far but I'm lacking one API: a function to get stack information from a generator object. Ideally it would return the name of the current generator function, the filename and the line number where it last yielded.
If I had this API I 'd be able to provide a complete trace of the stack of await calls when an exception is caught.
I raised the subject in the v8-users mailing list and Andy Wingo replied that this could be an interesting addition to the Reflect module and that I should post here.
I've got another less important suggestion regarding syntax but I'll post separately.