Error stack strawman
I am very interested. See the extended Causeway stack format in google/caja/blob/master/src/com/google/caja/ses/debug.js and the logic for scraping this information from the various stacktraces available on today's major platforms.
The "extended" is about the multiple source position layers present when an "eval" in one place evals a string that, at some position in the string, does the call. Both FF and Chrome provide these nested positions, but differently.
See the attachments for the information successfully scraped when visiting < rawgit.com/tvcutsem/es-lab/master/src/ses/contract.html> on each of
the platforms, converted to Causeway extended stack trace format, and then re-rendered in a common textual format. As you see, even with all the work at canonicalizing this information, it still differs substantially between platforms.
On Mon, Jan 11, 2016 at 10:05 PM, Mark S. Miller <erights at google.com> wrote:
I am very interested. See the extended Causeway stack format in google/caja/blob/master/src/com/google/caja/ses/debug.js
From that page, the JSON Schema-ish in an ad-hoc notation:
stacktrace ::= {calls: [frame*]}; frame ::= {name: functionName, source: source, span: [[startLine, startCol?], [endLine, endCol?]?]}; functionName ::= STRING; startLine, startCol, endLine, endCol ::= INTEGER source ::= STRING | frame;
and the logic for scraping this information from the various stacktraces
available on today's major platforms.
The "extended" is about the multiple source position layers present when an "eval" in one place evals a string that, at some position in the string, does the call. Both FF and Chrome provide these nested positions, but differently.
The nesting is represented as the recursion through "frame" in the above grammar.
Sorry for the delay, thank you for that response it was very useful.
Do you think it would make sense to open an API for Error stack that would be purely based upon user agent implementation? I'm not familiar with the internals of the traces but I have seen comments mentioning performance issues of complete parity across browsers due to their implementation differences.
It would however still be useful to standardise: lineNumber, columnNumber and fileName.
Perhaps that would be a simpler first step?
Thanks Jonathan
On Tue, Jan 19, 2016 at 6:35 PM, Jonathan Kingston <jonathan at jooped.co.uk>
wrote:
Hi Mark,
Sorry for the delay, thank you for that response it was very useful.
Do you think it would make sense to open an API for Error stack that would be purely based upon user agent implementation? I'm not familiar with the internals of the traces but I have seen comments mentioning performance issues of complete parity across browsers due to their implementation differences.
It would however still be useful to standardise: lineNumber, columnNumber and fileName.
Perhaps that would be a simpler first step?
I think I was too discouraging in my earlier email. I actually meant to be encouraging, though as always including weaknesses, cautions, requirements, and caveats. I did not mean to scare anyone off. The downside of starting with too small a part of the problem in isolation is that we are likely to paint ourselves into a corner that we cannot fix later. I think the useful thing to do is to standardize the extended Causeway stack trace format:
stacktrace ::= {calls: [frame*]};
frame ::= {name: functionName,
source: source,
span: [[startLine, startCol?], [endLine, endCol?]?]};
functionName ::= STRING;
startLine, startCol, endLine, endCol ::= INTEGER
source ::= STRING | frame;
that I showed in the previous message, as well as the APIs derived from < google/caja/blob/master/src/com/google/caja/ses/debug.js
:
System.getStack(error) -> extended-causeway-stack | undefined
Reflect.stackString(extended-causeway-stack) -> stacktrace-string
System.getStackString = error =>
Reflect.stackString(System.getStack(error));
This places the privilege-bearing operations on System, as do other proposals such as
System.global
System.defaultLoader
System.WeakRef
The only operation above that's fundamental is System.getStack, which, given an error object encapsulating stack trace info, returns that stack trace info in extended Causeway stacktrace format. debug.js shows how to polyfill a decent approximation of this by scraping this info from the divergent stack traces provided by today's major platforms.
Reflect.stackString renders an extended Causeway stack into a multiline string, one per stackframe. It is fully non-privileged and polyfillable, but I propose that the std provide it, in order to harmonize the current crazy divergence of these formats between platforms. The one provided by debug.js uses a vaguely v8-like concrete syntax, but that is merely a starting point of discussion. Anything we can all agree on, that reliably represents all the information from the extended Causeway stacktrace info, would be great. We put stackString on Reflect because by itself it provides no privilege.
System.getStackString is a mere convenience, composing together stackString with getStack. I expect that most callers will just call this and ignore the two lower level ones.
The hard part will be the escaping conventions needed in stackString, as this goes beyond any precedent set by current platforms. The scraping that the polyfill in debug.js does is grossly unreliable because it relies on characters such as "(" and "@" to recognize what part of the existing stackframe strings are filenames and function-or-method names. However, these characters can be present in filenames, function names, and method names, in which case this scraping will fail. Neither the current scraping nor rendering code in debug.js does anything yet about this need to escape significant characters.
Fortunately, once a reliable getStack is provided directly (instead of a polyfill scraping non-escaped strings), consumers of the extended Causeway format it returns will have reliable access to this structured data, without any more need to do their own scraping.
The other hard but necessary part is that any such standard should be coordinated -- ideally co-proposed -- with a proposed standard sourcemap < docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/edit?hl=en_US&pli=1&pli=1>,
which also needs to codify the meaning of at least filename and start/end line/col.
A transpiler translating esN+1 to esN, producing an esN to esN+1 sourcemap, should also provide to the esN+1 code a virtualized System.getStack and System.getStackString that maps back through that sourcemap, so that the esN+1 code sees stack trace info in terms of positions in its own esN+1 sources.
What all of this needs is someone who will take the lead writing a proposal. It is important, but I do not have the time. If someone does volunteer to take the lead, I will happily help, advise, ensure that all requirements are met, and to champion this proposal on the committee. Volunteers? Jonathan?
On 1/20/16 10:10 AM, Mark S. Miller wrote:
I think the useful thing to do is to standardize the extended Causeway stack trace format:
This format doesn't have an obvious way to represent "async stacks" [1], short of hacks with the "functionName" or the "source", right?
Gecko right now represents these precisely by hacking the function name, but that's in the context of backwards compat with existing stack-processing tools, and a string-only stack representation. It seems to me that moving to an object format should allow us to come up with a cleaner way to do this.
-Boris
[1] To see what I mean, try running this in a Firefox nightly:
<!DOCTYPE HTML> <script> function g() { console.log(new Error().stack); } function f() { setTimeout(g, 0); } f(); </script>
the output you will get is:
g at filename:4:17 setTimeout handler*f at filename:7:5 @filename:9:3
As in, the stack indicates the sequence of async steps that got you to the current source location.
It seems to me that moving to an object format should allow us to come up
with a cleaner way to do this.
Hello Boris,
I am the author of Causeway, a distributed debugger for programs built on the communicating event loops concurrency model. Causeway is the origin of the object format MarkM mentioned in his email.
To see what I mean, try running this in a Firefox nightly:
I ran a couple of simple test cases in FF nightly (1/20) and Chrome (latest general release).
Your code snippet using setTimeout and the same code using promises.
Breaking on console.log: FF debugger showed single turn call stack Chrome debugger with Async selected showed deep stack Console output: FF showed deep stack (as expected) Chrome showed single turn
So, the browsers do things differently but the output was as expected.
the output you will get is:
g at filename:4:17 setTimeout handler*f at filename:7:5 @filename:9:3
As in, the stack indicates the sequence of async steps that got you to the current source location.
This [object] format doesn't have an obvious way to represent "async
stacks" [1], short of hacks with the "functionName" or the "source", right?
You are correct. The extended stack trace format in MarkM's message is not enough to represent async stacks. The full Causeway format is more than what's needed for the async stacks you are asking about. Take a look at the attached screenshot or visit < rawgit.com/cocoonfx/causeway/master/src/js/com/teleometry/causalityGrid.html>[1]
and click on the red box in the lower right corner.
You can easily answer "how did I get here?". If tracking down a bug the graphical iconic view (happened-before relation) indicates events that might have contributed to the buggy state at the selected event. Dimmed events can be safely ignored.
If you start with the highlighted event in the outline view, and move up the message-order tree, you get essentially the deep stack you are asking about, stitching together the per-turn stacks across asynchronous causality. However, there's a crucial difference. Your deep stack is per vat (where a separate worker would be considered a separate vat). The deep stack you see here follows asynchronous causality as it flows across worker boundaries.
Causeway paper: www.hpl.hp.com/techreports/2009/HPL-2009-78.html
project: cocoonfx/causeway
It would be great to have browser support for such stitching together of intra-vat (page/worker) and cross-worker causality. Do the Causeway object formats wiki.erights.org/wiki/Causeway_Platform_Developer for
representing these look like an interesting start?
[1] Due to differences in cross-origin security under redirects, this page does not yet work on Safari, but does on FF, Chrome, and Edge.
On 1/22/16 4:51 PM, Terry Stanley wrote:
Breaking on console.log: FF debugger showed single turn call stack Chrome debugger with Async selected showed deep stack Console output: FF showed deep stack (as expected) Chrome showed single turn
Seems like people are incrementally introducing this stuff or just have bugs. :(
You are correct. The extended stack trace format in MarkM's message is not enough to represent async stacks.
OK, it's not just me. All I'm saying is we should go ahead and use a format that can represent them.
The full Causeway format is more than what's needed for the async stacks you are asking about.
Excellent. That reduces the problem to using a larger subset of the Causeway format, right?
The deep stack you see here follows asynchronous causality as it flows across worker boundaries.
That's pretty nifty. Being able to represent that too would be nice.
It would be great to have browser support for such stitching together of intra-vat (page/worker) and cross-worker causality. Do the Causeway object formats wiki.erights.org/wiki/Causeway_Platform_Developer for representing these look like an interesting start?
I haven't had a chance to study them in detail, but at first glance yes.
I believe a lot of additional work needs to be done for this proposal to make it works across realm. Consider the invocation chain:alpha, alpha0 (in realm A)beta, beta0 (in realm B)alpha -> alpha0 -> beta -> beta0 -> alpha ->alpha0 -> beta -> beta0 -> throws
Here comes the question. What should be included in the stack and what shouldn't?Same thing applies when calling native function. Such thing is severely lacking in current proposal. From: jonathan at jooped.co.uk Date: Tue, 12 Jan 2016 03:10:12 +0000 Subject: Error stack strawman To: es-discuss at mozilla.org
Hey all, Does anyone know if there is an active interest in revisiting the Error.stack strawman proposal? strawman:error_stack
I ask because CSP wanted an algorithm for getting the non standard properties defined in Error objects: w3c.github.io/webappsec-csp/#issue-f447ede5
A simpler start may be possible in standardising just: lineNumber, columnNumber and fileName
Kind Jonathan
On 1/25/16 1:55 PM, Gary Guo wrote:
I believe a lot of additional work needs to be done for this proposal to make it works across realm.
Consider the invocation chain: alpha, alpha0 (in realm A) beta, beta0 (in realm B) alpha -> alpha0 -> beta -> beta0 -> alpha ->alpha0 -> beta -> beta0 -> throws
Here comes the question. What should be included in the stack and what shouldn't?
What browsers will do is basically include everything, with some caveats.
The main caveat I'm aware of is that when the stack crosses between realms which have different effective script origins Gecko will sanitize the script view of the stack to only include the bits that the script asking for the stack is allowed to know about. We do record the entire stack, and can show the entire thing in devtools if we want to.
In practice, such stacks are quite hard to produce on the web; they require changing document.domain partway through the script execution.
Same thing applies when calling native function.
Yeah, this needs some thought. Browsers aren't even terribly consistent with each other or with themselves here. A imple example:
<script> var div = document.createElement("div"); div.onclick = function f() { console.log(new Error().stack); } div.dispatchEvent(new Event("click")); </script> <script> function g() { console.log(new Error().stack); } [0].map(g); </script>
this shows the following in Firefox:
f at filename:4:13 @filename:6:1
and
g at filename:10:17 @filename:12:3
Whereas in Chrome it shows:
at HTMLDivElement.f (filename1:4) at filename1:6
and
at g (filename2:10:17) at Array.map (native) at filename2:12:7
where filenam1 and filename2 are actually different: the latter is an absolute URI but the former is just the filename part of the URI...
Safari does:
f at filename:4:22 dispatchEvent@[native code] global code at filename:6:18
and
g at filename:10:26 map@[native code] global code at filename:12:10
which is what I was sort of hoping to see, possibly with better names for the built-ins (e.g. "Array.prototype.map" and "Node.prototype.dispatchEvent" or something).
One thing needs to be considered as well. What should be displayed for tail calls?
Just want to keep this going as I think mostly people agree it is worthy as an addition just the details are difficult etc.
@Boris does the main difficulties hinge on document.domain? There are discussions again in WebAppSec to deprecate that so perhaps the trace could also not work if document.domain has been changed.
On 2/10/16 4:52 AM, Jonathan Kingston wrote:
@Boris does the main difficulties hinge on document.domain?
Main difficulties with what?
Is implementing stack traces cross browser mostly difficult because of the use of document.domain or are there other issues blocking?
On 2/10/16 7:12 PM, Jonathan Kingston wrote:
Is implementing stack traces cross browser mostly difficult because of the use of document.domain or are there other issues blocking?
Ah, you mean in terms of interop?
I expect document.domain and built-ins to be the main thing that browsers will end up disagreeing if they just implement this willy-nilly; everything else is reasonably straightforward.
Deprecating document.domain is a noble goal, but given that sites that many people use (e.g. Facebook) make use of it, it's a pretty long-term goal at best...
Yeah interop issues of the output. (Sorry I'm not being clear) It seems the interop has been the main reason this has stalled as a proposal all these years.
Could the trace output just finish early at these origin boundaries perhaps rather than try and implement cross browser? With the right API this could be added in later if there was agreement.
The discussion at the moment just hinges around dropping it for new API features rather than break Facebook etc. w3c/webappsec-secure-contexts#10
On 2/10/16 9:47 PM, Jonathan Kingston wrote:
Could the trace output just finish early at these origin boundaries
The problem is that some browsers don't have origin boundaries at all in places where others do.
[+es-discuss] resending with es-discuss included.
The strawman looks very old, so I've created a new one.
Repo: nbdd0121/es-error-stack
I've collected many information about current implementation from IE, Edge, Chrome and Firefox, but missing Safari's. Many thanks if some one can collect these info and create a pull request.
I haven't write anything for API part, as you will see from the "concerns" part, there are many edge cases to be considered: cross-realm, native, global, eval, new Function, anonymous and tail call. All of these need to be resolved before we can trying to design an object representation of stack frame.
Personally I suggest "(global code)" for global, "(eval code)" for eval, "(Function code)" for new Function, "(anonymous function)" for anonymous function/lambda. For native call, we can simply replace filename & line & column by "(native)". For tail call I suggest add "(tail)" some where. I also suggest adding "(other realm)" or something alike to indicate realm boundary is crossed.
For object representation, I hope something like
{
name: 'string', // (global code), etc for special case, with parenthesis
source: 'url', // (native) for native code, with parenthesis
line: 'integer',
column: 'integer',
isTail: 'boolean'
}
And null entry indicating crossing realm. BTW, shall we add reference to function in the object representation?
Gary Guo
On Wed, Feb 17, 2016 at 4:19 PM, Gary Guo <nbdd0121 at hotmail.com> wrote:
The strawman looks very old, so I've created a new one.
Repo: nbdd0121/es-error-stack
I've collected many information about current implementation from IE, Edge, Chrome and Firefox, but missing Safari's. Many thanks if some one can collect these info and create a pull request.
I haven't write anything for API part, as you will see from the "concerns" part, there are many edge cases to be considered: cross-realm, native, global, eval, new Function, anonymous and tail call. All of these need to be resolved before we can trying to design an object representation of stack frame.
Personally I suggest "(global code)" for global, "(eval code)" for eval, "(Function code)" for new Function, "(anonymous function)" for anonymous function/lambda. For native call, we can simply replace filename & line & column by "(native)". For tail call I suggest add "(tail)" some where. I also suggest adding "(other realm)" or something alike to indicate realm boundary is crossed.
For object representation, I hope something like
{ name: 'string', // (global code), etc for special case, with parenthesis source: 'url', // (native) for native code, with parenthesis line: 'integer', column: 'integer', isTail: 'boolean' }
Unless the object representation is primary, we will need to agree on comprehensive escaping rules, and corresponding parsing rules, so that these stack strings can be unambiguously scraped even when file names and function names contain parens, slashes, angle brackets, at-signs, spaces, etc. Therefore, we should focus on the object representation first.
Your object representation above looks like a good start. It is similar to the extended Causeway stack format I mentioned earlier
stacktrace ::= {calls: [frame*]};
frame ::= {name: functionName,
source: source,
span: [[startLine,startCol?],[endLine,endCol?]?]};
functionName ::= STRING;
startLine, startCol, endLine, endCol ::= INTEGER
source ::= STRING | frame;
with the following differences:
-
You added an isTail. This is probably a good thing. I'd like to understand better what you have in mind.
-
Rather than have a single "span" property with a nested array of numbers as value, you define separate line and column property names. As long as we represent all that we need unambiguously, I'm indifferent to minor surface syntax differences.
-
Causeway's format has room for both start(line,col) and end(line,col). The format must include room for this, and I would hope any future standard would mandate that they be included. Such span information makes a huge usability improvement in reporting diagnostics.
-
The extended Causeway "source" field could be either a string as with your's, or a nested frame. This is necessary to preserve the information currently provided on both FF and Chrome of the nested positions in a single frame, when a call happens at position X in an eval string that was evaled by an eval call at position Y. (That is what the "extended" means. Causeway originally only has strings as the value of their "source" property.)
The proposed[1] API is:
System.getStack(err) -> stack-representation
Reflect.stackString(stack-representation) -> stack-string
System.getStackString(err) -> stack-string
where getStackString is just the obvious composition of getStack and stackString.
And null entry indicating crossing realm. BTW, shall we add reference to function in the object representation?
What do you mean by "reference to" above?
[1] Hopefully tc39/ecma262#395 will resolve in time that none of these need to be rooted in globals.
- isTail will be set when the frame indicates a frame created by tail call instead of normal function call. Caller's frame is already removed so we need some indication for that to help debugging.
- For span, I put only one pair of line/column there as it is the common implementation, but I agree that a starting position and a ending one is useful.
- For source, nested frame could be useful but it is not implemented by all implementations, and in fact we need an extra field to distinguish eval and new Function.
- By reference to function, I mean that shall we be able to retrieve the function object from the frame?
- I wonder if putting special cases in (), such as (native) will cause any problem. No one will have a file called "(native)" in reality, isn't it?
Gary Guo
On Wed, Feb 17, 2016 at 5:36 PM, Gary Guo <nbdd0121 at hotmail.com> wrote:
- isTail will be set when the frame indicates a frame created by tail call instead of normal function call. Caller's frame is already removed so we need some indication for that to help debugging.
Nice
For span, I put only one pair of line/column there as it is the common implementation, but I agree that a starting position and a ending one is useful.
For source, nested frame could be useful but it is not implemented by all implementations, and in fact we need an extra field to distinguish eval and new Function.
For eval vs Function (vs GeneratorFunction, vs AsyncFunction, etc), doesn't the name inside the nested frame already deal with that?
- By reference to function, I mean that shall we be able to retrieve the function object from the frame?
No, absolutely not. The stack rep should provide only info, not access.
- I wonder if putting special cases in (), such as (native) will cause any problem. No one will have a file called "(native)" in reality, isn't it?
If you do this, they will ;)
Mark Knichel has a lot of information regarding stacks and error handling in various browsers here: mknichel/javascript
On 18 February 2016 at 02:36, Gary Guo <nbdd0121 at hotmail.com> wrote:
- isTail will be set when the frame indicates a frame created by tail call instead of normal function call. Caller's frame is already removed so we need some indication for that to help debugging.
This would be fairly difficult to support by implementations. In V8, for example, we currently have no way of reconstructing that information, nor would it be easy or cheap to add that. A frame is created by the callee, but that does not know how it got called. Funnelling through that information would effectively require a hidden extra argument to every call.
Andreas <rossberg at google.com> wrote:
This would be fairly difficult to support by implementations. In V8, for example, we currently have no way of reconstructing that information, nor would it be easy or cheap to add that. A frame is created by the callee, but that does not know how it got called. Funnelling through that information would effectively require a hidden extra argument to every call.
Placing a boolean flag theoretically should not introduce too much overhead. If we are not going to indicate tail call some way, debugging might be extremely difficult, and the stack result might be making no sense at all.
As for how to distinguish between special code source, such as native, if tc39/ecma262#395 is resolved, we can use the built-in module name for the "source" field, and hopefully we can reserve some special built-in module name for native code, cross realm code, etc, and then we can have a unified representation of all frames.
On Thu, Feb 18, 2016 at 6:13 PM, Gary Guo <nbdd0121 at hotmail.com> wrote:
Andreas <rossberg at google.com> wrote:
This would be fairly difficult to support by implementations. In V8, for example, we currently have no way of reconstructing that information, nor would it be easy or cheap to add that. A frame is created by the callee, but that does not know how it got called. Funnelling through that information would effectively require a hidden extra argument to every call.
Placing a boolean flag theoretically should not introduce too much overhead. If we are not going to indicate tail call some way, debugging might be extremely difficult, and the stack result might be making no sense at all.
As for how to distinguish between special code source, such as native, if tc39/ecma262#395 is resolved, we can use the built-in module name for the "source" field, and hopefully we can reserve some special built-in module name for native code, cross realm code, etc, and then we can have a unified representation of all frames.
I like that! At least for the native case. I don't yet have coherent thoughts regarding the cross-realm issues.
On Thu, Feb 18, 2016 at 6:13 PM, Gary Guo <nbdd0121 at hotmail.com> wrote:
Andreas <rossberg at google.com> wrote:
This would be fairly difficult to support by implementations. In V8, for example, we currently have no way of reconstructing that information, nor would it be easy or cheap to add that. A frame is created by the callee, but that does not know how it got called. Funnelling through that information would effectively require a hidden extra argument to every call.
Placing a boolean flag theoretically should not introduce too much overhead. If we are not going to indicate tail call some way, debugging might be extremely difficult, and the stack result might be making no sense at all.
This smells wrong to me from a performance perspective and seems like a painful cross cutting concern. You would basically be trading the gains from tail-calls improves for complexity and cost everywhere else (an extra parameter for every call means a shorter stack). For instance, VM/native interfaces would always need a proxy to provide or be modified to carry this bit. If it were implemented as a extra parameter, this means at least one slot or stack push.
However, at the CPU level, it seems like you would be better pushing an return address for a special function that indicated the start of a sequence of tail calls. That way you trade only some complexity/performance for tail calls (an inspection of the last entry of the call stack) and some stack frame building complexity (to recognize this "special" frame).
It been a long time since I've worked directly with assembly, stack frames, and calling conventions and don't know how feasible this would be on modern/unique CPU/GPUs that are likely to have JS VMs.
On 19 February 2016 at 03:13, Gary Guo <nbdd0121 at hotmail.com> wrote:
Andreas <rossberg at google.com> wrote:
This would be fairly difficult to support by implementations. In V8, for example, we currently have no way of reconstructing that information, nor would it be easy or cheap to add that. A frame is created by the callee, but that does not know how it got called. Funnelling through that information would effectively require a hidden extra argument to every call.
Placing a boolean flag theoretically should not introduce too much overhead.
Practically, though, as I said, it would require passing an extra argument with every call, even if its information content is just a bit (or equivalently, having two entry points to every function, which would be "fun"). It also requires storing this bit in every stack frame, which, for V8 at least, would require allocating an additional word in every stack frame, because there is no other space left for it. So, it would likely be substantial overhead in calling conventions, in both time and space.
If we are not going to indicate tail call some way, debugging might be extremely difficult, and the stack result might be making no sense at all.
A tail call is a jump. Just like other jumps, you shouldn't expect their history to be visible in the continuation (which is what a stack trace represents). I agree that JS programmers might be surprised, and will have to relearn what they know. But wrt to debugging the situation is the same as for loops: you can't inspect their history either. (And functional programmers in fact see loops as just an ugly way to express self tail recursion. :) )
I'd argue that the real problem is that ES6 repurposed existing return syntax for tail calls. This would probably be much less of an issue if tail calls were a syntactically explicit feature. I wonder if we can still fix that...
On 19 February 2016 at 06:29, John Lenz <concavelenz at gmail.com> wrote:
However, at the CPU level, it seems like you would be better pushing an return address for a special function that indicated the start of a sequence of tail calls. That way you trade only some complexity/performance for tail calls (an inspection of the last entry of the call stack) and some stack frame building complexity (to recognize this "special" frame).
There is no way of knowing, neither statically nor dynamically, that you are at "the start of a sequence of tail calls". And doing it for every tail call would of course defeat tail calls.
On 19 February 2016 at 10:29, Andreas Rossberg <rossberg at google.com> wrote:
On 19 February 2016 at 06:29, John Lenz <concavelenz at gmail.com> wrote:
However, at the CPU level, it seems like you would be better pushing an return address for a special function that indicated the start of a sequence of tail calls. That way you trade only some complexity/performance for tail calls (an inspection of the last entry of the call stack) and some stack frame building complexity (to recognize this "special" frame).
There is no way of knowing, neither statically nor dynamically, that you are at "the start of a sequence of tail calls". And doing it for every tail call would of course defeat tail calls.
Or to put that differently: if there was, then the very problem wouldn't exist. ;)
Would it be possible to just define functions with an offset of exactly N bits before, with tail calls having a separate entry point than regular calls? Tail calls are already determined statically by spec, so you could add/subtract N bits to the code you generate for a function call in optimized code. And the offset could push a boolean to the stack with argc, etc., saying "this was called via tail call". When this function returns, tail call or not, this hidden boolean would be popped off.
This flag would be read during stack unwinding, and you could then add/subtract the offset if it's a tall call to get the real address of the function, and then do whatever.
As for interpreters, a single boolean flag can still be used. But that optimization would be a little more difficult.
(Yes, I saw that John Lenz suggested similar, but I'm not suggesting trying to detect this purely at runtime. That's the point of tail calls. I'm instead suggesting the entry point be different by a static offset, which can be determined statically.)
To be clear I meant, the VM knows statically whether to write code for a regular call or a tail call and that it is possible to do something clever by inspecting and rewriting the stack. Specifically, if you can inspect the stack for something that is inserted before performing the first tail call it is possible possible to distinguish. This is both complicated and potentially costly and very likely not worth the effort.
On 24 February 2016 at 02:01, John Lenz <concavelenz at gmail.com> wrote:
To be clear I meant, the VM knows statically whether to write code for a regular call or a tail call and that it is possible to do something clever by inspecting and rewriting the stack. Specifically, if you can inspect the stack for something that is inserted before performing the first tail call it is possible possible to distinguish. This is both complicated and potentially costly and very likely not worth the effort.
A function can only find out whether it is performing the "first" tail call if it can find out whether it was tail-called itself. So every tail call would have to inspect the stack to look for the special tail call frame, in order to decide whether it should push that special frame itself. So yes, that would make tail calls prohibitively expensive.
On 02/19/2016 01:26 AM, Andreas Rossberg wrote:
On 19 February 2016 at 03:13, Gary Guo <nbdd0121 at hotmail.com <mailto:nbdd0121 at hotmail.com>> wrote:
If we are not going to indicate tail call some way, debugging might be extremely difficult, and the stack result might be making no sense at all.A tail call is a jump. Just like other jumps, you shouldn't expect their history to be visible in the continuation (which is what a stack trace represents). I agree that JS programmers might be surprised, and will have to relearn what they know. But wrt to debugging the situation is the same as for loops: you can't inspect their history either. (And functional programmers in fact see loops as just an ugly way to express self tail recursion. :) )
To be even more pedantic: the stack trace isn't "the" continuation, it is one possible continuation. Other continuations are possible if you throw an exception. I guess you could say the stack trace plus the code allows you to statically derive the full set of possible continuations.
But I agree that it's worthwhile to remember the difference, since what is being requested for stacks really is a history, not a continuation. For example, it is desireable to encode "long stacks" or "async stacks" or whatever they're being called these days, where eg for an event handler you get the stack trace at the point the handler was installed. That is not a continuation, that is history. I would be very wary of mandating that full history be preserved, since it's easy for it to prevent optimizations or inadvertently leak details of the underlying implementation (tail calls, inlining, captured environments).
Does it work to specify something like "if and only if the information is available, it shall be encoded like this:..."? That can still leak information if not handled carefully, but at least it doesn't inhibit optimizations.
For a wild handwavy example of an information leak: say you do not include inlined calls in stack frames, and you only inline a call after the 10th invocation. Further assume that you self-host some JS feature. The caller can now learn something about how many times that self-hosted feature has been used. That feature might happen to be Math.something used only for processing non-latin1 characters in a password, or more likely just some feature used only if you are logged into a certain site. (Perhaps Error.stack is already specced to avoid this, by requiring all frames to be included whether inlined or not? Sorry, I don't know anything about it; I'm just posting to ask the question about what specifying stack formats encompasses.)
On Wed, Feb 24, 2016 at 11:40 AM, Steve Fink <sphink at gmail.com> wrote:
On 02/19/2016 01:26 AM, Andreas Rossberg wrote:
On 19 February 2016 at 03:13, Gary Guo <nbdd0121 at hotmail.com> wrote:
If we are not going to indicate tail call some way, debugging might be
extremely difficult, and the stack result might be making no sense at all.
A tail call is a jump. Just like other jumps, you shouldn't expect their history to be visible in the continuation (which is what a stack trace represents). I agree that JS programmers might be surprised, and will have to relearn what they know. But wrt to debugging the situation is the same as for loops: you can't inspect their history either. (And functional programmers in fact see loops as just an ugly way to express self tail recursion. :) )
To be even more pedantic: the stack trace isn't "the" continuation, it is one possible continuation. Other continuations are possible if you throw an exception. I guess you could say the stack trace plus the code allows you to statically derive the full set of possible continuations.
But I agree that it's worthwhile to remember the difference, since what is being requested for stacks really is a history, not a continuation. For example, it is desireable to encode "long stacks" or "async stacks" or whatever they're being called these days, where eg for an event handler you get the stack trace at the point the handler was installed. That is not a continuation, that is history. I would be very wary of mandating that full history be preserved, since it's easy for it to prevent optimizations or inadvertently leak details of the underlying implementation (tail calls, inlining, captured environments).
Does it work to specify something like "if and only if the information is available, it shall be encoded like this:..."? That can still leak information if not handled carefully, but at least it doesn't inhibit optimizations.
For a wild handwavy example of an information leak: say you do not include inlined calls in stack frames, and you only inline a call after the 10th invocation. Further assume that you self-host some JS feature. The caller can now learn something about how many times that self-hosted feature has been used. That feature might happen to be Math.something used only for processing non-latin1 characters in a password, or more likely just some feature used only if you are logged into a certain site. (Perhaps Error.stack is already specced to avoid this, by requiring all frames to be included whether inlined or not? Sorry, I don't know anything about it; I'm just posting to ask the question about what specifying stack formats encompasses.)
This information leak is not specific to tail-call variance. It is a problem with the whole error.stack interface < www.combex.com/papers/darpa-review/security-review.html#UniversalScope>.
Like makeWeakRef tc39/proposal-weakrefs, that is why
the getStack and getStackString functions won't be generally available to confined code, but must instead be virtualizable or deniable. Starting with KeyKOS, capability practitioners refer to this category as "closely held", as opposed to "ambient". Everything in the ES6 standard itself is ambient, since we postponed spec'ing anything that needs to be closely held. Now is that time tc39/ecma262#395.
Again, as with makeWeakRef, by treating getStack and getStackString as closely held, it is also less damaging[1] for them to leak implementation non-determinacy that may carry side channels or covert channels.
[1] It is less damaging when the leakage is limited to intra-realm, since one realm is not in a position to police another. For getStack and getStackString, this is yet another reason we need to think hard about cross-realm stacks. I, for one, have not yet done so.
Huh, weird. From the discussions that led to < claudepache/es-regexp-legacy-static-properties> being
proposed and the discussion on that issue, for me to take a consistent position, I must sadly concede the following:
We allowed a grossly leaky api (the RegExp statics, which leak non-local information from RegExp instances google/caja#531)
to be proposed for standardization as "normative optional", i.e., Annex B, and deletable. This way, and environment that deletes it still leaves a conforming system in its wake. Many people think they want error.stack because it seems to do something on all the browsers -- though their behaviors are tremendously different from each other. Nevertheless, whatever behavior we specify for getStackString, we could likewise spec a normative optional and deletable Error.prototype.stack accessor property, like Firefox's[1]. As with makeWeakRef and the leaky RegExp statics, we could specify the getter of this accessor only to work on Error objects from the same realm[2]. Since errors inherit from the Error.prototype of their own realm anyway, this does not impede normal usage. We need to impose the same restriction on getStack and getStackString anyway, for the same reasons.
FF demonstrates that it is already web compatible to provide error.stack only by providing a deletable Error.prototype.stack accessor.
On FF, my polyfill < tvcutsem/es-lab/blob/master/src/ses/debug.js> makes use
of the fact that Error.prototype.stack is an accessor, both to censor the property on initialization, be deleting it, and to reserve for itself the ability to inspect stacks, by utilizing the getter it took away. That is how it implements getStack without providing ambient access to the stack.
[1] For some reason the FF stack accessor has both getter and setter. I don't see any reason for a setter. [2] This solves only one of the cross-realm issue with stacks. It does nothing to address worries about cross-realm stacks.
On 2/24/16 4:30 PM, Mark S. Miller wrote:
[1] For some reason the FF stack accessor has both getter and setter. I don't see any reason for a setter.
We ran into code "in the wild" (more precisely, within our own test suite infrastructure, but we have no reason to believe this would not happen in the wild) where people were reading error.stack, doing some processing on it, and then setting error.stack on that same error, and expecting error.stack to be that thing after that. Clearly this Just Works with a value property, across all browsers at that point, so we made it work with the accessor too.
On 2/24/16 5:35 PM, Boris Zbarsky wrote:
Clearly this Just Works with a value property, across all browsers at that point, so we made it work with the accessor too.
Oh, and we made it work the way [Replaceable] stuff on window works: the setter defines a value property on the object which thereafter shadows the prototype's getter.
On Wed, Feb 24, 2016 at 2:37 PM, Boris Zbarsky <bzbarsky at mit.edu> wrote:
On 2/24/16 5:35 PM, Boris Zbarsky wrote:
Clearly this Just Works with a value property, across all browsers at that point, so we made it work with the accessor too.
Oh, and we made it work the way [Replaceable] stuff on window works: the setter defines a value property on the object which thereafter shadows the prototype's getter.
It is ugly so I would hope we don't need to preserve it. But if it is needed for web compat, as a normative-optional deletable accessor, I have no fundamental objection.
On 02/24/2016 01:30 PM, Mark S. Miller wrote:
[2] This solves only one of the cross-realm issue with stacks. It does nothing to address worries about cross-realm stacks.
We do have code in FF that handles cross-realm stacks, or at least a close moral equivalent to them. The stacks are stored internally as objects, and each frame records where it comes from, so a user will only see frames that it has privileges for. Obviously, once you convert to a string, you're past the point of control.
(Or at least, that's my understanding of what is going on. I'm not sure if that stuff is used for Error.stack yet.)
On 2/24/16 6:59 PM, Steve Fink wrote:
(Or at least, that's my understanding of what is going on. I'm not sure if that stuff is used for Error.stack yet.)
It is. See bugzilla.mozilla.org/show_bug.cgi?id=1038238
Things became more complicated when considering async & generators.
For async calls as mentioned before somewhere in the thread, only Firefox Nightly build includes histories of the frame, while all other browsers don't. It could be useful to include that in stack trace, but it will then be a special case to a general rule, so not all functions are treated as the same. Also, including this information can be costly.
Similar reasoning about usefulness can be extended to generators & async functions. Shall we include a copy of history frames at the time the function is called? (If we use FF's SavedFrame format, then in this case both parent and asyncParent will be set) Again, this will be costly as well.
In these two cases, it could be useful to developers if these information are available, but generally these will be costly to collect. Making these optional can be a choice, by simply not including unsupported fields in the returned object. I suppose isTailCall can be made optional as well.
On Thu, Feb 25, 2016 at 6:45 PM, Gary Guo <nbdd0121 at hotmail.com> wrote:
Things became more complicated when considering async & generators. For async calls as mentioned before somewhere in the thread, only Firefox Nightly build includes histories of the frame, while all other browsers don't. It could be useful to include that in stack trace, but it will then be a special case to a general rule, so not all functions are treated as the same. Also, including this information can be costly. Similar reasoning about usefulness can be extended to generators & async functions. Shall we include a copy of history frames at the time the function is called? (If we use FF's SavedFrame format, then in this case both
parentandasyncParentwill be set) Again, this will be costly as well. In these two cases, it could be useful to developers if these information are available, but generally these will be costly to collect. Making these optional can be a choice, by simply not including unsupported fields in the returned object. I supposeisTailCallcan be made optional as well.
All important issues. My current stance:
-
We should separate the issue of obtaining the shallow synchronous within-a-turn stacks from the gathering of multiple of these together into deep stacks or graphs of asynchronous causality.
-
The getStack and getStackString apis, and the corresponding normative optional deletable Error.prototype.stack accessor, should only provide shallow within-turn stacks.
In thinking about how Causeway stitches these separate shallow synchronous stacks into an overall picture of asynchronous causality, including deep stacks, I think the next steps are something like:
-
The stack object format should contain one additional thing -- a turn identifier, identifying the turn that stack occurred in. To avoid leaking unnecessary implementation details, say that this turn identifier is a large random number.
-
There be some way to declare that a certain scope of activity (realm?) is to be instrumented, i.e., that certain significant events generate notifications to some kind of logging apparatus. This issue already arises with onerror and unhandled rejection notifications.
-
When instrumented, an action in one turn that schedules something to happen in another turn also emits a log-notification containing:
- the stack of the scheduling action within the scheduling turn, including the turn identifier of the scheduling turn.
- the turn identifier of the scheduled turn.
When the scheduling turn and the scheduled turn both happen in the same vat, then stitching the shallow stacks together into a deep stack creates exactly the kinds of deep stack we see on FF. However, I purposely phrased that last bullet so that it also applies to inter-vat causality. For example, if instrumented worker A does a postMessage to instrumented worker B, the postMessage event in worker A logs, in A,
- the stack in A where A did the postMessage, including the identifier of the A turn in which this happened.
- the identifier of the turn in worker B where this message will be processed.
There is much more to be said and explored, but I think that would be a start. It naturally breaks up into two separable proposals -- the first two bullets and the rest.
Hey all,
Does anyone know if there is an active interest in revisiting the Error.stack strawman proposal?
strawman:error_stack
I ask because CSP wanted an algorithm for getting the non standard properties defined in Error objects: w3c.github.io/webappsec-csp/#issue-f447ede5
A simpler start may be possible in standardising just: lineNumber, columnNumber and fileName
Kind Jonathan