Reflect.parse (from RE:typeof null)
Maybe. It's too late for ES6 and different implementations have different concrete parse trees, I bet -- although perhaps they all agree on concrete, there's still the question of mapping to abstract. This needs more time to bake.
It's also a mouth to feed, not as much as a second built-in parser but it takes its maintenance toll feeding over time. More time needed to get implementor buy-in on this count.
Will Reflect.parse be standardized?
Maybe. It's too late for ES6 and different implementations have different concrete parse trees, I bet -- although perhaps they all agree on concrete, there's still the question of mapping to abstract. This needs more time to bake.
It's also a mouth to feed, not as much as a second built-in parser but it takes its maintenance toll feeding over time. More time needed to get implementor buy-in on this count.
In the absence of an ES standard for Reflect.parse, for reasons documented in this strawman page:
it appears that the SpiderMonkey AST has emerged as a de-facto standard.
There was just too much pressure for this not to happen, as one cannot even test a standalone parser without some way to represent its output. Parser implementors and parser users needed a target spec, and the SpiderMonkey AST was the best known, and the only one with engine-maker buy-in.
So, the concerns about not forcing different vendors to support a standard AST API have led to one vendor "owning" the standard..
Given the speed of parsers like esprima, and the performance and complexity implications of other engine's internal ASTs not matching the SpiderMonkey AST, it does seem right not to force engine makers to implement Reflect.parse.
The question is: how to move this de-facto standard forward?
For instance, there is an esprima branch implementing modules, does it have to wait for someone (Mozilla?) to extend the AST spec? Or does the feature get merged into the master, and everyone has to change when Mozilla implements? Similarly for other parts of the evolving ES6 spec.
Or, there are issues emerging from the increasing use of the AST API - will everyone have to apply to Mozilla for changing the spec? Will Mozilla do that if it means changing their implementations? Do they now have to pay the maintenance toll mentioned above?
It might be useful to decouple the SpiderMonkey AST API from its initial implementation, and to create a forum in which all users of the AST API spec (parser implementors, parser users) can discuss and influence the development. Then the question of whether or when Mozilla or others implement which parts of the spec would be separate from what the spec is. Such a forum and its spec could then be mentioned in the ES standard as an appendix, without including the AST spec itself in the ES standard.
Claus
On Thu, May 10, 2012 at 8:28 AM, Claus Reinke <claus.reinke at talk21.com> wrote:
Will Reflect.parse be standardized?
Maybe. It's too late for ES6 and different implementations have different concrete parse trees, I bet -- although perhaps they all agree on concrete, there's still the question of mapping to abstract. This needs more time to bake.
It's also a mouth to feed, not as much as a second built-in parser but it takes its maintenance toll feeding over time. More time needed to get implementor buy-in on this count.
In the absence of an ES standard for Reflect.parse, for reasons documented in this strawman page:
it appears that the SpiderMonkey AST has emerged as a de-facto standard.
Really? I guess there are many more users of Uglify and Esprima is growing fast. Personally I never heard of any non-Mozilla uses of the SpiderMonkey API, but perhaps you can point to some.
There was just too much pressure for this not to happen, as one cannot even test a standalone parser without some way to represent its output. Parser implementors and parser users needed a target spec, and the SpiderMonkey AST was the best known, and the only one with engine-maker buy-in.
So, the concerns about not forcing different vendors to support a standard AST API have led to one vendor "owning" the standard..
Given the speed of parsers like esprima, and the performance and complexity implications of other engine's internal ASTs not matching the SpiderMonkey AST, it does seem right not to force engine makers to implement Reflect.parse.
The question is: how to move this de-facto standard forward?
Extend esprima or compete with it.
For instance, there is an esprima branch implementing modules, does it have to wait for someone (Mozilla?) to extend the AST spec?
No, that's the beauty of non-standards.
Or does the feature get merged into the master, and everyone has to change when Mozilla implements?
Each project gets to make their own decisions.
Similarly for other parts of the evolving ES6 spec.
Or, there are issues emerging from the increasing use of the AST API - will everyone have to apply to Mozilla for changing the spec?
No, that is the beauty of de-facto standards.
Will Mozilla do that if it means changing their implementations?
We don't need to care.
Do they now have to pay the maintenance toll mentioned above?
It's their choice.
I think you are putting the cart in front of the horse. We don't have enough experience with JS dev tools to know which AST is best. We need to build some. We need some competition between esprima, uglify etc. Then we can make informed decisions about AST API. Really the AST API is not the biggest barrier to tools. Just pick one. I tried uglify, it's fast but the API takes some getting used to. I tried Traceur (because I know the traceur maintainer) and the API is wonderful, even if the parser is not as fast. However it supports error handling well. The Esprima API (based on the SpiderMonkey API) looks very similar to Traceur's and it's parser has a reputation of being fast but no error recovery. What is your experience?
Also: I think this might be a better topic for the js-tools groups.google.com/group/js-tools?pli=1
jjb
On May 10, 2012, at 8:28 AM, Claus Reinke wrote:
Will Reflect.parse be standardized?
Maybe. It's too late for ES6 and different implementations have different concrete parse trees, I bet -- although perhaps they all agree on concrete, there's still the question of mapping to abstract. This needs more time to bake. It's also a mouth to feed, not as much as a second built-in parser but it takes its maintenance toll feeding over time. More time needed to get implementor buy-in on this count.
In the absence of an ES standard for Reflect.parse, for reasons documented in this strawman page:
it appears that the SpiderMonkey AST has emerged as a de-facto standard.
Does any other browser provide a built-in API that produces that AST? If not, it isn't a de-facto browser standard. It may be a de-facto standard among JS front end tool builders, but that isn't the same thing.
There was just too much pressure for this not to happen, as one cannot even test a standalone parser without some way to represent its output. Parser implementors and parser users needed a target spec, and the SpiderMonkey AST was the best known, and the only one with engine-maker buy-in.
So, the concerns about not forcing different vendors to support a standard AST API have led to one vendor "owning" the standard..
Or, possibly, it isn't important enough for other venders to care. Is this really a "browser platform issue" or a language tool builder issue?
Given the speed of parsers like esprima, and the performance and complexity implications of other engine's internal ASTs not matching the SpiderMonkey AST, it does seem right not to force engine makers to implement Reflect.parse.
The question is: how to move this de-facto standard forward?
Basically, all standards development works approximately the same way. A group (people or organizations) with a common interest get together, write a specification, and get a standards organization to publish it. Whether or not it actually achieves broad adoption (see, for example, E4X) will depend upon many factors. TC39 is open to hosting such efforts. That's what happening right now with the ES Internationalization API. But the proponents need to provide the "feet on the street" to make such efforts actually happen.
For instance, there is an esprima branch implementing modules, does it have to wait for someone (Mozilla?) to extend the AST spec? Or does the feature get merged into the master, and everyone has to change when Mozilla implements? Similarly for other parts of the evolving ES6 spec.
It isn't clear to me why esprima should care about the Mozilla parser implementation. Why isn't Esprima free to do what ever its wants. While FF release turns aren't slow, certainly esprima can innovate faster than FF in this area. Get ahead, show Mozilla how to do it.
Or, there are issues emerging from the increasing use of the AST API - will everyone have to apply to Mozilla for changing the spec? Will Mozilla do that if it means changing their implementations? Do they now have to pay the maintenance toll mentioned above?
But what does it matter whether or not Mozilla does. If esprima turns out to be the best browser independent ES front-end infrastructure its AST is likely to turn into the de-facto standard so it doesn't matter what Mozilla does.
Basically, you win in the market, not by prematurely writing standards.
It might be useful to decouple the SpiderMonkey AST API from its initial implementation, and to create a forum in which all users of the AST API spec (parser implementors, parser users) can discuss and influence the development. Then the question of whether or when Mozilla or others implement which parts of the spec would be separate from what the spec is. Such a forum and its spec could then be mentioned in the ES standard as an appendix, without including the AST spec itself in the ES standard.
That's pretty much it...
However, it isn't clear if it is necessary or even desirable for this to been part of the ES standard, even by reference. We need to start thinking of the browser/web as "the platform", comparable to the role Window or Unix/Linux has had for the past couple decades. You will freeze innovation by making functionality prematurely a standardized immutable part of the platform. Providing functionality in a form that can can exist above the platform layer is should be seen as a goal rather than as a consolation prize.
it appears that the SpiderMonkey AST has emerged as a de-facto standard.
Really? I guess there are many more users of Uglify and Esprima is growing fast. Personally I never heard of any non-Mozilla uses of the SpiderMonkey API, but perhaps you can point to some.
I was merely stating my personal impression. It is difficult to pin down all the bits of information that contributed to that impression, but here are some of them:
-
not all JS parsers document their AST, but it appears that the SpiderMonkey AST has the largest number of (partial) implementations; most parsers have their own AST (jslint and jshint might share their AST by forking) but SpiderMonkey, esprima, PanPG, reflect.js, jstr all refer to the same AST document.
Of those, esprima seems the most active. The combination of Reflect.parse awareness and esprima being an efficient cross- browser implementation is certainly the reason why I think that this AST spec is in the process of winning.
-
even though I've moved to JS relatively recently, I've already worked with 3 different JS parsers and have written one myself;
it just isn't funny anymore: all of them had different ASTs and varying support, and I was still missing out on one popular JS parser and its community; so I'm in the process of consolidating my JS tool efforts on top of esprima, giving up or porting my own code to gain from collaboration;
-
some of the people working with or on esprima have been working with other JS parsers before, so I assume they, too, prefer convergence and sharing over their previous efforts; I expect more tools to be converted for use with esprima;
esprima doesn't try to be anything but an efficient, simple, ES parser. It doesn't define its own AST, special-purpose code has been moved out of the main branch, so it just aims to be a common building block for tool builders.
Having different standards compete in the market has its advantages, but not for the foundations on which tool-builders base and share their work. Code implementing the ES grammar is such a foundation.
There isn't much added value to be expected from parsing JS differently, we just need one good parser, one useable AST format. Depending on half a dozen parsers with slightly different ASTs, just because each was popular at a time when a different JS tool project was started, is slowing down progress, and adds to complexity and maintenance costs.
Since I used to be among those asking for an ES standard AST, I just wanted to join this thread by pointing out that de-facto standardizing on esprima (or on the Reflect.parse AST it uses) appears to offer a pragmatic solution to this problem. For my own tool building purposes, that is sufficient for the moment.
It would be good if the ES test suite included parser-only tests, though, so that one could test JS frontends that do not have a JS engine behind them for conformance to the ES spec grammar.
Also: I think this might be a better topic for the js-tools groups.google.com/group/js-tools?pli=1
As you know, that was one motivation for starting that group.
Claus
Will Reflect.parse be standardized?