Harmony transpilers
Caja code.google.com/p/google-caja includes an ES5-to-ES3
translator which already includes an emulation of ES-next proxies and a somewhat leaky emulation of WeakMaps. (Non-leaky WeakMaps cannot be practically provided by emulation, but our's are less leaky than you'd expect.)
On Tue, Oct 11, 2011 at 6:41 AM, Juan Ignacio Dopazo <dopazo.juan at gmail.com>wrote:
Hi! Is there anyone working on a Harmony "transpiler" besides Traceur?
I'd like to understand why Traceur is not suitable. As far as I understand it was written to study new directions in JS.
jjb
One of the reasons traceur is not suitable is that it's a product of google and thus not neutral.
I've actually asked traceur whether they intent to become a full es harmony compliant transpiler but there was no response.
And another reason would be that it currently implements some strawmans that conflict with harmony proposals.
For example it's class mechanism is not loyal to the current es harmony
proposal, another example would be traceur generators not using the
function* notation.
On Tue, Oct 11, 2011 at 9:10 AM, Jake Verbaten <raynos2 at gmail.com> wrote:
One of the reasons traceur is not suitable is that it's a product of google and thus not neutral.
Traceur is not a product, I think that is pretty obvious. It's just a JS to JS compiler written by some engineers very keen on their work, judging from the quality of the code.
I don't know what "neutral" means here. Traceur is open source with an Apache license. It has an open newsgroup and bug list.
I've actually asked traceur whether they intent to become a full es harmony compliant transpiler but there was no response.
I guess you mean you posted to the traceur newsgroup and there was no response. In my experience when this happens you can usually draw the obvious conclusion: the participants in the newsgroup have no answer. Based on what I have read on that newsgroup, there is one person contributing now, Eric. Based on his posts, his agenda is to improve the project.
And another reason would be that it currently implements some strawmans that conflict with harmony proposals.
For example it's class mechanism is not loyal to the current es harmony proposal, another example would be traceur generators not using the
function*notation.
Ok, so fix it. Return to the traceur newsgroup with your ideas and patches. Come up with some way to select different features by options. We don't know what the standard will be so we need some why to try out different features.
jjb
On Oct 11, 2011, at 12:49 PM, John J Barton wrote:
We don't know what the standard will be so we need some why to try out different features.
That's not how the committee has worked since 2008, and even before then (pre-Harmony), with a split committee, we still had two groups working on consensus by drafting proposals and prototyping them.
The way we try to work is to propose before implementing, including prototype features. Individual vendors don't deviate from proposals unilaterally and silently.
If old proposals were implemented, and then the proposals changed, we may have some work to do adjusting implementations (certainly true of SpiderMonkey, and we will do it). But we do not go off on our own, as individual vendors, and try out features never proposed on the wiki.
Also, your point in an earlier message about user testing was good. We can't "do science" to decide what features in the enormous feature-vector hyperspace to standardize. We won't search well, and user-testing gives little certain data. We can hope to find obvious usability problems that were not foreseen. Also interactions with other features (old and new) that were not anticipated.
User testing is good but if we're working in Harmony, the harmony:proposals are the ones to prototype-implement, and edit based on implementation and user testing. Ecma works by consensus.
It's hard to keep consensus or achieve it later by doing single-vendor experiments that seem to blindside the committee, and possibly even the community.
This does not mean design by committee. The designers are one or two champions working on individual proposals. But the proposal work happens in the open, in plain sight. Mark and Tom's Proxy work is an exemplar.
On Tue, Oct 11, 2011 at 12:42 PM, John J Barton <johnjbarton at johnjbarton.com
wrote:
On Tue, Oct 11, 2011 at 6:41 AM, Juan Ignacio Dopazo < dopazo.juan at gmail.com> wrote:
Hi! Is there anyone working on a Harmony "transpiler" besides Traceur?
I'd like to understand why Traceur is not suitable. As far as I understand it was written to study new directions in JS.
Traceur is very good! I'd just like to have something that compiles to ES5
without intermediate libraries, the way CoffeeScript works, so that it's easier to debug and use in NodeJS.
Traceur does this very well for certain features, like arrow functions and the rest operator. But classes still use library code and aren't very human readable. See:
class Person { constructor(name) { this.name = name; } }
... is compiled to:
var Person = traceur.runtime.createClass("Person", null, null, function(name) { this.name = name; }, undefined, { }, function $static() { }, null);
Compare it to CoffeeScript:
class Person constructor: (name) -> this.name = name sayHi: -> 'Hi!'
Coffee to JS:
var Person; Person = (function() { function Person(name) { this.name = name; } Person.prototype.sayHi = function() { return 'Hi!'; }; return Person; })();
Juan
On Tue, Oct 11, 2011 at 10:08 AM, Brendan Eich <brendan at mozilla.com> wrote:
On Oct 11, 2011, at 12:49 PM, John J Barton wrote:
We don't know what the standard will be so we need some why to try out different features.
That's not how the committee has worked since 2008, and even before then (pre-Harmony), with a split committee, we still had two groups working on consensus by drafting proposals and prototyping them.
The way we try to work is to propose before implementing, including prototype features. Individual vendors don't deviate from proposals unilaterally and silently.
If old proposals were implemented, and then the proposals changed, we may have some work to do adjusting implementations (certainly true of SpiderMonkey, and we will do it). But we do not go off on our own, as individual vendors, and try out features never proposed on the wiki.
You're making something out of nothing. To me Traceur is just exactly what it is: some open source code for a JS to JS compiler written by some good engineers who now seem to be doing something else. If you have a beef with how Google works with the committee, then contact a Google rep on the committee and complain to them. But don't paint the Traceur project as some weird plot to derail your work. It's just like a thousand other open source projects, a risky labor of love, a gift to the community of engineers.
Also, your point in an earlier message about user testing was good. We can't "do science" to decide what features in the enormous feature-vector hyperspace to standardize. We won't search well, and user-testing gives little certain data. We can hope to find obvious usability problems that were not foreseen. Also interactions with other features (old and new) that were not anticipated.
User testing is good but if we're working in Harmony, the harmony:proposals are the ones to prototype-implement, and edit based on implementation and user testing. Ecma works by consensus.
It's hard to keep consensus or achieve it later by doing single-vendor experiments that seem to blindside the committee, and possibly even the community.
This does not mean design by committee. The designers are one or two champions working on individual proposals. But the proposal work happens in the open, in plain sight. Mark and Tom's Proxy work is an exemplar.
Well we don't have a Mark and Tom for Traceur. We just have some great source code. If anyone here wants to try to match Traceur up to Ecma consensus, please step up.
jjb
On Oct 11, 2011, at 1:47 PM, John J Barton wrote:
On Tue, Oct 11, 2011 at 10:08 AM, Brendan Eich <brendan at mozilla.com> wrote: On Oct 11, 2011, at 12:49 PM, John J Barton wrote:
We don't know what the standard will be so we need some why to try out different features.
That's not how the committee has worked since 2008, and even before then (pre-Harmony), with a split committee, we still had two groups working on consensus by drafting proposals and prototyping them.
The way we try to work is to propose before implementing, including prototype features. Individual vendors don't deviate from proposals unilaterally and silently.
If old proposals were implemented, and then the proposals changed, we may have some work to do adjusting implementations (certainly true of SpiderMonkey, and we will do it). But we do not go off on our own, as individual vendors, and try out features never proposed on the wiki.
You're making something out of nothing.
Somehow I suspect the shoe would be on the other foot if other vendors did likewise.
But I'm tired of arguing in favor of keeping consensus. If it's not important, let's all go off and run our own experiments and see what happens. Biggest company wins, best two out of three in case of a order-of-magnitude market cap tie :-P. That will be fun.
If you have a beef with how Google works with the committee, then contact a Google rep on the committee and complain to them.
I already have, but that's not the topic here: you just wrote "We don't know what the standard will be so we need some why to try out different features" (cited above) and I wrote back arguing with you, not with anyone else.
But don't paint the Traceur project as some weird plot to derail your work.
I never said anything like that ("weird plot" -- come on!). I don't know why Traceur was developed the way it was, and then abandoned (I have some theories, but really, who cares?).
Maybe it's someone else's turn to maintain Traceur, and Google has done its fair share. The problem is no one else knows the code and Jake said he got no response when he asked (but maybe he asked in the wrong channel or something). It's hard for others to pick up where things left off.
It's just like a thousand other open source projects, a risky labor of love, a gift to the community of engineers.
There is a lot of abandon-ware in open source, but that's a very low bar to meet.
Well we don't have a Mark and Tom for Traceur. We just have some great source code. If anyone here wants to try to match Traceur up to Ecma consensus, please step up.
So you, another Googler, exhort anyone, or Jake, to "fix it" and "step up" (but thanks for the "please" the second time :-/).
I think it's fair to ask why anyone would do that, instead of choosing to work on other projects that seem to have active maintainers and open source communities going back to their genesis.
Beyond this, I'm still picking a fight with your "We don't know what the standard will be so we need some why to try out different features" line, which you have not defended. But we can table that, or forget about it if you prefer.
Something's wrong here. Jake cited specific concerns about Traceur and got a lecture to be grateful for it being open source, and to get to work fixing it. Is that really the best answer?
On Tue, Oct 11, 2011 at 11:14 AM, Brendan Eich <brendan at mozilla.com> wrote:
On Oct 11, 2011, at 1:47 PM, John J Barton wrote:
On Tue, Oct 11, 2011 at 10:08 AM, Brendan Eich <brendan at mozilla.com>wrote:
On Oct 11, 2011, at 12:49 PM, John J Barton wrote:
We don't know what the standard will be so we need some why to try out different features.
That's not how the committee has worked since 2008, and even before then (pre-Harmony), with a split committee, we still had two groups working on consensus by drafting proposals and prototyping them.
The way we try to work is to propose before implementing, including prototype features. Individual vendors don't deviate from proposals unilaterally and silently.
If old proposals were implemented, and then the proposals changed, we may have some work to do adjusting implementations (certainly true of SpiderMonkey, and we will do it). But we do not go off on our own, as individual vendors, and try out features never proposed on the wiki.
You're making something out of nothing.
Somehow I suspect the shoe would be on the other foot if other vendors did likewise.
But I'm tired of arguing in favor of keeping consensus. If it's not important, let's all go off and run our own experiments and see what happens. Biggest company wins, best two out of three in case of a order-of-magnitude market cap tie :-P. That will be fun.
If you have a beef with how Google works with the committee, then contact a Google rep on the committee and complain to them.
I already have, but that's not the topic here: you just wrote "We don't know what the standard will be so we need some why to try out different features" (cited above) and I wrote back arguing with you, not with anyone else.
But don't paint the Traceur project as some weird plot to derail your work.
I never said anything like that ("weird plot" -- come on!). I don't know why Traceur was developed the way it was, and then abandoned (I have some theories, but really, who cares?).
Maybe it's someone else's turn to maintain Traceur, and Google has done its fair share. The problem is no one else knows the code and Jake said he got no response when he asked (but maybe he asked in the wrong channel or something). It's hard for others to pick up where things left off.
I have quite a lot of first hand experience in picking up where things left off. No question it's hard, but you may get further by building on Traceur rather than starting over.
It's just like a thousand other open source projects, a risky labor of love, a gift to the community of engineers.
There is a lot of abandon-ware in open source, but that's a very low bar to meet.
Well we don't have a Mark and Tom for Traceur. We just have some great source code. If anyone here wants to try to match Traceur up to Ecma consensus, please step up.
So you, another Googler, exhort anyone, or Jake, to "fix it" and "step up" (but thanks for the "please" the second time :-/).
I think it's fair to ask why anyone would do that, instead of choosing to work on other projects that seem to have active maintainers and open source communities going back to their genesis.
Juan asked about Traceur. Advocates for other projects can reply if they like.
Beyond this, I'm still picking a fight with your "We don't know what the standard will be so we need some why to try out different features" line, which you have not defended. But we can table that, or forget about it if you prefer.
Well we don't know what the standard will be, that's just a fact. I happen to think that one needs to gain experience with language features by trying them out. I know you have a lot of experience so perhaps you don't need this step. I totally don't understand why you want to prevent Juan or Jake from trying out ideas related to JS.
Something's wrong here. Jake cited specific concerns about Traceur and got a lecture to be grateful for it being open source, and to get to work fixing it. Is that really the best answer?
I'm sorry if I came across as lecturing. I was taken aback by Jake's puzzling comments. I was just trying to being things back to reality. Traceur seems like a useful bit of code; it does not look like it will be maintained by the original authors. I was trying to encourage Jake and Juan to participate in taking it forward.
On Oct 11, 2011, at 2:40 PM, John J Barton wrote:
I have quite a lot of first hand experience in picking up where things left off. No question it's hard, but you may get further by building on Traceur rather than starting over.
It would be great to get Traceur active again and up to draft-spec. The desire for self-contained class desugaring, a la CoffeeScript, could be done too, probably by someone on the list.
On Tue, Oct 11, 2011 at 7:40 PM, John J Barton <johnjbarton at johnjbarton.com>wrote:
On Tue, Oct 11, 2011 at 11:14 AM, Brendan Eich <brendan at mozilla.com>wrote:
On Oct 11, 2011, at 1:47 PM, John J Barton wrote:
On Tue, Oct 11, 2011 at 10:08 AM, Brendan Eich <brendan at mozilla.com>wrote:
On Oct 11, 2011, at 12:49 PM, John J Barton wrote:
We don't know what the standard will be so we need some why to try out different features.
That's not how the committee has worked since 2008, and even before then (pre-Harmony), with a split committee, we still had two groups working on consensus by drafting proposals and prototyping them.
The way we try to work is to propose before implementing, including prototype features. Individual vendors don't deviate from proposals unilaterally and silently.
If old proposals were implemented, and then the proposals changed, we may have some work to do adjusting implementations (certainly true of SpiderMonkey, and we will do it). But we do not go off on our own, as individual vendors, and try out features never proposed on the wiki.
You're making something out of nothing.
Somehow I suspect the shoe would be on the other foot if other vendors did likewise.
But I'm tired of arguing in favor of keeping consensus. If it's not important, let's all go off and run our own experiments and see what happens. Biggest company wins, best two out of three in case of a order-of-magnitude market cap tie :-P. That will be fun.
If you have a beef with how Google works with the committee, then contact a Google rep on the committee and complain to them.
I already have, but that's not the topic here: you just wrote "We don't know what the standard will be so we need some why to try out different features" (cited above) and I wrote back arguing with you, not with anyone else.
But don't paint the Traceur project as some weird plot to derail your work.
I never said anything like that ("weird plot" -- come on!). I don't know why Traceur was developed the way it was, and then abandoned (I have some theories, but really, who cares?).
Maybe it's someone else's turn to maintain Traceur, and Google has done its fair share. The problem is no one else knows the code and Jake said he got no response when he asked (but maybe he asked in the wrong channel or something). It's hard for others to pick up where things left off.
I have quite a lot of first hand experience in picking up where things left off. No question it's hard, but you may get further by building on Traceur rather than starting over.
It's just like a thousand other open source projects, a risky labor of love, a gift to the community of engineers.
There is a lot of abandon-ware in open source, but that's a very low bar to meet.
Well we don't have a Mark and Tom for Traceur. We just have some great source code. If anyone here wants to try to match Traceur up to Ecma consensus, please step up.
So you, another Googler, exhort anyone, or Jake, to "fix it" and "step up" (but thanks for the "please" the second time :-/).
I think it's fair to ask why anyone would do that, instead of choosing to work on other projects that seem to have active maintainers and open source communities going back to their genesis.
Juan asked about Traceur. Advocates for other projects can reply if they like.
Beyond this, I'm still picking a fight with your "We don't know what the standard will be so we need some why to try out different features" line, which you have not defended. But we can table that, or forget about it if you prefer.
Well we don't know what the standard will be, that's just a fact. I happen to think that one needs to gain experience with language features by trying them out. I know you have a lot of experience so perhaps you don't need this step. I totally don't understand why you want to prevent Juan or Jake from trying out ideas related to JS.
Something's wrong here. Jake cited specific concerns about Traceur and got a lecture to be grateful for it being open source, and to get to work fixing it. Is that really the best answer?
I'm sorry if I came across as lecturing. I was taken aback by Jake's puzzling comments. I was just trying to being things back to reality. Traceur seems like a useful bit of code; it does not look like it will be maintained by the original authors. I was trying to encourage Jake and Juan to participate in taking it forward.
+1 for encouraging participation, traceur may be good starting point.
Sorry for seeing this thread so late.
I am the maintainer of Traceur at the moment.
We do want Traceur to align with the ES6 draft and with Harmony proposals. Any missmatch is a bug and both bug reports and patches are welcome.
The reason a lot of the generated code depends on a runtime library is mostly because it makes code size smaller (but it also makes generating the code easier). We should make it optional to include the required runtime functions in the generated code at least.
I've implemented several harmony proposals in es.next by expanding the grammar in pegjs (dmajda/pegjs/blob/master/examples/javascript.pegjs). The AST and generator is defined using es.next classes and modules. I was thinking of converting the derived classes in the ast and generator modules to use the proto operator |> rather than 'extends' since the LHS is desugared to a function. I'm using '@' for private names but this is a moving target and looks like it will be replaced by private names. It works similarly to coffeescript in that it generates plain javascript. It's been a side project I've worked on when I've had free time. It's in my private github repo and works with node using websockets (so you can load the parser into the browser). Could make it public if I have some time to spend on it this weekend. I like my approach because the grammar is easy to extend and I don't know of any other transpiler whose ast/generators are in the target language.
On Oct 11, 2011, at 9:10 AM, Jake Verbaten <raynos2 at gmail.com> wrote:
One of the reasons traceur is not suitable is that it's a product of google and thus not neutral.
I've actually asked traceur whether they intent to become a full es harmony compliant transpiler but there was no response.
And another reason would be that it currently implements some strawmans
that conflict with harmony proposals.
I have some thoughts about how to use Narcissus as a basis for a compiler to ES3 as well. It's obviously not necessary to do separately from Traceur, but it might be interesting to experiment with alternative implementation strategies. I haven't really done anything in earnest yet, including looking through Traceur's source, but during my vacation I played with an implementation strategy for generators that was pretty neat.
Traceur is very good! I'd just like to have something that compiles to ES5 without intermediate libraries, the way CoffeeScript works, so that it's easier to debug and use in NodeJS.
You aren't going to be able to match CoffeeScript's readability for many features, especially generators, private names, and proxies. Those require deep, whole-program compilation strategies.
Thanks for all the answers!
On Tue, Oct 11, 2011 at 3:40 PM, John J Barton <johnjbarton at johnjbarton.com> wrote:
I was trying to encourage Jake and Juan to participate in taking it forward.
I'd love to, but I'm afraid my interpreter-writing skills are somewhat null if not undefined. I'm interested in language design and interpreting but I'm barely scratching the surface with SICP for now.
On Tue, Oct 11, 2011 at 5:40 PM, Erik Arvidsson <erik.arvidsson at gmail.com> wrote:
We should make it optional to include the required runtime functions in the generated code at least.
Thanks! That would be very useful for targeting Node.
On Tue, Oct 11, 2011 at 6:29 PM, Kam Kasravi <kamkasravi at yahoo.com> wrote:
I've implemented several harmony proposals in es.next by expanding the grammar in pegjs ( dmajda/pegjs/blob/master/examples/javascript.pegjs). (..) I like my approach because the grammar is easy to extend and I don't know of any other transpiler whose ast/generators are in the target language.
pegjs looks great! I'm playing with the online version right now. I already
have a parser that interpretes ^ as Math.pow() happy newbie
On Tue, Oct 11, 2011 at 10:17 PM, David Herman <dherman at mozilla.com> wrote:
Traceur is very good! I'd just like to have something that compiles to ES5 without intermediate libraries, the way CoffeeScript works, so that it's easier to debug and use in NodeJS.
You aren't going to be able to match CoffeeScript's readability for many features, especially generators, private names, and proxies. Those require deep, whole-program compilation strategies.
Dave
I understand that. However proxies are already in V8 and SpiderMonkey.
WeakMaps are in SpiderMonkey as well. I'm already looking at using proxies in NodeJS to rewrite a MongoDB driver. So what I'm looking for is to try out all the syntax that's just sugar (classes, destructuring, rest, spread, etc) to see how it changes my everyday coding in my small projects that have room for experimentation.
Juan
On Tue, Oct 11, 2011 at 6:17 PM, David Herman <dherman at mozilla.com> wrote:
Traceur is very good! I'd just like to have something that compiles to ES5 without intermediate libraries, the way CoffeeScript works, so that it's easier to debug and use in NodeJS.
You aren't going to be able to match CoffeeScript's readability for many features, especially generators, private names, and proxies. Those require deep, whole-program compilation strategies.
I'm unclear, but are you saying: some features translate directly to simple JS but some features are more pervasive so their translation will not be as readable? So we need to develop new strategies for debugging these features? Or something else?
jjb
There is a concept of "expressiveness" in programming languages that has to do with the notion of whether a feature is "just sugar." A lightweight feature is one that can be described as a simple, local rewrite into some other feature. For example, the proposed method shorthand syntax:
{ f(x) { return x + 1 } }
can easily be translated to
{ f: function(x) { return x + 1 } }
By "local rewrite" I mean you can replace the AST node MethodInit(name, args, body) into an equivalent node PropertyInit(name, FunctionLiteral(args, body))) without modifying name, args, or body, and the program will behave identically.
Intuitively, sugar doesn't increase the expressiveness of a language; it just makes for nice little syntactic abstractions for common patterns. This doesn't really change the power of the language because there's nothing you can't do with the new feature that you couldn't already do with only minor revisions to the code. Conversely, a feature that can't be implemented exclusively with local rewrites is one that can potentially replace massive amounts of boilerplate. This is a feature that can fundamentally increase the expressiveness of the language.
CoffeeScript is, by design, a language that is designed to have the same expressiveness as JavaScript -- all its features are defined as syntactic sugar. Harmony is, by design, a language that is designed to have more expressiveness than ES5 -- it contains some features that can't be defined as syntactic sugar (in addition to some that can).
But there are costs to adding expressiveness, such as the need to adapt tools like debuggers, as you mention, and the risk of features that are subject to abuse or unwieldy to program with. That's why we have been very conservative about only introducing a few such features, using concepts that are well-studied from other languages, and even then limiting their expressiveness. For example, generators are strictly less expressive than full coroutines or continuations. How so? You can expressive the former as sugar for the latter, but not vice versa.
Is there anyone working on a Harmony "transpiler" besides Traceur? It'd be really useful to have a transpiler that justs desugars (what's possible to desugar) without using a library like Closure, the way CoffeeScript is working nowadays.
Thanks,
Juan