Shepherd.js - Implementing Harmony modules for today's browsers
This is great! I've been considering doing the same thing, but I haven't found the time. When you say it is compatible with CommonJS modules, does that mean that you can do an import using ES6 syntax and have the result do a CommonJS require?
The big thing I'm noticing is that all of the examples are inside of comments, and your module definitions including exports are all separate from the actual code for those exports. I'm assuming that means somebody couldn't actually take the ES6 module examples and make them work, for example. I appreciate the difference in difficulty level. Parsing just the module syntax vs being an ES5 compliant parser + modules is a much different task. The problem is, I don't see myself using it unless it actually used full module syntax. Do you plan on going in that direction?
Thanks for your comments, I'm glad you liked it. About CommonJS, the compatibility is the other way round. Shepherd can load commonJS modules without the addition of the in-comment syntax declaration. In such a case, require is wrapped to load whether an already loaded ES6 module or use commonJS's require. furthermore, exports or module.exports is used as the public API of the ES6 module (which is basically the definition :) ). This is really useful for 3rd party modules.
Removing the comments would be possible, but it would require a much heavier parsing phase, and would probably require a full-blown JS parser, such as Esprima. it would be probably computationnally too heavy for the browser (read: for the user) without a systematic and automatic code-rewrite to ES5, optionnaly accompagned with some minification. On the other hand, the great advantage of using comments is that they act as placeholders for the syntax declaration, and are very easy to locate with a single regular expression.
More realistically, I plan on allowing multiple module declarations in a single file (currently one only), where module implementations would be syntactically separated by the numerous definitions (ie, the implementation of a module ends at the beginning of the next module declaration).
But yes, I am looking for an efficient way to remove the comments, which would be the only way Shepherd could be used as an efficient Harmony:modules polyfill. Maybe I'll have to play around with more minimalist placeholders, such as:
//s6d module myModule { import x from X; export a, b, c, d; } //-s6d
...though the ending comment doesn't seem necessary.
And please don't blame me for the possible terrible idea, I'm thinking aloud on this one! ;)
The tradeoff to be found is the following: As of today, no browser has a stable release that allows harmony modules (yeah, the latest V8 has an option, well...). Whatsmore, the syntax is just a proposal which is updated almost every quarter. So do we want to allow files that work on today's engines and can be enhanced with Harmony:modules' features thankd to Shepherd, or do we want Harmony compliant files that won't run natively on any stable engine we can find today?
I chose the first path, but the discussion remains open. I'll be happy to hear your thoughts on that point.
Xavier
It's great to see this project! I'll take a closer look soon. I will see if I can help contribute to this.
This is quite interesting. I really feel that the comment syntax is pretty ugly. Parsing JS is non trivial but it is not a performance issue. If you expect this to have some uptake I would expect it to use the "real" module syntax and not rely on comments.
By coincidence I landed import support to Traceur last night. You might want want to check it out. It also doesn't use the latest syntax since the current BNF on the wiki is incomplete.
code.google.com/p/traceur-compiler/source/browse/#git%2Ftest%2Ffeature%2FModules, code.google.com/p/traceur-compiler/source/detail?r=f4f8788860f624ca1b02883890325cbb4ee9c1eb
Traceur does have a CodeLoader that allows loading external js files but it is not very convenient to use at this point. We have an open bug to allow offline compilation of modules with external dependancies.
@David: I am all ears to your comments!
@Erik: As you may have read in this thread, it seems that the major request is for the removal of comments. So, here I announce it: the feature has just bumped to the top of the TODO list :) Thanks for the pointers. I'll probably have a closer look at the functional coverage of Traceur, as it has clearly evolved since I last watched it.
On Wed, Apr 18, 2012 at 10:21 AM, Xavier CAMBAR <xcambar at gmail.com> wrote:
Thanks for your comments, I'm glad you liked it. About CommonJS, the compatibility is the other way round. Shepherd can load commonJS modules without the addition of the in-comment syntax declaration. In such a case, require is wrapped to load whether an already loaded ES6 module or use commonJS's require. furthermore, exports or module.exports is used as the public API of the ES6 module (which is basically the definition :) ). This is really useful for 3rd party modules.
Removing the comments would be possible, but it would require a much heavier parsing phase, and would probably require a full-blown JS parser, such as Esprima. it would be probably computationnally too heavy for the browser (read: for the user) without a systematic and automatic code-rewrite to ES5, optionnaly accompagned with some minification.
Yes, that's what I was getting at in my response. I know it would be a much bigger effort, which is why I understand the approach. If I were to do the project, though, I would assume that if it were done in the browser, that would be purely for development - it should all be precompiled for production. You could even use the module imports to do script concatenation.
On the other hand, the great advantage of using comments is that they act as placeholders for the syntax declaration, and are very easy to locate with a single regular expression.
This is beneficial for the implementation, but not the user.
More realistically, I plan on allowing multiple module declarations in a single file (currently one only), where module implementations would be syntactically separated by the numerous definitions (ie, the implementation of a module ends at the beginning of the next module declaration).
But yes, I am looking for an efficient way to remove the comments, which would be the only way Shepherd could be used as an efficient Harmony:modules polyfill. Maybe I'll have to play around with more minimalist placeholders, such as:
Personally, I think you'll want the full polyfill as a target. You can get there incrementally by removing comments, but until it matches the full spec, I don't think you'll see widespread adoption. IMHO anyway.
//s6d module myModule { import x from X; export a, b, c, d; } //-s6d
...though the ending comment doesn't seem necessary.
And please don't blame me for the possible terrible idea, I'm thinking aloud on this one! ;)
I think I would actually prefer comments over this. I wrote some code long before module proposals that basically made my JavaScript files work like Java files (Java is the other language we use), basically import statements, exports were just based on the filename matching the class or object definition inside. I would probably change it to be more like modules if I did it now, maybe even how you did it. Anyway, point is, I put the imports in the comments just like you for two reasons.
- easier to find with regexes, didn't have to parse full js
- didn't break existing JavaScript tooling i.e. syntax highlighting
Its the second reason that I would prefer your current comment based solution over the new one you propose. Unless you support writing modules the way I expect them to be written, I would rather have them in comments.
The tradeoff to be found is the following: As of today, no browser has a stable release that allows harmony modules (yeah, the latest V8 has an option, well...). Whatsmore, the syntax is just a proposal which is updated almost every quarter. So do we want to allow files that work on today's engines and can be enhanced with Harmony:modules' features thankd to Shepherd, or do we want Harmony compliant files that won't run natively on any stable engine we can find today?
I chose the first path, but the discussion remains open. I'll be happy to hear your thoughts on that point.
Support is coming. I look at it this way. Some day relatively soon, ES6 modules will be in node. Soon after that they will start showing up in browsers. Over the course of the next year, I bet you'll see module support in at least firefox and chrome. If you set your sights on full modules now, you'll have code that doesn't need to be rewritten a year from now. A polyfill will happen, do you want to write it?
You've made solid points here!
The various comments I've received from here and there show the same direction: someday, modules will be available, and Shepherd will then be unnecessary for supported engines. Polyfilling remains. If it ever gains some momentum, it is highly related to how it follows the specifications and remains usable in supported engines.
It appears then that removing comments is a must-have. Keeping backward-compatibility maybe was a bad idea from the very beginning after all. If you look at the example of the home page ( xcambar.github.com/shepherd-js), who will seriously consider developing a module with the relevant code architecture in mind, and then proclaim "Hey, you can also use it as a good old module-less JS file, but it will put crap all over you global scope by exposing its inner parts ?" I hope no one will.
As a conclusion, Shepherd seems to be going its way to a compliant polyfill for Harmony modules. I should change the script type to text="harmony" to reflect this :p
But you shouldn't expect the tools you use not to break. Unless they parse the modules syntax, of course! This is why I hardly understand your point on the proposed new syntax. It's much more lightweight, is parsed easily (I am the only one happy, but that matters at some point ;) ) and works natively on module-enabled engines. It may IMO constitute a valid first step, as it reduces the effort for the user drastically.
Xavier
On Thu, Apr 19, 2012 at 9:11 AM, Russell Leggett <russell.leggett at gmail.com>wrote:
[snip] Support is coming. I look at it this way. Some day relatively soon, ES6 modules will be in node. Soon after that they will start showing up in browsers. Over the course of the next year, I bet you'll see module support in at least firefox and chrome.
In Chrome Canary, with Harmony flag enabled...
module Foo {}
(Foo in this); // true
:)
But, Foo is still undefined.
Russell, I've worked on this branch during the evening: xcambar/shepherd-js/tree/20120420_CommentRemoval, Currently, single line comments must still wrap the module declaration, but this version offers a good comparison point in to the master branch.
Rick, I've seen this, it is encouraging that Google considers adding support. But there's a long way to go before a stable release. By the way, the V8 engine what will come in Node 0.8 has Harmony flags, and one of them is about modules.
Xavier
I wanted to announce that I've been working on a project called Shepherd ( xcambar.github.com/shepherd-js), a pure Javascript implementation of Harmony modules.
Why such a project ? Fun first. Second, I was really looking forward to use harmony modules. Third, I wanted an efficient way to use my modules on the server and the client. And it seems to me that current module loaders and APIs available will be, at the end, superseded by the module syntax being defined at ECMA for the future versions of ECMAScript.
The syntax used is as of 2012-02-27 ( doku.php?id=harmony:modules&rev=1330363672), I couldn't find the time to implement the latest proposal (besides I would have had to choose one of the two variants), but apart from the syntax, it is usable both on the client and the server, and tested.
The parser/lexer has been developped using JISON ( zaach.github.com/jison) and is available as a separate project ( xcambar/harmony-parser).
For backward-compatibility, the module declarations have to be put into comments (which you will discover in the examples provided on the site), it is compatible with CommonJS modules (user-defined as well as native modules in Node.js), and, although not critical to the project, a compatibility wrapper for the AMD API is on its way. Regarding production-level requirements, an optimizer has been implemented, but it still requires testing before being released.
I've had and I'm still having a really good time working on this project and I would really appreciate if I could have some feedback from the readers and contributors of the mailing-list. A mailing list has been created for the project, still empty of messages simply because it has been created yesterday ;)
,