Simple Modules and Current Modules
On Thu, Nov 4, 2010 at 9:22 AM, Kris Zyp <kris at sitepen.com> wrote:
I believe it should be a requirement that harmony modules (or at least a subset of them) should be desugarable into ES5 code, such that the desugared code is still a working module in harmony engines that support native modules.
I think that this requirement would mean that modules in Harmony would be unable to provide all of the things we want - in particular, true lexical scoping of modules and the names they export and import.
[lots on the integration of modules with existing systems]
I think the area of integration with the work people are already doing in RequireJS, CommonJS, Dojo, and other systems is very important. As the Simple Modules proposal continues to evolve, Dave and I will work hard to make this integration and future migration as easy and transparent as possible.
But,
At least in the area of modules, if we focused on small composable feature(s) that are truly needed to build module systems on top of EcmaScript, we can introduce a much simpler addition, and ensure that we get it right, yet still provide the build blocks for modules with security.
I strongly disagree with this. There is no consensus whatsoever as to what those "small composable features" are. Further, there is no precedent in other programming languages for building modules out of other features, especially not using runtime evaluation. I think that trying to be way out in front of the rest of programming language world in this area would be a mistake.
Namely, the true nugget from the module system is the ability to evaluate scripts with lexical scoping, and ensure a real filename is associated with the script (for real stack traces, debugging, etc).
I disagree that this is the "true nugget". Modules are about much more than 'eval', and certainly much more than associating a "real filename" (hopeless in the general case anyway).
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/4/2010 8:02 AM, Sam Tobin-Hochstadt wrote:
On Thu, Nov 4, 2010 at 9:22 AM, Kris Zyp <kris at sitepen.com> wrote:
I believe it should be a requirement that harmony modules (or at least a subset of them) should be desugarable into ES5 code, such that the desugared code is still a working module in harmony engines that support native modules.
I think that this requirement would mean that modules in Harmony would be unable to provide all of the things we want - in particular, true lexical scoping of modules and the names they export and import.
For imported names, I don't understand why lexical scoping fails to work just because a function call is involved. For exports, I thought the key to linking was freezing objects, which is readily available in ES5.
[lots on the integration of modules with existing systems]
I think the area of integration with the work people are already doing in RequireJS, CommonJS, Dojo, and other systems is very important. As the Simple Modules proposal continues to evolve, Dave and I will work hard to make this integration and future migration as easy and transparent as possible.
That's to great to hear. My objection below is due to the fact that I currently don't see how this proposal brings an integration path with real benefits to existing module systems or provides a compelling alternative to them (other than the security benefits of lexical scoping and improved eval, which is awesome). If it I could be illuminated on the benefits and how to deal with the problems I mentioned, I am sure I would be more favorable towards it.
But,
At least in the area of modules, if we focused on small composable feature(s) that are truly needed to build module systems on top of EcmaScript, we can introduce a much simpler addition, and ensure that we get it right, yet still provide the build blocks for modules with security.
I strongly disagree with this. There is no consensus whatsoever as to what those "small composable features" are.
Of course not, I just suggested it! Are you suggesting that reaching consensus on large complicated features is easier than small simple features?
Further, there is no precedent in other programming languages for building modules out of other features, especially not using runtime evaluation.
What about EcmaScript!? All current JS module systems build on existing features.
I think that trying to be way out in front of the rest of programming language world in this area would be a mistake.
This isn't out in front, quite the opposite, I am suggesting conservative steps forward.
Namely, the true nugget from the module system is the ability to evaluate scripts with lexical scoping, and ensure a real filename is associated with the script (for real stack traces, debugging, etc).
I disagree that this is the "true nugget". Modules are about much more than 'eval', and certainly much more than associating a "real filename" (hopeless in the general case anyway).
Of course there is much more. But from real-world use of today's module systems, this is one of the key missing build blocks.
On Nov 4, 2010, at 6:22 AM, Kris Zyp wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I've been meaning to make some comments about simple modules and in last minute with Brendan, he noted that he wanted to work with CommonJS to make it really good, so... I think one of the key needs that seems to be unaddressed from the current simple modules proposal is the need to have a feasible transition from today's modules to native modules.
I disagree. The design of the new system should not be predicated on what current practice looks like. We should use the entire available design space to construct the best module system for JavaScript possible. If that's incompatible with current code, that's OK.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/4/2010 11:13 AM, Alex Russell wrote:
On Nov 4, 2010, at 6:22 AM, Kris Zyp wrote:
I've been meaning to make some comments about simple modules and in last minute with Brendan, he noted that he wanted to work with CommonJS to make it really good, so... I think one of the key needs that seems to be unaddressed from the current simple modules proposal is the need to have a feasible transition from today's modules to native modules.
I disagree. The design of the new system should not be predicated on what current practice looks like. We should use the entire available design space to construct the best module system for JavaScript possible. If that's incompatible with current code, that's OK.
I agree. Well, except that we shouldn't be blind to what has been necessary to build real world applications in the past, I hope you are not suggesting that. To be clear though, I don't have any objections to new syntax, and I don't have any objection to taking advantage of the entire design space. If that is incompatible, so be it. But if the current proposal can desugar and provide a smooth transition, why shouldn't it? So far I am not seeing the essential incompatibility. Thanks, Kris -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - enigmail.mozdev.org
iEYEARECAAYFAkzS60UACgkQ9VpNnHc4zAwbeQCfaNPSNaF9mdjFYrh7dZsBvKzt 2FIAn1NkHGsxTTgC14rXnDcAjh3+VWN/ =ZsSE -----END PGP SIGNATURE-----
On Nov 4, 2010, at 10:20 AM, Kris Zyp wrote:
the entire design space. If that is incompatible, so be it. But if the current proposal can desugar and provide a smooth transition, why shouldn't it? So far I am not seeing the essential incompatibility.
I repeated these points in AMWB, to what I was afraid was an excessive degree. I guess it wasn't :-/.
There is no way to "desugar" a require("foo") into a prefetch of that URI before the script in which the require call occurs has loaded and is being evaluated.
There is no way to remove the global object from the scope chain in ES5.
It is currently difficult to control visibility of outer names as viewed by an externally-loaded module body.
Arbitrarily complicated compilation or preprocessing is not "desugaring".
I don't see why you came out swinging hard, with lots of words, against Harmony modules, if you now agree with Alex.
In Harmony, we are not mainly codifying what can be done in the language already using first-class objects and functions. Some de-jure codification of de-facto standards will happen, as with JSON, array extras, etc. in ES5. But that is not the focus and we do not want to overdo it before de-facto standards are ripe.
In Harmony, we are aiming to fill semantic gaps that cannot be filled by in-language means, as well as improve syntax all around. This definitely applies to the module system.
Objecting to this with loaded language about "being blind", while making an idol of ill-defined and (when taken too far) unsuitable "desugaring", is not helpful. It misses the purpose of evolving the language beyond adding syntactic conveniences.
Yes, we want to keep kernel semantics as simple as possible and (truly, locally) desugar where possible. No, that principle does not circumscribe our efforts to fill real gaps in the language.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/4/2010 11:42 AM, Brendan Eich wrote:
On Nov 4, 2010, at 10:20 AM, Kris Zyp wrote:
the entire design space. If that is incompatible, so be it. But if the current proposal can desugar and provide a smooth transition, why shouldn't it? So far I am not seeing the essential incompatibility.
I repeated these points in AMWB, to what I was afraid was an excessive degree. I guess it wasn't :-/.
There is no way to "desugar" a require("foo") into a prefetch of that URI before the script in which the require call occurs has loaded and is being evaluated.
I was not suggesting desugaring into CommonJS require() calls. The transport-style API I referred to actually works similar to the interaction module loader API [1] in terms of providing a set of dependencies ([1] uses a call to a load function), provides a callback for executing the script when the modules dependencies are available (like provideSource() call from [1]). Based on the fact this is currently used mechanism in the simple modules proposal, it doesn't seem impossible to use API that it is suitable for ES5 code to work within.
It may well be feasible to that deal with the issues I raised by creating custom module loaders. Is that what you would recommend? The options so far have felt awkward, but I could certainly be missing something. My objection is really based on my inability to see how we use this new system will work with a mix of modules written for ES5, text-based resources, and so on.
There is no way to remove the global object from the scope chain in ES5.
It is possible to write ES5 that doesn't access the global object. Enforcing this as part of simple modules is awesome.
It is currently difficult to control visibility of outer names as viewed by an externally-loaded module body.
Arbitrarily complicated compilation or preprocessing is not "desugaring".
I don't see why you came out swinging hard, with lots of words, against Harmony modules, if you now agree with Alex.
Sorry, I didn't mean to come across harshly. Scaling back was just one of the options I suggested. There is certainly goodness in simple modules. If simple modules are really usable with real web applications, that is great. If it can't meet modern requirements, the simpler building blocks of goodness should be extracted. I assume that is not very controversial. I am definitely open to simple modules being a part of future ecmascript, but right now the ROI hasn't convinced me.
On Nov 4, 2010, at 11:18 AM, Kris Zyp wrote:
I was not suggesting desugaring into CommonJS require() calls.
I know, I said "to ES5". You have to desugar to something in the current ES5 + browser APIs target.
The transport-style API I referred to actually works similar to the interaction module loader API [1] in terms of providing a set of dependencies ([1] uses a call to a load function), provides a callback for executing the script when the modules dependencies are available (like provideSource() call from [1]).
Providing only dynamic module loading, which is a good tool but not enough and not right for the common case, breaks the run-to-completion, apparently single-threaded JS execution model. People use require as if it were a special form, and indeed Caja and other systems (last I heard) preprocess accordingly, and prefetch to preserve run-to-completion.
Based on the fact this is currently used mechanism in the simple modules proposal, it doesn't seem impossible to use API that it is suitable for ES5 code to work within.
This Harmony script:
<script type="harmony">
var invariant = 99; module Foo = "foo"; import Foo.bar; bar(invariant); </script>
is not equivalent to anything like this:
<script>
var invariant = 99; requestModule("foo", function (Foo) { Foo.bar(invariant); }); </script>
because invariant could be varied by a later script or event handler.
This is precisely what I mean by "not possibly in the language currently".
On Nov 4, 2010, at 11:32 AM, Brendan Eich wrote:
This Harmony script:
<script type="harmony"> var invariant = 99; module Foo = "foo"; import Foo.bar; bar(invariant); </script>
is not equivalent to anything like this:
<script> var invariant = 99; requestModule("foo", function (Foo) { Foo.bar(invariant); }); </script>
because invariant could be varied by a later script or event handler.
This is precisely what I mean by "not possibly in the language currently".
Never mind the lack of TCP inherent in desugaring using function (no, we aren't adding Tennent-preserving lambdas).
I suggest not making cargo-cult idols of "what can be done in the language currently". It's great to innovate on the language. It's a terrible idea to make this the only possible future of the language, with only syntactic improvements.
The idol seems to have wings and a propeller, but it is made of palm fronds and it will not fly.
In most languages it's great to innovate on them. In JS the equation is not the same. In JS, any innovation won't be usable for 5+ years, optimistically, because it will take that long for IEs that don't support it to fade to a low enough percentage of use.
On Nov 4, 2010, at 12:04 PM, Nathan Stott wrote:
In most languages it's great to innovate on them. In JS the equation is not the same. In JS, any innovation won't be usable for 5+ years, optimistically, because it will take that long for IEs that don't support it to fade to a low enough percentage of use.
I'm not sure (sorry if my usage of it was unclear) what you mean by "innovate on them".
When I wrote:
It's great to innovate on the language. It's a terrible idea to make this the only possible future of the language, with only syntactic improvements.
I meant "innovate on top of the existing language, using only what its kernel semantics (and any APIs in the relevant embedding) can express."
That is not fine for evolving the language into the future, not by our lights in TC39, nor by anyone looking at the big picture and the long haul. If you took that view in 1994, there would be no JS. In 1995, there would be no ES2 or ES3. The web does evolve, including in the kernel semantics of its core languages. TC39 is looking out for the big picture and the long haul, as best we can.
As for "five years", that's not true historically, and it need not be true going forward. IE4 added ES3-like regexps and sites used them, because Netscape 4 had them too. IE4 did not add try/catch/finally, however, and that was held back for more than five years. It all depends on what browser vendors do to innovate and upgrade their installed bases.
Some IE6 and IE8 (the new IE6?) problems do stick. But they haven't prevented continuing evolution, and developers are either shunning IE6 or targeting it with no-JS or less-JS content. This costs but it's not insuperable.
For Harmony extensions, we in TC39 (including all the major browser vendors save Opera) are prototyping before we standardize. Same as was done, of necessity outside of TC39 and without Microsoft, in the bad old days, when we (Mozilla) added getters and setters. All it took was live.com launching in 2005 with its "Atlas" JS library assuming if (!document.all) that getters and setters were present, and pretty quickly (five months, more like -- not five years) Safari and Opera supported getters and setters too.
Beyond extensions being prototyped and user-tested, and therefore becoming available in leading-edge browser releases, people are developing translation tools. These aren't free lunches either but they beat writing code twice.
And in the long run, however many years it may take assuming browser vendors cooperate in TC39 and continue to work on Chrome-style always-up-to-date user upgrades, we will reach a point, as we did with ES3 and the modern "JS" dialect interoperably implemented, where the new features are there in almost all browsers hitting your server.
ES5 is rolling out now. It won't show up on Windows XP in IE8 of course, unless Microsoft backports Chakra. But all the other four top browser vendors support XP in their current releases, AFAIK.
None of this is perfect, but it is not all as bad as five years minimum. It's also no reason not to evolve the language.
To be very concrete about the modules topic, mapping Harmony modules to ES3 or ES5 is doable without using an async dynamic-loading API, by pre-processing and prefetching dependencies before the scirpt in which the out-of-line module declarations lie is evaluated. This is what Caja, et al, do today.
On Thu, Nov 4, 2010 Kris Zyp wrote:
I've been meaning to make some comments about simple modules and in last minute with Brendan, he noted that he wanted to work with CommonJS to make it really good, so...
Hi, Kris! I saw the same MWB and thought, "Hey, it would be great to have some interaction with TC-39 et al" but never thought to post here. Thanks for taking the initiative!
Incidentally -- if there are es-discuss participants interested in interacting with CommonJS on a more limited basis (reading the mailing list is a lot like drinking from a firehose this quarter) -- feel free to propose something. I'll volunteer to act as a coordinator/organizer if required.
I think one of the key needs that seems to be unaddressed from the current simple modules proposal is the need to have a feasible transition from today's modules to native modules.
I agree in principle, but I don't necessarily agree that we need to maintain exactly the same semantics, nor that ES-Harmony modules need to be shimmable into ES5. I believe that machine-translation of modules will be sufficient, provided that the machine translation can reasonably be written without implementing a full ECMAScript parser.... and heck, even if a full parser is necessary, we could probably leverage Narcissus or something.
This means that we don't need to worry about ES-Harmony module syntax so much as the semantics. The module semantics are fairly well understood; modules are independent units of code which provide exports when asked to do so. If Harmony modules do this, and introduce only syntax relating to the edges of the module system (exports, module definition, module load, import) then I think we'll be okay for machine translation.
Additionally, we have found with CommonJS that current ECMAScript syntax is simply insufficient, in particular with respect to scope chain pollution, dependency resolution and asynchronous module loading. The proposed Harmony primitives should allow a more robust system.
CommonJS modules are based on specifying dependencies using a
require() call which synchronously returns an object providing a module exports.
As you know, there is work underway at CommonJS to try and repair this deficiency. The require() function, while adequate for blocking-friendly environments like Server-Side JavaScript environments is, by itself, insufficient for the browser environment.
Require in CommonJS Modules/1.1 performs three main tasks:
- Dereference the hidden module memo and return the exports object (singleton)
- Load the module
- Initialize the module (compile it)
Work underway decouples these requirements. In particular, module loading is delegated to a browser-friendly API which invokes a callback when the module is ready -- this includes loading its dependencies. Require() will still dereference the module memo, maintaining compatibility with Modules/1.1 module code bodies.
This work requires changing the syntax of the module declaration away from a free-form group of ECMAScript expressions bounded by a file to a group of ECMAScript expressions bounded by a function. This function groups the module's dependency list and the module's factory function together as a unit - making the dependency list available to the ECMAScript host environment without invoking the module's factory function... nor parsing the module's source code.
It may also be possible to decouple initialization of the module from the require() function, allowing module loaders to eagerly compile modules. This would allow a bit better separation of module-system from runtime-code, but I fear that lazy exports evaluation may be too ingrained in the CommonJS idiom at this point. This problem occurs because modules in CommonJS are more than just a collection of exports; invoking a module's factory function can have observable side-effects, such as the binary module which alters String.prototype.
(Note to non-CommonJS observers -- there is no guarantee that the work I am describing (I have been calling it "Modules/2.0") will be recommended by the group; It is subject to a vote, and changing the module format in incompatible way will be a long uphill battle)
RequireJS has become a highly popular library for module loading, and is based on the CommonJS proposal [1] for script tag based modules.
The footnote might not be clear to this group - RequireJS does not actually run CommonJS modules, although it is straightforward to refactor a CommonJS module into the format used by RequireJS.
The difficulties encountered with RequireJS's implementation led directly to the recent innovations around the CommonJS module format.
The RequireJS folks initially advocated changing the module format to allow modules to be written in a such a way that they could be loaded directly (no server-side component) via DOM SCRIPT tag injection. Discussion around this issue led to the realization that module loading and robust dependency resolution without regard to the environment's concurrency module was a bigger problem; a nice bonus is that all these problems have a strong solution-space intersection.
Another aspect of modules worth considering from practical module use in Dojo is that modules don't depend solely on other JavaScript modules.
This is a really excellent point -- one which I had not previously considered. I'm not sure that an ECMAScript module system is the right place to solve it, however.
Is it likely that the W3C will give us DOM events that can notify us of resource availability any time in the near future?
I also don't think extending the native module loader to load non-script resources is such a bad thing, although clearly assistance from the DOM will be required to, say, load a style sheet as part of a module. Your comments about module-system bootstrapping are on point, thoug
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Another idea that might provide the extensibility to deal with loading various resources and alternate/older modules, what if we added the ability to specify the loader to use for a module:
module myApp { module Template = load "template.html" with "text-loader.js"; // load a template text module Messages = load "messages.json" with "i18n.js"; // load some messages module OldModule = load "old-module.js" with "commonjs-loader.js"; module NewShiny = load "a-harmony-module.js"; // use the default native harmony module loader ...
I realize this may involve providing first-class objects to a second-class module system, but that doesn't seem impossible. Thanks, Kris -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - enigmail.mozdev.org
iEYEARECAAYFAkzUTCcACgkQ9VpNnHc4zAxWFQCdGFHfs2ZuI8EUYX/KzHNW/2I9 bigAnj3bS09KAYR0GzaPUs7P4YgTkfoy =59GQ -----END PGP SIGNATURE-----
On Fri, Nov 5, 2010 at 2:25 PM, Kris Zyp <kris at sitepen.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Another idea that might provide the extensibility to deal with loading various resources and alternate/older modules, what if we added the ability to specify the loader to use for a module:
This is of course possible with a module loader, and these are interesting and valuable use cases.
module myApp { module Template = load "template.html" with "text-loader.js"; // load a template text module Messages = load "messages.json" with "i18n.js"; // load some messages module OldModule = load "old-module.js" with "commonjs-loader.js"; module NewShiny = load "a-harmony-module.js"; // use the default native harmony module loader ...
But I don't think using the declarative syntax for this is a good idea
- there's no static scope here; instead, there's arbitrary code execution and evaluation. Additionally, this makes everything synchronous. We want to keep the module syntax for statically-resolved parts of the program, so that programmers and implementations can reason in straightforward ways. When you need more dynamic behavior, that's what module loaders are for.
I realize this may involve providing first-class objects to a second-class module system, but that doesn't seem impossible.
I don't understand this sentence. Note that Simple Modules can already be reified as module instance objects, but I don't think that's what you're referring to here.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/5/2010 12:31 PM, Sam Tobin-Hochstadt wrote:
On Fri, Nov 5, 2010 at 2:25 PM, Kris Zyp <kris at sitepen.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Another idea that might provide the extensibility to deal with loading various resources and alternate/older modules, what if we added the ability to specify the loader to use for a module:
This is of course possible with a module loader, and these are interesting and valuable use cases.
Right, but how does a module express a dependence on a particular module loader? The problem is that it requires knowledge of what module loader is needed, and this must be manually loaded before hand, which is problem that modules is supposed to solve.
module myApp { module Template = load "template.html" with "text-loader.js"; // load a template text module Messages = load "messages.json" with "i18n.js"; // load some messages module OldModule = load "old-module.js" with "commonjs-loader.js"; module NewShiny = load "a-harmony-module.js"; // use the default native harmony module loader ...
But I don't think using the declarative syntax for this is a good idea - there's no static scope here; instead, there's arbitrary code execution and evaluation. Additionally, this makes everything synchronous.
Doesn't the default module loader get executed normally? Why do alternate module loaders change that? Why does it have to be synchronous, the module loader API is asynchronous?
On Nov 5, 2010, at 11:47 AM, Kris Zyp wrote:
[I rewrapped so this is readable, the lines were overlong and citing mangled them. /be]
module myApp { // load a template text module Template = load "template.html" with "text-loader.js";
// load some messages module Messages = load "messages.json" with "i18n.js"; module OldModule = load "old-module.js" with "commonjs-loader.js";
// use the default native harmony module loader module NewShiny = load "a-harmony-module.js"; ... };
[samth here:]
But I don't think using the declarative syntax for this is a good idea - there's no static scope here; instead, there's arbitrary code execution and evaluation. Additionally, this makes everything synchronous.
Doesn't the default module loader get executed normally? Why do alternate module loaders change that?
Why does it have to be synchronous, the module loader API is asynchronous?
The declarative syntax, as I pointed out earlier in this thread, is used to prefetch, so there is no violation of JS's run-to-completion model.
What you seem to be thinking, in your module myApp example, is that everything in between the nested module Template, module NewShiny, etc. lines is either run before those modules load (so cannot use anything loaded, so violates the JS execution model), or is automatically transformed into callbacks or even threaded, so that one could call OldModule.foo(); right after the module OldModule = ... line (btw, the "load" pseudo-keyword is being dropped), and not get a reference error.
But Harmony module syntax exists precisely to preserve the step at a time, no hidden threading or CPS'ing or event loop nesting, execution model, by enabling prefetching of static dependencies.
Dynamic dependencies require use of the module loader API, as Sam said. The static module declarations use the default module loader under the hood, but ahead of execution of the script that contains those module declarations. The script is parsed and the modules prefetched before execution starts.
On Fri, Nov 5, 2010 at 2:47 PM, Kris Zyp <kris at sitepen.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/5/2010 12:31 PM, Sam Tobin-Hochstadt wrote:
On Fri, Nov 5, 2010 at 2:25 PM, Kris Zyp <kris at sitepen.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Another idea that might provide the extensibility to deal with loading various resources and alternate/older modules, what if we added the ability to specify the loader to use for a module:
This is of course possible with a module loader, and these are interesting and valuable use cases.
Right, but how does a module express a dependence on a particular module loader? The problem is that it requires knowledge of what module loader is needed, and this must be manually loaded before hand, which is problem that modules is supposed to solve.
Modules don't and shouldn't get to specify their loader - the whole point of module loaders is for the loader, not the loadee, to specify the semantics.
If you want to load a data file using a special loader, you'd do it like this:
let ld = new TextModuleLoader(); ld.loadModule(['Template', 'template.html'] function (templ) { ... });
If you need to obtain TextModuleLoader async as well, that can be arranged in the same fashion.
module myApp { module Template = load "template.html" with "text-loader.js"; // load a template text module Messages = load "messages.json" with "i18n.js"; // load some messages module OldModule = load "old-module.js" with "commonjs-loader.js"; module NewShiny = load "a-harmony-module.js"; // use the default native harmony module loader ...
But I don't think using the declarative syntax for this is a good idea - there's no static scope here; instead, there's arbitrary code execution and evaluation. Additionally, this makes everything synchronous.
Doesn't the default module loader get executed normally? Why do alternate module loaders change that? Why does it have to be synchronous, the module loader API is asynchronous?
The default module loader gets executed if you say "load". Alternate module loaders don't change that - using module loaders explicitly allows changing that.
The declarative syntax is synchronous because the JS run-to-completion semantics prohibits any other behavior.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/5/2010 12:58 PM, Brendan Eich wrote:
[snip] The declarative syntax, as I pointed out earlier in this thread, is
used to prefetch, so there is no violation of JS's run-to-completion model.
What you seem to be thinking, in your module myApp example, is that
everything in between the nested module Template, module NewShiny, etc. lines is either run before those modules load (so cannot use anything loaded, so violates the JS execution model), or is automatically transformed into callbacks or even threaded, so that one could call OldModule.foo(); right after the module OldModule = ... line (btw, the "load" pseudo-keyword is being dropped), and not get a reference error.
But Harmony module syntax exists precisely to preserve the step at a
time, no hidden threading or CPS'ing or event loop nesting, execution model, by enabling prefetching of static dependencies.
Dynamic dependencies require use of the module loader API, as Sam said.
The static module declarations use the default module loader under the hood, but ahead of execution of the script that contains those module declarations. The script is parsed and the modules prefetched before execution starts.
The prefetching is not performed by the default module loader provided by the host? Can't that default module loader be overriden? When you use a custom module loader why isn't arbitrary code executed when the JS VM requests a module needed by the loading module?
On Nov 5, 2010, at 12:08 PM, Kris Zyp wrote:
The prefetching is not performed by the default module loader provided by the host?
No, it is. But that didn't seem to be the point, compared to the execution model.
Can't that default module loader be overriden?
In the browser, maybe. I'll let Sam take this one, but obviously different embeddings have different trust models and constraints on loaders.
When you use a custom module loader why isn't arbitrary code executed when the JS VM requests a module needed by the loading module?
Yes, but all that racy prefetching happens before the script starts executing, and the modules are parsed but not run.
The execution semantics remain sequential, apparently single-threaded.
On Fri, Nov 5, 2010 at 3:27 PM, Brendan Eich <brendan at mozilla.com> wrote:
Can't that default module loader be overriden?
In the browser, maybe. I'll let Sam take this one, but obviously different embeddings have different trust models and constraints on loaders.
Again, you can create and use new (or existing) module loaders, but you can't change the behavior of your context. That is, a program can't change what restrictions are imposed on it, but it can impose additional restrictions on new contexts that it creates.
On Nov 5, 2010, at 12:33 PM, Sam Tobin-Hochstadt wrote:
On Fri, Nov 5, 2010 at 3:27 PM, Brendan Eich <brendan at mozilla.com> wrote:
Can't that default module loader be overriden?
In the browser, maybe. I'll let Sam take this one, but obviously different embeddings have different trust models and constraints on loaders.
Again, you can create and use new (or existing) module loaders, but you can't change the behavior of your context. That is, a program can't change what restrictions are imposed on it, but it can impose additional restrictions on new contexts that it creates.
Thanks, this is an important point to keep clear.
There definitely is interest in support secure subsets with whitelisting module loaders, e.g. As you say, their confined world of differently-typed scripts will need to be bootstrapped from a <script> tag and that has a loader than can't be rewired.
It seems plausible to me that the web page/app context's script-type-to-module-loader mapping could be configured for types other than the built-in ones. Then you could write
<script type="application/ses"> ... </script>
and use a non-default "Secure ECMAScript" module loader. But I haven't thought through all the security angles, by a long shot.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I've been meaning to make some comments about simple modules and in last minute with Brendan, he noted that he wanted to work with CommonJS to make it really good, so... I think one of the key needs that seems to be unaddressed from the current simple modules proposal is the need to have a feasible transition from today's modules to native modules. I believe it should be a requirement that harmony modules (or at least a subset of them) should be desugarable into ES5 code, such that the desugared code is still a working module in harmony engines that support native modules. This provides a transition by allowing developers to start coding against a harmony module API very soon, using module loader shims for ES5 engines, and dropping them for future harmony-based native loaders. If harmony modules can only be written with new syntax, the harmony-based native module loaders can only be really used once harmony engines are available everywhere, which creates a much slower more painful transition.
CommonJS modules are based on specifying dependencies using a require() call which synchronously returns an object providing a module exports. From the browser, asynchronous module loader is critical, which requires a module loader to determine dependencies prior to executing the module. This can be done by static analysis + evals, but this generally has performance issues and stack traces on eval's are miserable, so browser-focused developers have predominantly moved towards wrapping modules such that modules can be loaded dynamically via script tags. RequireJS has become a highly popular library for module loading, and is based on the CommonJS proposal [1] for script tag based modules. This has proposal has quickly reached broad adoption, with a node-based loader, a server side wrapper, and Dojo has even refactored its code-base in 1.6 to follow this format. I believe we have purposed to evolve the language advised by common patterns seen in the wild, and so this seems like a reasonable starting point for a desugaring of harmony simple modules.
Another aspect of modules worth considering from practical module use in Dojo is that modules don't depend solely on other JavaScript modules. Modules depend on other resources like template files (HTML/text files), NLS bundles (set of JSON files selected based on locale), and possibly CSS files (Dojo doesn't actually support CSS as a dependency, partly because there isn't any good mechanism for determining when a CSS file is loaded due to the lack an onload event for stylesheets in Firefox, but we may in the future). Taking Dojo as an example, there are several ways that Dojo could achieve support for non-JavaScript resources as dependencies support:
This isn't a rare problem. Developers with complex dependencies are exactly the target of simple modules. Ignoring their needs kind of misses the main developers that we are trying to help with simple modules. Handling very simple dependencies is relatively easy to do by hand, and developers with such systems are typically quite content with just script tags.
Frankly, I think that trying to take on a module system for ES6, at least at this point, might not be wise. The harmony "simple modules" is already anything but simple, and it is still very incomplete and immature for real-world module needs. ES5 was a success because it backed down from large complicated new features and focused on small incremental changes that were desperately needed to compose important functionality. Maybe the same principle should be applied again. At least in the area of modules, if we focused on small composable feature(s) that are truly needed to build module systems on top of EcmaScript, we can introduce a much simpler addition, and ensure that we get it right, yet still provide the build blocks for modules with security. Namely, the true nugget from the module system is the ability to evaluate scripts with lexical scoping, and ensure a real filename is associated with the script (for real stack traces, debugging, etc). If ES6 simply had such API, it would be tremendous step forward for loading securable modules in EcmaScript, with minimal complexity cost, no new syntax, smooth transitioning, and the composibility to build real-life module systems. When considering the implementation cost of "simple modules" plus truly critical new features like proxies, weakmaps, and so on, the complexity adds up. A simple lexical scoping improved evaluation API could provide a much better return on our investment, at least for any near future edition.
Anyway, I thought I would throw out these thoughts, if more formal proposals on anything would help, I'd be glad to put something together.
[1] wiki.commonjs.org/wiki/Modules/AsynchronousDefinition