ES modules: syntax import vs preprocessing cs plugins
On Sat, Jun 30, 2012 at 2:20 AM, Claus Reinke <claus.reinke at talk21.com>wrote:
When reading Dave's post on "Static module resolution" [1], the section on "Future-compatibility for macros" struck me as a case where users/proponents of different module systems seem to be talking past each other. All agree that there is an important feature, but the approaches differ so much that the common purpose may not be obvious.
IMO this feature should be staged for addition after we get modules in ES.
jjb
When reading Dave's post on "Static module resolution" [1], the section on "Future-compatibility for macros" struck me as a case where users/proponents of different module systems seem to be talking past each other. All agree that there is an important feature, but the approaches differ so much that the common purpose may not be obvious.
IMO this feature should be staged for addition after we get modules in ES.
Can we agree what the feature is, though? Macros are not scheduled for this round, loader plugins are in current use with requirejs/AMD, as is require.extensions in node.
Looking closer at
there are interesting options for the Loader constructor, namely the resolve, fetch and translate hooks. Those look as if the existing features of current module systems could be emulated. An extended loader could be chained with the system loader.
However, I see two problems:
-
is there a way to use extended loaders with the module syntax for import/export? Some way to extend the system loader, before syntactic module loading commences?
-
since Loader.prototype.load is asynchronous, how does one export from a dynamically loaded module, and how does one integrate a dynamically loaded module in the normal module dependency resolution?
Claus
On Tue, Jul 3, 2012 at 4:22 PM, Claus Reinke <claus.reinke at talk21.com> wrote:
When reading Dave's post on "Static module resolution" [1], the section on "Future-compatibility for macros" struck me as a case where users/proponents of different module systems seem to be talking past each other. All agree that there is an important feature, but the approaches differ so much that the common purpose may not be obvious.
IMO this feature should be staged for addition after we get modules in ES.
Can we agree what the feature is, though? Macros are not scheduled for this round, loader plugins are in current use with requirejs/AMD, as is require.extensions in node.
Looking closer at
there are interesting options for the Loader constructor, namely the resolve, fetch and translate hooks. Those look as if the existing features of current module systems could be emulated. An extended loader could be chained with the system loader.
However, I see two problems:
[problems snipped]
There are two issues here. One is what code evaluated in nested
loaders looks like. That's entirely up to the loader, but it
certainly can use import and export and all of the other features
of the module system. Those references are resolved by the loader
being used.
The second question is how we specify a particular loader to use.
This is easy to do in JS code: myLoader.load(url, callback). If we
want convenient syntax for using this in a particular environment,
then that would require an extension to the environment, such as an
HTML declaration to use a particular loader for the rest of the page,
or for a particular script tag. This is certainly doable, but
requires coordination outside of TC39.
On Jul 3, 2012, at 3:36 PM, Sam Tobin-Hochstadt wrote:
...
[problems snipped]
There are two issues here. One is what code evaluated in nested loaders looks like. That's entirely up to the loader, but it certainly can use
importandexportand all of the other features of the module system. Those references are resolved by the loader being used.The second question is how we specify a particular loader to use. This is easy to do in JS code:
myLoader.load(url, callback). If we want convenient syntax for using this in a particular environment, then that would require an extension to the environment, such as an HTML declaration to use a particular loader for the rest of the page, or for a particular script tag. This is certainly doable, but requires coordination outside of TC39.
Sam, Isn't it also the case that the full characteristics of the default module loader used by browsers still remain to be specified? This might be somewhat out of scope for TC39 put practically speaking it's something we will need (and want) to be involved with.
On Tue, Jul 3, 2012 at 7:19 PM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
On Jul 3, 2012, at 3:36 PM, Sam Tobin-Hochstadt wrote:
...
[problems snipped]
There are two issues here. One is what code evaluated in nested loaders looks like. That's entirely up to the loader, but it certainly can use
importandexportand all of the other features of the module system. Those references are resolved by the loader being used.The second question is how we specify a particular loader to use. This is easy to do in JS code:
myLoader.load(url, callback). If we want convenient syntax for using this in a particular environment, then that would require an extension to the environment, such as an HTML declaration to use a particular loader for the rest of the page, or for a particular script tag. This is certainly doable, but requires coordination outside of TC39.Sam, Isn't it also the case that the full characteristics of the default module loader used by browsers still remain to be specified? This might be somewhat out of scope for TC39 put practically speaking it's something we will need (and want) to be involved with.
Yes, this needs to be fully specified, but Dave and I have thought a bunch about this particular issue, and I think the issues here are better understood, because they're similar to other ES/HTML integration issues. As an example, where the system loader looks for JS source specified with a relative path should be related to how the browser does this for script tags.
There are two issues here. One is what code evaluated in nested loaders looks like. That's entirely up to the loader, but it certainly can use
importandexportand all of the other features of the module system. Those references are resolved by the loader being used.The second question is how we specify a particular loader to use. This is easy to do in JS code:
myLoader.load(url, callback). If we want convenient syntax for using this in a particular environment, then that would require an extension to the environment, such as an HTML declaration to use a particular loader for the rest of the page, or for a particular script tag. This is certainly doable, but requires coordination outside of TC39.
I think there has been a disconnect in how you and I think about using the new module loaders. Since you're involved in the design, its probably my view that needs adjustment, but I'd appreciate if your view could be made clearer on the module loader page. At the moment, I do not have sufficient information to decide whether the proposed functionality will be sufficient in practice.
To outline the disconnect, consider this simple static module chain for a dummy project ab I might like to use:
// a.js
export var a = "a";
// b.js
import {a} from 'a';
export var b = a+"b";
To use this in my main code, I again use static import
// main.js (or perhaps a script element in index.html)
import {b} from 'b'; // early failure-protection here
.. other imports .. // executed in sequence, but
// can be loaded in parallel
.. lots of code that uses imports ..
// if this code is part of a HTML
// source, we can use imports
// to fill static HTML syntax here
Now, if I need to use module loader hooks while importing the ab project, I thought there would be a way to extend the default loader, then keep the rest of main.js unchanged.
Instead, you seem to say that any use of module loader functionality for ab will require a rewrite of main.js. The module loader API gives us two options for this rewrite. The obvious option is to move main.js code into the callback
// main.js
let myLoader = new Loader(...);
myLoader.load('b',function(bMO){
let {b} = bMO; // no more early failure here
.. other imports .. // these have to be rewritten,
// cannot use module syntax,
// are now loaded in sequence
// (instead of in paralell)
.. lots of code that uses imports ..
// import API failures can happen
// at any time now;
// no static context functionality
can be // used here (eg, if main was originally // part of an HTML file, and the imports // were used in <body>) }); // there is no easy way to use the imports at this point
If the callback represents the dynamic continuation of .load, there is also the option of extending the import chain instead, which gives us a static continuation (one in which we can use the static module syntax in code that can use the imports)
// main.js
let myLoader = new Loader(...);
myLoader.load('old-main',function(mainMO){
let {...} = mainMO;
// can connect mainMO exports to original context here
});
// cannot easily use mainMO or other imports here
// old-main.js
import {b} from 'b'; // early failure-protection here
.. other imports .. // executed in sequence, but
// can be loaded in parallel
.. lots of code that uses imports ..
// cannot use context functionality
here export ... // can return something to // importer
These are the two options that come to mind. Have I missed
anything? Is it the second option that you are thinking of when
you say "it certainly can use import and export and all of the
other features of the module system. Those references are
resolved by the loader being used."?
Claus
On Wed, Jul 4, 2012 at 4:50 AM, Claus Reinke <claus.reinke at talk21.com> wrote:
I think there has been a disconnect in how you and I think about using the new module loaders. Since you're involved in the design, its probably my view that needs adjustment, but I'd appreciate if your view could be made clearer on the module loader page. At the moment, I do not have sufficient information to decide whether the proposed functionality will be sufficient in practice.
To outline the disconnect, consider this simple static module chain for a dummy project ab I might like to use:
// a.js export var a = "a";
// b.js import {a} from 'a'; export var b = a+"b";
To use this in my main code, I again use static import
// main.js (or perhaps a script element in index.html) import {b} from 'b'; // early failure-protection here .. other imports .. // executed in sequence, but // can be loaded in parallel .. lots of code that uses imports .. // if this code is part of a HTML // source, we can use imports // to fill static HTML syntax here
Now, if I need to use module loader hooks while importing the ab project, I thought there would be a way to extend the default loader, then keep the rest of main.js unchanged.
In what way do you want to extend the default loader? Some changes
don't require modifying main.js at all. For example, we might have:
<script>
System.set('jquery', {...}); </script> <script> // main.js import {b} from 'b'; import $ from 'jquery'; ... </script>
On the other hand, if you want to set up a new loader that translates CoffeeScript to JS, then someone needs to invoke that new loader.
If you can give an example of what you want to change about the default loader, then perhaps I can be more specific.
Both of your snipped examples are reasonable uses of the loader API.
Some of your problems with them seem to be general complains about
asynchrony, rather than issues with the module loading system. For
example, making the module available outside the callback passed to
.load could involve promises, or yield with task.js, or something
else, but we can't just add synchronous loading.
To outline the disconnect, consider this simple static module chain for a dummy project ab I might like to use:
// a.js export var a = "a";
// b.js import {a} from 'a'; export var b = a+"b";
To use this in my main code, I again use static import
// main.js (or perhaps a script element in index.html) import {b} from 'b'; // early failure-protection here .. other imports .. // executed in sequence, but // can be loaded in parallel .. lots of code that uses imports .. // if this code is part of a HTML // source, we can use imports // to fill static HTML syntax here
Now, if I need to use module loader hooks while importing the ab project, I thought there would be a way to extend the default loader, then keep the rest of main.js unchanged.
In what way do you want to extend the default loader?
Similar to the dummy project sources, the nature of the loader is not important to the concerns expressed in my message (in the comments).
What is important is loader usage and, in particular, the requirement to rewrite mostly synchronous and syntactic module usage into mostly asynchronous and dynamic module usage, only to accommodate a single added loader- dependent import.
Both of your snipped examples are reasonable uses of the loader API.
Thanks, so I did not misread the spec, at least. However, I do not consider the need to rewrite the original example into one of the snipped variants a reasonable requirement in practice.
Meanwhile, a third variant occurred to me: we might be able to make use of the staged loading to limit the impact of the rewrite, and to recover from asynchronous loading and dynamic usage back to synchronous loading and module syntax. This is going to be a variant of my second example, in a web page context:
// old-main.js
import {b} from 'b';
.. other imports ..
.. lots of code that uses imports ..
export ...
// index.html
<script>
let myLoader = new Loader(...);
myLoader.load('old-main',function(mainMO){
System.set('main',mainMO),
});
</script>
<script>
// this second script element will be resolved in a
// separate 'phase', with 'main' available, right?
import {...} from 'main';
</script>
.. can we now use imports from 'main' in the <body>?
But even if this works out in a web page (does it?), what is the equivalent mechanism server-side?
Claus
On Thu, Jul 5, 2012 at 5:19 PM, Claus Reinke <claus.reinke at talk21.com> wrote:
To outline the disconnect, consider this simple static module chain for a dummy project ab I might like to use:
// a.js export var a = "a";
// b.js import {a} from 'a'; export var b = a+"b";
To use this in my main code, I again use static import
// main.js (or perhaps a script element in index.html) import {b} from 'b'; // early failure-protection here .. other imports .. // executed in sequence, but // can be loaded in parallel .. lots of code that uses imports .. // if this code is part of a HTML // source, we can use imports // to fill static HTML syntax here
Now, if I need to use module loader hooks while importing the ab project, I thought there would be a way to extend the default loader, then keep the rest of main.js unchanged.
In what way do you want to extend the default loader?
Similar to the dummy project sources, the nature of the loader is not important to the concerns expressed in my message (in the comments).
I disagree -- it's not possible to know what the requirements are, how else they could be solved, what the current state-of-the-art is, and how common this will be without further details.
Meanwhile, a third variant occurred to me: we might be able to make use of the staged loading to limit the impact of the rewrite, and to recover from asynchronous loading and dynamic usage back to synchronous loading and module syntax. This is going to be a variant of my second example, in a web page context:
// old-main.js import {b} from 'b'; .. other imports ..
.. lots of code that uses imports .. export ...
// index.html <script>
let myLoader = new Loader(...); myLoader.load('old-main',function(mainMO){ System.set('main',mainMO), }); </script> <script> // this second script element will be resolved in a // separate 'phase', with 'main' available, right? import {...} from 'main'; </script> .. can we now use imports from 'main' in the <body>?
But even if this works out in a web page (does it?),
Yes, this should work fine.
what is the equivalent mechanism server-side?
That's a choice that the designers of server-side JS frameworks can make. The boundary between script tags could be translated into the boundary between files, or statements, or forms typed into the REPL, or something else.
Similar to the dummy project sources, the nature of the loader is not important to the concerns expressed in my message (in the comments).
I disagree -- it's not possible to know what the requirements are, how else they could be solved, what the current state-of-the-art is, and how common this will be without further details.
Well, I got useful answers (thanks!) without distractions by unnecessary details;-)
But if you insist, consider the following scenario: adding a project dependency which is written in current node style (require/export; in a subset that can be translated into ES modules by using a translate hook). And doing so without rewriting the existing code, in either the dependency or the importer.
Meanwhile, a third variant occurred to me: we might be able to make use of the staged loading to limit the impact of the rewrite, and to recover from asynchronous loading and dynamic usage back to synchronous loading and module syntax. .. Yes, this should work fine.
Okay, that solves part of the problem for client-side apps. We can use extended loaders without having to switch every import to callback style.
Would this, in combination with the resolve hook, also allow to make dynamic decisions about which dependency to import (or rather, which code to return when a certain dependency is imported)? This seemed to be one of the features James was requesting.
It is still more complex than I'd like (I was hoping for something like requirejs.config [*], with added loader configuration), but at least this works.
[*] jrburke/requirejs/wiki/Upgrading-to-RequireJS-2.0
It would be useful to have examples of adding loader-based dependencies to a project which already has other imports, on the wiki page. Please feel free to reuse the three examples if you like.
what is the equivalent mechanism server-side?
That's a choice that the designers of server-side JS frameworks can make. The boundary between script tags could be translated into the boundary between files, or statements, or forms typed into the REPL, or something else.
This, however, I cannot support. Yes, there are decisions which tc39 needs to leave to the bodies standardizing the ES host environments. Nevertheless, it should be possible to write cross-platform ES code without reference to the host environment.
In other words, I would like to see an in-ES solution to the issue of using module loaders without dropping out of synchronous loading and module syntax.
Then the host environment bodies could provide additional, simpler ways to use this in-ES functionality.
If a project has to be rewritten when taking it out of client-side host environment, then a big part of the advantage of having an ES standard module system is gone.
Thanks for engaging in these discussions. I think they are important to find and smooth out the rough edges of the ES modules spec.
Claus
On Thu, Jul 5, 2012 at 6:13 PM, Claus Reinke <claus.reinke at talk21.com> wrote:
Similar to the dummy project sources, the nature of the loader is not important to the concerns expressed in my message (in the comments).
I disagree -- it's not possible to know what the requirements are, how else they could be solved, what the current state-of-the-art is, and how common this will be without further details.
Well, I got useful answers (thanks!) without distractions by unnecessary details;-) But if you insist, consider the following scenario: adding a project dependency which is written in current node style (require/export; in a subset that can be translated into ES modules by using a translate hook). And doing so without rewriting the existing code, in either the dependency or the importer.
Ok, this will work fine using the multiple script tags approach you showed.
Meanwhile, a third variant occurred to me: we might be able to make use of the staged loading to limit the impact of the rewrite, and to recover from asynchronous loading and dynamic usage back to synchronous loading and module syntax. ..
Yes, this should work fine.
Okay, that solves part of the problem for client-side apps. We can use extended loaders without having to switch every import to callback style.
Would this, in combination with the resolve hook, also allow to make dynamic decisions about which dependency to import (or rather, which code to return when a certain dependency is imported)? This seemed to be one of the features James was requesting.
Yes, in the code you showed, you were just writing JS code, so obviously it could have conditionals in it.
what is the equivalent mechanism server-side?
That's a choice that the designers of server-side JS frameworks can make. The boundary between script tags could be translated into the boundary between files, or statements, or forms typed into the REPL, or something else.
This, however, I cannot support. Yes, there are decisions which tc39 needs to leave to the bodies standardizing the ES host environments. Nevertheless, it should be possible to write cross-platform ES code without reference to the host environment.
In so far as the ES spec makes no reference to files, script tags, or REPLs, this would be a big step in tc39 taking over the responsibility of designing other people's systems. For browsers, tc39 has most of the vendors in the room, but that's not the case for other kinds of JS embeddings.
If you have a suggestion for how this could work, I'm all ears, but so far, this sounds unlikely.
Ok, this will work fine using the multiple script tags approach you showed.
Actually, I no longer think that third approach will work: there is no reason why the load callback in the first script element should run before the code in the second script element, so 'main' will not be available there.
Okay, that solves part of the problem for client-side apps. We can use extended loaders without having to switch every import to callback style.
That means I'm back to square one: no way to use extended loaders without rewriting every module construct to callbacks.
.. there are decisions which tc39 needs to leave to the bodies standardizing the ES host environments. Nevertheless, it should be possible to write cross-platform ES code without reference to the host environment.
In so far as the ES spec makes no reference to files, script tags, or REPLs, this would be a big step in tc39 taking over the responsibility of designing other people's systems. For browsers, tc39 has most of the vendors in the room, but that's not the case for other kinds of JS embeddings.
If you have a suggestion for how this could work, I'm all ears, but so far, this sounds unlikely.
It should not be necessary to reference external elements any more than in the rest of the modules spec. As a fallback approach (if we don't find anything else), we could take a leaf from the AMD loader plugins (things like "text!resource"):
import {b} from 'b' using 'myLoader.js';
.. other imports ..
.. lots of code that uses imports ..
Since modules are resolved before code is run, we cannot easily get myLoader from the current scope (in HTML, it could come from a previous script element, but how would that work in node?). So the idea is to fetch loaders themselves as modules:
If 'myLoader.js' contains a module with an export interface that corresponds to a module loader, then the import-from-using line could use that loader to do the import.
Claus
When reading Dave's post on "Static module resolution" [1], the section on "Future-compatibility for macros" struck me as a case where users/proponents of different module systems seem to be talking past each other. All agree that there is an important feature, but the approaches differ so much that the common purpose may not be obvious.
ES modules: compile-time imports allow for syntax import, once syntax declarations/exports/macros are introduced
node modules: require.extensions [2] (long time undocumented) allows registering alternative loaders for preprocessing alternative syntax (coffeescript, streamline, ..)
require.js/AMD: loader plugins allow to handle alternative resource formats as module dependencies [3,4]
I just wanted to make sure that participants in this discussion are aware of these similarities. [4] does briefly compare AMD's loader plugins with node's require.extensions.
Btw: for all the usefulness of plugins, they do seem to express syntax dependencies (which plugin to use) differently than code dependencies (which modules to load).
Claus clausreinke.github.com
[1] calculist.org/blog/2012/06/29/static-module-resolution [2] nodejs.org/docs/latest/api/globals.html#globals_require_extensions [3] www.sitepen.com/blog/2012/06/25/amd-the-definitive-source [4] tagneto.blogspot.ca/2012/06/es-modules-suggestions-for-improvement.html
When reading Dave's post on "Static module resolution" [1], the section on "Future-compatibility for macros" struck me as a case where users/proponents of different module systems seem to be talking past each other. All agree that there is an important feature, but the approaches differ so much that the common purpose may not be obvious. - ES modules: compile-time imports allow for syntax import, once syntax declarations/exports/macros are introduced - node modules: require.extensions [2] (long time undocumented) allows registering alternative loaders for preprocessing alternative syntax (coffeescript, streamline, ..) - require.js/AMD: loader plugins allow to handle alternative resource formats as module dependencies [3,4] I just wanted to make sure that participants in this discussion are aware of these similarities. [4] does briefly compare AMD's loader plugins with node's require.extensions. Btw: for all the usefulness of plugins, they do seem to express syntax dependencies (which plugin to use) differently than code dependencies (which modules to load). Claus http://clausreinke.github.com/ [1] http://calculist.org/blog/2012/06/29/static-module-resolution/ [2] http://nodejs.org/docs/latest/api/globals.html#globals_require_extensions [3] http://www.sitepen.com/blog/2012/06/25/amd-the-definitive-source/ [4] http://tagneto.blogspot.ca/2012/06/es-modules-suggestions-for-improvement.html