Discuss
  • Modules feedback from March 2013 meeting

Modules feedback from March 2013 meeting

# James Burke (12 years ago)

I expect the module champions to be busy, so I am not expecting a response. This is just some feedback to consider or discard at their discretion. I'll wait for the next public update on modules to see where things end up. In general, sounds promising.

I'm going off the meeting notes from here (thanks Rick and all who make these possible!): rwldrn/tc39-notes/blob/master/es6/2013-03/mar-12.md#42-modules

Single Anonymous Export

The latest update was more about semantics, but some thoughts on how single anonymous export might work:

Just use export for the single anonymous export:

module "m" { export function calculate() {} }

where calculate is just the local name for use internally by the module, but calculate is not visible to outside modules, they just import that single anonymous export.

For exporting a named property:

module "n" { export calculate: function () {} }

This would still result in a local calculate, let-equivalent local name, but then also allows for other modules to import calculate from this module.

Single anonymous export of something that is not a function:

module "crypto" { export let c = { encrypt: function () {}, decrypt: function () {} } }

Inside this module, c is just the local name within the module, not visible to the outside world.

Syntax is hard though, so I will not be surprised if this falls down.

Import

If the above holds together:

For importing single anonymous export, using the "m" above:

import calc from "m";

This module gets a handle on the single anonymous export and calls it calc locally. The "n" example:

import { calculate } from "n";

---- start extremely speculative section:

This next part is very speculative, and the most likely of this feedback to be a waste of your time:

"crypto" is a bit more interesting. It would be neat to allow:

let { encrypt } from "crypto";

which is shorthand for:

import temp from "crypto"; let { encrypt } = temp;

This could all work out because from would still be restricted to the top level of a module (not nested in control structures). from would be the parse hook for finding dependencies, and not import.

If refutable matching: harmony:refutable_matching

applied to destructuring allowed some sort of "throw if property is not there" semantics (making this up, assume a ! prefix for that):

let { !encrypt } from "crypto";

This would give a similar validity check to what import { namedExport } would give. It may happen later in the lifecycle of the module (so when the code was run, vs linking time) but since from is top level, it would seem difficult to observe the difference.

Going one step further:

With that capability, it may be possible to go without import at all, at least at this stage of ES (macros later may require it). The one case where I think import may help are cycles, but if the cycle parts are placed in separate modules with a single export, it may still work out. Using the assumption of single anonymous export and the "odd even" example from the doku wiki:

module 'E' { let odd from 'O'; export function even(n) { return n == 0 || odd(n - 1); } }

module 'O' { let even from 'E'; export function odd(n) { return n != 0 && even(n - 1); } }

Going even further: then the export publicName: value syntax may not be needed either.

My gut says getting to this point may not be possible. Maybe it is for the short term/ES 6, but macros may require import and export publicName:. When documenting the final design decisions, it would be good to address where these speculative steps fall down, as I expect there are some folks in the node community that would also take this train of thought.

--- end extremely speculative section

Loader pipeline hooks

I know more needs to be specified for this, so this feedback may be too early.

The examples look like they assign to the hooks:

System.translate = function () {}

Are these additive assignments? If more than one thing wants to translate, is this more like addEventListener?

I recommend allowing something like AMD loader plugins, since they allow participating in the pipeline without needing to be loaded first before any other modules. It also allows the caller to decide what hooks/transforms should be done, instead of some global handler sneaking in and making the decision.

So, if a dependency is "text at some.html" (AMD loader plugins use !, "text!some.html", pick whatever you all think works as a separator):

  • load the text module.
  • Grab the export value. If the export value has a property (or an explicit export property) that matches one of the pipeline names, normalize, resolve, fetch, translate, etc…, use those when trying to process "some.html".

The main feedback here is to not expect to load all pipeline hooks up front. This would break dependency encapsulation. It seems fine to also allow those hooks to be specified up front before module loading begins, but that should not be the only mechanism.

Declarative loader configuration

Related to System.ondemand: I suggest considering the declarative config worked out by AMD loaders: amdjs/amdjs-api/wiki/Common-Config

I would probably use "alias" instead of "map" now, and I will likely add a "layers" config that is similar to what "ondemand" is: given this layer module ID, here are the fully qualified, exact match module IDs in that file.

"shim" config has proved to be very useful when bridging the gap with legacy code.

Nested modules

This was explicitly stated as a non-goal, but it is something that shows up in the AMD module world (library builds with almond wrapped with a UMD style boilerplate for use by the outside world):

module "publicThing" { module "j" {} module "k" {}

export …. //something visible outside publicThing }

The idea being that internally "publicThing" uses some modules, but they are only for its internal use, not for use outside of "publicThing".

It would be nice to allow a chain of ID lookup tables, favoring a local lookup table/closer tables and working up the chain to find a match. If no match, a module is fetched/loaded and placed in the top level table.

If something like the declarative "map" config from AMD is supported in the loader, it would allow resolving possible top level ID/version conflicts.

Legacy opt-in use case

I did not see this explicitly mentioned in the use cases, but I believe it would help the bootstrapping phase for ES modules if the default loader also parses for System.get and System.set usage in addition to the import or from new syntax. This would allow JS libraries that need to operate in pre-ES6 worlds to still stick with ES3-5 syntax but then be usable in a project that can just use ES6:

//Some base library that needs to be in ES5 syntax: var dep1, dep2 if (typeof System !== 'undefined' && System.get) { //ES6 module loader. The loader will fetch and process //these dependencies before executing this file dep1 = System.get('dep1'); dep2 = System.get('dep2'); } else { //browser globals case, assume the scripts have already loaded dep1 = global.dep1; dep2 = global.dep2; }

The above, coupled with a declarative "shim" config would mean ES6 users could start using ES6 without needing another "script loader" library to use their base libraries that will need to stay in ES3-5-land for a while. This will avoid confusion about a "built in" ES6 feature needing a helper script to use code that exists today.

However, for processing node/commonjs/AMD modules, it is fine to say a helper script is needed that sets up the module loader hooks (for node I expect it will be built-in anyway to their bootstrapping code).

James

# Sam Tobin-Hochstadt (12 years ago)

On Mon, Mar 25, 2013 at 1:31 PM, James Burke <jrburke at gmail.com> wrote:

I expect the module champions to be busy, so I am not expecting a response. This is just some feedback to consider or discard at their discretion. I'll wait for the next public update on modules to see where things end up. In general, sounds promising.

I'm only going to respond to part of this right now, since there's a lot here.

Single Anonymous Export

[snip]

Syntax is hard though, so I will not be surprised if this falls down.

I'm going to leave off responding to syntax discussions at the moment ...

let { encrypt } from "crypto";

... other than to point out that this is a form of import, just with different checks performed on the bindings.

With that capability, it may be possible to go without import at all, at least at this stage of ES (macros later may require it). The one case where I think import may help are cycles, but if the cycle parts are placed in separate modules with a single export, it may still work out. Using the assumption of single anonymous export and the "odd even" example from the doku wiki:

Therefore, these changes aren't really simplifying things.

Loader pipeline hooks

I know more needs to be specified for this, so this feedback may be too early.

The examples look like they assign to the hooks:

System.translate = function () {}

Are these additive assignments? If more than one thing wants to translate, is this more like addEventListener?

These are assignments just like you'd expect -- assigning overwrites the previous value.

New hook implementations will most likely want to either save the existing value and defer to it, or use a more complex library that implements a dispatch/plugin/etc system.

I recommend allowing something like AMD loader plugins, since they allow participating in the pipeline without needing to be loaded first before any other modules. It also allows the caller to decide what hooks/transforms should be done, instead of some global handler sneaking in and making the decision.

So, if a dependency is "text at some.html" (AMD loader plugins use !, "text!some.html", pick whatever you all think works as a separator):

  • load the text module.
  • Grab the export value. If the export value has a property (or an explicit export property) that matches one of the pipeline names, normalize, resolve, fetch, translate, etc…, use those when trying to process "some.html".

This is certainly implementable in the current system, and Dave presented an example of how it works on slide 18 of his presentation.

The system doesn't build AMD-style plugins into the core of the module proposal, however. They're neither fully-general (you could configure based on something other than a prefix) nor used everywhere in existing JS, and we don't want to prematurely standardize on one system.

The main feedback here is to not expect to load all pipeline hooks up front. This would break dependency encapsulation. It seems fine to also allow those hooks to be specified up front before module loading begins, but that should not be the only mechanism.

You can certainly load whatever code you want in loader hooks, subject to synchrony restrictions.

Declarative loader configuration

Related to System.ondemand: I suggest considering the declarative config worked out by AMD loaders: amdjs/amdjs-api/wiki/Common-Config

I would probably use "alias" instead of "map" now, and I will likely add a "layers" config that is similar to what "ondemand" is: given this layer module ID, here are the fully qualified, exact match module IDs in that file.

"shim" config has proved to be very useful when bridging the gap with legacy code.

Fortunately, it looks like the "shim" config is very similar to what the link hook supports.

Nested modules

This was explicitly stated as a non-goal, but it is something that shows up in the AMD module world (library builds with almond wrapped with a UMD style boilerplate for use by the outside world):

module "publicThing" { module "j" {} module "k" {}

export …. //something visible outside publicThing }

The idea being that internally "publicThing" uses some modules, but they are only for its internal use, not for use outside of "publicThing".

You could express this as:

module "publicThing/j" {} module "publicThing/k" {}

module "publicThing" {

export …. //something visible outside publicThing }

This doesn't enforce the privacy of the other modules, but if privacy is required, then you can use internal bindings in "publicThing" that aren't exported.

It would be nice to allow a chain of ID lookup tables, favoring a local lookup table/closer tables and working up the chain to find a match. If no match, a module is fetched/loaded and placed in the top level table.

If something like the declarative "map" config from AMD is supported in the loader, it would allow resolving possible top level ID/version conflicts.

I don't really understand what you're suggesting here.

Legacy opt-in use case

I did not see this explicitly mentioned in the use cases, but I believe it would help the bootstrapping phase for ES modules if the default loader also parses for System.get and System.set usage in addition to the import or from new syntax. This would allow JS libraries that need to operate in pre-ES6 worlds to still stick with ES3-5 syntax but then be usable in a project that can just use ES6:

//Some base library that needs to be in ES5 syntax: var dep1, dep2 if (typeof System !== 'undefined' && System.get) { //ES6 module loader. The loader will fetch and process //these dependencies before executing this file dep1 = System.get('dep1'); dep2 = System.get('dep2'); } else { //browser globals case, assume the scripts have already loaded dep1 = global.dep1; dep2 = global.dep2; }

In this setting, you could just run the code exactly as you wrote it, without changing the default loader at all, and it would work provided that the dependencies were, in fact, already loaded, just the way it's assumed in the browser globals case. I imagine that lots of libraries will work exactly like this, the same way jQuery plugins expect jQuery to already be loaded today.

Adding a build step that performs this analysis explicitly seems like a much better idea than building an ad-hoc analysis that changes the semantics of these methods.

# James Burke (12 years ago)

I really was not expecting a reply, as it was a lot of feedback. Just wanted to get some things in the "to be considered at some point/use case" queue. Some clarifications, but I do not think it is worth continuing discussion here given the breadth of the feedback and the stage of the spec development:

On Mon, Mar 25, 2013 at 12:56 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:

With that capability, it may be possible to go without import at all, at least at this stage of ES (macros later may require it). The one case where I think import may help are cycles, but if the cycle parts are placed in separate modules with a single export, it may still work out. Using the assumption of single anonymous export and the "odd even" example from the doku wiki:

Therefore, these changes aren't really simplifying things.

Does it complicate things more though? While import checking may not change much, hopefully the simplifications are:

  • removal of an import syntax keyword (just have from)
  • possibly reducing the scope of export syntax
  • possible improvement in general destructuring, even when a module is not the target.

Maybe this makes some things much more complicated. If so, it would be good to document why/include in the use cases at some point.

I believe this is at the heart of some of the "this seems complicated" feedback, but hopefully expressed in a more precise, targeted way as to what needs to be explained to someone who might think that. It does not need to be explained now, just calling out a candidate for the documentation/use case queue.

Loader pipeline hooks

The system doesn't build AMD-style plugins into the core of the module proposal, however. They're neither fully-general (you could configure based on something other than a prefix) nor used everywhere in existing JS, and we don't want to prematurely standardize on one system.

AMD-style plugins would purposely not be fully general in this system, that is the job of the loader pipeline hooks.

It does not have to be AMD-style directly, but something where I could specify a module ID that could handle a type of resource ID, that module gets loaded (with its dependencies), and it gets automatically wired into the pipeline if it exposes a property whose name matches a pipeline hook name.

This is also why I suggest more declarative config, like shim vs an imperative link hook. It is still useful to have the loader hooks as they are, but just like "ondemand" is being considered, there are others that have been proven useful without prone-to-error imperative overrides of hooks (don't forget to check for a previous one and call it (before, after?) you run your code).

Since the loader pipeline stuff is still under development though, this feedback may be much to early.

Nested modules

You could express this as:

module "publicThing/j" {} module "publicThing/k" {}

module "publicThing" {

export …. //something visible outside publicThing }

I am sorry, I mixed how the code would be on disk with how it may be organized later conceptually after loading. How the example would be on disk:

//In publicThing.js module "j" {} module "k" {} export …. //something visible outside this module

Then, this module is imported by some other module via the "publicThing" name. So, publicThing.js does not know its final ID, and "j" and "k" are not meant to be exposed as public modules, they are just for publicThing's internal use.

The ID lookup tables I visualized more like the "maps with prototypes" with the prototype being a parent module space map. Unfortunately we probably do not share a common vocabulary here, so I will stop trying to suggest a solution and just point out the use case.

Legacy opt-in use case

//Some base library that needs to be in ES5 syntax: var dep1, dep2 if (typeof System !== 'undefined' && System.get) { //ES6 module loader. The loader will fetch and process //these dependencies before executing this file dep1 = System.get('dep1'); dep2 = System.get('dep2'); } else { //browser globals case, assume the scripts have already loaded dep1 = global.dep1; dep2 = global.dep2; }

In this setting, you could just run the code exactly as you wrote it, without changing the default loader at all, and it would work provided that the dependencies were, in fact, already loaded, just the way it's assumed in the browser globals case. I imagine that lots of libraries will work exactly like this, the same way jQuery plugins expect jQuery to already be loaded today.

We have found in the AMD world that once the developer has a module loader API, they want to avoid loading scripts in some manually constructed order specified outside of JS, in HTML. If the user needs a third party script loader to do this on top of ES modules, that seems redundant.

Adding a build step that performs this analysis explicitly seems like a much better idea than building an ad-hoc analysis that changes the semantics of these methods.

I may be misunderstanding "ad-hoc analysis", but I am only suggesting looking for System.get("StringLiteral") just as the system looks for from "StringLiteral". Specifically, the analysis should not be expanded to node/commonjs/amd APIs.

I am not sure what what semantics you reference will change. If you mean things like "script may not be executed like it is when loaded via an HTML script tag", library authors have adapted, some patterns are here:

umdjs/umd

So library authors are already used to "this module system is in play, the global could be different, I need to behave differently in this way if this module system is available". Getting wider use of their library in other systems is usually worth doing the change for them. That, or letting people know what kind of declarative "shim config" works for it. However, changing their lib to use ES6 syntax is likely not an option for a couple years.

James

# Andreas Rossberg (12 years ago)

On 25 March 2013 18:31, James Burke <jrburke at gmail.com> wrote:

Single Anonymous Export

The latest update was more about semantics, but some thoughts on how single anonymous export might work:

Just use export for the single anonymous export:

module "m" { export function calculate() {} }

where calculate is just the local name for use internally by the module, but calculate is not visible to outside modules, they just import that single anonymous export.

For exporting a named property:

module "n" { export calculate: function () {} }

This would still result in a local calculate, let-equivalent local name, but then also allows for other modules to import calculate from this module.

And how about const declarations, class declarations, module declarations, and any other declaration form that might enter the language at some point? I can't see how your suggestion scales to that.

Also, optimising the entire syntax for one special use case while uglifying all regular ones will be a hard sell.

---- start extremely speculative section:

This next part is very speculative, and the most likely of this feedback to be a waste of your time:

"crypto" is a bit more interesting. It would be neat to allow:

let { encrypt } from "crypto";

which is shorthand for:

import temp from "crypto"; let { encrypt } = temp;

You are trying very hard to avoid using ES6 modules the way they are meant to be used. Just define your crypto module as

module "crypto" { export function encrypt() {} export function decrypt() {} }

and the existing import destructuring will work like a charm, while having made your module both prettier and more concise. ;)

With that capability, it may be possible to go without import at all, at least at this stage of ES (macros later may require it).

As I have explained earlier on this list, destructuring import and destructuring let are not the same. The former introduces aliases, not new stateful bindings. This is relevant if you want to be able to export mutable entities. So no, we cannot drop import.

Going even further: then the export publicName: value syntax may not be needed either.

Only when you're willing to throw all static checking (not just of imports but any module access M.x) out the window.

Nested modules

This was explicitly stated as a non-goal, but it is something that shows up in the AMD module world (library builds with almond wrapped with a UMD style boilerplate for use by the outside world):

module "publicThing" { module "j" {} module "k" {}

export …. //something visible outside publicThing }

The idea being that internally "publicThing" uses some modules, but they are only for its internal use, not for use outside of "publicThing".

It would be nice to allow a chain of ID lookup tables, favoring a local lookup table/closer tables and working up the chain to find a match. If no match, a module is fetched/loaded and placed in the top level table.

I agree with your goal, and that is why I still maintain my point of view that modules should be denoted by regular lexically scoped identifiers, like any good language citizen. Then we'd get the right rules for free, in a clean, declarative manner, and wouldn't need to reinvent the wheel continuously, through more and more operational hacks.

# Kevin Smith (12 years ago)

I agree with your goal, and that is why I still maintain my point of view that modules should be denoted by regular lexically scoped identifiers, like any good language citizen.

I agree with this. I don't think that the concatenation scenario provides a good reason to hold off on lexically scoped nested modules, particularly since much of the design work has already been done.

# James Burke (12 years ago)

On Tue, Mar 26, 2013 at 3:23 AM, Andreas Rossberg <rossberg at google.com> wrote:

On 25 March 2013 18:31, James Burke <jrburke at gmail.com> wrote:

Single Anonymous Export

Also, optimising the entire syntax for one special use case while uglifying all regular ones will be a hard sell.

I believe this is one of the points of disconnect, at least with people in the node and AMD communities. Single exports is regular form, multiple export are seen as the special case.

But my main point with this section was: I was hoping that by turning the syntax around (export for single anonymous, some other export with label for multiple export) maybe that opened up some syntax options. But syntax is hard, and I do not envy TC39's job to sort it all out. Sorry if this was just noise.

As I have explained earlier on this list, destructuring import and destructuring let are not the same. The former introduces aliases, not new stateful bindings. This is relevant if you want to be able to export mutable entities. So no, we cannot drop import.

Right, thank you for the reminder of the previous thread. I can see mutable entities helping cycle cases. I am curious to know what else it helps. But cycle cases are important, so that alone is nice.

I was hoping that with single export, since a mutable entity was a special, new thing, it had more freedom to write the rules around it. However, it seems different enough that it cannot not fit in with let or var.

I wonder if this implies later assignment to the import name is not allowed:

import { decrypt } from 'crypto'; //This would be an error? decrypt = function () {};

If so, that really drives home that it is a new special kind of thing.

In any case, thanks for your response. With that information, this is the kind of summary I would give to node and AMD users:

  • import exists because it creates something new in the language, a reference to a mutable slot. This is really important for cycle resolution. let and var cannot handle this type of mutable slot.

  • multiple exports exists because it allows for better static checking, and due to how import/export works with mutable slots, allows cycles with those exports. While your community may not prefer a multiple export style, there are others that do. Also, in some cases there are "roll up" modules that aggregate an interface to multiple module exports, and the multiple exports allows that to work even with cycles.

  • single anonymous export will be supported, so you can code all your modules in that style and it all works out, and you even get better cycle support when non-function exports are involved. (I have seen cycles in node rely on function hoisting and strategically placed require/module.exports assignment to work -- non-function exports are harder to support with that pattern)

  • node's imperative require is not deterministic enough for a general loading solution, particularly for the web and network fetching. The ES spec solves this by using string literals for dependencies that are language-enforced to be top level, with System.load() for anything that is computed dependency. The mutable slots provided by import give robust cycle support.

  • there are enough hooks in the Module Loader spec to allow node to internally maintain its synchronous require, so it does not have to force all modules to upgrade to a new syntax, and a good level of interop with ES6 modules is possible.

  • AMD's dependency resolution has the right amount of determinism, but suffers from weak cycle support. It also less clear semantically since require('StringLiteral') can be used in control structures like if/else, but operates more like System.get('StringLiteral') --- it just returns the cached module value, it does not trigger conditional code loading. All require('StringLiteral') calls are effectively "hoisted" to the top level for module loading purposes, which can be surprising to the end user, particularly when coming from node.

  • since the ES6 Module Loader can load scripts with the same browser security rules as script tags (load cross domain without CORS, avoids problems with eval, like CSP restrictions) then the need in AMD for a function wrapper in single module per file source form goes away, and you recover a level of indent.

  • there are enough hooks in the Module Loader spec that a hybrid AMD/ES6 loader can be made, so no need to force upgrade all your AMD modules, it can be done over time. Since AMD's execution model aligns pretty well with the ES6 model, it will be easy to write conversion scripts.

Nested modules

I agree with your goal, and that is why I still maintain my point of view that modules should be denoted by regular lexically scoped identifiers, like any good language citizen. Then we'd get the right rules for free, in a clean, declarative manner, and wouldn't need to reinvent the wheel continuously, through more and more operational hacks.

Some dependencies are referenced via strings, from "some/thing". Those string names need to be there for modules not built into the local collection of modules. Having two ways to refer to modules seems like one too many.

System.get() looks up via string IDs. Once in string space, it is best to stay in string space. Otherwise, code rewriting rules are needed, that rewritten code moves further away from the source form of the code, and it still seems that a "string ID to lexical ID" lookup table would be needed, to handle all the cases of imperative System.get() usage.

James