Discuss
  • Modules feedback, proposal

Modules feedback, proposal

# James Burke (13 years ago)

First I'll give some smaller bits of feedback, and then at the end propose something that integrates all the feedback.

I hope they can considered independently: if you do not like the proposal, feel free to just take in the feedback. I tried to order the feedback in order of "craziness" with the first item being least crazy.

======== Feedback

Some of these just may be drift between current thinking and what is on the wiki. Sorry for anything that is already covered by current thinking, I mostly just have the wiki as a reference:

  1. Multiple mrls for System.load():

Dave Herman's latest post on "Synchronous module loading in ES6": calculist.org/blog/2012/03/29/synchronous-module-loading-in-es6

mentions only allowing System.load() in inline HTML script tags. I'm assuming that is similar to the "Loader.prototype.load( mrl, callback, errback )" API in: harmony:module_loaders

Since System.load() is the only allowed option in HTML script tags, it should allow specifying more than one mrl in the call. Either:

System.load("jquery.js", "underscore.js", function ($, _) {}) or System.load(["jquery.js", "underscore.js"], function ($, _) {})

  1. Default module path resolution

In the "module path resolution" thread, Dave mentioned that the module_loaders resolve API should allow for some custom path resolution logic. However it would be good to specify the default logic.

For the default logic, it would be great to see it be:

baseUrl + ID + ".js"

unless the ID contains a protocol, or starts with a "/", then ID is assumed to be an URL and used as-is. I'm sure the real default behavior needs a stronger definition, and I would like to see the equivalent of AMD loaders's "paths" config as an option for default config, but the main point is that in code, the IDs for importing/referencing would look like this:

import $ from "jquery";

instead of this:

import $ from "jquery.js";

I am just using an arbitrary import example, not sure what the most current import syntax is. Nothing else implied here for "import", but the ability to use "jquery" for the name and it works by default given the rules above.

This is a similar module ID-to-path resolution used by AMD loaders today, and it has worked out well for us because it allows setting up references to "implementation providers" vs. particular URLs. This is nice when using a few libraries that depend on the same piece of functionality. A concrete example:

I use jQuery in my app, and I use Backbone, which also uses jQuery. Ideally my script that directly uses jQuery and the Backbone module can both do:

import $ from "jquery";

and it gets resolved to the same URL. With the "paths" config that is possible in AMD loaders, we can map that "jquery" to be a CDN version, or a local version, or a local version that is actually Zepto, just something that can stand in for jQuery. I like the idea of having a simpler "paths" config for that instead of requiring the developer to implement a resolver function or worse include a library to set up the resolver (might as well just use an AMD loader then :).

  1. Modules IDs as strings

This item of feedback is assuming that the way to get better optimized code delivery is to "concatenate" module files together. However, even if that is not the case for browser delivery, I still believe that allowing a way to combine a set of modules together in a file helps just with code sharing -- it is easier to handle and pass around a single JS file for distribution, but the library dev may still want to work with the modules separately on disk.

With that assumption, module IDs should always be strings. This makes it easier to combine modules together and match the module to the string used in the import call:

module "jquery" { }

module "foo" { import $ from "jquery"; }

This means that "foo" will get a handle on the "jquery" module. By using string IDs even in the "module {}" part, it makes it easier to match the references to the module provider, particularly for combined files.

  1. export call

There was mention in "simpler, sweeter syntax for modules" thread by Dave that maybe with syntax like this: import $ from "jquery.js";

that having a way to export a function may not be needed.

However, I still feel some sort of "export call" is needed, or some way to export a function as the module value, to give parity with the runtime calls:

System.load('jquery.js', function ($) { });

It would be awkward to do this:

System.load('jquery.js', function (something) { var $ = something.$; });

  1. Compile time linking

There is a tension between the runtime calls like System.load and the compile time linking of the module/import syntax. The runtime capabilities cannot be removed. However, I believe it would simplify the story for an end user if the the compile time linking is removed.

While the compile time linking may give some kinds of type checks/export name checking, it is only one level deep, it does not help with this sort of checking:

//Compile time checking can make sure //'jquery.js does export a $ import $ from 'jquery.js';

//However, it cannot help check if foo is //a real property $.foo();

Similar story for prototypical properties on constructor functions.

New possibilities open up if this the compile time stuff is removed, and I believe it simplifies the end user's day-to-day interaction with modules (more below).

  1. Import syntax

If the compile time linking/checking is removed, module referencing gets simpler. No more import stuff, just use "from" to indicate the value is fetched from a module:

let $ from "jquery"; let {search, fetch} from "service"; let {*} from "math";

Normal var/let and destructuring is used. The only new thing is "from" which indicates a module source. It only allow a string literal for the module name.

Not sure if that last * example is valid, but maybe allow it to be valid to support the "import *" cases. If it is untenable, then throw it out. Node and AMD users have done just fine without having *.

  1. Object destructuring

Not module-specific, but it comes up in modules. In the "simpler, sweeter syntax for modules", Brendan wrapped up that mini-thread it up by it just needs to be documented and taught, but I cannot help calling it out again, particularly since I do not post very often to this list.

No need to respond to this item, I just want to put my voice behind the backwards-ness of:

let { draw: drawWidget } = widget;

It will always stick out because object literals are constructed the other way, and object literals show up in the code next to these destructuring calls. Even if you know the rules, scanning code always requires a mental discontinuity jump to parse out what is going on. Yes, there is logic behind the way it is, but it scans wrong, takes the developer out of the flow. Sorry, had to get that out.

======== Proposal

Here is how module syntax might look given the feedback above:

module "foo" { let $ from "jquery", {search, fetch} from "service";

//Use return to specify the exports, or optionally
//use exports.foo = to assign values to properties
//on an exports object, for use in circular dependencies
return function () {};

}

The module "foo" {} wrapper would not be needed in a file that is just declaring a module (think how node modules are constructed today). It is just for inlined module declarations, like when a bunch of modules are combined together.

For the unwrapped case, it means allowing "return" for unwrapped modules in a place that normally has not been allowed. Hopefully that can be worked out if a module is in play and not a Program (I am not a grammarian, may have that wrong).

To allow existing libraries to work in non ES.next and ES.next environments, perhaps also allow using an API form of the above:

module("foo", function (from, exports) { var $ = from("jquery"), service = from("service"), search = service.search, fetch = service.fetch;

//Use return to specify the exports, or optionally
//use exports.foo = to assign values to properties
//on an exports object, for use in circular dependencies
return function () {};

});

If "module" does not work for an API name, this last form is close enough that the AMD "define" could be used instead, along with swapping "require" for "from". But that is a minor naming bikeshed. I would try to match the ES syntax names if possible.

The main point is to have an API that could be runtime-checked for libraries that want to live in both the old and new world.

James

# Sebastian Markbåge (13 years ago)

I'd like to add some counter points.

  1. Multiple mrls for System.load():

I think it makes sense to have this convenience, even though it's probably better practice to use the static syntax as much as possible. For performance and tooling support. There's some precedence in Web Workers.

  1. Default module path resolution

There should be a way to define completely custom URL resolution for a specific loader, yes. The resolver could add a "canonicalize" method that can resolve a path relatively to the current file.

I don't think it makes sense for ECMAScript to define the default scheme at all. That should be up to the host environment. The browser environment should default to the same resolution algorithm as script tags or importScripts in Web Workers.

For more complex schemes that require configuration, you cannot statically resolve the configuration (unless you add the complexity of static configuration as well). Therefore you have to use the dynamic loader. Then you already have the option to override the resolution scheme when you call the loader.

Trying to standardize a more complex default scheme is likely to open up a lot of bike shedding.

  1. Modules IDs as strings

What you're proposing require concensus on path resolution (i.e. module IDs). I think that's difficult on it's own.

When both "jquery" and "foo" is in the same file; a compiler can easily exchange the string references to unique symbols when combining them into a single file.

When they're in different files; you have a possible race condition. You would have to make sure you load them sequentially. This means you need to use the loader API to control this sequence.

The current loader API draft lets you have your package containing jQuery define the module on the loader.

module jQuery { } System.set("jquery", jQuery);

This way, the subsequent call to load "foo", will use the already define "jquery" module instead of loading it from another source.

Therefore I don't think it's necessary to change module IDs to strings. It'll likely bring up more bike shedding surrounding point 2.

  1. export call

This would be nice to have, but it does open up a can of worms. I don't have a nice idea how to do this in a simple and clean way, while retaining the requirement of static export names.

  1. Compile time linking

Yea, it's true it's just one level. Ultimately you'd hope that engine and tool implementors can dynamically link any property reference to frozen objects. At least this one level certainly makes it easier to predict this behavior for that single level.

More importantly, compile time linking facilitate the import * syntax. Without named exports, it would be not be possible to give early warnings on misspelled variables, refactor etc. I.e. it's a better alternative than the with statement.

  1. Import syntax

Obviously I certainly don't agree that Node and AMD users have done just fine without having *. That's one of the reasons I created Labeled Modules. labeledmodules/labeled-modules-spec/wiki

Aliasing every single API at the top of the file quickly gets bloated. The alternative is to put them into an object as a namespace. Now you have a level of indirection that obstructs the readability of your code.

I.e. Math.abs(Math.ceil(x) + Math.floor(y)) instead of abs(ceil(x) + floor(y))

It also makes it easier to move APIs between modules and group them into packages without changing your code.

As for your proposal to use let instead of import. I'd suggest just letting import be used as an expression. That way you can also pass a module instance directly.

let foo = import "foo"; const bar = import "bar"; let {search, fetch} = import "service";

pass(import "A", import "B");

This would not let you reference foo as static module from other import statements, if compile time linking is enforced.

For fun here's a multi-file loader using evalSync and import as an expression:

function load(files, callback){ evalAsync( '[' + files.map(file => 'import "' + file + '"') + ']',

result => callback(...result)) ); }

In conclusion, I like the current proposal but I'd like to add one of my concerns that I've mentioned to David Herman already.

  1. Export syntax

The current export syntax will break any ES.old parser. This means that it's impossible to create universal libraries that work in both ES.old and ES.next. Other than by passing the module system completely.

I'd like to propose that we need an alternate way to export variables that's compatible with ES.old. Otherwise we'll have an unnecessary split between ES.old and ES.next when it could be perceived as a single language.

I like the label approach. It couldn't break existing code and still allow for static exports.

# James Burke (13 years ago)

On Sat, Mar 31, 2012 at 8:54 AM, Sebastian Markbåge <sebastian at calyptus.eu> wrote:

Hi James,

I'd like to add some counter points.

  1. Multiple mrls for System.load():

I think it makes sense to have this convenience, even though it's probably better practice to use the static syntax as much as possible. For performance and tooling support. There's some precedence in Web Workers.

In Dave's post: calculist.org/blog/2012/03/29/synchronous-module-loading-in-es6

the only way to use modules in inline script tags will be via System.load(), no static syntax allowed. For simple test pages, like jsbins/jsfiddles, the developer will need a way to pull in a few scripts at once via System.load. Nested calls would be ugly.

  1. Default module path resolution

There should be a way to define completely custom URL resolution for a specific loader, yes. The resolver could add a "canonicalize" method that can resolve a path relatively to the current file.

I don't think it makes sense for ECMAScript to define the default scheme at all. That should be up to the host environment. The browser environment should default to the same resolution algorithm as script tags or importScripts in Web Workers.

The problem is that with ES.next, scripts are using a JS API to specify dependencies. This is something new. The existing script tags/importScripts relies on the developer to properly sequence the order of the tags.

When Backbone and my app script want to use "jquery" there needs to be a way to resolve that to the same resource. It seems awkward if Backbone uses "jquery.js" which by default would map directly to local disk instead of the CDN.

But say this is what is encouraged: "jquery.js" and the developer is told to use the resolve API on the Module Loader to map always map "jquery.js" to the CDN location.

The resolve API is a function. This means the developer has to code up logic to do if tests on MRL names to see if it should switch the resolution to a CDN URL.

That if() test will be brittle/wordy to always manually code, so the developer will code a "helper lib" that allows them to pass a config object that is used by the helper lib.

Now they have a script loader library dependency. More below.

For more complex schemes that require configuration, you cannot statically resolve the configuration (unless you add the complexity of static configuration as well). Therefore you have to use the dynamic loader. Then you already have the option to override the resolution scheme when you call the loader.

Trying to standardize a more complex default scheme is likely to open up a lot of bike shedding.

There does not need to be a bikeshed. There is already an existing scheme that is not complex used by AMD module loaders (baseUrl and paths config), and it meshes well with other envs like Node, which had implemented the equivalent of its own resolve logic.

The alternative is to say "we don't want to try to figure it out" will mean that "config libraries" pop up for doing declarative config, because that is mostly what is needed.

So now to use modules, I need to include what is the equivalent of a module loader script. This looks like no net improvement for the end developer who has to use a module loader today -- they still need to manage and load script that needs to be called before the rest of their modules are run.

That loader script that sets the resolve logic will then need to expose an API that accepts a list of modules to load, since the configuration needs to be set before the rest of the loading occurs.

It starts to just look like requirejs then. I want script/module loaders like requirejs to go away. The "just use URLs" approach is not enough to do that.

  1. Compile time linking

Yea, it's true it's just one level. Ultimately you'd hope that engine and tool implementors can dynamically link any property reference to frozen objects. At least this one level certainly makes it easier to predict this behavior for that single level.

The unfortunate thing is that it is a very shallow level, see jQuery-type APIs or constructor functions. All the action happens that second level down. Ideally whatever solution could be found for that second level could also apply to that top level module. I definitely want to have that kind of checking generically though.

The compile time module linking is not enough, and it makes other higher value features harder or impossible.

  1. Import syntax

Obviously I certainly don't agree that Node and AMD users have done just fine without having *. That's one of the reasons I created Labeled Modules. labeledmodules/labeled-modules-spec/wiki

We should try for the equivalent of import *, sorry if I was too short on this point. I'll expand what I meant:

I think it should be a runtime-like thing (probably using the wrong word there), and just allow it for any object via destructuring:

let {} from "Math"; let {} from shape;

However, I'm OK dismissing * if it meant:

  1. This * logic is only possible for compile time linking of modules.
  2. and that compile time module linking prevents current libs from doing a runtime opt-in to ES.next modules.

To me #2 is an order of magnitude more important than the compile time features. If libraries do not have a way to opt in to both the old world and the new module world, it will make it harder to accept ES.next.

However, maybe I'm wrong on that. It would be good to hear arguments on how I misread that or how that transition can be managed well.

  1. Export syntax

I'd like to propose that we need an alternate way to export variables that's compatible with ES.old. Otherwise we'll have an unnecessary split between ES.old and ES.next when it could be perceived as a single language.

This is my primary concern. I want a way to specify dependencies and exports that works with ES.old, and ES.next. I believe the compile-time linking is the thing preventing that, but maybe I am wrong.

I like the label approach. It couldn't break existing code and still allow for static exports.

Labeled modules do not allow exporting a function as the module value. I believe it will be very difficult to sell that to the Node and AMD communities.

Labeled modules also do not allow creating a local name for the imported dependency, and always does the equivalent of an import *. It seems very hard to match up the right module property to use if you have two dependencies specified.

James

# David Herman (13 years ago)
  1. Multiple mrls for System.load():

System.load("jquery.js", "underscore.js", function ($, _) {}) or System.load(["jquery.js", "underscore.js"], function ($, _) {})

Makes sense. The former is a bit muckier but more convenient than the latter. I'll play with the alternatives, thanks.

  1. Default module path resolution

However it would be good to specify the default logic.

We'll have to in the browser context, but we also have to leave the default logic unspecified enough to accommodate the very different constraints in the browser and server contexts. For example, Node searches the filesystem, which is not something we'd do in the browser.

For the default logic, it would be great to see it be:

baseUrl + ID + ".js"

Yeah, I've thought about auto-appending ".js". I think you're right that it opens up the possibility to be a little more abstract.

I like the idea of having a simpler "paths" config for that instead of requiring the developer to implement a resolver function or worse include a library to set up the resolver (might as well just use an AMD loader then :).

I think we may want something like this, but I'm still thinking about it.

  1. Modules IDs as strings

This item of feedback is assuming that the way to get better optimized code delivery is to "concatenate" module files together. ...

With that assumption, module IDs should always be strings.

I agree that we should make sure it's easy to refactor between a module being defined out-of-band vs in-band. But I disagree with your conclusion that module definitions must have string names to achieve this. Let me chew on this question for a bit, but I think the simpler mechanism will be loader configuration/installation.

  1. export call

However, I still feel some sort of "export call" is needed, or some way to export a function as the module value, to give parity with the runtime calls:

System.load('jquery.js', function ($) { });

It would be awkward to do this:

System.load('jquery.js', function (something) { var $ = something.$; });

You're forgetting about destructuring:

System.load('jquery.js', function({$}) { });

In the context of built-in modules for JS, callable modules are more trouble than they're worth. In ES3, they're important because you want to minimize the number of global definitions you create, you often want your main workhorse of a library to be a function, and there's no easy way for clients to destructure. But in ES6, with destructuring and import, and with modules no longer polluting the global scope, the pressures are simply not the same. And the downsides to callable modules are that it complicates the semantic model, interferes with a bunch of other design constraints (like not mixing up the export namespace with the Function.prototype namespace, and with static linking), and closes off future possibilities like parameterized modules.

  1. Compile time linking

While the compile time linking may give some kinds of type checks/export name checking, it is only one level deep, it does not help with this sort of checking:

That's just unfair. What you're saying is that static scope checking doesn't do type checking. Well, of course not! And type checking doesn't prove your programs are bug-free, and compilers don't write your software for you. But they're still useful tools. Static linking provides us with a bunch of possibilities: in the short term, error checking for some of the the most common errors, as well as enabling conveniences like import *. In the longer term, additional compile-time constructs.

And we already have mechanisms for doing dynamic things. JS has objects and functions and those go a long way. You should know, you've built some incredibly powerful software with them! :)

  1. Object destructuring

No need to respond to this item, I just want to put my voice behind the backwards-ness of:

let { draw: drawWidget } = widget;

I know it throws people off, but the fact is that with full destructuring, the syntax just isn't symmetric. Imagine you're destructuring a property as an array of three elements:

let { [foo, bar, baz]: draw } = widget;

That just doesn't make any sense. I'm not even sure if it could be defined unambiguously.

The way to understand destructuring is that the thing to the left of the colon is the label, and the thing to the right is the structured part. When you construct an object, you are creating an object with labelled property names and computed property values; when you deconstruct an object, you are requested the labelled property names and destructuring the structured property value.

# David Herman (13 years ago)

On Mar 31, 2012, at 11:11 AM, James Burke wrote:

We should try for the equivalent of import *, sorry if I was too short on this point. I'll expand what I meant:

I think it should be a runtime-like thing (probably using the wrong word there), and just allow it for any object via destructuring:

let {} from "Math"; let {} from shape;

This is with. This is why you can't just dismiss static semantics.

# Wes Garland (13 years ago)

On 31 March 2012 21:30, David Herman <dherman at mozilla.com> wrote:

Yeah, I've thought about auto-appending ".js". I think you're right that it opens up the possibility to be a little more abstract.

FWIW -- the CommonJS convention of auto-appending .js has been a real boon to us when writing GPSEE modules. We took it a step farther, and decided we should search the filesystem for modules ending in .so also, giving us the ability to have modules which are written in JS, C/C++, or both -- completely transparently to the JS user.

I am considering allowing modules written in CoffeeScript, and we'll do the same thing, compiling the CoffeeScript code transparently to a CommonJS module interface, and again, the JS application developer never needs to know what language the application is written in.

A similar pattern might be useful if another language on the web ever emerges which needs to co-exist with ES within the browser -- provided this new language could present an ES6 module interface, your loader could detect which compiler to send it to, based on the MIME type, which the web server would figure out based on the extension which it finds by searching the filesystem. You could trivially implement the web-server part today with Apache's content negotiation stuff.

# Wes Garland (13 years ago)

On 1 April 2012 08:22, Wes Garland <wes at page.ca> wrote:

the JS application developer never needs to know what language the application is written in.

Sorry, what language the module is written in.

# John J Barton (13 years ago)

On Sat, Mar 31, 2012 at 6:30 PM, David Herman <dherman at mozilla.com> wrote:

baseUrl + ID + ".js"

Yeah, I've thought about auto-appending ".js". I think you're right that it opens up the possibility to be a little more abstract.

Auto-appending makes the API less abstract:the arguments must be JS. That in turn requires some new type-signaling string to opt out of JS if you want to expand the loader to other kinds of dependents. RequireJS uses a type-signaling prefix, 'text!notJS' and it has different rules for things that end do or do not end in '.js'. All of this has value at the cost of complexity. I'd like to see the standard file extension naming used by default.

In the code: System.load(["jquery.js", "underscore.js"], function ($, _) {}) the string "jquery.js" and "underscore.js" are combined with path info to create URLs. They identifies a resource, not a module; they should hew to standard URL naming conventions. (The tokens $ and _ refer to modules).

jjb

# Brendan Eich (13 years ago)

I agree with you, auto-appending is too concrete and JS-only. We can't use auto-append as a fallback since it's ambiguous (foo and foo.js could exist) and we shouldn't prefer foo based on existence (we'd have to GET twice or otherwise probe for existence). We want names not to depend on whether the URI denotes an existing "file".

I don't agree with reinventing MIME, er, IANA Media, Types, though. The rules for content typing from HTML, specifically for <script src=>,

should be adapted with as few changes as possible for out-of-line modules.

# John J Barton (13 years ago)

On Sun, Apr 1, 2012 at 11:33 AM, Brendan Eich <brendan at mozilla.org> wrote:

I agree with you, auto-appending is too concrete and JS-only. We can't use auto-append as a fallback since it's ambiguous (foo and foo.js could exist) and we shouldn't prefer foo based on existence (we'd have to GET twice or otherwise probe for existence). We want names not to depend on whether the URI denotes an existing "file".

I don't agree with reinventing MIME, er, IANA Media, Types, though.

The System.load() exists at the interface between JS and the browser. JS devs need features similar to the 'text!someCSS' and require.toUrl() to load dependencies into JS programs using the same path system that maps URLs to module resources. While I understand that ES may not want to tackle this aspect, this needs to happen in a way the devs can rely upon across browsers.

The rules for content typing from HTML, specifically for <script src=>, should be adapted with as few changes as possible for out-of-line modules.

Yes, even more so if ES delegates to another group for the rest.

jjb