Discuss
  • Questions about Harmony Modules

Questions about Harmony Modules

# James Burke (14 years ago)

I was looking over the harmony proposals around modules[1], module loaders[2], and the module examples[3], and have some questions. These questions come from making a module system/loader that works in today's browsers/Node/Rhino, via AMD[4] and RequireJS[5].

  1. Files as modules do not need module wrapper ==================== Just to confirm, if a JS file contains only a module definition, the module X{} wrapper is not needed? So I could do the following in the browser:

module JSON = require('json.org/modules/json2.js');

and json.js could just look like:

export stringify = ...; export parse = ...;

Correct?

  1. Set module export value ====================

Is it possible to support setting the value of the module to be a function or some other value? The Node implementation of CommonJS modules allows setting the exported value via module.exports = and in the AMD API, "return value".

This is useful because many modules are just one function, for instance, a constructor function, and having to create a property name that is the same name as the module is very awkward, particularly since module names are not fixed:

import (require('PersonV2')).{Person}

vs.

//This should probably be "module" //instead of "var", but see question #3 var Person = require('PersonV2');

What about the following rule in a module:

"If return is used in the module, then it sets the value of the module, and export cannot be used. If export is used in the module, then return cannot be used".

I know that means a possible syntax issue that is not allowed in traditional scripts: in the module examples wiki page, a module can just be a file with no module wrapper, and in that case and with the rule above, there could be a "return" at the top level. However, maybe that can be allowed if it is a module?

  1. Are "module" and "import" needed inside a module? ====================

I can see that "module" may be needed when defining a module, like:

module Math {}

However, inside a module definition, is it possible to get by without needing "module" and "import". For instance, instead of this:

module foo {    import Geometry.{draw: drawShape};    module a = require('a');

a.bar();    drawShape(); }

what about using var or let instead of module, and use destructuring for the import:

module foo {    var {draw: drawShape} = Geometry;    var a = require('a');

a.bar();    drawShape(); }

Maybe these are allowed, and the examples need to be expanded? But if they are allowed, then I do not see the need to specify the use of "module" and "import" inside a module. The only advantage for import may be "import Geometry.*" but I would rather see a generalized syntax for doing that for any object, not just a module, something like:

let {} = Geometry;

That is just an example, I am not a syntax expert, I am not proposing syntax, and that example probably would not work. I am just trying to demonstrate the general usefulness of "create a local variable for each property on this object".

The end goal is trying limit the number of new special things in the language the developer needs to learn for modules. Ideally it would be: "use 'module' to declare an inline module, then use require('') inside it to get handles on other modules, and use var/let and destructuring as you normally would".

  1. How to optimize ====================

How can I combine some modules for optimized code delivery? Is that out of scope? Is the hope that something like resource packages[6] take hold? There is value in distributing a single file with many modules, both for web performance and for command line tools: giving the developer one file to execute on the command-line is better than having to ship them a directory.

  1. Loader plugins ====================

Some of the AMD loaders support loader plugins[7]. This allows an AMD module to load other types of resources, and a module can treat them as a dependency. For instance:

//The 'text' module knows how to load text files,    //this one loads text for an HTML template    var template = require('text!templates/one.html');

//The 'cs' module knows how to load CoffeeScript-formatted    //modules via XHR, then transforms the    //CoffeeScript text to regular JavaScript and    //then executes that code to define the module.    var Widget = require('cs!Widget');

Would something like this be possible, perhaps via the module loaders API? It looks like the resolver API may come close, but I am not sure it is enough. This may be difficult if the module loader wants to parse all the require() calls fetch those resources before doing any module evaluation, where loader plugins are modules that implement an API and need to be evaluated and allow them to process the dependency string to satisfy the dependency.

  1. Loader API: multiple modules ====================

The loader API seems to only allow loading one module at a time with a callback (looking at Loader.load). Is there a way to say "load these three modules, and call this callback when all three are loaded"?

  1. Cross-domain script loading ====================

Given the modules examples page, it looks like it will be possible to load scripts from other domains, just confirming that it will be allowed, particularly in the browser.

I apologize for the long email. If some items need longer discussion, I'll break them out into separate threads.

[1] harmony:modules [2] harmony:module_loaders [3] harmony:modules_examples [4] wiki.commonjs.org/wiki/Modules/AsynchronousDefinition [5] requirejs.org [6] limi.net/articles/resource-packages

James

# David Herman (14 years ago)
  1. Files as modules do not need module wrapper ==================== Just to confirm, if a JS file contains only a module definition, the module X{} wrapper is not needed?

That's correct.

  1. Set module export value ====================

Is it possible to support setting the value of the module to be a function or some other value?

Currently, there's nothing like this in the proposal, but I remember you blogged about this before. Harmony modules are static entities, and setting their "value" dynamically would clash with the design goals of static importing and exporting.

That said, we could consider adding functionality to make a module callable, to address the use case you describe. Thanks for bringing this up.

  1. Are "module" and "import" needed inside a module? ====================

There are two questions here:

3a) Is "module" needed inside a module?

Yes. A module declaration declares a binding to a static module. A let/var declaration declares a binding to a dynamic value. You only get compile-time linking and scoping with static modules.

3b) Is "import" needed inside a module?

I think the answer here is technically no. You're right that you could generally use let/var and destructuring, except for the import M.* case. If we allowed you to destructure a runtime expression and dynamically bind all its variables, we would be re-introducing `with' into Harmony, which we will never, ever, ever do. :)

The other difference -- at least in the current state of the proposal -- is that you can't use a require' form in an ordinary expression, only in the RHS of an import/module declaration. This doesn't show up in the Geometry example you cited, but if you wanted to import from an external module usinglet', you would have to use the callback API:

ModuleLoader.load("geometry.js", function(Geometry) {
    let { draw } = Geometry;
    ...
});

(We've talked a little bit about generalizing the `require' form to be an expression operator that does a static module load, but I'm not sure whether it hangs together.)

The end goal is trying limit the number of new special things in the language the developer needs to learn for modules. Ideally it would be: "use 'module' to declare an inline module, then use require('') inside it to get handles on other modules, and use var/let and destructuring as you normally would".

For many cases, I think that's fine. But .* is an important convenience for scripting. Also, FWIW, import is already reserved and has symmetry with export, so it seems like a wasted opportunity to fill a natural space in the grammar.

  1. How to optimize ====================

I think this is beyond the scope of ECMAScript.next. People are still figuring out how to optimize delivery, and web performance patterns are still shaking out. ISTM there will be situations where it's more performant to combine many modules into one (which can be done with nested module declarations) and others where it's more performant to conditionally/lazily load modules separately (which can be done with module loaders). I don't currently have a clear picture of how we could build syntactic conveniences or abstractions for doing this, but at least the pieces are there so programmers have the building blocks to start constructing their own tools for doing this.

  1. Loader plugins ====================

Some of the AMD loaders support loader plugins[7].

Missing a reference in your bibliography. :)

This allows an AMD module to load other types of resources, and a module can treat them as a dependency.

I don't know-- I'd be interested to read more on this topic.

  1. Loader API: multiple modules ====================

The loader API seems to only allow loading one module at a time with a callback (looking at Loader.load). Is there a way to say "load these three modules, and call this callback when all three are loaded"?

Perhaps. We're trying to get the building blocks in place first. We can figure out what conveniences are needed on top of that.

BTW, what you're asking for is essentially a concurrent join, which is convenient to express in a new library I'm working on called jsTask:

let [m1, m2, m3] = yield join(load("m1.js"), load("m2.js"), load("m3.js"));
  1. Cross-domain script loading ====================

Given the modules examples page, it looks like it will be possible to load scripts from other domains, just confirming that it will be allowed, particularly in the browser.

Yep!

Thanks for the questions,

# Brendan Eich (14 years ago)

On Apr 2, 2011, at 4:33 PM, David Herman wrote:

(We've talked a little bit about generalizing the `require' form to be an expression operator that does a static module load, but I'm not sure whether it hangs together.)

I don't see how we can reserve 'require' as a static (compile-time) pseudo-function or operator, given that identifier's uses in extant JS.

For Harmony migration we want early errors, but some uses of require would "look ok" even though coded to call something other than the new built-in.

The end goal is trying limit the number of new special things in the language the developer needs to learn for modules. Ideally it would be: "use 'module' to declare an inline module, then use require('') inside it to get handles on other modules, and use var/let and destructuring as you normally would".

For many cases, I think that's fine. But .* is an important convenience for scripting. Also, FWIW, import is already reserved and has symmetry with export, so it seems like a wasted opportunity to fill a natural space in the grammar.

I would like to make a stronger case for 'import': it is a static binding form new in Harmony that we need for two reasons:

  1. To make 'require' on the right a contextual keyword, if used (see above).

  2. To make a static binding and fail if there is already a binding of the same name.

None of the existing binding forms do both of these. 'const' is closest to 'import' and satisfies (2), plus it is reserved already -- and variously implemented (IIRC Opera equates it with 'var'!). But 'const' does not satisfy (1).

Does this sound right?

# Dave Herman (14 years ago)

Fully agree on all counts.

# Dmitry A. Soshnikov (14 years ago)

On 03.04.2011 3:33, David Herman wrote:

Hi James,

  1. Files as modules do not need module wrapper ==================== Just to confirm, if a JS file contains only a module definition, the module X{} wrapper is not needed? That's correct.
  2. Set module export value ====================

Is it possible to support setting the value of the module to be a function or some other value? Currently, there's nothing like this in the proposal, but I remember you blogged about this before. Harmony modules are static entities, and setting their "value" dynamically would clash with the design goals of static importing and exporting.

That said, we could consider adding functionality to make a module callable, to address the use case you describe. Thanks for bringing this up.

  1. Are "module" and "import" needed inside a module? ==================== There are two questions here:

3a) Is "module" needed inside a module?

Yes. A module declaration declares a binding to a static module. A let/var declaration declares a binding to a dynamic value. You only get compile-time linking and scoping with static modules.

3b) Is "import" needed inside a module?

I think the answer here is technically no. You're right that you could generally use let/var and destructuring, except for the import M.* case. If we allowed you to destructure a runtime expression and dynamically bind all its variables, we would be re-introducing `with' into Harmony, which we will never, ever, ever do. :)

Yes, since technically it's possible to get a value of an exported binding directly via:

let foo = Mod.foo;

then it should be possible to use destructuring to get references for several bindings.

However, as noticed, such imported via assignment bindings aren't static. I.e. it's possible e.g to assign to them in contrast if the binding is imported. Here I showed this difference: gist.github.com/885534 While this example is shown on prototyped modules in Narcissus, actually technically for real modules implementation there can be overloaded operator of assignment which calls defineProperty instead to make the binding the same as it was imported.

Regarding, "import M.*" via destructuring, it's also arguable whether we don't need it since it "looks like a with". Besides, if we can import all via "partial destructuring", i.e.:

module Mod { export foo var = 10; export bar var = 20; }

let {foo, bar} = Mod;

then (theoretically!) why not to have let {} = Mod (or any other syntax, e.g. let * = Mod). But, IMO it just looks more complicated than import Mod.*. So I would disallow let {} = Mod not because of with, but because of the oddness of the construction which sounds as "destruct the module Mod into the empty object".

The other difference -- at least in the current state of the proposal -- is that you can't use a require' form in an ordinary expression, only in the RHS of an import/module declaration. This doesn't show up in the Geometry example you cited, but if you wanted to import from an external module usinglet', you would have to use the callback API:

 ModuleLoader.load("geometry.js", function(Geometry) {
     let { draw } = Geometry;
     ...
 });

(We've talked a little bit about generalizing the `require' form to be an expression operator that does a static module load, but I'm not sure whether it hangs together.)

This is the question of allowing expression which we talked in the Narcissus list (mail.mozilla.org/pipermail/narcissus/2011-March/000043.html ; btw, is it worth to put the discussing on duplications on this list?).

Though the other question is: why do we need require at all? Why not just:

// loading from the local file system

module $ = "./selector.js";

// loading from Net

module Widgets = "mysite.com/widgets.cs";

Though, I see the only case with destructuring (assuming that the module isn't on local system):

let {foo, bar} = require "mysite.com/widgets.cs"

But since you notice that require won't be allowed in such expression, I don't see why it's needed at all.

Besides, I can imagine something like this:

import "widgets/panel"

which loads file "panel.js" from the "widgets" directory and either creates the module binding name as Panel (i.e corresponding to the file name), or imports all bindings from the module. Another variant:

import "widgets/panel" {foo, myBar: bar}

or

import {foo, myBar: bar} from "widgets/panel"

Dmitry.

# James Burke (14 years ago)

On Sat, Apr 2, 2011 at 4:33 PM, David Herman <dherman at mozilla.com> wrote:

  1. Set module export value ==================== That said, we could consider adding functionality to make a module callable, to address the use case you describe. Thanks for bringing this up.

Allowing a callable module would go a long way towards bridging the gap with "settings exports" as that is the primary use of that feature, although it has been useful for text plugins to set the value of an AMD module to a text string (see loader plugins section).

3a) Is "module" needed inside a module?

Yes. A module declaration declares a binding to a static module. A let/var declaration declares a binding to a dynamic value. You only get compile-time linking and scoping with static modules.

In that case, I can see the appeal for what Dmitry mentions in his reply, just reducing it to:

module thing = "some/thing";

Ideally, you could have multiple ones with one module word:

module datePicker = "datePicker", thing = "some/thing", Q = "Q";

That would really help with the typing cost seen with traditional CommonJS modules (require is typed a lot), and better "local identifier to module name" alignment than what happens in AMD, where the dependencies and local var names are separate lists :

define(["datePicker", some/thing", "Q"], function (datePicker, thing, Q({

});

I have gone to aligning the lists to avoid positioning problems, but I can see how people could find it ugly. Example: mozilla/f1/blob/master/web/dev/share/panel/index.js

3b) Is "import" needed inside a module? For many cases, I think that's fine. But .* is an important convenience for scripting. Also, FWIW, import is already reserved and has symmetry with export, so it seems like a wasted opportunity to fill a natural space in the grammar.

Understood. Thanks to you and Brendan for running that one down. I like the idea of a let {} = someObject; where it only grabs the own properties of that object at the time of the call, but I can appreciate if that is difficult to work out and if it skates too close to "with".

  1. How to optimize ====================

I think this is beyond the scope of ECMAScript.next. People are still figuring out how to optimize delivery, and web performance patterns are still shaking out. ISTM there will be situations where it's more performant to combine many modules into one (which can be done with nested module declarations) and others where it's more performant to conditionally/lazily load modules separately (which can be done with module loaders). I don't currently have a clear picture of how we could build syntactic conveniences or abstractions for doing this, but at least the pieces are there so programmers have the building blocks to start constructing their own tools for doing this.

The experience in RequireJS/AMD and in Dojo land is that different people/projects want different things: sometimes build all modules into one file, sometimes build some together in one or a couple of files and have those sets of modules be usable by some other modules that are loaded separately.

When you say a built file would be possible with nested module declarations, that makes it sound like those nested modules may not be usable/visible by other modules that are not loaded as part of that built file. It would be interesting to explore that more at some point.

Using string names as the module names in AMD has helped make it possible to meet the optimization expectations we have seen so far. So a module that has a 'some/thing' dependency:

define("my/thing", ["some/thing", function (thing){})

when 'some/thing' module is built into the optimized file it has its string name as the first arg:

define('some/thing', function () {});

I can see this as trickier in Harmony modules, at least for the examples I have seen, where 'some/thing' needs to be an identifier like:

module someThing {}

Maybe allow strings instead for the names (I say with blissful syntax ignorance). Just trying to figure out how to match up string references to modules in inside a module to a named thing in a built layer.

  1. Loader plugins ====================

Some of the AMD loaders support loader plugins[7].

Missing a reference in your bibliography. :)

Sorry, a link would be helpful: requirejs.org/docs/plugins.html

  1. Loader API: multiple modules Perhaps. We're trying to get the building blocks in place first. We can figure out what conveniences are needed on top of that.

BTW, what you're asking for is essentially a concurrent join, which is convenient to express in a new library I'm working on called jsTask:

let [m1, m2, m3] = yield join(load("m1.js"), load("m2.js"), load("m3.js"));

I like the ideas behind jsTask, although I would rather not type load that much:

let [m1, m2, m3] = yield load(["m1.js", "m2.js", "m3.js"]);

Array of modules via require([], function (){}) has been useful in RequireJS, and has a nice parity with the define([], function (){}) dependency list too, although not applicable here. Fair enough if you want to consider sugar later.

Thanks for your time, James

# David Herman (14 years ago)

Regarding, "import M.*" via destructuring, it's also arguable whether we don't need it since it "looks like a with".

I don't see any sense in which it looks like a |with|. It's both syntactically and semantically different. Syntactically, because it's a global (or module-global) declaration rather than a local statement form, and semantically, because it imports all the bindings at compile-time.

then (theoretically!) why not to have let {} = Mod

because that is ambiguous with existing destructuring syntax

(or any other syntax, e.g. let * = Mod)

because that would be dynamically evaluating the module expression and dynamically importing all the bindings, and hence tantamount to |with|.

But, IMO it just looks more complicated than import Mod.*.

I agree. Part of the point of import is to create a distinct syntactic context for module expressions. Mixing this in with ordinary expressions and destructuring confuses the issue.

(We've talked a little bit about generalizing the `require' form to be an expression operator that does a static module load, but I'm not sure whether it hangs together.)

This is the question of allowing expression which we talked in the Narcissus list

No, it's a different issue. You were talking about export declarations. Here I was talking about allowing you to statically load a module in any expression.

(mail.mozilla.org/pipermail/narcissus/2011-March/000043.html ; btw, is it worth to put the discussing on duplications on this list?).

Not really. It's a pretty specific detail and we'll sort it out. It's not worth spending time discussing.

Though the other question is: why do we need require at all?

Purely for syntactic/practical reasons. Let's not bikeshed.

# David Herman (14 years ago)

Allowing a callable module would go a long way towards bridging the gap with "settings exports" as that is the primary use of that feature, although it has been useful for text plugins to set the value of an AMD module to a text string (see loader plugins section).

I'll try to think up some lightweight ways to make a module callable.

In that case, I can see the appeal for what Dmitry mentions in his reply, just reducing it to:

module thing = "some/thing";

That's been one of the alternative surface syntaxen we've considered. As I said to Dmitry, let's avoid bikeshedding on-list.

Ideally, you could have multiple ones with one module word:

module datePicker = "datePicker", thing = "some/thing", Q = "Q";

This is actually already part of the proposal.

BTW, what you're asking for is essentially a concurrent join, which is convenient to express in a new library I'm working on called jsTask:

let [m1, m2, m3] = yield join(load("m1.js"), load("m2.js"), load("m3.js"));

I like the ideas behind jsTask, although I would rather not type load that much:

let [m1, m2, m3] = yield load(["m1.js", "m2.js", "m3.js"]);

Sure, and that's easy enough to implement. You could even simplify it further by making load variable-arity:

let [m1, m2, m3] = yield load("m1.js", "m2.js", "m3.js");

Array of modules via require([], function (){}) has been useful in RequireJS, and has a nice parity with the define([], function (){}) dependency list too, although not applicable here. Fair enough if you want to consider sugar later.

Yeah, we'll get there. This is something I feel pretty confident can be made convenient without affecting the core of the system, so I think it can wait till later.

Thanks again for your input,

# Dmitry A. Soshnikov (14 years ago)

On 04.04.2011 18:40, David Herman wrote:

Though the other question is: why do we need require at all? Purely for syntactic/practical reasons. Let's not bikeshed.

Why I was asking -- because I saw it in your talk on ES.next, where you used exactly this approach, i.e. module Foo = "modules.com/foo.js" -- without any require. That's it. (should I fix my following presentation also? I used the same as in yours)

Agreed on rest points.

Dmitry.

# David Herman (14 years ago)

Why I was asking -- because I saw it in your talk on ES.next, where you used exactly this approach, i.e. module Foo = "modules.com/foo.js" -- without any require. That's it.

No problem, I didn't mean to chastise. Just trying to keep focussed.

(should I fix my following presentation also? I used the same as in yours)

Sure. Surface syntax isn't set in stone, but we aren't likely to go back to just the string literal, since it looks too much like the module is being assigned a string value. The current syntax is:

module Foo = require "http://modules.com/foo.js";
import Foo.bar;

where you can also parenthesize the whole require expression:

module Foo = (require "http://modules.com/foo.js");
import Foo.bar;

or more simply:

import (require "http://modules.com/foo.js").bar;
# James Burke (14 years ago)

On Wed, Apr 6, 2011 at 8:25 AM, David Herman <dherman at mozilla.com> wrote:

Sure. Surface syntax isn't set in stone, but we aren't likely to go back to just the string literal, since it looks too much like the module is being assigned a string value.

I know you do not want to get into bikeshedding at this point, and I will not follow this up any more unless you explicitly ask me, but I strongly encourage just using the string value, sans require. It is much more concise, and OK given that modules are special (compile-time processing, special injection via import).

If string IDs are allowed for inline module declarations to allow optimization bundling:

module "some/thing" {}

that would help feed the consistency when seeing: module M = "some/thing".

James