Module system strawpersons
2010/1/14 <ihab.awad at gmail.com>:
Hi folks,
Kris Kowal and I have written up our proposals for a module system, to be discussed during the Jan 28-29 meeting. The proposals are as follows:
I skimmed it and came up with a few questions that might make it easier to read in depth if answered.
Are these proposals mutually exclusive or complementary. If (1) and (2) are exclusive, is there a place to collect use cases for a module-off?
I don't see the word module anywhere in the packages proposal besides footnotes. What is the relationship between packages and modules.
Module proposal, by Kris, focusing on management of primordials.
Maybe mention in the Context definition that Context is invokable as a constructor. Is the global instanceof global.Context? In the grammar for ImportWith, can id be replaced with the object literal production from the native JSON portion of the spec? Can Import and ImportWith only appear in a module function body? If not, where are the semantics of ImportWIth and Import specified? In Module, when you say "load clause", does that bear any relationship to the load_opt parameter?
Module proposal, by Ihab, focusing on EMaker-style invocation semantics.
"First class objects. To the extent possible, module system components must be represented as first-class ECMAScript objects." Does object exclude primitives such as strings, or is the goal to exclude abstractions like LexicalScopes and References?
"import" is added to the reserved keyword list?
"return { getX: function() { return x; }, getY: function() { return y; }," indentation
"In the module’s code, this is initialized to a new object that is instanceof the module function" Does this mean that access to a module instance conveys authority to create new module instances?
"An import expression evaluates immediately to a module function despite the fact that the specified module may have been fetched by asynchronous means (e.g., nonblocking network operations). This is done by taking advantage of the fact that import is a special form." Does this require "import" to be a special form instead of a desugaring? Or can the special form be boiled down to a function which when called, delays until another function is invoked, presumably with a valid FunctionExpr string as its sole argument?
On Thu, Jan 14, 2010 at 1:29 PM, Mike Samuel <mikesamuel at gmail.com> wrote:
Are these proposals mutually exclusive or complementary. If (1) and (2) are exclusive, is there a place to collect use cases for a module-off?
For my part, I think (1) and (2) are counterproposals, but there may exist a bunch of cross-pollination. For example, I will probably end up stealing Kris's "import with" syntax sometime.
I don't see the word module anywhere in the packages proposal besides footnotes. What is the relationship between packages and modules.
The packages proposal was written with module proposal (1) in mind, but is compatible in the broad sense with either. It does say in the beginning that it "... is intended to satisfy the Uniform location and retrieval goal of modules_emaker_style". There are more details under "Use in import" (where I just updated the syntax a little bit).
Module proposal, by Ihab, focusing on EMaker-style invocation semantics.
"First class objects. To the extent possible, module system components must be represented as first-class ECMAScript objects." Does object exclude primitives such as strings, or is the goal to exclude abstractions like LexicalScopes and References?
The point is that module system components are made to appear, to the extent possible, as first-class objects rather than unmentionable "static" magic. E.g., contrast Java where, if Foo is a class, the expression "Foo" by itself is not an object; generic class parameters are not objects; etc. -- which makes higher-order programming somewhat harder.
"import" is added to the reserved keyword list?
It should be reserved now, and is for sure reserved according to this proposal.
"return { getX: function() { return x; }, getY: function() { return y; }," indentation
Fixed, tx.
"In the module’s code, this is initialized to a new object that is instanceof the module function" Does this mean that access to a module instance conveys authority to create new module instances?
Yes. Just like any other ES function where it's return value is instanceof that function.
"An import expression evaluates immediately to a module function despite the fact that the specified module may have been fetched by asynchronous means (e.g., nonblocking network operations). This is done by taking advantage of the fact that import is a special form." Does this require "import" to be a special form instead of a desugaring? Or can the special form be boiled down to a function which when called, delays until another function is invoked, presumably with a valid FunctionExpr string as its sole argument?
Would that require a continuation-passing transform of the code after the desugared "import"? If so, that may not be semantics preserving.
Ihab
Ok, so let me try to make sure I understand the special form in strawman:modules_emaker_style, by defining a mini-strawman and knocking it down.
Consider a special form (f, definer) = defineDelayedFunction(); where f is a function that will block on first call until definer has been called with an arguments array and a function body upon which point it will behave as if it were defined via new Function(...) with those arguments and function body. definer raises an Error on second and subsequent calls.
defineDelayedFunction could be used with an async channel like XMLHttpRequest to define a module system, but all module definitions would happen at execute time, so it cannot be used to define a module system that resolves definitions prior to body evaluation.
In modules_emaker_style, imports are statically determinable, so the set of dependencies is fully satisfied before the Program is evaluated, and this can be done recursively to make full use of sockets. A browser could optimistically find import definitions and start downloading before the full script body is loaded.
defineDelayedFunction would impose lazy fetching, but modules_emaker_style allows eager fetching.
2010/1/14 <ihab.awad at gmail.com>:
On Thu, Jan 14, 2010 at 2:29 PM, Mike Samuel <mikesamuel at gmail.com> wrote:
defineDelayedFunction could be used with an async channel like XMLHttpRequest to define a module system, but all module definitions would happen at execute time, so it cannot be used to define a module system that resolves definitions prior to body evaluation.
Yes, and if the definitions are resolved while the body is being evaluated, and the thread of execution in which the body is being evaluated blocks, this will freeze the UI of the app or otherwise prevent it from servicing requests while the requested module is being resolved.
defineDelayedFunction would impose lazy fetching, but modules_emaker_style allows eager fetching.
Yes. Thanks for the clarification.
Ihab
My proposal is attempting to present two alternative ways to implement "require" on top of the "import" keyword. The first option, as you point out:
On Fri, Jan 15, 2010 at 9:13 AM, kkasravi <kkasravi at me.com> wrote:
Within the Layered systems, the wrapper require looks like require('util/pointUtils')
needs "require" itself to be recognized as a special form in the input language, which involves parsing the input code. The second form:
However the proposal also suggests that require could be require(import 'util/pointUtils')
does not, and as you point out:
Since import 'util/pointUtils' returns a Function Object and require needs a string literal, I'm wonder how the second require (above) would work.
the second "require" would not comply with the existing CommonJS "require", but would be close to it.
Ihab
Thanks for the clarification, I misunderstood it as being replaceable with the current commonjs require.
Kam
2010/1/15 kkasravi <kkasravi at me.com>:
Hi Mike, Ihab:
One factor that would influence eager or lazy fetching is where the import keyword may appear within the ecmascript grammar.
The proposal suggests it could appear anywhere an identifier could appear. For example:
I believe it cannot appear where a LeftHandSideExpression is expected because a function cannot be assigned to. So no replacing foo.bar = baz with (import ...) = baz
- function find(name) { var name = foo.bar.Baz.find(name); // identifier 'foo.bar.Baz' ... }
Replacing foo.bar with import would yield
- function find(name) { var name = (import 'foo/bar').Baz.find(name);
Provisioning is separate from importing. The above is looking up Baz on a function instance which normally would be the same as var name = Function.prototype.Baz.find(name)
I think normal usage would look like var name = (import ...)(module, instance, parameters).Baz.find(name)
... }
Replacing foo.bar.Baz with import would yield
- function find(name) { var name = (import 'foo/bar/Baz').find(name); ... }
I provided 2) and 3) because its not clear where a module definition begins. The presumption by naming convention is that foo.bar.Baz can be found within a relative resource named ./foo/bar.js on some server. This metadata needs to be somehow made available to the javascript runtime. So in some ways module's need package info. Let's assume the correct one is 2).
Looking at 2), current javascript behavior is always lazy fetching because the above function find
fetching can be eager, but evaluation happens whenever the ImportExpr is reached, possibly not at all.
may not be called immediately or not at all. The javascript runtime would not complain about identifiers that are not immediately accessed or not accessed at all.
Doing eager fetching means that we would need to fetch all imports prior to eval. Impacts are:
- We fetch more than necessary and immediately. User perception is the new js runtime is much slower.
Eager fetching can be done, or can be delayed, or multiple module definitions could be fetched in the same transaction. The degree of eagerness is up to the interpreter.
Thanks Mike, all bang-on. One minor comment:
On Fri, Jan 15, 2010 at 11:20 AM, Mike Samuel <mikesamuel at gmail.com> wrote:
I think normal usage would look like var name = (import ...)(module, instance, parameters).Baz.find(name)
Actually the module function takes a map of free variable names to bindings. So it would be more like:
var name = (import ...)({ bufferSize: 3, mapMaker: myMapMaker}).Baz.find(name);
Ihab
Ihab, Mike - Thanks for the comments.
Yes, my examples confused definition with declaration, thanks for the corrections.
I think I still have some unanswered questions which I'll try and clarify as:
-
I'm still not clear what the module identifier represents. Something like 'a/b/c' could be a URI or associated with a URI. Commonjs implicitly adds a '.js' which suggests this maps to a REST construct. I think it would be helpful to clarify the above literal, the module.id, any relation to javascript namespace and a mapping to server resources since there is such an overlap with package management.
-
If the eagerness of the interpreter is determined by its implementation we could have lots of indeterminism across interpreters. There should probably be another construct that defines when resources are fetched. I'm not sure if the interpreter shouldn't be influenced by the developer.
Brendan, do you recall es4 where the concept of a unit was introduced? There was also some discussion where management and security where implicitly mapped to the '.' between identifier parts. I cannot find that reference.
Kam
2010/1/15 Kam Kasravi <kamkasravi at yahoo.com>:
Ihab, Mike - Thanks for the comments. Yes, my examples confused definition with declaration, thanks for the corrections. I think I still have some unanswered questions which I'll try and clarify as:
I'm still not clear what the module identifier represents. Something like 'a/b/c' could be a URI or associated with a URI. Commonjs implicitly adds a '.js' which suggests this maps to a REST construct. I think it would be helpful to clarify the above literal, the module.id, any relation to javascript namespace and a mapping to server resources since there is such an overlap with package management.
If the eagerness of the interpreter is determined by its implementation we could have lots of indeterminism across interpreters. There should
Which sources of indeterminism do you foresee? Is this indeterminism as apparent to a network observer or as apparent to code running in the same interpreter?
probably be another construct that defines when resources are fetched. I'm not sure if the interpreter shouldn't be influenced by the developer.
Could you rephrase this last sentence, please?
comments below...
2010/1/15 Kam Kasravi <kamkasravi at yahoo.com>:
comments below...
From: Mike Samuel <mikesamuel at gmail.com> To: Kam Kasravi <kamkasravi at yahoo.com> Cc: kkasravi <kkasravi at me.com>; es-discuss <es-discuss at mozilla.org> Sent: Fri, January 15, 2010 1:45:52 PM Subject: Re: Module system strawpersons
2010/1/15 Kam Kasravi <kamkasravi at yahoo.com>:
Ihab, Mike - Thanks for the comments. Yes, my examples confused definition with declaration, thanks for the corrections. I think I still have some unanswered questions which I'll try and clarify as:
I'm still not clear what the module identifier represents. Something like 'a/b/c' could be a URI or associated with a URI. Commonjs implicitly adds a '.js' which suggests this maps to a REST construct. I think it would be helpful to clarify the above literal, the module.id, any relation to javascript namespace and a mapping to server resources since there is such an overlap with package management.
If the eagerness of the interpreter is determined by its implementation we could have lots of indeterminism across interpreters. There should
Which sources of indeterminism do you foresee? Is this indeterminism as apparent to a network observer or as apparent to code running in the same interpreter? [Kam] the network observer. A given interpreter may end up making lots of fetches or a few depending when it decided to resolve the import's.
probably be another construct that defines when resources are fetched. I'm not sure if the interpreter shouldn't be influenced by the developer.
Could you rephrase this last sentence, please? [Kam] as an example. lets say a module contains 3 adapters: one that uses jQuery, one that uses Ext and one that uses YUI. A eager interpreter may fetch all three when evaluating the module. Yet if the choice of what framework to use (jQuery, Ext or YUI) was determined by the developer/user early on, then the interpreter would fetch 2 frameworks without needing to. If the developer had a way to indicating to the interpreter that 2 of the frameworks weren't needed then the behavior of the interpreter would be predictable.
If the module needs all 3, then it needs to fetch all 3.
But if only one adapter is needed, then there is presumably some code somewhere which chooses which adapter. Let's call it the bootstrapper. Then the module relationship looks like:
bootstrapper imports YUI-adapter
YUI adapter imports YUI
bootstrapper passes adapter to framework as a module parameter
Since modules are first class, and are parameterizable, dependencies can be injected without requiring over-import. By not importing the adapters that are not needed, the other libraries are not imported.
I guess the problem with examples is their analysis may not surface a particular use case. I agree at runtime this example may allow the interpreter to not over import. The original question about how an interpreter may be non-deterministic is whether the rules that determine whether the interpreter is lazy or eager have not been defined yet. The example was an attempt on my part to have the developer drive how aggressive the interpreter is in prefetching resources. To summarize, there should be some rules that make each vendors interpreter behave like the next one, and there should be a way for the developer to make the interpreter prefetch even if the interpreter sees no reason to do so.
On Jan 15, 2010, at 4:11 PM, Mike Samuel <mikesamuel at gmail.com> wrote:
2010/1/15 Kam Kasravi <kamkasravi at yahoo.com>:
comments below...
2010/1/15 Kam Kasravi <kamkasravi at yahoo.com>:
I guess the problem with examples is their analysis may not surface a particular use case. I agree at runtime this example may allow the interpreter to not over import. The original question about how an interpreter may be non-deterministic is whether the rules that determine whether the interpreter is lazy or eager have not been defined yet. The example was an attempt on my part to have the developer drive how aggressive the interpreter is in prefetching resources. To summarize, there should be some rules that make each vendors interpreter behave like the next one, and there should be a way for the developer to make the interpreter prefetch even if the interpreter sees no reason to do so.
I think determinism at the network level is out of scope of a module system, and may be out of scope of the Ecmascript group.
Even ignoring network conditions, browsers' wildly different behavior around deferred scripts, the particulars of request headers, ordering of scripts, number of simultaneous open connections per host; can all affect the order in which requests are issued. Anything that requires deterministic behavior as apparent to a network observer seems like a non-starter to me.
If you want a clear description of which resources are and are not fetched, then that's probably specifiable, but GET requests are supposed to be idempotent, so I don't know what that would gain assuming fetches are via GET.
Java gets around restrictive definitions on class loading by separating the loading phase from the initialization phase. Loading can involve network requests, and can be optimistic. Initialization happens at well-defined times. Would such a split satisfy the use cases you have in mind?
2010/1/15 Mike Samuel <mikesamuel at gmail.com>:
2010/1/15 Kam Kasravi <kamkasravi at yahoo.com>:
I guess the problem with examples is their analysis may not surface a particular use case. I agree at runtime this example may allow the interpreter to not over import. The original question about how an interpreter may be non-deterministic is whether the rules that determine whether the interpreter is lazy or eager have not been defined yet. The example was an attempt on my part to have the developer drive how aggressive the interpreter is in prefetching resources. To summarize, there should be some rules that make each vendors interpreter behave like the next one, and there should be a way for the developer to make the interpreter prefetch even if the interpreter sees no reason to do so.
I think determinism at the network level is out of scope of a module system, and may be out of scope of the Ecmascript group.
Even ignoring network conditions, browsers' wildly different behavior around deferred scripts, the particulars of request headers, ordering of scripts, number of simultaneous open connections per host; can all affect the order in which requests are issued. Anything that requires deterministic behavior as apparent to a network observer seems like a non-starter to me.
If you want a clear description of which resources are and are not fetched, then that's probably specifiable, but GET requests are supposed to be idempotent, so I don't know what that would gain assuming fetches are via GET.
Actually, I'm dumb. idempotence only holds for # of requests >= 1.
Obviously, requesting the first time may convey significant information.
I'm sorry Mike, the network viewpoint is not what I meant, my bad. Rather, at what point does the interpreter evaluate multiple import's. I think Es4 suggested that eager fetching should be done if 'use strict' were specified. This is one way to answer your argument below of modules_emaker_style vs defineDelayedFunction. We are both asking the same question: how much static analysis is done and when.
Btw, someone will ask at some point if import literal's or expressions should accommodate regex's. A different topic, but worth clarifying.
thx Kam
On Sat, Jan 16, 2010 at 12:10 AM, Kam Kasravi <kamkasravi at yahoo.com> wrote:
I'm sorry Mike, the network viewpoint is not what I meant, my bad. Rather, at what point does the interpreter evaluate multiple import's. I think Es4 suggested that eager fetching should be done if 'use strict' were specified. This is one way to answer your argument below of modules_emaker_style vs defineDelayedFunction. We are both asking the same question: how much static analysis is done and when.
Btw, someone will ask at some point if import literal's or expressions should accommodate regex's. A different topic, but worth clarifying.
They should not. I still believe they should only be strings. But I can live with the proposed notion that they can be a JSON literal (i.e., an ES expression in the subset of ES corresponding to JSON). I think this is already more flexibility than we need in that position. I see no reason to go any farther.
To clarify:
On Sat, Jan 16, 2010 at 8:03 AM, Mark Miller <erights at gmail.com> wrote:
They should not. I still believe they should only be strings. But I can live with the proposed notion that they can be a JSON literal (i.e., an ES expression in the subset of ES corresponding to JSON). I think this is already more flexibility than we need in that position. I see no reason to go any farther.
There are two forms:
import <String>
for the non-packaged case and, if packages and catalogs are accepted:
import <String> from <JSONLiteral> import <String> from <String>
Ihab
Ihab, Mark:
Thanks for responding to the regex question. I am still befuddled by the relationships between module id's, namespaces, server resources and package management. Would you be able to provide commentary on the following example? Given import './a/b/c', it looks like this is an implicit mapping to the URI ./a/b/c.js. (true/false). Within this module it's namespace may be appended to by adding attibutes to 'this' or setting attributes on the function. (true/false).Once this module returns there is a module with id 'a.b.c', there may or may not be a namespace that includes a.b.c.(true/false). If this module with id 'a.b.c' wanted to reference a module instance with id 'a.b' it could not do so directly but would need to do an (import 'a/b')(...). (true/false). Presumably the developer would need to know the arguments needed and whether the module function was intended to be an object. (true/false) Assuming module with id 'a.b.c.d' wanted to share the same module instance 'a.b' it would also do a similar import. (true/false) Given the common use case today of accessing a library framework by it's global instance, the import construct would need to be used or the instance would need to be provided as a free variable to the module function (true/false)
The motivation for the questions above is to understand how existing common use cases would change
Thx Kam
On Jan 16, 2010, at 8:03 AM, Mark Miller <erights at gmail.com> wrote:
On Sat, Jan 16, 2010 at 12:10 AM, Kam Kasravi <kamkasravi at yahoo.com> wrote:
I'm sorry Mike, the network viewpoint is not what I meant, my bad. Rather, at what point does the interpreter evaluate multiple import's. I think Es4 suggested that eager fetching should be done if 'use strict' were specified. This is one way to answer your argument below of modules_emaker_style vs defineDelayedFunction. We are both asking the same question: how much static analysis is done and when.
Btw, someone will ask at some point if import literal's or expressions should accommodate regex's. A different topic, but worth clarifying.
They should not. I still believe they should only be strings. But I can live with the proposed notion that they can be a JSON literal (i.e., an ES expression in the subset of ES corresponding to JSON). I think this is already more flexibility than we need in that position. I see no reason to go any farther.
thx Kam
From: Mike Samuel <mikesamuel at gmail.com>
To: Kam Kasravi <kamkasravi at yahoo.com>
Cc: kkasravi <kkasravi at me.com>; es-discuss <es-discuss at mozilla.org>
Sent: Fri, January 15, 2010 7:33:24 PM
Subject: Re: Module system strawpersons
2010/1/15 Mike Samuel <mikesamuel at gmail.com>:
2010/1/15 Kam Kasravi <kamkasravi at yahoo.com>:
I guess the problem with examples is their analysis may not surface a particular use case. I agree at runtime this example may allow the interpreter to not over import. The original question about how an interpreter may be non-deterministic is whether the rules that determine whether the interpreter is lazy or eager have not been defined yet. The example was an attempt on my part to have the developer drive how aggressive the interpreter is in prefetching resources. To summarize, there should be some rules that make each vendors interpreter behave like the next one, and there should be a way for the developer to make the interpreter prefetch even if the interpreter sees no reason to do so.
I think determinism at the network level is out of scope of a module system, and may be out of scope of the Ecmascript group.
Even ignoring network conditions, browsers' wildly different behavior around deferred scripts, the particulars of request headers, ordering of scripts, number of simultaneous open connections per host; can all affect the order in which requests are issued. Anything that requires deterministic behavior as apparent to a network observer seems like a non-starter to me.
If you want a clear description of which resources are and are not fetched, then that's probably specifiable, but GET requests are supposed to be idempotent, so I don't know what that would gain assuming fetches are via GET.
Actually, I'm dumb. idempotence only holds for # of requests >= 1.
Obviously, requesting the first time may convey significant information.
On Thu, Jan 14, 2010 at 1:58 PM, <ihab.awad at gmail.com> wrote:
Hi Mike,
On Thu, Jan 14, 2010 at 1:29 PM, Mike Samuel <mikesamuel at gmail.com> wrote:
Are these proposals mutually exclusive or complementary. If (1) and (2) are exclusive, is there a place to collect use cases for a module-off?
For my part, I think (1) and (2) are counterproposals, but there may exist a bunch of cross-pollination. For example, I will probably end up stealing Kris's "import with" syntax sometime.
Are they really? The primitive proposal says: "This proposal specifies support for both “loading” dependencies as module factory functions (Makers) ..."
I can see how the Makers will work, but I think to convince ourselves, it would be good to provide example #4 of implementing Makers on top of Context() (proposal 2 on top of proposal 1). I like the approach of providing a primitive Context() and then being able to implement Makers-style modules in those semantics (i.e. with pure JS code).
For makers, what you're essentially proposing is "parameterized modules", no? The modules are parameterized by their dependencies. That is something I strongly support, but is unusual compared say Python or Ruby's module systems. In those systems, dependencies are essentially hard-coded. I think making this clear in the proposals is useful.
Most of these papers are about this distinction (especially the Newspeak papers, and the Scala one): lambda-the-ultimate.org/node/3735
The way I think of it is that there are "bound" and "unbound" modules -- you bind the dependencies ("requires") of the Module by calling its module function with concrete dependencies.
I've been working on some code which does allows this extra degree of flexibility for CommonJS modules:
code.google.com/p/gelatin-js (needs docs, coming shortly)
Basically you say require('file') in the source code. And then at build time, bind 2 different implementations to that "requirement", depending on where you want to deploy. You can think of this as "module metaprogramming at build time", whereas the makers proposal is module metaprogramming at runtime, in JavaScript.
In one of the Newspeak papers he talks about not conflating "module definition" and "module configuration" (e.g. which Python/Ruby/Perl do). I agree with this -- there is one mechanism to define modules, and then you can configure the modules at build time or runtime.
JavaScript especially needs this feature because there are so many different execution environments, and no standard library (yet).
So I really like the general ideas of these 2 proposals, but some of the details are unclear. I think relating things to CommonJS modules is good because they are already implemented quite widely.
thanks, Andy
Within that body of literature there is one called Higher Order Functors which describes in this context modules being initialized with modules all of them parameritized. It looks like this is possible and powerful.
Kam
On Jan 16, 2010, at 11:47 AM, Andy Chu <andy at chubot.org> wrote:
On Thu, Jan 14, 2010 at 1:58 PM, <ihab.awad at gmail.com> wrote:
Hi Mike,
On Thu, Jan 14, 2010 at 1:29 PM, Mike Samuel <mikesamuel at gmail.com> wrote:
Are these proposals mutually exclusive or complementary. If (1) and (2) are exclusive, is there a place to collect use cases for a module-off?
For my part, I think (1) and (2) are counterproposals, but there may exist a bunch of cross-pollination. For example, I will probably end up stealing Kris's "import with" syntax sometime.
Are they really? The primitive proposal says: "This proposal specifies support for both “loading” dependencies as module factory functions (Makers) ..."
I can see how the Makers will work, but I think to convince ourselves, it would be good to provide example #4 of implementing Makers on top of Context() (proposal 2 on top of proposal 1). I like the approach of providing a primitive Context() and then being able to implement Makers-style modules in those semantics (i.e. with pure JS code).
For makers, what you're essentially proposing is "parameterized modules", no? The modules are parameterized by their dependencies. That is something I strongly support, but is unusual compared say Python or Ruby's module systems. In those systems, dependencies are essentially hard-coded. I think making this clear in the proposals is useful.
Most of these papers are about this distinction (especially the Newspeak papers, and the Scala one): lambda-the-ultimate.org/node/3735
The way I think of it is that there are "bound" and "unbound" modules -- you bind the dependencies ("requires") of the Module by calling its module function with concrete dependencies.
I've been working on some code which does allows this extra degree of flexibility for CommonJS modules:
code.google.com/p/gelatin-js (needs docs, coming shortly)
Basically you say require('file') in the source code. And then at build time, bind 2 different implementations to that "requirement", depending on where you want to deploy. You can think of this as "module metaprogramming at build time", whereas the makers proposal is module metaprogramming at runtime, in JavaScript.
In one of the Newspeak papers he talks about not conflating "module definition" and "module configuration" (e.g. which Python/Ruby/Perl do). I agree with this -- there is one mechanism to define modules, and then you can configure the modules at build time or runtime.
JavaScript especially needs this feature because there are so many different execution environments, and no standard library (yet).
So I really like the general ideas of these 2 proposals, but some of the details are unclear. I think relating things to CommonJS modules is good because they are already implemented quite widely.
thanks, Andy
On Thu, Jan 14, 2010 at 1:06 PM, <ihab.awad at gmail.com> wrote:
Module proposal, by Kris, focusing on management of primordials.
I've revised this first proposal to support all of the "import id [from package] [with scope]" forms necessary for Ihab's second and third proposals, and restricted the "id" expressions to Strings, while leaving the "package" expression open to any JSON. Assuming that package bindings are managed externally (as they are presently proposed to be at CommonJS), this could be further constrained to Strings; I do not have an opinion on the matter. In any case, I'm attempting to line things up better so that proposal 2 can stack on proposal 1, as Andy suggested.
Also, with regard to externally managing module instantiation, these primordials could support several possible systems. The first is to construct sandboxes with primed module memos with manually instantiated modules. The second is to interpose a loader that provides alternate module functions which in turn instantiate the desired modules on demand. The third is to not use "require" at all, and simply to presume that certain free variables have been provided with instances of the modules a module depends upon. Both have advantages and disadvantages and both can be built on the recommended primoridals.
Kris Kowal
On Jan 15, 2010, at 1:32 PM, Kam Kasravi wrote:
Brendan, do you recall es4 where the concept of a unit was introduced? There was also some discussion where management and security where implicitly mapped to the '.' between identifier parts. I
cannot find that reference.
Was it this?
On Jan 14, 2010, at 1:29 PM, Mike Samuel wrote:
"import" is added to the reserved keyword list?
Already in ES5, see 7.6.1.2:
FutureReservedWord :: one of class enum extends super const export Import or in strict mode code one of implements let private public interface package protected static yield
Yes, that looks right, I also remember a reference to the 'dot' notation where namespace access would be arbitrated implicitly by objects representing the '.' It may have been in one of Lars Hansen papers on gradual typing. In any case there was some good discussion on meta-level objects and controlling capabilities or access.
Was there any discussion in the past about parameterized modules or units where recursively nested modules/units were brought up? If two modules refer to each other and all imports are resolved eagerly then I think this would be a problem where use cases would abound.
thx Kam
On Mon, Jan 18, 2010 at 10:29 AM, Kam Kasravi <kamkasravi at yahoo.com> wrote:
Yes, that looks right, I also remember a reference to the 'dot' notation where namespace access would be arbitrated implicitly by objects representing the '.' It may have been in one of Lars Hansen papers on gradual typing. In any case there was some good discussion on meta-level objects and controlling capabilities or access.
Was there any discussion in the past about parameterized modules or units where recursively nested modules/units were brought up? If two modules refer to each other and all imports are resolved eagerly then I think this would be a problem where use cases would abound.
What are you referring to here? I think you are hinting at the problem of circular dependencies using "import" semantics like Python and CommonJS. Right now if you have mutually recursive modules, you get partially initialized modules, because the "program counter" just follows every require(). The CommonJS spec specifically allows the partially initialized modules I think.
Personally I try to break up circular dependencies, but this behavior is not very friendly, so I think it would be better if Harmony supported circular dependencies in a more "correct" way. So this is an advantage of the "makers" style semantics (separating module definition and configuration).
The Newspeak paper talks explicitly about this: scholar.google.com/scholar?hl=en&source=hp&q=objects as modules in newspeak&um=1&ie=UTF-8&sa=N&tab=ws
If you are initializing modules in topological order, and there is a circular dependency between A and B, he says to just make a dummy proxy object for B to pass into A, initialize A, and then initialize B.
That also reminds me that a good use case for catchalls is making module-like objects. I haven't seen any discussions of the 2 together but it's definitely worth thinking about how they interact.
Andy
Andy
yes I was referring to the circular dependency issue. Catch-alls would be useful for this, though it would be nice if they were automated, eg returning a proxy until the module was used. Doing it manually by developer would be too difficult. I realize modules are intended to just satisfy a function body and can be initialized as objects or called as functions, but since they have strong container semantics I wonder if they should have more than just an 'id' especially if they provide a gateway to what else is in the directory as in commonjs.
Kam
I don't quite understand what is meant by circular dependencies in this context. Are you talking about the graph of imports, or the graph of module instances?
If the former, how is it a problem?
If the latter, how could it occur without explicit user intent : a module explicitly passing itself to a dependency?
E.g. in the former // fib.js var self = (import "fib.js"); if (n < 2) { return n; } return self({ n: n - 2 }) + self({ n : n - 1 }); a module imports itself, and recursively invokes separate instances which is stupid but does not result in a module being observed in a partially initialized state.
In the latter: // foo.js var otherModuleMaker = (import "bar.js")({ baz: this }); module foo is explicitly allowing otherModuleMaker to observe it in an uninitialized state.
2010/1/18 Kam Kasravi <kamkasravi at yahoo.com>:
On Mon, Jan 18, 2010 at 4:27 PM, Kam Kasravi <kamkasravi at yahoo.com> wrote:
Andy
yes I was referring to the circular dependency issue. Catch-alls would be useful for this, though it would be nice if they were automated, eg returning a proxy until the module was used. Doing it manually by developer would be too difficult. I realize modules are intended to just satisfy a function body and can be initialized as objects or called as functions, but since they have strong container semantics I wonder if they should have more than just an 'id' especially if they provide a gateway to what else is in the directory as in commonjs.
Can you clarify what you mean by "provide a gateway"? AFAIK the only gateway a commonjs module provides is that which is explicitly added its exports object, which can be required modules from anywhere in require.paths. Also, in the context of commonjs SecurableModules, as the spec stands, modules * cannot* be called as functions. However if Harmony were to bless a module system (or at least a means to create module systems) it would be wonderful if this oft-requested feature would be considered.
The former but not in the way you've described. Rather
module A: this.getB= function() { return new (import 'B')(); }
module B: this.getA = function() { return new (import 'A')(); }
If modules eagerly resolve imports then the above circular dependency would be a problem. If modules lazily resolve imports then the above circular dependency would be ok. If catch-alls were used then proxy's would be used and the circular dependency would be ok. Current js has many examples of circular dependencies which are resolved by lazy evaluation.
Kam
2010/1/18 Kam Kasravi <kamkasravi at yahoo.com>:
Hi Mike
The former but not in the way you've described. Rather
module A: this.getB= function() { return new (import 'B')(); }
module B: this.getA = function() { return new (import 'A')(); }
If modules eagerly resolve imports then the above circular dependency would be a problem. If modules lazily resolve imports then the above circular dependency would be ok. If catch-alls were used then proxy's would be used and the circular dependency would be ok. Current js has many examples of circular dependencies which are resolved by lazy evaluation.
Ok. Ihab's proposal allows eager fetching, and mandates lazy instantiation, so the above example is not a problem with Ihab's proposal.
Yes I was talking about the commonjs pattern where objects are extended or aggregated by modifying their exports. I was thinking of an API where a module could be queried for dependencies, its 'public' methods or other types of meta info via a common API. It would be a good place to define policies. Depending on the caller/context, different things could be returned, wrapped or delegated - eg friends (c++), proxies (catchalls), strategy patterns etc.
Kam
Ok, cool! So the import framework could go about fetching A then B or B then A for any 3rd object that was going to use them. I know that YUI 3 does this type of loadiong explicitly, in this case would the import framework would take care of it or would there still need to be something the developer would do to get A and B early? I wasn't sure going thru the 'pim' function in Ihab's strawman where it was finding the dependencies or was getting help from the js runtime or something. thx Kam
My understanding is that in a simplified example: module A: function getB() { return (import 'B')(); } module B: function getA() { return (import 'A')(); } main : window.onload = (import 'A')().getB; where "main" is a module loaded via <script>(import 'main')()</script>, the following happens (1) browser parses script body, sees import of 'main' (2) browser starts fetch of 'main' and suspends script execution until 'main' is satisfied (3) browser receives 'main' content and sees import of 'A'. browser has the option of starting fetching 'A' (4) browser marks 'main' satisfied. (5) browser starts instantiating 'main', and reaches (import 'A'). It must suspend execution if 'A' has not been satisfied ; i.e. if 'A' is not already in the cache due to prefetching or previous network activity. (6) once 'A' has been satisfied, it can start instantiating 'A', at which point it could optionally prefetch 'B'. (7) at some point the onload event fires which will suspend execution until 'B' has been satisfied (8) 'B' can be satisfied without any further prefetching because 'A' has already been satisfied.
So the prefetching can be done or not depending on how many network connections are filled with other things, but the instantiation is done only when control reaches the "import" operator.
2010/1/18 Kam Kasravi <kamkasravi at yahoo.com>:
On Mon, Jan 18, 2010 at 2:24 PM, Kam Kasravi <kamkasravi at yahoo.com> wrote:
Hi Mike
The former but not in the way you've described. Rather
module A: this.getB= function() { return new (import 'B')(); }
module B: this.getA = function() { return new (import 'A')(); }
In CommonJS, this doesn't seem like a special problem. The imports are not executed until the functions are called, so everything is dandy. I was referring to this situation:
module A: import B
B.foo()
module B: import A
A.bar()
Do the given proposals allow or disallow this?
Andy
So A.bar() would be a function A with a function bar attached. I believe Ihab's proposal allows this.
In order to satisfy something like (import B).foo() the browser would need to import B prior to reaching this statement or suspend execution at the '.' so the resources could be fetched. I believe we're talking about the former - getting all resources early.
On Jan 18, 2010, at 4:48 PM, Kam Kasravi wrote:
In order to satisfy something like (import B).foo() the browser would need to import B prior to reaching this statement or suspend execution at the '.' so the resources could be fetched. I believe we're talking about the former - getting all resources
early.
That's right -- no races, execution still seems single-threaded.
Wherefore the special form, although if this line is in a seldom-
executed branch you still have the hit of loading B up front.
On Mon, Jan 18, 2010 at 4:54 PM, Brendan Eich <brendan at mozilla.com> wrote:
That's right -- no races, execution still seems single-threaded. Wherefore the special form, although if this line is in a seldom-executed branch you still have the hit of loading B up front.
Right. The optimization in this case is that the runtime could delay loading B and, if B is encountered, block the currently running event loop and fetch B.
Ihab
On Jan 18, 2010, at 5:20 PM, ihab.awad at gmail.com wrote:
On Mon, Jan 18, 2010 at 4:54 PM, Brendan Eich <brendan at mozilla.com>
wrote:That's right -- no races, execution still seems single-threaded.
Wherefore the special form, although if this line is in a seldom-executed
branch you still have the hit of loading B up front.Right. The optimization in this case is that the runtime could delay loading B and, if B is encountered, block the currently running event loop and fetch B.
So the spec is that import could nest an event loop? What scripts or
event handlers/timeouts/etc. could run at that point? In general
breaking the run-to-completion execution model means the importing
script and its containing window could be destroyed, a zombie.
On Jan 18, 2010, at 5:41 PM, Brendan Eich wrote:
On Jan 18, 2010, at 5:20 PM, ihab.awad at gmail.com wrote:
On Mon, Jan 18, 2010 at 4:54 PM, Brendan Eich <brendan at mozilla.com> wrote:
That's right -- no races, execution still seems single-threaded. Wherefore the special form, although if this line is in a seldom-executed branch you still have the hit of loading B up front.
Right. The optimization in this case is that the runtime could delay loading B and, if B is encountered, block the currently running event loop and fetch B.
So the spec is that import could nest an event loop? What scripts or event handlers/timeouts/etc. could run at that point? In general breaking the run-to-completion execution model means the importing script and its containing window could be destroyed, a zombie.
It's also horrific in the context of a browser as it's effectively an arbitrary length call blocked on IO.
On Mon, Jan 18, 2010 at 5:41 PM, Brendan Eich <brendan at mozilla.com> wrote:
So the spec is that import could nest an event loop? What scripts or event handlers/timeouts/etc. could run at that point? In general breaking the run-to-completion execution model means the importing script and its containing window could be destroyed, a zombie.
No, my proposal was that, if the system can determine that an import is infrequently used or unlikely to be used, it can block the entire event loop natively while fetching that delayed import. From the viewpoint of the running code, nothing happened; it's just that wall clock time ran forward a bit.
But maybe this is a bad idea. Never mind. :)
Ihab
On Mon, Jan 18, 2010 at 5:49 PM, <ihab.awad at gmail.com> wrote:
On Mon, Jan 18, 2010 at 5:41 PM, Brendan Eich <brendan at mozilla.com> wrote:
So the spec is that import could nest an event loop? What scripts or event handlers/timeouts/etc. could run at that point? In general breaking the run-to-completion execution model means the importing script and its containing window could be destroyed, a zombie.
No, my proposal was that, if the system can determine that an import is infrequently used or unlikely to be used, it can block the entire event loop natively while fetching that delayed import. From the viewpoint of the running code, nothing happened; it's just that wall clock time ran forward a bit.
But maybe this is a bad idea. Never mind. :)
I concur that it is a bad idea. Such optimization opportunities should only
be at async load points. We cannot optimize the sync load points without violating non-blocking event-loop concurrency, so we shouldn't.
On Mon, Jan 18, 2010 at 5:43 PM, Oliver Hunt <oliver at apple.com> wrote:
It's also horrific in the context of a browser as it's effectively an arbitrary length call blocked on IO.
That's a good description of what I was proposing. Yes it would block the event loop. Perhaps it could be useful if (say) the module were available in a local disk cache but not loaded into memory: the cost of I/O to a disk cache may be reasonable, but not the cost of a completely new fetch from the network.
Ihab
On Jan 18, 2010, at 5:52 PM, ihab.awad at gmail.com wrote:
On Mon, Jan 18, 2010 at 5:43 PM, Oliver Hunt <oliver at apple.com> wrote:
It's also horrific in the context of a browser as it's effectively
an arbitrary length call blocked on IO.That's a good description of what I was proposing. Yes it would block the event loop. Perhaps it could be useful if (say) the module were available in a local disk cache but not loaded into memory: the cost of I/O to a disk cache may be reasonable, but not the cost of a completely new fetch from the network.
Even disk can be painfully slow-ish, and what if there's a cache miss?
I guess Brendan is saying that if a continuation was natively built ar the '.' then the risk is that not acquiring the resource would result in a zombie. But if the runtime could build a continuation at the '.' then it could come back to it when the resource was acquired. Brendan is that layman's description or did I get it wrong?
Kam
On Jan 18, 2010, at 5:52 PM, ihab.awad at gmail.com wrote:
On Mon, Jan 18, 2010 at 5:43 PM, Oliver Hunt <oliver at apple.com> wrote:
It's also horrific in the context of a browser as it's effectively an arbitrary length call blocked on IO.
That's a good description of what I was proposing. Yes it would block the event loop. Perhaps it could be useful if (say) the module were available in a local disk cache but not loaded into memory: the cost of I/O to a disk cache may be reasonable, but not the cost of a completely new fetch from the network.
Ihab
On Jan 18, 2010, at 6:14 PM, Kam Kasravi wrote:
I guess Brendan is saying that if a continuation was natively built
ar the '.' then the risk is that not acquiring the resource would
result in a zombie. But if the runtime could build a continuation at
the '.' then it could come back to it when the resource was
acquired. Brendan is that layman's description or did I get it wrong?
That's not wrong, but it does not consider the preemption point.
Scripts in browsers today (event handler or timeout functions, same
deal) run to completion. They do not race or nest event loops (bugs
notwithstanding). So you can be sure a global variable you've just set
to 42 will still be 42 after a call to a function.
We could break this model at the . whose left-hand side was (import M)
but should we?
Neil Mix's NarrativeJS used -> instead, to signify the difference. I
do not propose we break the run-to-completion model implicitly, or add
explicit syntax for doing so. At least not without a lot more
discussion.
On Jan 18, 2010, at 8:40 PM, Brendan Eich wrote:
On Jan 18, 2010, at 6:14 PM, Kam Kasravi wrote:
I guess Brendan is saying that if a continuation was natively built ar the '.' then the risk is that not acquiring the resource would result in a zombie. But if the runtime could build a continuation at the '.' then it could come back to it when the resource was acquired. Brendan is that layman's description or did I get it wrong?
That's not wrong, but it does not consider the preemption point. Scripts in browsers today (event handler or timeout functions, same deal) run to completion. They do not race or nest event loops (bugs notwithstanding). So you can be sure a global variable you've just set to 42 will still be 42 after a call to a function.
We could break this model at the . whose left-hand side was (import M) but should we?
Neil Mix's NarrativeJS used -> instead, to signify the difference. I do not propose we break the run-to-completion model implicitly, or add explicit syntax for doing so. At least not without a lot more discussion.
One thing i still do not understand is how a module is loaded -- there seems to be much reference to "module A" and "module B" but where (in the browser context) are we expecting them to be loaded from? How are module names resolved? To what extent are module instances expected to be shared (given you have multiple concurrent-ish JS contexts in the browser)? Or are we primarily interested in defining the semantics for a module specifically, and leaving the actual import mechanism up to the embedding environment.
Understood, yes I'm familiar with Neil Mix's NarrativeJS. Sorry for the misinterpretation of your comments as well as the import strawman. Lazily fetching imports in this way could force preemption in many places. Were there any es4 strawman's that suggested a similar deviation from the run-to-completion model?
Kam
On Mon, Jan 18, 2010 at 10:04 PM, Oliver Hunt <oliver at apple.com> wrote:
One thing i still do not understand is how a module is loaded -- there seems to be much reference to "module A" and "module B" but where (in the browser context) are we expecting them to be loaded from?
I deliberately left this open in order to allow enough wiggle room to bring in the package proposal.
In the simplest case, imagine that "A" and "B" are URLs.
To what extent are module instances expected to be shared ...
A module instance is shared if and only if the instance is supplied as an object directly to the two participants in the sharing.
Or are we primarily interested in defining the semantics for a module specifically, and leaving the actual import mechanism up to the embedding environment.
It is my hope that we would standardize this, but I didn't want to overconstrain it and instead hoped for the discussion about what a module ID "means" to happen with regard to the package proposal.
Ihab
On Mon, Jan 18, 2010 at 6:05 PM, Brendan Eich <brendan at mozilla.com> wrote:
Even disk can be painfully slow-ish, and what if there's a cache miss?
Ok. Conversation reset! :)
I proposed this merely as a way to deal with a specific problem -- the problem that an infrequently used module must be loaded anyway.
This may be a bad idea and it is confusing the core issue, of how the proposal I made is intended to be used and implemented. Let me restate the base proposal:
* * * * *
In the simple case, (import "foo") evaluates immediately to an already-loaded module function.
To do asynchronous loading, I propose that the programmer use a function, provided by the embedding, that allows them to initiate a graph of "import"-ed modules and receive the root-most module function eventually.
It is an open question whether we should standardize the asynchronous form as well.
Ihab
Kris Kowal and I have written up our proposals for a module system, to be discussed during the Jan 28-29 meeting. The proposals are as follows:
strawman:modules_primordials
Module proposal, by Kris, focusing on management of primordials.
strawman:modules_emaker_style
Module proposal, by Ihab, focusing on EMaker-style invocation semantics.
strawman:modules_packages
Packages proposal, by Ihab, focusing on how code may be identified and distributed.
Ihab
Hi folks, Kris Kowal and I have written up our proposals for a module system, to be discussed during the Jan 28-29 meeting. The proposals are as follows: 1. http://wiki.ecmascript.org/doku.php?id=strawman:modules_primordials Module proposal, by Kris, focusing on management of primordials. 2. http://wiki.ecmascript.org/doku.php?id=strawman:modules_emaker_style Module proposal, by Ihab, focusing on EMaker-style invocation semantics. 3. http://wiki.ecmascript.org/doku.php?id=strawman:modules_packages Packages proposal, by Ihab, focusing on how code may be identified and distributed. Ihab -- Ihab A.B. Awad, Palo Alto, CA