Minor questions on new module BNF
so shouldn't
ExportDeclarationbe added toScriptElement? Or perhaps toStatement?
Upon reflection, Statement is obviously incorrect, as it prevents static determination of a module's exports. ScriptElement still seems good though.
On Sun, Jun 2, 2013 at 11:59 PM, Domenic Denicola <domenic at domenicdenicola.com> wrote:
Sam was saying that harmony:modules is up to date. If so, I'm seeing a few things missing from the BNF. It would be great to get these clarified so people can start writing accurate transpilers.
- It looks like
ExportDeclarationcan only occur insideModuleBody, which itself can only occur as part of the namedModuleDeclarationproduction. I thought we would be able to export things without explicit named module declaration, so shouldn'tExportDeclarationbe added toScriptElement? Or perhaps toStatement? At harmony:modules_examples the "compiler/Lexer.js" example contains such a situation, which seems invalid from the current grammar.
The idea here is that modules will typically be written in files like
"compiler/Lexer.js", where the starting grammar production is
ModuleBody.
- Is there any way to simply "execute" a module that does not export anything? This is a common technique in both client and server-side JavaScript today, e.g.
I would just write import {} from "someModule";
From: samth0 at gmail.com [mailto:samth0 at gmail.com] On Behalf Of Sam Tobin-Hochstadt
The idea here is that modules will typically be written in files like "compiler/Lexer.js", where the starting grammar production is
ModuleBody.
Ah, that makes sense! It's a nice way of prohibiting <script>export function foo() { }</script> as well, assuming inline <script> corresponds to Script. It would be helpful to update the wiki with this, or more generally to show how this grammar will integrate with the rest of the grammar.
I would just write
import {} from "someModule";
That appears to be disallowed; I believe
"{" ImportSpecifier ("," ImportSpecifier)* ","? "}"
requires at least one ImportSpecifier. (It's also sad and ugly; any reason not to allow import "someModule";?)
On Mon, Jun 3, 2013 at 12:24 AM, Domenic Denicola < domenic at domenicdenicola.com> wrote:
From: samth0 at gmail.com [mailto:samth0 at gmail.com] On Behalf Of Sam Tobin-Hochstadt
The idea here is that modules will typically be written in files like "compiler/Lexer.js", where the starting grammar production is
ModuleBody.Ah, that makes sense! It's a nice way of prohibiting
<script>export function foo() { }</script>as well, assuming inline<script>corresponds toScript. It would be helpful to update the wiki with this, or more generally to show how this grammar will integrate with the rest of the grammar.I would just write
import {} from "someModule";That appears to be disallowed; I believe
"{" ImportSpecifier ("," ImportSpecifier)* ","? "}"requires at least one
ImportSpecifier. (It's also sad and ugly; any reason not to allowimport "someModule";?)
I've advocated for this in the past. I believe it should be allowed.
Separately, I would like this form to be specified as deferring execution
until bindings are explicitly imported (from another module), or a
synchronous System.get call is made.
This would make it possible to guarantee that a synchronous System.get
will succeed, without being forced to execute the module first.
Yehuda Katz wrote:
I've advocated for this in the past. I believe it should be allowed.
Separately, I would like this form to be specified as deferring execution until bindings are explicitly imported (from another module), or a synchronous
System.getcall is made.This would make it possible to guarantee that a synchronous
System.getwill succeed, without being forced to execute the module first.
+1.
Big +1 on being able to defer execution until explicit import. It definitely seems useful to allow for control over dependency specifications separately from [potential] dependency execution. More concretely, I don't think you could do something like the following with the current module spec (correct if wrong):
exports.foo = function() { var bar = require('bar'); bar.baz(); }
What about specifying some form of hoisting for ImportDeclarations so that dependencies for a module can be declared separately from their execution? This would still allow for compile-time resolution and linking while also addressing concerns about runtime-conditional dependencies...
On Jun 3, 2013, at 9:33 AM, Yehuda Katz <wycats at gmail.com> wrote:
On Mon, Jun 3, 2013 at 12:24 AM, Domenic Denicola <domenic at domenicdenicola.com> wrote: From: samth0 at gmail.com [mailto:samth0 at gmail.com] On Behalf Of Sam Tobin-Hochstadt
I would just write
import {} from "someModule";That appears to be disallowed; I believe
"{" ImportSpecifier ("," ImportSpecifier)* ","? "}"requires at least one
ImportSpecifier.
Oversight. Fixed.
(It's also sad and ugly; any reason not to allow
import "someModule";?)I've advocated for this in the past. I believe it should be allowed.
I've always liked this too, and just hadn't really gotten to it. But I've added it to the wiki page too. I'll work with Jason to add this to the reference library; it should be a small addition.
Separately, I would like this form to be specified as deferring execution until bindings are explicitly imported (from another module), or a synchronous
System.getcall is made.This would make it possible to guarantee that a synchronous
System.getwill succeed, without being forced to execute the module first.
Yep, agreed. (To be pedantic, it's not that it defers execution so much as that it doesn't force execution.)
On Jun 3, 2013, at 12:24 AM, Domenic Denicola <domenic at domenicdenicola.com> wrote:
Ah, that makes sense! It's a nice way of prohibiting
<script>export function foo() { }</script>as well, assuming inline<script>corresponds toScript. It would be helpful to update the wiki with this, or more generally to show how this grammar will integrate with the rest of the grammar.
I added a note at the end of the grammar section. Do you think that helps make it a little clearer?
Thanks,
Yep, agreed. (To be pedantic, it's not that it defers execution so much as that it doesn't force execution.)
Can you explain? I would naively expect for this:
// someModule
console.log("someModule");
// main
import "someModule";
console.log("main");
when executing main, to output:
> someModule
> mainThat appears to be disallowed; I believe
"{" ImportSpecifier ("," ImportSpecifier)* ","? "}"requires at least one
ImportSpecifier.Oversight. Fixed.
Looks good, but I'm thinking that this should probably not be allowed:
import {,} from "x";On Tue, Jun 4, 2013 at 6:29 AM, Kevin Smith <zenparsing at gmail.com> wrote:
Yep, agreed. (To be pedantic, it's not that it defers execution so much as that it doesn't force execution.)
Can you explain? I would naively expect for this:
// someModule console.log("someModule"); // main import "someModule"; console.log("main");when executing main, to output:
> someModule > main
In general, expectations about side-effects that happen during module loading are really edge-cases. I would go as far as to say that modules that produce side effects during initial execution are "doing it wrong", and are likely to produce sadness.
In this case, the import statement is just asking the module loader to
download "someModule", but allowing the app to move on with life and not
bother executing it. This would allow an app to depend on a bunch of
top-level modules that got executed only once the user entered a particular
area, saving on initial boot time.
From: Yehuda Katz [wycats at gmail.com]
In general, expectations about side-effects that happen during module loading are really edge-cases. I would go as far as to say that modules that produce side effects during initial execution are "doing it wrong", and are likely to produce sadness.
In this case, the
importstatement is just asking the module loader to download "someModule", but allowing the app to move on with life and not bother executing it. This would allow an app to depend on a bunch of top-level modules that got executed only once the user entered a particular area, saving on initial boot time.
I don't think this is correct. It is strongly counter to current practice, at the very least, and I offered some examples up-thread which I thought were pretty compelling in showing how such side-effecting code is fairly widely used today.
This isn't a terribly important thing, to be sure. But IMO it will be very surprising if
import x from "x";
executes the module "x", producing side effects, but
import "x";
does not. It's surprising precisely because it's in that second case that side effects are desired, whereas I'd agree that for modules whose purpose is to export things producing side effects is "doing it wrong."
In this case, the
importstatement is just asking the module loader to download "someModule", but allowing the app to move on with life and not bother executing it. This would allow an app to depend on a bunch of top-level modules that got executed only once the user entered a particular area, saving on initial boot time.
That explains it, but I wonder how much time is saved. The browser would have to fetch the module and parse the module text in order to pick up dependencies. If modules don't have side-effects, then I presume that they don't really do much of anything when executed, other than initialize bindings.
Thanks again!
On Mon, Jun 3, 2013 at 11:33 AM, Yehuda Katz <wycats at gmail.com> wrote:
I've advocated for this in the past. I believe it should be allowed.
Separately, I would like this form to be specified as deferring execution until bindings are explicitly imported (from another module), or a synchronous
System.getcall is made.
It makes the static import language a bit more expressive, but why is it necessary?
For performance? The alternative, if a module has expensive initialization, would be to have it initialize itself more lazily.
This would make it possible to guarantee that a synchronous System.get
will succeed, without being forced to execute the module first.
It would definitely make that possible, but what System.get() use case
are you looking to support? To my mind System.get() is like examining
$LOADED_FEATURES in Ruby or sys.modules in Python; those use cases
exist, but the kind of code you're writing when you touch those is
typically either an egregious hack or it's generic across all modules,
right? They don't need or merit syntactic support.
I want to understand the motivation, because Domenic asked for syntax that just loads and runs a module, without bindings. It seems like we could support that feature with a tweak of the grammar, but you're proposing taking that exact syntax and using it for something else. I expect people trying to load and run a module will greatly outnumber people trying to load and not run a module. Wouldn't we be astonishing more people by doing it your way?
On Tue, Jun 4, 2013 at 8:11 AM, Jason Orendorff <jason.orendorff at gmail.com>wrote:
On Mon, Jun 3, 2013 at 11:33 AM, Yehuda Katz <wycats at gmail.com> wrote:
I've advocated for this in the past. I believe it should be allowed.
Separately, I would like this form to be specified as deferring execution until bindings are explicitly imported (from another module), or a synchronous
System.getcall is made.It makes the static import language a bit more expressive, but why is it necessary?
For performance? The alternative, if a module has expensive initialization, would be to have it initialize itself more lazily.
This would make it possible to guarantee that a synchronous
System.getwill succeed, without being forced to execute the module first.
It would definitely make that possible, but what
System.get()use case are you looking to support? To my mindSystem.get()is like examining$LOADED_FEATURESin Ruby orsys.modulesin Python; those use cases exist, but the kind of code you're writing when you touch those is typically either an egregious hack or it's generic across all modules, right? They don't need or merit syntactic support.I want to understand the motivation, because Domenic asked for syntax that just loads and runs a module, without bindings. It seems like we could support that feature with a tweak of the grammar, but you're proposing taking that exact syntax and using it for something else. I expect people trying to load and run a module will greatly outnumber people trying to load and not run a module. Wouldn't we be astonishing more people by doing it your way?
I'm happy to give in here, but I'm nervous about how strongly Domenic wants to connect the timing of imports with execution in people's minds.
If people actually write code like that, it's going to be a bad time.
On Tuesday, June 4, 2013, Yehuda Katz wrote:
On Tue, Jun 4, 2013 at 8:11 AM, Jason Orendorff <jason.orendorff at gmail.com<javascript:_e({}, 'cvml', 'jason.orendorff at gmail.com');>
wrote:
On Mon, Jun 3, 2013 at 11:33 AM, Yehuda Katz <wycats at gmail.com<javascript:_e({}, 'cvml', 'wycats at gmail.com');>
wrote:
I've advocated for this in the past. I believe it should be allowed.
Separately, I would like this form to be specified as deferring execution until bindings are explicitly imported (from another module), or a synchronous
System.getcall is made.It makes the static import language a bit more expressive, but why is it necessary?
For performance? The alternative, if a module has expensive initialization, would be to have it initialize itself more lazily.
This would make it possible to guarantee that a synchronous
System.getwill succeed, without being forced to execute the module first.It would definitely make that possible, but what
System.get()use case are you looking to support? To my mindSystem.get()is like examining$LOADED_FEATURESin Ruby orsys.modulesin Python; those use cases exist, but the kind of code you're writing when you touch those is typically either an egregious hack or it's generic across all modules, right? They don't need or merit syntactic support.I want to understand the motivation, because Domenic asked for syntax that just loads and runs a module, without bindings. It seems like we could support that feature with a tweak of the grammar, but you're proposing taking that exact syntax and using it for something else. I expect people trying to load and run a module will greatly outnumber people trying to load and not run a module. Wouldn't we be astonishing more people by doing it your way?
I'm happy to give in here, but I'm nervous about how strongly Domenic wants to connect the timing of imports with execution in people's minds.
If people actually write code like that, it's going to be a bad time.
This is possibly a faux pas, but what you just wrote reminded me of php's include behavior—which is a very bad time.
I still kinda like the idea of allowing ImportDeclarations to be expressed anywhere inside a ScriptElement (including in its children), and then hoisting the declaration for compilation/linking, but only executing the dependency when the statement is reached at runtime. It is then a parse error if an ImportDeclaration is outside of a ScriptElement context.
The System.get() paradigm is kind of awkward and indistinguishable from the import syntax in terms of its expressive relation. If its all we can get, so be it -- but it feels pretty unpolished.
FWIW this is what we have been doing at Facebook for a while now (over a year), and it's worked pretty well for us.
We use a require() function for pulling in dependencies. We then statically extract all require() callsites for a given module during our build step, and use that to identify the dependency-graph of a file (for packaging, etc).
On Tue, Jun 4, 2013 at 11:37 AM, Jeff Morrison <lbljeffmo at gmail.com> wrote:
I still kinda like the idea of allowing ImportDeclarations to be expressed anywhere inside a ScriptElement (including in its children), and then hoisting the declaration for compilation/linking, but only executing the dependency when the statement is reached at runtime. It is then a parse error if an ImportDeclaration is outside of a ScriptElement context.
That sounds nice. But again, why? Is it a performance thing? It seems like we're saying: yes, eagerly load, parse, and link these modules, but don't execute them eagerly because that would be too slow. I would expect loading to be the rate-determining step. What kind of module are we talking about?
The System.get() paradigm is kind of awkward and indistinguishable from the
import syntax in terms of its expressive relation. If its all we can get, so be it -- but it feels pretty unpolished.
Yeah, I don't like the idea of using System.get() for everyday programming
at all. Let's make import do whatever it is that's being asked for...
On Tue, Jun 4, 2013 at 11:44 AM, Jeff Morrison <lbljeffmo at gmail.com> wrote:
FWIW this is what we have been doing at Facebook for a while now (over a year), and it's worked pretty well for us.
We use a require() function for pulling in dependencies. We then statically extract all require() callsites for a given module during our build step, and use that to identify the dependency-graph of a file (for packaging, etc).
Thanks, this is a useful data point. Does your require() execute stuff lazily, on demand? How much time does that save on startup (and are there other benefits)?
On Tue, Jun 4, 2013 at 8:02 AM, Domenic Denicola <domenic at domenicdenicola.com> wrote:
From: Yehuda Katz [wycats at gmail.com]
In general, expectations about side-effects that happen during module loading are really edge-cases. I would go as far as to say that modules that produce side effects during initial execution are "doing it wrong", and are likely to produce sadness.
In this case, the
importstatement is just asking the module loader to download "someModule", but allowing the app to move on with life and not bother executing it. This would allow an app to depend on a bunch of top-level modules that got executed only once the user entered a particular area, saving on initial boot time.I don't think this is correct. It is strongly counter to current practice, at the very least, and I offered some examples up-thread which I thought were pretty compelling in showing how such side-effecting code is fairly widely used today.
This isn't a terribly important thing, to be sure. But IMO it will be very surprising if
import x from "x";executes the module "x", producing side effects, but
import "x";does not. It's surprising precisely because it's in that second case that side effects are desired, whereas I'd agree that for modules whose purpose is to export things producing side effects is "doing it wrong."
Agreed, and this is at least what is expected in AMD code today. Not
all scripts export something, but are still part of a dependency
relationship (may be event listeners/emitters). The import "x"
expresses that relationship.
I do like the idea of module bodies not being executed (or even parsed?) if it is not part of an explicit System.load or import chain. For code that wanted to delay the execution of some modules though, I expect that trick to be worked out via a delayed System.load() call than something to do with an import "x" combined with a System.get().
This is how it works in AMD today: define()'d modules are not executed until part of a require chain. Some folks use this to deliver define()'d modules in a bundle, but only trigger their execution on some later runtime event, and then do a require(["x"]) call (which is like System.load) that would then execute the define()'d "x" module.
So: yes to delayed execution (and even delayed parse), but not via import "x" + System.get(), just via System.load(), and all import forms doing the same thing for module body execution.
James
On 6/4/13 9:52 AM, Jason Orendorff wrote:
On Tue, Jun 4, 2013 at 11:44 AM, Jeff Morrison <lbljeffmo at gmail.com <mailto:lbljeffmo at gmail.com>> wrote:
FWIW this is what we have been doing at Facebook for a while now (over a year), and it's worked pretty well for us. We use a require() function for pulling in dependencies. We then statically extract all require() callsites for a given module during our build step, and use that to identify the dependency-graph of a file (for packaging, etc).Thanks, this is a useful data point. Does your require() execute stuff lazily, on demand? How much time does that save on startup (and are there other benefits)?
require() is a synchronous call in to our module system, and we don't execute a given module until all of its require() dependencies have been downloaded.
90% of the time we place all of our require()s at the top of the file (for grokkability/style reasons mostly), but we have done several experiments that show that evaluation time of JS in certain critical scenarios is a measurable bottleneck at scale. We've also shown engagement wins from optimizing said scenarios by deferring execution of 'less' important dependencies until they are needed. Contrived example:
var importantDep = require('importantDep');
class Foo extends importantDep { bar() { var lazyDep = require('lazyDep'); lazyDep.baz(); } }
exports.Foo = Foo;
Here, the loader understands that this module has two dependencies "importantDep" and "lazyDep" and it downloads both before executing the module. However, we don't "force execution" of the lazyDep module until someone comes along and calls the Foo.prototype.bar function.
More realistic scenarios where we have employed this kind of optimization include initial page loads (especially on slower devices such as older mobile phones), our platform widgets such as the like button, etc.
In general, it is true that it should not be the common case to optimize these kinds things, but the language should still allow for such optimizations when they arise.
Jeff Morrison wrote:
require() is a synchronous call in to our module system, and we don't execute a given module until all of its require() dependencies have been downloaded.
Hi Jeff, catching up (sorry if I missed it): do you do a browserify-style analysis for require("...") calls and hoist|uniq them, then prefetch them?
I haven't played with Browserify too much myself (shame on me), but from my own understanding of it and after talking to a friend who is more familiar, yes I believe so.
We "link" our modules statically on the server to build a dependency graph before we push to production. This linking step is based on require() call expressions found in the module, no matter where the call expression is located in the module. If we find one, it becomes a blocking dependency for the file. (we also have a separate API for non-blocking dependencies, but that's not too pertinent to this thread)
So the following two modules will become "available for execution" at the same time -- when (or some point after) both Dep1 and Dep2 have been downloaded but not necessarily evaluated:
/**
- ModuleA */ var Dep1 = require('Dep1'); // evaluate Dep1 synchronously if we haven't already var Dep2 = require('Dep2'); // evaluate Dep2 synchronously if we haven't already function foo() { Dep1.blah(); } function bar() { Dep2.blah(); } export.foo = foo; export.bar = bar;
/**
- ModuleB */ function foo() { require('Dep1').blah(); // evaluate Dep1 synchronously if we haven't already } function bar() { require('Dep2').blah(); // evaluate Dep2 synchronously if we haven't already }
The only difference between the two modules is that ModuleA will force evaluattion of both Dep1 and Dep2 when it runs, and ModuleB will not force execution until runtime invocation in one of the foo/bar functions.
Sam was saying that harmony:modules is up to date. If so, I'm seeing a few things missing from the BNF. It would be great to get these clarified so people can start writing accurate transpilers.
It looks like
ExportDeclarationcan only occur insideModuleBody, which itself can only occur as part of the namedModuleDeclarationproduction. I thought we would be able to export things without explicit named module declaration, so shouldn'tExportDeclarationbe added toScriptElement? Or perhaps toStatement? At harmony:modules_examples the "compiler/Lexer.js" example contains such a situation, which seems invalid from the current grammar.Is there any way to simply "execute" a module that does not export anything? This is a common technique in both client and server-side JavaScript today, e.g.
// initSubsystem1.js console.log("initializing subsystem 1..."); // initSubsystem2.js console.log("initializing subsystem 2..."); // init.js require("./initSubsystem1"); require("./initSubsystem2");On the client side this is common for e.g. plugins:
var Backbone = require("backbone"); require("backbone.layoutmanager"); // now `Backbone` has been augmented with a `Backbone.LayoutView` property we can use.I don't see any way to do this, except perhaps via
import defaultExportWillBeUndefinedSoThisBindingIsPointless from "./initSubsystem1"; import thisOneTooOhMyPoorScopeIsBeingPolluted from "./initSubsystem2";A production of the form
"import" ModuleSpecifier ";"would probably help this.Sam was saying that http://wiki.ecmascript.org/doku.php?id=harmony:modules is up to date. If so, I'm seeing a few things missing from the BNF. It would be great to get these clarified so people can start writing accurate transpilers. 1. It looks like `ExportDeclaration` can only occur inside `ModuleBody`, which itself can only occur as part of the named `ModuleDeclaration` production. I thought we would be able to export things without explicit named module declaration, so shouldn't `ExportDeclaration` be added to `ScriptElement`? Or perhaps to `Statement`? At http://wiki.ecmascript.org/doku.php?id=harmony:modules_examples the "compiler/Lexer.js" example contains such a situation, which seems invalid from the current grammar. 2. Is there any way to simply "execute" a module that does not export anything? This is a common technique in both client and server-side JavaScript today, e.g. ```js // initSubsystem1.js console.log("initializing subsystem 1..."); // initSubsystem2.js console.log("initializing subsystem 2..."); // init.js require("./initSubsystem1"); require("./initSubsystem2"); ``` On the client side this is common for e.g. plugins: ```js var Backbone = require("backbone"); require("backbone.layoutmanager"); // now `Backbone` has been augmented with a `Backbone.LayoutView` property we can use. ``` I don't see any way to do this, except perhaps via ```js import defaultExportWillBeUndefinedSoThisBindingIsPointless from "./initSubsystem1"; import thisOneTooOhMyPoorScopeIsBeingPolluted from "./initSubsystem2"; ``` A production of the form `"import" ModuleSpecifier ";"` would probably help this.