Module naming and declarations
I've been following es-discuss for a short amount of time. I'm a JS dev working on a significant code base, this biases how I perceive ES6 issues.
From my viewpoint by far the most important advancements provided by ES6, eclipsing all others, are modules and classes. This feeling is widely shared among the developers I work with.
So I'm somewhat surprised at the lack of response to Andreas email. Firstly I agree with Andreas point about there being an issue with the naming and declaring of modules.
In the original module proposal, modules were declared via ordinary lexical identifiers, but could also be imported from external sources denoted by strings. In November, the proposal was changed to use string-valued "module ids" for both. The motivation was to simplify the language, and to provide better support for configuration and concatenation.
This seems an odd change to make, a backward one in fact.
I presume that the aim of modules is to provide clean scopes/environments, to prevent global state pollution and to aid in structuring/separating code. Therefore you would wish modules to provide an abstract identifier as opposed to a concrete file path string as its identifier.
module topLevelNamespace.subNamespace {
export MyClass {
}
}
Nobody would suggest to use file paths in place of variable identifiers internally. Yet, that is almost exactly what the proposal does for modules!
Indeed. Working on a large code base containing hundreds of JS classes I think it's cleaner to deal with abstract identifiers which to correspond to namespaces as opposed to file locations on disk.
As various discussions show, people want and expect scope chain behaviour, and for very good reasons: e.g. nesting modules, confining modules to local scope, convenient local names, etc.
Yes. If we create private packages in our frameworks/libraries I see no reason for any end consumer to have access to these internal artifacts. This is all about working with large code bases, privacy and integrity are very helpful in those situations.
(The path semantics is inherited from legacy module frameworks for JS, such as AMD. It is a fine solution under the constraints that these frameworks have to operate in -- in particular, the inability to extend syntax or add new primitives. However, for ES6 most of these constraints don't apply, and there is no particular reason to limit the design to making the same compromises.)
To produce such a design purely to serve the needs of old module systems seems a poor choice to me. Over time the standard mechanism will eclipse all other module systems even if it provides no upgrade path for the old systems purely because it is the standard system. As long as it is not a totally broken design. The only time I would be willing to invest learning about the old module systems would be when I want to convert an old module to the new system so I can dump the old system.
There is far more code that is not using modules than code that is.
The focus should be on creating the best possible module system not the best possible system that smoothly accommodates AMD modules!
A custom loader can, in principle, perform arbitrary interpretation or rewriting of URLs. In particular, this could be used to implement interop to absolute repository paths a la AMD or Node, e.g. by interpreting an "amd:" schema for importing AMD modules that are relative to a separately configured base URL. In other words, you'd write
import M1 from "a/b"; // native ES6 import, relative path import M2 from "amd:c/d"; // import of AMD module, relative to AMD base URL
Schema handlers for non standard resources seems like an excellent idea. Neat, clean and easy to identify. I would support that, it could also be used to load non JS resources.
Thank you Andreas for highlighting these niggles I do hope there is more interest in them then the lack of response indicates.
The focus should be on creating the best possible module system not the best possible system that smoothly accommodates AMD modules!
Amen to that! I would add "or Node modules" to your last sentence above. The Node ship sailed, by my count, about three years ago. Competing feature-by-feature with AMD/Node modules should be an explicit non-goal. As should interoperability, for reasons which we could go into in another post.
On Apr 25, 2013 4:00 AM, "Brian Di Palma" <offler at gmail.com> wrote:
I've been following es-discuss for a short amount of time. I'm a JS dev working on a significant code base, this biases how I perceive ES6 issues.
From my viewpoint by far the most important advancements provided by ES6, eclipsing all others, are modules and classes. This feeling is widely shared among the developers I work with.
So I'm somewhat surprised at the lack of response to Andreas email.
On this point, Andreas wrote a long email, which I'm sure he spent time on. A response deserves similar consideration. I'm sure you'll see some soon.
On 25 April 2013 14:56, Kevin Smith <zenparsing at gmail.com> wrote:
The focus should be on creating the best possible module system not the best possible system that smoothly accommodates AMD modules!
Amen to that! I would add "or Node modules" to your last sentence above. The Node ship sailed, by my count, about three years ago. Competing feature-by-feature with AMD/Node modules should be an explicit non-goal. As should interoperability, for reasons which we could go into in another post.
I would actually disagree with that. I think interoperability is an important goal. But making interoperability maximally convenient less so, and in particular, it should not be a reason to heavily compromise on the design.
Having said that, interoperability with existing module systems was not the main motivation for the change in the proposal, as far as I can tell. It rather was simplicity (which is debatable, as I hope I've made clear), and convenient support for common configuration and concatenation use cases. You could count the latter as "easy interoperability with an existing deployment tool like `cat'", if you are so inclined. Notably, though, that doesn't work with AMD either.
Sam, I hope that will be the case, that's why I mentioned that I've only been following es-discuss for a short amount of time. I wasn't sure about the speed of response to such a substantial argument. I waited a day to see if anyone was interested or if it generated a discussion.
I agree that interoperability with older module systems is desirable and that it should not come at the cost of a better standard ES6 module system.
B.
First, I appreciate you setting your thoughts down in detail. I think this will help us move forward in the discussion.
You write in a later message:
Having said that, interoperability with existing module systems was not the main motivation for the change in the proposal, as far as I can tell. It rather was simplicity (which is debatable, as I hope I've made clear), and convenient support for common configuration and concatenation use cases.
I don't think this is right, and I think this is the heart of the issue. Let me back up a bit to explain this.
Module names play a role in three processes, in general:
- As a way to identify local components.
- As a way to find the physical resource that is the source code (or object code) of the module.
- As a way for two separately developed components to coordinate about which module they mean.
In the current design, the internal names (eg, "jquery") serve role 1, and URLs (as generated by the loader hooks) serve role 2. The coordination role is played by internal names in a shared registry.
To pick another example (at random :), take SML with CM [1]. Here, lexically-bound module name serve role 1, and CM serves role 2. Coordination is managed by the ML toplevel, and files are typically open in the sense that they refer to modules that will ultimately be bound at the top level during the compilation process.
The system you outline at the end of your message, and the similar system that Dave and I originally proposed, doesn't have anything that really fulfills role 3. That's why we changed the design.
This is perhaps easiest to understand with an example. Imagine you're developing a library, which will depend on jQuery, but you're not shipping jQuery with your library (this is how Ember works, as do a lot of other libraries). Then you have to write something like this:
// in ember.js
import "jquery" as $;
That's what you'd write in the current design.
In the design you're proposing, there are a few choices. You can give up on allowing people choose where they load jQuery from, and hardcode a dependency on a specific URL in the Ember source code. This is obviously not ok.
You could decide that instead there's a single well-known URL that everyone is going to identify with jQuery. This is basically using the URL namespace as keys in the module registry of the current design, except without separating logical names from URLs. Then your program looks like this:
// in ember.js
module jquery = "http://code.jquery.com/jquery-1.9.1.js";
import jquery as $;
Of course, even though that looks like a URL, it isn't being used to fetch anything, and everyone has to use some form of loader hook to map it to where they actually keep jQuery. Moreover, not everyone will have the resources or want to take the trouble to register a domain to publish their library.
Or you could settle on a local URL to use in the same way, again as basically a registry key.
// in ember.js
module jquery = "scripts/jquery.js";
import jquery as $;
Again, this thing that looks like a URL isn't being fetched, instead it will have to be rewritten to some other URL. And again, we have the same division into internal names that are in a registry, and external URLs that are actually for fetching the resource, except now because your proposal tries to abolish this distinction, it's much harder to manage.
In other words, we are now manually simulating the registry, and it's more painful and problematic for exactly the reasons that manually simulating things on top of abstractions that don't support them always is.
In contrast, the current design factors these two concerns, so that this configuration is about the mapping from internal names (like "jquery") to external names (like "http://....").
The key takeaway
The key takeaway here is this: in the current design, different modules from different developers can coordinate based on the internal names of their dependencies. In your lexical proposal, it's not possible to coordinate globally on internal names, because they're lexical. So instead developers would have to coordinate on external names. This is fundamentally flawed, because external names are about where to get bits, not which abstraction a name represents.
Supporting this use case properly is what led us to realize that the earlier lexically-named proposal was flawed.
Note that none of this is about concatenation. We've made some particular design decisions where concatenation played a role, but it wasn't a part of the reason we moved away from lexical modules.
Some smaller points
- As an external naming mechanisms, it violates standard relative path/URL semantics.
This isn't done particularly to be compatible, and node or AMD could have required "/" in front of absolute paths. But since that's an important use case, we don't think it's a good idea to tax it with extra typing. This convention has become popular in JS for a reason.
- The shared name space between internal and external modules can lead to incoherent programs.
I've already pointed out above how these namespaces aren't shared, and in fact their separation is an important reason for the current design. Also, the incoherent programs you refer to are ruled out by a suggestion (of yours!) that we adopted at the last meeting: declarative module forms that define a module that already exists are a static error.
- Likewise, a single global name space for all internally defined modules can lead to incoherent programs.
I would be more worried about this if (a) we didn't provide convenient ways to structure this name space and (b) it wasn't already an existing successful approach in real-world JS systems.
- "Local" references are globally overridable.
This is only true in the sense that executing first lets you define the meaning of particular modules. But this is a feature. This is just another way to describing configuration. It's not reasonable to think that we can decide for everyone what will need configuration and what won't.
- Internal module definition is coupled with external module registration, and modules cannot be renamed or re-registered.
These are not "external" names, but it is true that declaring a module registers it. It's possible to take it out of the table afterward, or give it a new name, or have it share multiple names, all by using the loader API in very easy ways. I don't think these will be common operations, and this seems like a reasonable tradeoff.
- Language-level naming semantics interferes with file system/URL naming semantics.
While it's sad that everyone doesn't use the same filesystem ;), this is inevitable unless we force everyone to write both internal and external names explicitly for every module; that's clearly more user-hostile than the alternative. Lots of languages (Java, Node, ...) manage a default mapping to the file system while making Windows and Mac filesystems work sensibly.
- Bundling ("concatenation") generally would require embedding arbitrary string resources, not syntactic modules.
The reason to use concatenation is to avoid consuming excessive client resources -- in this setting of course you won't want to run translation on the client side. Translation hooks are important (a) in less perf-sensitive settings like development and (b) for isolating remotely-loaded code, neither of which require concatenation.
- Module "declarations" are not a declarative mechanism, but an operational one.
This comes back to my original point. Registration of module names is about coordination, and thus this is an import feature, not a problem.
This is again long, but I hope it's clarified why the current design is the way it is, and why the alternative you propose doesn't solve the use cases we need to address.
Sam
[1] www.smlnj.org/doc/CM for those following along at home.
Module names play a role in three processes, in general:
- As a way to identify local components.
- As a way to find the physical resource that is the source code (or object code) of the module.
- As a way for two separately developed components to coordinate about which module they mean.
In the current design, the internal names (eg, "jquery") serve role 1, and URLs (as generated by the loader hooks) serve role 2. The coordination role is played by internal names in a shared registry.
You argue for a two-level system of non-lexical names to support configuration - okay. But why does that imply you have to drop the lexical naming altogether, instead of using a three-level system (from external to internal to lexical names)?
Also, in a two-level system of external and lexical names, could one not model the coordination level by a registry/configuration module?
// using loose syntax module registry { module jquery = <external remote or local path>
export jquery }
module client { import {jquery: $} from registry }
Claus
On Apr 25, 2013, at 4:08 PM, Claus Reinke <claus.reinke at talk21.com> wrote:
You argue for a two-level system of non-lexical names to support configuration - okay. But why does that imply you have to drop the lexical naming altogether, instead of using a three-level system (from external to internal to lexical names)?
You don't, it's an orthogonal concern. Note that Sam was not arguing against the existence of lexical modules.
But it's not nearly as important as the rest of the core system -- as Sam describes, coordination and separate development are the most important piece that the module system needs to address. We dropped lexical modules mostly in the interest of working out the core and eliminating parts that weren't necessary for ES6. Kevin's been urging us to reconsider dropping them, and I'm open to that in principle. In practice, however, we have to ship ES6.
But let's keep the question of having lexical private modules separate from this thread, which is about Andreas's suggestion to have lexical modules be the central way to define public modules.
Also, in a two-level system of external and lexical names, could one not model the coordination level by a registry/configuration module?
No, it would be too hard to get this expressive enough to satisfy the web platform's polyfilling needs.
You argue for a two-level system of non-lexical names to support configuration - okay. But why does that imply you have to drop the lexical naming altogether, instead of using a three-level system (from external to internal to lexical names)?
You don't, it's an orthogonal concern. Note that Sam was not arguing against the existence of lexical modules.
Good to hear that confirmed.
But it's not nearly as important as the rest of the core system -- as Sam describes, coordination and separate development are the most important piece that the module system needs to address. We dropped lexical modules mostly in the interest of working out the core and eliminating parts that weren't necessary for ES6. Kevin's been urging us to reconsider dropping them, and I'm open to that in principle. In practice, however, we have to ship ES6.
But let's keep the question of having lexical private modules separate from this thread, which is about Andreas's suggestion to have lexical modules be the central way to define public modules.
There are a couple of problems I see with that: it proposes adding yet another imperative API to JS where a declarative API would do (adding modules to the internal registry instead of the local scope); and it misses the big-rewrite-barrier that is going to accompany ES6 introduction - modules are the most urgent of ES6 improvements, but do you think users are going to rewrite their code bases twice just because modules are going to be delivered in two stages?
You believe you have worked out the core parts that caused you to postpone lexical modules, and you had a lexical module proposal before that. What is standing in the way of re-joining those two parts?
Claus
Thanks for this long explanation. I have several thoughts, but I'd like to ask one thing in particular.
What you propose, with "logical names", is a global namespace of short human-readable names with no conflict resolution authority. How do you see that working? From a namespace perspective, how is that any different than hanging identifiers off of the global object, as we do today? I'm not understanding how this strategy will facilitate namespace coordination. I can only see it leading to namespace confusion.
From: Kevin Smith [zenparsing at gmail.com]
What you propose, with "logical names", is a global namespace of short human-readable names with no conflict resolution authority. How do you see that working? From a namespace perspective, how is that any different than hanging identifiers off of the global object, as we do today? I'm not understanding how this strategy will facilitate namespace coordination. I can only see it leading to namespace confusion.
Indeed, I must second this question. (This causes me some cognitive dissonance, since from a Node.js/AMD perspective that I work in daily, string IDs seem great, so I'm not sure what I'm arguing for ^_^.)
The way I think of this is that it's conflating packages and modules. In current practice, packages have names that are registered in a global registry, and they are represented in the module system by a "main module" for that package. The wiring that allows "${packageName}" to resolve to the URL for the main module of packageName is part of the loader; in Node.js for example it does directory climbing through node_modules and inspects package.jsons for their "main" values.
In this way, modules do not have any globally-known names at all; their names are derived entirely from URLs (in the form of relative file paths).
I made this point in a recent presentation on client-side packages, slides 6 and 7.
Curious if others agree with this or if I'm totally off base as to how this fits into the discussion...
On Apr 26, 2013, at 7:20 AM, Claus Reinke <claus.reinke at talk21.com> wrote:
You believe you have worked out the core parts that caused you to postpone lexical modules,
We're still on the hook to finish a "wiki-complete" design of the core by the May meeting. I'm a busy guy.
On Apr 26, 2013, at 7:27 AM, Kevin Smith <zenparsing at gmail.com> wrote:
What you propose, with "logical names", is a global namespace of short human-readable names with no conflict resolution authority. How do you see that working? From a namespace perspective, how is that any different than hanging identifiers off of the global object, as we do today? I'm not understanding how this strategy will facilitate namespace coordination. I can only see it leading to namespace confusion.
Well first, it's much cleaner than the global object, because it does not involve the mess of prototype chains and sharing space with a DOM object's properties and methods.
There is, of course, a need for people to agree on common naming for shared modules. If they want to use conventions to avoid name collisions, there's in fact nothing preventing them from doing something like Java's reverse-DNS:
import spawn from "org/calculist/taskjs";
import Promise from "io/tilde/RSVP";
And note that Java also does not mandate reverse-DNS, it's just a convention. But in fact, that convention is really annoying and people hate it. Node uses much simpler global names that are reserved via NPM. This does lead to collisions and some people don't like that; an alternative system could use usernames. These are all viable alternatives, and what will really be needed will be package management systems like NPM for the web. What we are creating here is the basic semantics that provides a way for people to refer to shared modules. People can and should build package management systems, including tools, servers, and web sites, on top of this.
On Fri, Apr 26, 2013 at 7:17 PM, David Herman <dherman at mozilla.com> wrote:
On Apr 26, 2013, at 7:27 AM, Kevin Smith <zenparsing at gmail.com> wrote:
I'm not understanding how this strategy will facilitate namespace coordination. I can only see it leading to namespace confusion.
There is, of course, a need for people to agree on common naming for shared modules.
One additional point on this topic. Even on the web, where there isn't something like NPM as an arbiter for names, the JS community has managed to use shared resources like the global object and the AMD module name space effectively. And as Dave points out, building tools like NPM for web JavaScript will improve upon this further. We shouldn't bake in a harder-to-use system up front when JS is doing well as it is.
Claus Reinke wrote:
but do you think users are going to rewrite their code bases twice just because modules are going to be delivered in two stages?
What are you talking about?
People are not going to rewrite more than once. Current NPM/AMD modules do not nest, so there's no basis for asserting they'll be rewritten twice, first to ES6-as-proposed modules, then to add lexical naming and nesting.
And note that Java also does not mandate reverse-DNS, it's just a convention. But in fact, that convention is really annoying and people hate it. Node uses much simpler global names that are reserved via NPM. This does lead to collisions and some people don't like that; an alternative system could use usernames. These are all viable alternatives, and what will really be needed will be package management systems like NPM for the web. What we are creating here is the basic semantics that provides a way for people to refer to shared modules. People can and should build package management systems, including tools, servers, and web sites, on top of this.
Let see how we might build package systems on top of this. Let's say that I want to create a package management system named "browserpm". Now, since I don't want any of my names conflicting with some other PM's names, the logical name for modules within my registry will need to include the registry name itself:
import something from "browserpm/taskjs";
This looks almost like a real URL. Why not just use a URL instead?
import something from "//browserpm.org/taskjs";
I could even serve the source code directly from the "browserpm.org" site. If we want to load scripts from our local server, we would just override the module URL resolution algorithm:
"//browserpm.org/taskjs" => "/modules/browserpm.org/taskjs";
It seems to me that, long term, URLs provide all of the advantages of "logical names", without the downside of sacrificing one of the core principles of the web: namely, that resources are represented by URLs.
One additional point on this topic. Even on the web, where there isn't something like NPM as an arbiter for names, the JS community has managed to use shared resources like the global object and the AMD module name space effectively.
Facility of scale. If done correctly, we ought to hope that the global module namespace will be far larger than anything AMD or the global object has had to deal with. We can't assume that solutions which are effective at small scales will be effective at larger scales.
On Sat, Apr 27, 2013 at 12:22 AM, Kevin Smith <zenparsing at gmail.com> wrote:
And note that Java also does not mandate reverse-DNS, it's just a convention. But in fact, that convention is really annoying and people hate it. Node uses much simpler global names that are reserved via NPM. This does lead to collisions and some people don't like that; an alternative system could use usernames. These are all viable alternatives, and what will really be needed will be package management systems like NPM for the web. What we are creating here is the basic semantics that provides a way for people to refer to shared modules. People can and should build package management systems, including tools, servers, and web sites, on top of this.
Let see how we might build package systems on top of this. Let's say that I want to create a package management system named "browserpm". Now, since I don't want any of my names conflicting with some other PM's names, the logical name for modules within my registry will need to include the registry name itself:
import something from "browserpm/taskjs";
Why would you need to worry about this, though? I've never seen a language (or other software infrastructure) where people built systems using multiple package managers at once. For example, Macs have an unfortunate multiplicity of package systems, but you don't have a brew package depend on a macports package. The point of a package system is to fetch and setup the code, not to sit there in the names of your modules. This is made especially clear if you think about the source of a library. Is every library going to be rewritten for every package manager?
Certainly, existing JS package management systems don't work this way.
I've in fact worked with a system that was a lot like this, in that the package manager was part of the import statement. We didn't like it, and we're replacing it with a more NPM-like system.
This looks almost like a real URL. Why not just use a URL instead?
import something from "//browserpm.org/taskjs";I could even serve the source code directly from the "browserpm.org" site. If we want to load scripts from our local server, we would just override the module URL resolution algorithm:
"//browserpm.org/taskjs" => "/modules/browserpm.org/taskjs";It seems to me that, long term, URLs provide all of the advantages of "logical names", without the downside of sacrificing one of the core principles of the web: namely, that resources are represented by URLs.
The URLs you're proposing here just are logical names, and they aren't in most cases being dereferenced to produce resources, which is the core point of URLs on the web. They're just inconvenient logical names.
users are going to rewrite their code bases twice just because modules are going to be delivered in two stages?
What are you talking about?
People are not going to rewrite more than once. Current NPM/AMD modules do not nest, so there's no basis for asserting they'll be rewritten twice, first to ES6-as-proposed modules, then to add lexical naming and nesting.
Talking for myself, I've been using node modules, AMD modules, my own module loader, and have even tried, on occasion, to make my code loadable in two module systems (though I've shied away from the full complexity of UMD). I'm tired of that needless complexity - I want to build on modules, not fight with them (and I don't want tool builders having to guess what kind of module system a given code base might be using and what its configuration rules might be).
I have high hopes for getting to use ES6 modules early, via transpilers, but that cannot happen until that spec settles down - we have users of implementations that are waiting for that spec to tell them what to converge on.
As soon as the dust settles, I'll try to stop using legacy modules directly, switching to ES6 modules, transpiled to whatever (until the engines catch up).
But what I really want are lexical modules as the base case. The use cases that led to the new design are important, so a design not covering them would be incomplete, but if ES6 modules are not lexical, I'll be rewriting my code again once ES7 true modules come out. That is twice for me, and I doubt there is anything untypical about me in this situation.
I understand that David is snowed under (he has an unfortunate habit of taking on too much interesting work?-) but given the importance of this particular feature, perhaps more of tc39 could give a helping hand? The earlier and the more complete that spec is, the earlier there will be user feedback, and the greater the chance that ES6, or at least ES6.1, will have a module system that works in practice.
Claus
Claus Reinke wrote:
But what I really want are lexical modules as the base case.
But you don't have them now, with AMD, etc. So you'll have to wait no matter what (ES6, ES7). There's no "rewrite twice" if you assume AMD non-lexically-nested modules. Only by up'ing the ante independent of what you do today, do you get that "twice" -- and that is not a problem of ES6 per se.
So yes, you want lexical modules. There ought to be a strawman for them soon. At that point, prototyping, especially by trans-compiler writers, will be fair game.
The URLs you're proposing here just are logical names, and they aren't in most cases being dereferenced to produce resources, which is the core point of URLs on the web. They're just inconvenient logical names.
No. In my hypothetical scenario they are simply URLs which when dereferenced produce the required source code. They are "cannonical", in a sense, and can be remapped to other URLs, but they are URLs, nonetheless.
You have not demonstrated that a URL-based semantics would not work. Only that "Sam doesn't like it". But I thought that was one of the original design goals of the module loaders API: to allow users to apply whatever arbitrary URL resolution semantics they like.
More to follow...
I understand this design now. At scale, it depends upon an implicit, centralized naming authority to manage naming conflicts. The namespacing scheme of this authority will necessarily be flat because it will be seeded with names like "jquery" and "ember".
Who's authority will this be? Google's? Mozilla's? Apple's? Twitter's? Node's? Who will be responsible for maintaining it? What will the conflict-resolution strategy be? Will names be immortal? Will there be any standard conventions? Will it support versioning?
All of these questions will be left unspecified (because ES6 will surely not specify them), and as with all unspecified needs, a path-dependent and quite possibly sub-optimal solution will emerge. Javascript, to some degree, will be bound to this autonomous naming authority, like it or not.
I think some accounting would be helpful here.
As far as I can tell, the proposed resolution semantics would take about 10 lines of code to write using a module loader API. Let's be generous and say that it really comes out to 20 lines.
By baking in these semantics as the default, what do we get?
Assets
- We save the user, at most, from having to include 20 lines of code.
Liabilities
- We are drawn inescapably toward an unspecified central naming authority whose policies we cannot foresee or control.
- We break a central tenet of the web (1): that external resources are represented by URLs.
- We break a central tenet of the web (2): that naming authority is decentralized using DNS.
- As with all things on the web, we will be stuck with these semantics for a very long time.
This is a terrible deal. It is one-sided, risky, and (worst of all) it flippantly dis the conceptual integrity of the host platform.
On the bright side, we have an excellent design that we can return to: Sam and Dave's pre-November lexical modules.
On 26 April 2013 00:27, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
First, I appreciate you setting your thoughts down in detail. I think this will help us move forward in the discussion.
You write in a later message:
Having said that, interoperability with existing module systems was not the main motivation for the change in the proposal, as far as I can tell. It rather was simplicity (which is debatable, as I hope I've made clear), and convenient support for common configuration and concatenation use cases.
I don't think this is right, and I think this is the heart of the issue. Let me back up a bit to explain this.
Thank you for your explanation. I apologize if I have misrepresented your intention with the above. I seem to remember that it, roughly, is the motivation one of you explained to me at some point, but my memory may be unreliable.
Let me try to group my rebuttal to the technical content of your reply somewhat.
- The Package Manager Assumption
Frankly, I still didn't really understand from your reply how the design hangs together until you clarified in a later answer to Kevin that you are generally assuming the use of some package manager. This is a fairly significant assumption, that I wished I had heard being made explicit in any of the previous discussions. AFAICT, it is key, and without it, your proposal cannot fly. Not in a world with tens of thousands of module releases, anyway.
As for whether that is a good assumption to make, I have my serious doubts. As Kevin points out, it creates a lot of open questions and potential liabilities. In particular, I highly question whether it is wise to make that the exclusive mechanism for importing modules.
In other words, package managers are great, and the module system should definitely support them. But it should not, de facto, require them -- especially not on the web. Neither should the language prescribe how such a manager chooses to address packages, which, again, the current proposal does de facto.
- Naming
More specifically, everything should still play nice with standard web mechanisms. For me, the web platform implies that I should be able to address modules as remote resources (it's a choice not to use that mechanism). That requires that the system allows proper URLs as module identifiers. And at that point, you absolutely want your logical names to integrate nicely into the URL name space semantics, which currently, they do not do at all. (The proposal calls module ids "URLs", and syntactically, they form a sublanguage. But semantically, this sublanguage is divorced and interpreted in a completely incompatible way.)
That brings me to your repeated assertion that URLs are not appropriate for logical names, because they are, well, logical, not physical. Of course, I should have said URI, and as usual, that's what I really meant. :) URIs generally are logical names for resources. Arguably, the inherent abstraction from logical to physical (or virtual) is one main reason for their existence. So, from my perspective, URIs are exactly the appropriate tool.
That does not mean that logical names have to become unreadable or awful to type. Just that they are slightly more explicit. A schema prefix -- say, jsp: for a random strawman -- is good enough and should not offend anybody in terms of verbosity. (If it did, then I'd have little hope for the evolution of the web platform. :) )
- Module declarations
There are a few issues you haven't touched at all in your reply. One is (id-named) module declarations. What's their role in all this?
You say that lexical declarations are not precluded by your proposal. While technically true, much of my point was that id-named module declarations cannot be the only form of declaration without creating serious problems. They shouldn't even be the primary form. Internal naming, and therefor lexical declarations, are necessary to make the design complete. And once you have them, id-named declarations become fairly moot, except that they create extra confusion.
As far as I can see, they are not relevant for anything but concatenation. And given the assumption of a package management system, concatenation becomes even less relevant a concern -- you'll use a tool for that anyway (as, e.g., AMD's optimizer), and that works perfectly well (presumably, better) on top of lexical declarations. (And a tool handling the simple, manager-less cases for which a naive 'cat' would currently be enough is really straightforward.)
From my POV, the discussion about package managers also reveals
another conflation in your design: namely, between modules (as individual, language-level objects), and packages (a mechanism for deploying one, or a collection of, modules). A logical naming mechanism like the one you envision is for addressing packages, not modules. That makes it all the more dubious to bake in external names, which are really package names, with module definitions.
- Summary
So I stand by my proposal. We should:
- add lexical module declarations to provide internal naming;
- use proper URIs for external naming (including logical names);
- get rid of id-named module declarations, which don't serve much purpose;
- prototype a simple bundling tool to replace naive 'cat';
- prototype a simple package manager, to verify that it actually integrates properly.
I don't think we can ship the design without taking these steps. It would neither be a complete nor a future-proof system.
- Some specific replies
Module names play a role in three processes, in general:
- As a way to identify local components.
- As a way to find the physical resource that is the source code (or object code) of the module.
- As a way for two separately developed components to coordinate about which module they mean.
In the current design, the internal names (eg, "jquery") serve role 1, and URLs (as generated by the loader hooks) serve role 2. The coordination role is played by internal names in a shared registry.
In the current design, an import from a name like "jquery", by default, is supposed to fall back to a file/URL, just like in AMD and friends, doesn't it? By all practical means that makes it a direct reference to a resource. Given that, I don't see any conceivable way in which you can argue that "jquery" is an internal name in your system. From my perspective, there are no actual internal names in the system, which is part of the problem I'm trying to address.
Re URL, see above. Once you consider general URIs, they are also the appropriate tool for (3).
To pick another example (at random :), take SML with CM [1]. Here, lexically-bound module name serve role 1, and CM serves role 2. Coordination is managed by the ML toplevel, and files are typically open in the sense that they refer to modules that will ultimately be bound at the top level during the compilation process.
Interesting you'd mention CM, since that actually uses fully scoped, lexical name spacing, with no global name space at all. And it strictly separates module names from file names. ;)
Having said that, I would not recommend CM for comparison in the context of this discussion, since it only cares about batch compilation of a set of statically known source files. If you are inclined to compare to some ML equivalent, then I can surely suggest one which happens to have a dynamic module system almost exactly like the one we are discussing here (but substantially more powerful). You know which one. ;)
Or you could settle on a local URL to use in the same way, again as basically a registry key.
// in ember.js module jquery = "scripts/jquery.js"; import jquery as $;
Of course, that would simply be
import $ from "scripts/jquery";
(and similarly for your other examples).
And yes, as I said above, something close to this is the only web-compatible solution. More specifically, I assume you will have to work with some package manager if you want logical names, and the packages installed by this manager should be accessed in a manner that adheres to standard web practice. That is, using a URI that properly denotes either an absolute path, or even nicer, a custom schema:
import $ from "jsp:jquery"
That makes clear who the authority for the name is, and it is something entirely different from
import $ from "jquery"
which refers to a relative path -- at least everywhere else on the web!
The key takeaway
The key takeaway here is this: in the current design, different modules from different developers can coordinate based on the internal names of their dependencies. In your lexical proposal, it's not possible to coordinate globally on internal names, because they're lexical. So instead developers would have to coordinate on external names. This is fundamentally flawed, because external names are about where to get bits, not which abstraction a name represents.
We seem to have some misunderstanding about the nature of internal names. As far as I'm concerned, in both approaches, coordination is going through external names. The difference, as far as I can tell, is that you seem to suggest abusing the URI mechanism for short logical names that violate URI semantics, whereas I am saying that conformant integration into the syntactic and semantic structure of URIs is vital.
Supporting this use case properly is what led us to realize that the earlier lexically-named proposal was flawed.
Note that none of this is about concatenation. We've made some particular design decisions where concatenation played a role, but it wasn't a part of the reason we moved away from lexical modules.
Concatenation is the only reason I can see for the non-lexical module declarations in the current proposal, and if I remember correctly, that was the main motivation you or Dave gave at the last meeting. Is that correct?
Some smaller points
- As an external naming mechanisms, it violates standard relative path/URL semantics.
This isn't done particularly to be compatible, and node or AMD could have required "/" in front of absolute paths. But since that's an important use case, we don't think it's a good idea to tax it with extra typing. This convention has become popular in JS for a reason.
- The shared name space between internal and external modules can lead to incoherent programs.
I've already pointed out above how these namespaces aren't shared, and in fact their separation is an important reason for the current design.
I'm not following you here. How are they not shared?
Also, the incoherent programs you refer to are ruled out by a suggestion (of yours!) that we adopted at the last meeting: declarative module forms that define a module that already exists are a static error.
The suggestion you mention rather deals with the next case (clashes between different internally defined modules). It does not generally help when embedded resources clash with externally defined resources.
- Likewise, a single global name space for all internally defined modules can lead to incoherent programs.
I would be more worried about this if (a) we didn't provide convenient ways to structure this name space and (b) it wasn't already an existing successful approach in real-world JS systems.
- "Local" references are globally overridable.
This is only true in the sense that executing first lets you define the meaning of particular modules. But this is a feature. This is just another way to describing configuration. It's not reasonable to think that we can decide for everyone what will need configuration and what won't.
Er, I'm not saying that we should decide it. I'm saying that the implementer of a module should be able to make that choice. In the current proposal, he doesn't have a choice at all, he is forced to make each and every module definition essentially "mutable" by the rest of the world.
So I'm arguing for more choice, not less.
- Internal module definition is coupled with external module registration, and modules cannot be renamed or re-registered.
These are not "external" names, but it is true that declaring a module registers it. It's possible to take it out of the table afterward, or give it a new name, or have it share multiple names, all by using the loader API in very easy ways. I don't think these will be common operations, and this seems like a reasonable tradeoff.
Again you seem to be using "internal" in a different sense than I did. A name that can be referenced from, or shared with, the outside of a script's scope is an external name by my book. That goes for all names in the system you proposed.
I think that defining a module without wanting to register it is the common case, at least in real code. Explicit registration should only be needed for more ad-hoc configuration scenarios, or generated by tools creating bundles. In all other cases, what you refer to with external names are actual external resources (including logical package names).
- Language-level naming semantics interferes with file system/URL naming semantics.
While it's sad that everyone doesn't use the same filesystem ;), this is inevitable unless we force everyone to write both internal and external names explicitly for every module; that's clearly more user-hostile than the alternative.
But you do not want to do that for every module! In fact, you rarely need to explicitly define a module with an external name at all, at least as I envision it. You only want to do that for a module that you want to "export" from your current script, so to speak. Such "export" (i.e., globally registering a module you defined textually) should be a very rare operation to write manually (as mentioned, you don't usually do the equivalent in AMD, for example).
- Bundling ("concatenation") generally would require embedding arbitrary string resources, not syntactic modules.
The reason to use concatenation is to avoid consuming excessive client resources -- in this setting of course you won't want to run translation on the client side. Translation hooks are important (a) in less perf-sensitive settings like development and (b) for isolating remotely-loaded code, neither of which require concatenation.
I don't believe it's as clear cut, and I can imagine a number of cool use cases for translation that do not necessarily fall into this simple pattern.
- Module "declarations" are not a declarative mechanism, but an operational one.
This comes back to my original point. Registration of module names is about coordination, and thus this is an import feature, not a problem.
I agree it's an important feature -- but one already provided by the loader API. I am not convinced that there is a need to also provide it in declarative disguise (without really being declarative) -- and certainly not to conflate it with the (actually declarative) notion of module definition.
On 26 April 2013 01:20, David Herman <dherman at mozilla.com> wrote:
But let's keep the question of having lexical private modules separate from this thread, which is about Andreas's suggestion to have lexical modules be the central way to define public modules.
No, that's not what I suggested -- but I fully admit that my actual suggestion (and motivation) may have been overly obfuscated by my longish post. :)
I do think lexical names are essential to provide internal naming, which is vital. Thus having lexical modules was the first half of my suggestion. "Public" modules however are externally visible, so are naturally named by external names. The second half of my suggestion was that we should use proper URIs for that. The "third half" was about properly separating these concerns.
See also my latest reply to Sam for some more details.
Also, in a two-level system of external and lexical names, could one not model the coordination level by a registry/configuration module?
No, it would be too hard to get this expressive enough to satisfy the web platform's polyfilling needs.
I'm not sure I get that, what has coordination to do with polyfilling?
On 27 April 2013 01:17, David Herman <dherman at mozilla.com> wrote:
On Apr 26, 2013, at 7:27 AM, Kevin Smith <zenparsing at gmail.com> wrote:
What you propose, with "logical names", is a global namespace of short human-readable names with no conflict resolution authority. How do you see that working? From a namespace perspective, how is that any different than hanging identifiers off of the global object, as we do today? I'm not understanding how this strategy will facilitate namespace coordination. I can only see it leading to namespace confusion.
Well first, it's much cleaner than the global object, because it does not involve the mess of prototype chains and sharing space with a DOM object's properties and methods.
That actually is a moot point, however, since we had long decided that module declarations would live in the lexical top-level scope, so not inherit any of the global object craziness. And given that, and the assumption of using a package manager that could do the set-up, I think Kevin's is a valid question. If all you allow are simple logical path names, then I see no substantial advantage of the current design over sharing modules via global identifiers.
That said, I don't see it as sufficient either. But I conjecture that if you have lexical modules + a loader + the ability to use the full URI name space for imports, then everything else can be programmed on top in whatever way anybody prefers. No need to bake anything more specific into the language.
While this is starting to make a lot of sense to me, especially the package-vs.-module concerns, I worry about trying to get it in ES6. Also, as someone with an ES5 background, I don't see the value of lexically-named modules, and so am happy to postpone them to ES7.
Taken together, I think the minimal solution would be to remove explicit module declaration entirely: that is, ES6 would just have import/export, without any module.
But this leaves the question of how to support the concatenative use case if we don't have the ability to declare multiple modules in the same file. This brings to mind two approaches:
-
Does SPDY alone solve the problem concatenation is meant to solve? I don't know very much about SPDY but it's reasonable to assume browsers that support ES6 modules will support SPDY, given current trends, and I know it makes some vague moves in this direction.
-
Can the loader API be used instead? E.g. instead of the concatenated file being
module "foo" { ... }
module "bar" { ... }
it could instead be
System.set("npm:foo", `...`);
System.set("npm:bar", `...`);
On 29 April 2013 16:24, Domenic Denicola <domenic at domenicdenicola.com> wrote:
While this is starting to make a lot of sense to me, especially the package-vs.-module concerns, I worry about trying to get it in ES6. Also, as someone with an ES5 background, I don't see the value of lexically-named modules, and so am happy to postpone them to ES7.
Taken together, I think the minimal solution would be to remove explicit module declaration entirely: that is, ES6 would just have
import/export, without anymodule.But this leaves the question of how to support the concatenative use case if we don't have the ability to declare multiple modules in the same file. This brings to mind two approaches:
Does SPDY alone solve the problem concatenation is meant to solve? I don't know very much about SPDY but it's reasonable to assume browsers that support ES6 modules will support SPDY, given current trends, and I know it makes some vague moves in this direction.
Can the loader API be used instead? E.g. instead of the concatenated file being
module "foo" { ... } module "bar" { ... }it could instead be
System.set("npm:foo", `...`); System.set("npm:bar", `...`);
I brought up this very suggestion as a "min-max" sort of compromise at the March meeting, but it did not get much love. And I agree that it isn't pretty, although it actually has some technical advantages, e.g. enabling parallel parsing.
As for a SPDY-like mechanism, I absolutely believe that something along these lines is the way to go, for two reasons:
-
My prediction is that in the mid-term future, concatenation will actually become harmful to performance when the sources are already in the browser cache, because it prevents a lot of parallelisation, caching and/or work deferring that would otherwise be possible in the VM.
-
The bundling and transmission issue is a problem far beyond JS files, and needs to be solved on a more general level. Large web applications like Gmail currently go to great length to optimize it manually, across file types, and I believe that a purely JS modules focused approach to bundling will not help them a iota, whatever we do.
[Responding to these two emails together]
On Mon, Apr 29, 2013 at 6:40 AM, Kevin Smith <zenparsing at gmail.com> wrote:
The URLs you're proposing here just are logical names, and they aren't in most cases being dereferenced to produce resources, which is the core point of URLs on the web. They're just inconvenient logical names.
No. In my hypothetical scenario they are simply URLs which when dereferenced produce the required source code. They are "canonical", in a sense, and can be remapped to other URLs, but they are URLs, nonetheless.
To make this concrete, what you suggest is that any code that depends on jQuery will need to write:
import $ from "http://code.jquery.com/jquery-1.9.1.js";
Because that's the canonical URL for the current version of jQuery. Then, anyone who wants a version of jQuery that is not that version will have to set up a URL rewriting scheme, probably as part of a loader. Of course, absolutely everyone will want to use a different version of the code than this; it's not even minified, let alone served on a CDN, or over HTTPS, or on a local site, or any of the other concerns people will have in production. Even worse, this bakes in a particular version. If I want to upgrade to 1.9.2 when it comes out, what do I do?
Further, you can't load a module that defines jQuery with a script tag -- how would that define the same module that the URL above specifies?
You have not demonstrated that a URL-based semantics would not work. Only that "Sam doesn't like it".
I've spent a lot of time in the discussion spelling out precisely the technical problems with the URL-only approach. I assume you can tell the difference between that and "Sam doesn't like it".
But I thought that was one of the original design goals of the module loaders API: to allow users to apply whatever arbitrary URL resolution semantics they like.
It is a goal of the module loaders API to allow users to configure, restrict, redirect, etc the URL retrieval behavior. This is important for caching, bulk loading, security restrictions, etc. It is not a goal to force every user to use the module loaders API just to host some libraries locally.
On Mon, Apr 29, 2013 at 6:44 AM, Kevin Smith <zenparsing at gmail.com> wrote:
I understand this design now.
This does not appear to be the case.
At scale, it depends upon an implicit, centralized naming authority to manage naming conflicts. The namespacing scheme of this authority will necessarily be flat because it will be seeded with names like "jquery" and "ember".
This is false. We do not assume any naming authority, and have never said that we did. Similarly, neither the AMD module namespace nor the JS global object have a "naming authority" to manage conflicts.
How did you get the impression that this was required?
In another domain, there's no "global authority" to manage the file system on, say, linux systems, although there are multiple package managers.
I assume that package managers for the browser will appear; in fact, some already exist. That doesn't mean there will be a central authority.
Who's authority will this be? Google's? Mozilla's? Apple's? Twitter's? Node's? Who will be responsible for maintaining it? What will the conflict-resolution strategy be? Will names be immortal? Will there be any standard conventions? Will it support versioning?
Again, no one will be required to use any package manager. Google already ships a tool that manages a namespace (the Closure compiler), but no one has to use it to use the global object.
All of these questions will be left unspecified (because ES6 will surely not specify them), and as with all unspecified needs, a path-dependent and quite possibly sub-optimal solution will emerge. Javascript, to some degree, will be bound to this autonomous naming authority, like it or not.
If JS develops package systems, then I'm assuming that they'll competitive, and that people will work hard to develop good ones. Certainly systems like NPM are excellent, and we should not assume that the JS community will do worse on the web. But JS will not be bound by it.
I think some accounting would be helpful here.
As far as I can tell, the proposed resolution semantics would take about 10 lines of code to write using a module loader API. Let's be generous and say that it really comes out to 20 lines.
What "resolution" semantics are you talking about? I can't think of anything in the semantics that answers to this description.
- We are drawn inescapably toward an unspecified central naming authority whose policies we cannot foresee or control.
An actual argument for this, rather than repeated assertion, would be useful.
- We break a central tenet of the web (1): that external resources are represented by URLs.
- We break a central tenet of the web (2): that naming authority is decentralized using DNS.
This is no more true of the module system than that the HTML id
attribute violates this principle.
On the bright side, we have an excellent design that we can return to: Sam and Dave's pre-November lexical modules.
As I explained in some detail in my response to Andreas, this design has a significant flaw, which we fixed, leading to the current design.
I was wondering how versioning was expected to work in this module system.
Let's take the example of a large institution with many separate development groups. These groups do not collaborate and do wish to depend on each other. They all produce components that are to be integrated into a large single page application.
Group 1, let's call them "FX" create a component that uses JQuery v1.8. Group 2, "Metals" create a component that uses JQuery 1.9. Group 3, "Equities" create one that uses JQuery 2.0.
Now in each teams code base JQuery is imported like so:
import $ from "jquery";
That's great as long as they are developing separately from each other. What I'm wondering is how their components are meant to be integrated into a single app.
With nested modules maybe you could bundle them like so
module Metals { module jquery { //jquery 1.9. } }
module FX { module jquery { //jquery 1.8. } }
module Equities { module jquery { //jquery 2.0. } }
Everyone has their own jquery local reference and everything is OK.
Without nested modules though how should this work? The first jquery module grabs that identifier and the rest end up throwing an error from what I can tell.
I suppose the bundler tool is meant to be clever enough to rewrite all the module identifier when creating the single page app deployment artifact?
So we end up with
module "jquery:1.8.0" {}
module "jquery:1.9.0" {}
module "jquery:2.0.0" {}
This also means that all the code from each team needs to be rewritten by the bundling tool to
import $ from "jquery:1.8.0"
The other possibility of having each import have a hard coded version in it is frankly a non-flier. What I mean is having each import statement in the form "identifier:version" as convention. That might work if you have a few packages that you are working with but if you are writing class based code were every class is a module and you have hundreds of classes you are going to have some really ugly issues with doing a global search and replace of versions to newer version as you might end upgrading some classes/modules that you never meant to.
e.g.
import MyLittleFxTicket from "class:1.0.0"; import MyUtility from "otherclass:1.0.0";
You may want to update all uses of the classes in the package that contains "class" as one of it's classes but not "otherclass". So hard coding versions into the identifiers is just a no go.
Has any thought been put into this version issue? Large companies would be likely to find this a pain point.
Also what is the actual issue with reverse domain names as a form of identifier, it seems a perfectly acceptable one. Some comments were made about how Java uses it and it's not acceptable/good enough but there was no evidence provided to back that claim up. Personally I've never had issues with it.
For organizations it's excellent
com.megabank.metals / com.megabank.fx
is a clean, simple way to avoid clashes. What's wrong with
import $ from "org.jquery";
?
B.
On Mon, Apr 29, 2013 at 4:50 PM, Brian Di Palma <offler at gmail.com> wrote:
I was wondering how versioning was expected to work in this module system. [...] Now in each teams code base JQuery is imported like so:
import $ from "jquery";
That's great as long as they are developing separately from each other. What I'm wondering is how their components are meant to be integrated into a single app.
The simplest thing would be to make a jquery module in your component:
module "megabank/metals/jquery" {
import $ from "jquery-1.9";
export $;
}
// other modules under megabank/metals/ would write:
import $ from "./jquery";
That's the way that's built in the system, and that's what I would probably do.
Loaders are customizable, so one alternative is to add versioning to the system loader: gist.github.com/jorendorff/5489886
With nested modules maybe you could bundle them like so
module Metals { module jquery { //jquery 1.9. } }
Do you imagine the whole Metals module being in a single file, including jquery? Or would lexical modules be "open", like C++ namespaces and Ruby classes? Or would they be put together some other way?
Having to paste jquery code into your project seems unfortunate. You'd rather use the files unchanged, right? Also, if jquery imports something, and the name happens to collide with something you've declared in Metals, wouldn't jquery break? (Well, OK, the real jQuery doesn't import anything and is extremely careful about which global names it uses. But one goal of a module system is not to have to be quite so paranoid.)
Closing thought: All use cases are welcome! That said, using multiple versions of a library in a project is icky enough that an occasional project-wide search-and-replace at the margin isn't much worse.
I'm probably going to display my ignorance of the modules proposal here...
Good suggestions, thank you, they seem to have highlighted some features in the module system that were new to me. It does look like the wiki is out of date or not showing the full range of functionality provided by modules.
For example the first suggestion you make seems to suggest module imports are relative to the module you're currently executing in. I had presumed all imports where from a "root" location.
In other words I thought any import using a specific identifier ( "./jquery" ) would always return the same value regardless of where the import statement is located. My reading of your first suggestion leads me to believe I was wrong.
If you have
module "com/megabank/metals" { import $ from "./jquery"; }
The import of "./jquery" is actually relative to "com/megabank/metals" ? It's in fact importing "com/megabank/metals/jquery" ?
Therefore if I had a large concatenated JS bundle with
module "com/megabank/metals/jquery" { //code... export $; }
in it then the "./jquery" import in "com/megabank/metals" would pull the module above.
If there was no such module a request would be fired off by the browser to the "%%PAGE_ROOT%%/com/megabank/metals/jquery.js" URL ?
Funnily enough I think I prefer the second suggestion. The reason being that I think I'd rather not have location embedded in my module identifier. That might be fine for websites but for large code bases I feel it would make moving resources around more painful/brittle.
The second version would allow you to have a cleaner, logical identifier that is the same in all classes that use jquery. So imagine I have a few classes that use jquery in my metals components set.
module "com/megabank/metals/orderticket" { import $ from "jquery";
export class OrderTicketPresentationModel {
//code in here uses jquery.
}
}
module "com/megabank/metals/orderticket/settlement" { import $ from "jquery";
export class SettlementPresentationNode {
//code in here uses jquery.
}
}
module "com/megabank/metals/spotticket" { import $ from "jquery";
export class SpotTicketPresentationModel {
//code in here uses jquery.
}
}
Each of these classes would be in their own files and concatenated by a bundler when loading up the application. Could you imagine packing all those classes with relative import statements to jquery? Moving modules around becomes a total pain, surely I have this very wrong? Is the expect future code base really meant to be litered with "../../jquery" and "../jquery" references everywhere?
I've not given a very good overview of our code structure or development environment but I think that's probably necessary to understand where I'm coming from. We sell a framework that allows banks to build their own trading applications. As we are targeting a large number of different banks with different needs our framework has to be very modular and easy to use as the banks themselves do a lot of their own development. To achive this we built a server side framework that uses conventions to provide quick and easy development. This short video shows how we configure a trade ticket ( vimeo.com/49064357 ) but it also gives a glimpse of our conventions.
Actually the video shows how you shouldn't develop as he's working in a full application and we recommend working in what we call a workbench ( isolated from the app with fake services ). Each component can have a "src", "resources" and "themes" directories and inside the "resources" you can have "xml", "html" and "i18n".
We bundle all required resources on the fly when developing, we have a single URL request for each resource. One "bundle.js" request basically bundles all the JS you are using - and only the JS you are using, no more. We do that by reading some seed files and the files they pull in as our JS is all namespaced according to the folder/filename like Java. The code "new novox.package.MyClass();" would make us bundle and analyze that code to pull its depedencies. You add a new class creation to your code, save, press F5 and voila you have the code you referenced when before you didn't.
It's surprisingly quick to do even for large code bases and for production we create actual bundle files that are the same as the ones in development. The difference between dev and prod is quite minor.
Anyway we develop like you would in Java, with lots of small classes and then the server side pulls them all together for us when we need the code in the browser.
What I'd like to know is, are modules open?
Can I have package like declarations in all my files exporting the one class in the file and when I concatanate the code together will it work?
From what I can tell I can't do that.
I will instead have to have a module per class I suppose?
In my GridView.js file I can't do.
module "novox/grid" { export class GridView { } }
and in the same directory in my GridDecorator.js file I can't then do.
module "novox/grid" { export class GridDecorator { } }
That will throw an error won't it? I've declared the same module identifier twice and that will trigger an error.
I'd be forced to write
module "novox/grid/GridDecorator" { export class GridDecorator { } }
So my module identifier are namespaced by including the leaf node ( the class I'm exporting ).
import { GridView } from "novox/grid/GridView";
Pity to have that redundancy but it seems we have no choice.
Also must I have the module declaration in each class/file. I think not, correct?
module "novox/grid/GridDecorator" { //I think this line is not required? export class GridDecorator { } }
When I concatanate I will need to insert that line though?
Yes, I wasn't planning to paste any jQuery code into any file I was thinking more about how our bundler would output the code.
I know that multiple versions of libraries is an awful requirement but that's what some of the tier 1 banks have been asking for. They have large, seperate teams and they don't coordinate in any way...
" Also, if jquery imports something, and the name happens to collide with something you've declared in Metals, wouldn't jquery break? "
OK so what I think you mean is if one of our classes imports something and the names collide... That's not an issue for us, we enforce namespacing of our resources and so you can't have two resources with the same identifier. i.e. we blow up if you have two components with the same src directory structure or IDs in the XML/HTML resources.
Or do you mean what happens if we use a 3rd-party library and they import something like
import {MyClass} from "novox/grid/GridView";
Highly unlikely as we stick to the reverse domain, Java-like namespace approach. So novox would be either our company name or one of our clients.
Not many OSS libraries would call their packages after banks I'd say.
Not sure you could fix that beyond changing the actual source code of the library I'd guess?
B.
On Apr 29, 2013, at 6:34 AM, Andreas Rossberg <rossberg at google.com> wrote:
...you are generally assuming the use of some package manager. This is a fairly significant assumption, that I wished I had heard being made explicit in any of the previous discussions. AFAICT, it is key, and without it, your proposal cannot fly. Not in a world with tens of thousands of module releases, anyway.
Not only is this wrong -- it'll be just fine to use ES6 without package managers -- but your proposal does not make any material difference in the relative dependence on package managers.
In other words, package managers are great, and the module system should definitely support them. But it should not, de facto, require them -- especially not on the web.
I believe package managers will become a common part of the JS development experience. This is true for almost every modern programming environment, from Ruby to Scala to Node. I also think multiple package managers will exist, and that's fine too. There's nothing centralized about this that makes it inappropriate for the web. In fact, it's the nature of decentralization to leave the space open.
Finally, plenty of people will continue to use the system without centralization. For many packages, particularly popular ones, people will choose globally unique names like "backbone" and no reasonable software package will step on their toes.
As an analogy, consider the Unix file system. You can build software and deploy it without using a package manager at all, and everyone knows not to take the name "vi" (except actual vi implementations). But almost everyone uses a package manager in practice, there are multiple package managers, and they don't require any new semantics from the file system.
More specifically, everything should still play nice with standard web mechanisms. For me, the web platform implies that I should be able to address modules as remote resources (it's a choice not to use that mechanism). That requires that the system allows proper URLs as module identifiers.
Fully agreed, and if it wasn't clear, the current system does completely support this. You can use a URL as a module name. In the default browser loader, you can choose to import a module from an explicit URL:
import { $ } from "http://my.cdn.com/jquery/1.9";
and it will use that URL both as the logical name and as the place to go and fetch the bits. If you don't use an absolute URL, it treats it as a path, which defaults to loading as a source file from a relative URL derived from the module name.
And at that point, you absolutely want your logical names to integrate nicely into the URL name space semantics, which currently, they do not do at all. (The proposal calls module ids "URLs", and syntactically, they form a sublanguage. But semantically, this sublanguage is divorced and interpreted in a completely incompatible way.)
Maybe there's some confusion caused by misleading or out-of-date wording on the wiki here. I apologize if that's what's happened -- I'll have the wiki updated and clarified before the May meeting.
The sublanguages are fully compatible. In fact, note that they're only two sublanguages for the browser loader. In the browser loader, if you use an http:// URL, it will fetch the module from the web. But the core semantics simply treats module names as uninterpreted strings.
That brings me to your repeated assertion that URLs are not appropriate for logical names, because they are, well, logical, not physical.
No, that's not the argument. The argument is that URI's are not appropriate (I wasn't using the right terminology either!), because they don't work well in practice.
Of course, I should have said URI, and as usual, that's what I really meant. :) URIs generally are logical names for resources. Arguably, the inherent abstraction from logical to physical (or virtual) is one main reason for their existence. So, from my perspective, URIs are exactly the appropriate tool.
This idea is attractive enough that it comes up a lot, but it just doesn't ever seem to work out in practice.
History is not on your side here. The W3C has moved away from using URI's as unique names or namespaces:
http://dev.w3.org/2006/webapi/DOM-Level-3-Events/html/DOM3-Events.html#changes-drafts
URI-based DTD declarations and XML namespaces failed and have been replaced with simple identifiers. The modern DTD for HTML5 is <!doctype html>. The modern way to embed data in HTML tags is with data-<id> attributes where the <id> is any user-chosen identifier:
http://www.w3.org/html/wg/drafts/html/master/dom.html#embedding-custom-non-visible-data-with-the-data-*-attributes
You've been arguing we should use time-tested, well-understood, and successful mechanisms. The problem is, URI's as logical names, while time-tested and well-understood, are a failure.
That does not mean that logical names have to become unreadable or awful to type. Just that they are slightly more explicit. A schema prefix -- say, jsp: for a random strawman -- is good enough and should not offend anybody in terms of verbosity. (If it did, then I'd have little hope for the evolution of the web platform. :) )
First, let's be clear: this is literally an argument over the surface syntax of public module names, nothing more. You're arguing for the universal addition of a few characters of boilerplate to all registered module names. That's just bad ergonomics for no gain. Not only that, it hurts interoperability with existing JavaScript module systems.
There's a reason why the existing systems favor lightweight name systems. They're nicely designed to make the common case syntactically lighter-weight. And they absolutely can co-exist with naming modules directly with URL's.
On Mon, Apr 29, 2013 at 9:34 AM, Andreas Rossberg <rossberg at google.com> wrote:
On 26 April 2013 00:27, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
Dave has already responded about URIs, URLs, and external names, so I'll focus on lexical modules and module declarations below. This includes response to a couple other of your emails inline.
The key takeaway
The key takeaway here is this: in the current design, different modules from different developers can coordinate based on the internal names of their dependencies. In your lexical proposal, it's not possible to coordinate globally on internal names, because they're lexical. So instead developers would have to coordinate on external names. This is fundamentally flawed, because external names are about where to get bits, not which abstraction a name represents.
We seem to have some misunderstanding about the nature of internal names. As far as I'm concerned, in both approaches, coordination is going through external names. The difference, as far as I can tell, is that you seem to suggest abusing the URI mechanism for short logical names that violate URI semantics, whereas I am saying that conformant integration into the syntactic and semantic structure of URIs is vital.
Ok, I think I was misunderstanding your use of terminology. The names mapping in the registry in the current system I was calling internal names, and you think of them as external names.
From now on, I'll stick with "logical name" for a name that goes in
the registry ("jquery/ui") and "URL" for something where the physical source is retrieved from the internet ("jquery.com/modules/jquery/ui.js").
Supporting this use case properly is what led us to realize that the earlier lexically-named proposal was flawed.
Note that none of this is about concatenation. We've made some particular design decisions where concatenation played a role, but it wasn't a part of the reason we moved away from lexical modules.
Concatenation is the only reason I can see for the non-lexical module declarations in the current proposal, and if I remember correctly, that was the main motivation you or Dave gave at the last meeting. Is that correct?
I brought up [removing module declarations] as a "min-max" sort of compromise at the March meeting, but it did not get much love. And I agree that it isn't pretty, although it actually has some technical advantages, e.g. enabling parallel parsing.
No, that isn't correct. We've been over this a few times at this point, but once more into the breach ...
There are several reasons why module declarations are important.
-
Right now, every JS construct can be written at the REPL, in a script tag, in a bunch of files in a Node package, or anywhere else. This is a very important property for the language, and one we should maintain. Without module declarations except as separate files, we would lose this uniformity.
-
Being able to define multiple modules concisely in a single file is simply more convenient than having to separate them into multiple files. Even clunky old Java allows this simple convenience!
-
Serving modules as script tags, and defining modules in inline script tags, is likely to be important for a wide variety of use cases.
-
Concatenation.
As for a SPDY-like mechanism, I absolutely believe that something along these lines is the way to go, for two reasons:
My prediction is that in the mid-term future, concatenation will actually become harmful to performance when the sources are already in the browser cache, because it prevents a lot of parallelisation, caching and/or work deferring that would otherwise be possible in the VM.
The bundling and transmission issue is a problem far beyond JS files, and needs to be solved on a more general level. Large web applications like Gmail currently go to great length to optimize it manually, across file types, and I believe that a purely JS modules focused approach to bundling will not help them a iota, whatever we do.
I realize that you don't seem to find concatenation an important use case, but I think this is wrong. For the medium-term, http requests are expensive, and concatenation is a reality. Browsers haven't made extra requests free, let alone cheaper, and it would be a big mistake to build a system that assumes this.
Large web apps use concatenation as part of their approach to performance today, despite its limitations.
[From another email]
I do think lexical names are essential to provide internal naming, which is vital. Thus having lexical modules was the first half of my suggestion. "Public" modules however are externally visible, so are naturally named by external names. The second half of my suggestion was that we should use proper URIs for that. The "third half" was about properly separating these concerns.
Using your terminology, the current design uses names like "jquery/ui" for internal naming. You're suggesting that we really need another level of internal naming, based on lexically-scoped identifiers.
This would be a coherent addition to the system, but is not a necessary one. Below, you point out some issues that would be potentially addressed by adding them. However, multiple JS module systems are in use today, and with the partial exception of the TypeScript namespaces feature (which is a significantly different design), none of them has lexically-named modules in the sense you advocate.
Some smaller points
Also, the incoherent programs you refer to are ruled out by a suggestion (of yours!) that we adopted at the last meeting: declarative module forms that define a module that already exists are a static error.
The suggestion you mention rather deals with the next case (clashes between different internally defined modules). It does not generally help when embedded resources clash with externally defined resources.
When the second resource loaded is a declarative form, then this addresses this case as well.
- Likewise, a single global name space for all internally defined modules can lead to incoherent programs.
I would be more worried about this if (a) we didn't provide convenient ways to structure this name space and (b) it wasn't already an existing successful approach in real-world JS systems.
- "Local" references are globally overridable.
This is only true in the sense that executing first lets you define the meaning of particular modules. But this is a feature. This is just another way to describing configuration. It's not reasonable to think that we can decide for everyone what will need configuration and what won't.
Er, I'm not saying that we should decide it. I'm saying that the implementer of a module should be able to make that choice. In the current proposal, he doesn't have a choice at all, he is forced to make each and every module definition essentially "mutable" by the rest of the world.
There are two possible scenarios here. If your module is loaded inside a loader that you don't trust, you have zero semantic guarantees. The translate and link hooks give the loader arbitrary power, lexical modules or no. The initial loader in the JS environment is mutable and thus if you are in a mutually-untrusting situation, you can't rely on anything. Nonetheless, this is the right default for the system we're designing, and it can be configured and new loaders built for other situations.
In contrast, if you can trust the loader, and the loader isn't exposed to arbitrary mutation, then you're in a much better position. After two modules are linked, the link never changes. Linking is a configurable process, but its results are immutable. This is in contrast to how JS programs that use the top level as a namespace work today, for example. Thus, two modules in a script tag or file that are linked together are just as bound as they would be in a system with lexical modules.
- Internal module definition is coupled with external module registration, and modules cannot be renamed or re-registered.
These are not "external" names, but it is true that declaring a module registers it. It's possible to take it out of the table afterward, or give it a new name, or have it share multiple names, all by using the loader API in very easy ways. I don't think these will be common operations, and this seems like a reasonable tradeoff.
I think that defining a module without wanting to register it is the common case, at least in real code. Explicit registration should only be needed for more ad-hoc configuration scenarios, or generated by tools creating bundles. In all other cases, what you refer to with external names are actual external resources (including logical package names).
Needing to explicitly register a module will certainly be common in a node-like system, where small modules are the norm. Which is the common case in general isn't a question we can answer today. But I don't think that needing to avoid registering will be common.
- Language-level naming semantics interferes with file system/URL naming semantics.
While it's sad that everyone doesn't use the same filesystem ;), this is inevitable unless we force everyone to write both internal and external names explicitly for every module; that's clearly more user-hostile than the alternative.
But you do not want to do that for every module! In fact, you rarely need to explicitly define a module with an external name at all, at least as I envision it. You only want to do that for a module that you want to "export" from your current script, so to speak. Such "export" (i.e., globally registering a module you defined textually) should be a very rare operation to write manually (as mentioned, you don't usually do the equivalent in AMD, for example).
Anonymous module defintions are often used in AMD for modules that are implicitly named by the file they're in. In ES6, that simply doesn't require a wrapper.
- Bundling ("concatenation") generally would require embedding arbitrary string resources, not syntactic modules.
The reason to use concatenation is to avoid consuming excessive client resources -- in this setting of course you won't want to run translation on the client side. Translation hooks are important (a) in less perf-sensitive settings like development and (b) for isolating remotely-loaded code, neither of which require concatenation.
I don't believe it's as clear cut, and I can imagine a number of cool use cases for translation that do not necessarily fall into this simple pattern.
Can you say more about these use cases?
- Module "declarations" are not a declarative mechanism, but an operational one.
This comes back to my original point. Registration of module names is about coordination, and thus this is an import feature, not a problem.
I agree it's an important feature -- but one already provided by the loader API. I am not convinced that there is a need to also provide it in declarative disguise (without really being declarative) -- and certainly not to conflate it with the (actually declarative) notion of module definition.
Using the loader API should not be required for declaring a simple module that other modules import. This makes life harder for compilers, tools, and most of all, developers.
Some brief, general observations:
-
Dave, your argument that URI's as a naming mechanism is a "failure" cherry-picks cases where URIs were obviously overkill. You have not shown that URIs are overkill in this situation. In order to do so, you would need to posit a centralized naming authority, or a dependency management system (e.g. package manager) with attendant dependency metadata. Or else you would need to show that for any reasonable selected subgraph of the global module (really package) graph, the chance of name collision is near-zero.
-
To repeat my last argument, the module URL resolution semantics proposed in this design (that strings such as "jquery" are resolved relative to some base URL, and that a ".js" extension is added), can be coded in about 10 lines, by my guess. Are we seriously to accept these sloppy semantics should be the default for the web, for all time? How is that anything but a non-starter, given that it conflicts with all other resolution semantics on the web?
-
It seems that every response to Andreas comes with a new requirement for modules that we have not yet seen on es-discuss. How can we possibly have a meaningful discussion of a solution when we don't even know what the goals are? Can someone provide a list of the requirements?
On Wed, May 1, 2013 at 7:18 AM, Kevin Smith <zenparsing at gmail.com> wrote:
- Dave, your argument that URI's as a naming mechanism is a "failure" cherry-picks cases where URIs were obviously overkill. You have not shown that URIs are overkill in this situation. In order to do so, you would need to posit a centralized naming authority, or a dependency management system (e.g. package manager) with attendant dependency metadata. Or else you would need to show that for any reasonable selected subgraph of the global module (really package) graph, the chance of name collision is near-zero.
Or, we can simply point to real-world examples of large-scale custom namespaces, where there is no central authority, the chance of collision is very low and can be largely predicted and worked around ahead of time, and in the case of collisions, the author can manually work around the issue without too much difficulty.
See: jQuery modules, python modules, and many others. Some of these have a central distribution authority which prevents name conflicts for people using it, but it's optional to use.
Central naming authorities are only necessary if you need complete machine-verifiable consistency without collisions. As long as humans are in the loop, they tend to do a pretty good job of avoiding collisions, and managing them when they do happen.
On Wed, May 1, 2013 at 8:28 AM, Tab Atkins Jr. <jackalmage at gmail.com> wrote:
Central naming authorities are only necessary if you need complete machine-verifiable consistency without collisions. As long as humans are in the loop, they tend to do a pretty good job of avoiding collisions, and managing them when they do happen.
I would go further: because humans are involved, requiring a central naming authority, like an URL, for module IDs are a worse choice. There are subcultures that ascribe slightly different meanings to identifiers, but still want to use code that mostly fits that identifier but is from another subculture.
The current approach to module IDs in ES modules allows for that fuzzy meaning very well, with resolution against any global locations occurring at dependency install/configure time, when the subculture and context is known. It would require more config, generate more friction and more typing with runtime resolution of URL IDs.
Examples from the AMD module world:
- Some projects want to use jQuery with some plugins already wired up to it. They can set up 'jquery' to be a module that imports the real jQuery and all the plugins they want, and then return that modified jQuery as the value for 'jquery'. Any third party code that asks for 'jquery' still gets a valid value for that dependency.
With ES modules in their current form, they could do this without needing any Module Loader configuration, and all the modules use a short 'jquery' module ID.
- A project developer want to use jQuery from the project's CDN. A third party module may need jQuery as a dependency, but the author of that third party module specified a specific version range that does not match the current project. However, the project developer knows it will work out fine.
The human that specified the version range in that third party module did not have enough context to adequately express the version range or the URL location. The best the library author can express is "I know it probably works with this version range of jQuery".
If all the modules just use 'jquery' for the ID, the project developer just needs one top level, app config to point 'jquery' to the project's CDN, and it all works out.
An URL ID approach, particularly when version ranges are in play, would mean much more configuration that is needed for the runtime code. All the IDs would require more typing, particularly if version ranges are to be expressed in the URLs.
Summary:
It is best if the suggestions on where to fetch a dependency from a globally unique location and what version range is applicable are done in separate metadata, like package.json, bower.json, component.json. But note that these are just suggestions, not requirements, and the suggestions may vary based on the project context. For example, browser-based vs. server-based, or mobile vs. desktop. Only the end consumer has enough knowledge to do the final resolution.
It would be awkward to try to encode all of the version and context choices in the module IDs used for import statements as some kind of URL. Even if it was attempted, it could not be complete on its own -- end context is only known by the consumer. So it would lead to large config blocks that need to be loaded by the runtime to resolve the URL IDs to the actual location.
With short names like 'jquery', there is a chance to just use convention based layout, which means no runtime config at all, and if there is config, a much smaller size to that config than what would be needed for URL-based IDs. Any resolution against global locations happens once, when the dependency is brought into the project, and not needed for every runtime execution. Plus less typing is needed for the module IDs in the actual JS source.
In addition, the shorter names have been used successfully in real world systems, examples mentioned already on this thread, so they have been proven to scale.
James
Thanks James, for your input. As an aside, I want to make perfectly clear that I don't think the AMD approach to module IDs is necessary a bad thing, or that full "http" URLs are necessarily any better for dependency naming.
I am arguing that AMD resolution semantics should not be baked in.
I am arguing that URLs should be the only default means of specifying external modules in the browser, with standard URL semantics.
I am arguing that we should not bake any package dependency resolution strategy into the core module system. Especially not something as "fuzzy" as module IDs.
On Wed, May 1, 2013 at 7:08 PM, Jason Orendorff <jason.orendorff at gmail.com>wrote:
On Wed, May 1, 2013 at 9:18 AM, Kevin Smith <zenparsing at gmail.com> wrote:
- Dave, your argument that URI's as a naming mechanism is a "failure" cherry-picks cases where URIs were obviously overkill.
What counterexamples should David have mentioned?
Well, it's Dave's job to come up with one, not mine ; P
Actually, I'm starting to think that the "URI vs. unstructured names" argument is a bit of a tangent. These are the kinds of arguments to have when designing a packaging system, not the core module system.
A clean separation between modules and packages will give us the freedom to experiment with different approaches to inter-package dependency resolution (IPDR, so I don't have to repeat it later). At the base level, we just want URLs. Loader hooks can then be used to give special semantics to URI subsets, or even to provide AMD-style URL overloading.*
We don't need to solve the IPDR problem with the core module system. Instead, we can provide the primitives and see what flourishes.
{ Kevin }
*Note that CommonJS always blurred the distinction between packages and modules.
On 1 May 2013 00:01, David Herman <dherman at mozilla.com> wrote:
On Apr 29, 2013, at 6:34 AM, Andreas Rossberg <rossberg at google.com> wrote:
...you are generally assuming the use of some package manager. This is a fairly significant assumption, that I wished I had heard being made explicit in any of the previous discussions. AFAICT, it is key, and without it, your proposal cannot fly. Not in a world with tens of thousands of module releases, anyway.
Not only is this wrong -- it'll be just fine to use ES6 without package managers -- but your proposal does not make any material difference in the relative dependence on package managers.
I don't see how logical names can possibly make sense without at least a rudimentary manager that maps them. Otherwise they are just physical names.
My point on the topic of external naming is that the language (1) should not prescribe any specific naming scheme; (2) should not willfully violate URI semantics; (3) should properly separate it from internal naming.
Package managers are great, and I agree that they will (and should) become part of the eco system. But leave design decisions to them, and don't limit their options a priori. I'm not sure why we even disagree on that.
More specifically, everything should still play nice with standard web mechanisms. For me, the web platform implies that I should be able to address modules as remote resources (it's a choice not to use that mechanism). That requires that the system allows proper URLs as module identifiers.
Fully agreed, and if it wasn't clear, the current system does completely support this. You can use a URL as a module name. In the default browser loader, you can choose to import a module from an explicit URL:
import { $ } from "http://my.cdn.com/jquery/1.9";and it will use that URL both as the logical name and as the place to go and fetch the bits.
OK, great! Can I also use such URLs for module definitions then?
If you don't use an absolute URL, it treats it as a path, which defaults to loading as a source file from a relative URL derived from the module name.
And here we get to the core of the problem. Namely, how do you interpret relative and absolute here? URLs have a very clear semantics: everything that isn't absolute is relative to the referrer. So far, everything I have seen in any of the recent module specs or presentations is a blunt violation of these semantics.
In particular, as I mentioned before, you cannot make "a" mean something different than "./a" without violating URLs [1,2]. Yet that is not only what you envision, it is what you de facto prescribe with your proposal. I think that's simply a total no-go.
[1] tools.ietf.org/html/rfc3986#section-4.2 [2] tools.ietf.org/html/rfc3986#section-5.2.2
That does not mean that logical names have to become unreadable or awful to type. Just that they are slightly more explicit. A schema prefix -- say, jsp: for a random strawman -- is good enough and should not offend anybody in terms of verbosity. (If it did, then I'd have little hope for the evolution of the web platform. :) )
First, let's be clear: this is literally an argument over the surface syntax of public module names, nothing more. You're arguing for the universal addition of a few characters of boilerplate to all registered module names. That's just bad ergonomics for no gain. Not only that, it hurts interoperability with existing JavaScript module systems.
I don't see how it hurts interop. It just means that you have to find a convenient and proper way to embed a logical name space into URLs. I proposed using a schema, but anything that is compatible with URL semantics is OK. Just taking AMD-style strings verbatim is not.
It is not something which you can justify with AMD/Node precedence either, since AFAICT these systems do not have the notational overloading between logical paths and URLs that you propose.
There's a reason why the existing systems favor lightweight name systems. They're nicely designed to make the common case syntactically lighter-weight. And they absolutely can co-exist with naming modules directly with URL's.
Sure they can coexist. But please not by bastardizing URL semantics.
On 1 May 2013 22:02, Kevin Smith <zenparsing at gmail.com> wrote:
Thanks James, for your input. As an aside, I want to make perfectly clear that I don't think the AMD approach to module IDs is necessary a bad thing, or that full "http" URLs are necessarily any better for dependency naming.
I am arguing that AMD resolution semantics should not be baked in.
I am arguing that URLs should be the only default means of specifying external modules in the browser, with standard URL semantics.
I am arguing that we should not bake any package dependency resolution strategy into the core module system. Especially not something as "fuzzy" as module IDs.
Exactly.
We don't need to solve the IPDR problem with the core module system. Instead, we can provide the primitives and see what flourishes.
And to bring this back around full-circle, once we let go of the requirement to solve the IPDR problem, then the need for lexical modules come back into focus. Not because users need them right away (although they may find them irresistible soon enough! 1), but because lexical modules provide us with the model though which we can understand external modules and indeed the entire module graph.
On 1 May 2013 01:15, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Mon, Apr 29, 2013 at 9:34 AM, Andreas Rossberg <rossberg at google.com> wrote:
I brought up [removing module declarations] as a "min-max" sort of compromise at the March meeting, but it did not get much love. And I agree that it isn't pretty, although it actually has some technical advantages, e.g. enabling parallel parsing.
No, that isn't correct.
What isn't correct?
There are several reasons why module declarations are important.
Right now, every JS construct can be written at the REPL, in a script tag, in a bunch of files in a Node package, or anywhere else. This is a very important property for the language, and one we should maintain. Without module declarations except as separate files, we would lose this uniformity.
Being able to define multiple modules concisely in a single file is simply more convenient than having to separate them into multiple files. Even clunky old Java allows this simple convenience!
Serving modules as script tags, and defining modules in inline script tags, is likely to be important for a wide variety of use cases.
Concatenation.
I generally sympathise with (1). However, modules arguably are a borderline case. They fall into the same category as multiple scripts -- a very module-ish "construct" that cannot accurately be expressed in pure JavaScript either.
As for the others, shrug. All valid points, and I agree that that it's suboptimal if you cannot do that. On the other hand, nothing will break, nothing essential would be missing. That's the nature of a min-max path.
But my broader point, of course, is that lexical modules address all these cases just fine.
I do think lexical names are essential to provide internal naming, which is vital. Thus having lexical modules was the first half of my suggestion. "Public" modules however are externally visible, so are naturally named by external names. The second half of my suggestion was that we should use proper URIs for that. The "third half" was about properly separating these concerns.
Using your terminology, the current design uses names like "jquery/ui" for internal naming. You're suggesting that we really need another level of internal naming, based on lexically-scoped identifiers.
Yes, because the current design necessitates (ab)using external names for internal naming -- with all the problems that implies.
This would be a coherent addition to the system, but is not a necessary one. Below, you point out some issues that would be potentially addressed by adding them. However, multiple JS module systems are in use today, and with the partial exception of the TypeScript namespaces feature (which is a significantly different design), none of them has lexically-named modules in the sense you advocate.
I do not think this observation is particularly relevant. As I said already, emulating a module system within JS necessarily has much more severe limitations than designing it as a language feature. In particular, I'm not sure you even could build a lexical module system under those constraints. There is no reason to apply those limitations to ES6 modules, though.
Also, the incoherent programs you refer to are ruled out by a suggestion (of yours!) that we adopted at the last meeting: declarative module forms that define a module that already exists are a static error.
The suggestion you mention rather deals with the next case (clashes between different internally defined modules). It does not generally help when embedded resources clash with externally defined resources.
When the second resource loaded is a declarative form, then this addresses this case as well.
Yes, but not the other way round.
- "Local" references are globally overridable.
This is only true in the sense that executing first lets you define the meaning of particular modules. But this is a feature. This is just another way to describing configuration. It's not reasonable to think that we can decide for everyone what will need configuration and what won't.
Er, I'm not saying that we should decide it. I'm saying that the implementer of a module should be able to make that choice. In the current proposal, he doesn't have a choice at all, he is forced to make each and every module definition essentially "mutable" by the rest of the world.
There are two possible scenarios here. If your module is loaded inside a loader that you don't trust, you have zero semantic guarantees. The translate and link hooks give the loader arbitrary power, lexical modules or no.
Sure.
The initial loader in the JS environment is mutable and thus if you are in a mutually-untrusting situation, you can't rely on anything.
Well, in practice it's not all-or-nothing. In this scenario, I worry about accident more than malice. Once modular JS applications grow in scale, and we get more interesting towers of libraries, I'm sure this will become a problem.
In contrast, if you can trust the loader, and the loader isn't exposed to arbitrary mutation, then you're in a much better position. After two modules are linked, the link never changes. Linking is a configurable process, but its results are immutable. This is in contrast to how JS programs that use the top level as a namespace work today, for example. Thus, two modules in a script tag or file that are linked together are just as bound as they would be in a system with lexical modules.
True. Except that in a larger application, loading will likely be delayed for many modules, so the difference becomes a bit more blurry.
Needing to explicitly register a module will certainly be common in a node-like system, where small modules are the norm. Which is the common case in general isn't a question we can answer today. But I don't think that needing to avoid registering will be common.
Pro JS code today certainly goes to great length to avoid polluting the global name space, by using anonymous functions. I think the situation is much more similar than you'd like it to be.
But ultimately, it's simply a question of the right default. Making everything externally visible, let alone overridable, is, plain and simple, the wrong default. I'm still puzzled why you'd argue otherwise, since for very good reasons, you made the opposite choice one level down, for the contents of a module. I fail to see the qualitative difference.
While it's sad that everyone doesn't use the same filesystem ;), this is inevitable unless we force everyone to write both internal and external names explicitly for every module; that's clearly more user-hostile than the alternative.
But you do not want to do that for every module! In fact, you rarely need to explicitly define a module with an external name at all, at least as I envision it. You only want to do that for a module that you want to "export" from your current script, so to speak. Such "export" (i.e., globally registering a module you defined textually) should be a very rare operation to write manually (as mentioned, you don't usually do the equivalent in AMD, for example).
Anonymous module defintions are often used in AMD for modules that are implicitly named by the file they're in. In ES6, that simply doesn't require a wrapper.
Yes, my point? Thing is, in AMD experience, named module definitions are neither common, nor recommended. You use separate files. Why expect something different for ES6?
The reason to use concatenation is to avoid consuming excessive client resources -- in this setting of course you won't want to run translation on the client side. Translation hooks are important (a) in less perf-sensitive settings like development and (b) for isolating remotely-loaded code, neither of which require concatenation.
I don't believe it's as clear cut, and I can imagine a number of cool use cases for translation that do not necessarily fall into this simple pattern.
Can you say more about these use cases?
For example, imagine statically generating (the definition of) a non-trivial JS data structure from a domain-specific description language and some dynamic information. Or things like FFTW. In a similar vein, you could implement a semi-dynamic macro system on top of JS.
On Wed, May 1, 2013 at 11:59 PM, Kevin Smith wrote:
On Wed, May 1, 2013 at 7:08 PM, Jason Orendorff wrote:
On Wed, May 1, 2013 at 9:18 AM, Kevin Smith wrote:
- Dave, your argument that URI's as a naming mechanism is a "failure" cherry-picks cases where URIs were obviously overkill.
What counterexamples should David have mentioned?
Well, it's Dave's job to come up with one, not mine ; P
Wait, we've had a misunderstanding somewhere. If you think David's argument cherry-picks cases, then you must have in mind some cases he didn't mention that contradict his point. That's what I'm asking for.
A clean separation between modules and packages will give us the freedom to experiment with different approaches to inter-package dependency resolution (IPDR, so I don't have to repeat it later). At the base level, we just want URLs.
Do we? It seems to me that "filenames as import-specifiers" is a design option that has been available to, and rejected by, all of the JS module systems and all recent programming languages, except PHP.* And the reason seems obvious enough: hard-coding a library's location into every import site is bad.
Loader hooks can then be used to give special semantics to URI subsets, or even to provide AMD-style URL overloading.
If we don't expect loaders to treat these names as URLs or respect URL resolution semantics, then they aren't URLs, and it's bogus to call them URLs.
-j
*In fairness to PHP, I'm told include "filename.php" is now
considered bad style. Autoloaders are preferred.
On Thu, May 2, 2013 at 9:47 AM, Andreas Rossberg <rossberg at google.com> wrote:
I don't see how logical names can possibly make sense without at least a rudimentary manager that maps them. Otherwise they are just physical names.
Yes. A rudimentary way of mapping logical names to URLs is built into the proposal: the default resolve behavior.
-
No effort: modules are loaded relative to the document base url, with ".js" appended. So
import "jquery"maps to the relative URL "jquery.js". -
One line of code: you can set the root URL from which all imports are resolved. If you set
System.baseURL = "https://example.com/scripts/"thenimport "jquery"maps to the URL "example.com/scripts/jquery.js". -
A few lines: you can use System.ondemand() to set the URL for each module you use. If you call
System.ondemand({"https://example.com/jquery-1.9.1.js": "jquery"})thenimport "jquery"maps to the URL you specified (imports for modules that aren't in the table will fall back on the loader's baseURL). -
Beyond that: write a resolve hook and locate modules however you want.
Wait, we've had a misunderstanding somewhere. If you think David's argument cherry-picks cases, then you must have in mind some cases he didn't mention that contradict his point. That's what I'm asking for.
I gotcha - that's fair. I concede the point though as not central to my position.
A clean separation between modules and packages will give us the freedom to experiment with different approaches to inter-package dependency resolution (IPDR, so I don't have to repeat it later). At the base level, we just want URLs.
Do we? It seems to me that "filenames as import-specifiers" is a design option that has been available to, and rejected by, all of the JS module systems and all recent programming languages, except PHP.* And the reason seems obvious enough: hard-coding a library's location into every import site is bad.
First, I assume that you're just referring to inter-package dependencies. Relative paths are fine for a library's internal modules.
Second, I'm well aware of the issues you point out with using URLs or file paths for inter-package dependencies. I think URLs can perhaps do some interesting things for us here which we haven't seen yet, but I'm not arguing against using a flat namespace.
But (and this is the key) such a flat namespace must be layered on top of URLs. It cannot overload URLs as the current design attempts to do. Why not? Because such an overloaded semantics would conflict with the resolution semantics as defined by the platform in HTML, CSS, etc. Truth, beauty conceptual integrity, and all that good stuff. : )
The Dart designers came to a similar conclusion. External files are specified by URL, with a custom schema designating the "package" root.
import 'package:mylib/mylib.dart';
www.dartlang.org/docs/dart-up-and-running/contents/ch02.html
Although note the lack of file-extension magic, and the fact that Dart does not even pretend these are "logical names".
Should javascript do something similar? Maybe. Maybe that's something for the platform to decide. But in any event, that's not a discussion that we can begin to have until after we have consensus on modules.
Loader hooks can then be used to give special semantics to URI subsets, or even to provide AMD-style URL overloading.
If we don't expect loaders to treat these names as URLs or respect URL resolution semantics, then they aren't URLs, and it's bogus to call them URLs.
By design, you can implement any kind of madness you want with loader hooks, but by default: URLs, baby ; )
If we don't expect loaders to treat these names as URLs or respect URL
resolution semantics, then they aren't URLs, and it's bogus to call them URLs.
I didn't really answer you here.
I think that, considering the fact that "module IDs" look like URLs, and effectively perform the same function as URLs (i.e. locating external resources), it is bogus to call them anything but bogus URLs. : )
No effort: modules are loaded relative to the document base url, with ".js" appended. So
import "jquery"maps to the relative URL"jquery.js".A few lines: you can use
System.ondemand()to set the URL for each module you use. If you callSystem.ondemand({"https://example.com/jquery-1.9.1.js": "jquery"})thenimport "jquery"maps to the URL you specified (imports for modules that aren't in the table will fall back on the loader's baseURL).
I think part of Andreas' concerns was that you now have a conflict
between import "jquery" referring to a relative (0.) or registered (2.)
thing, because all names just look URL-ish. Another part was that
both times, the import may look URL-ish but doesn't behave like one.
Using something like import "registered:jquery" for 2 would
remove the conflict, without changing the functionality. That would
still leave the implicit rewriting involved in 0 - perhaps one could
specify that every protocol-free name refers to a module (with
rewriting) and names with protocol prefixes refer to URLs?
On 4 May 2013 01:46, Jason Orendorff <jason.orendorff at gmail.com> wrote:
On Thu, May 2, 2013 at 9:47 AM, Andreas Rossberg <rossberg at google.com> wrote:
I don't see how logical names can possibly make sense without at least a rudimentary manager that maps them. Otherwise they are just physical names.
Yes. A rudimentary way of mapping logical names to URLs is built into the proposal: the default resolve behavior.
No effort: modules are loaded relative to the document base url, with ".js" appended. So
import "jquery"maps to the relative URL"jquery.js".One line of code: you can set the root URL from which all imports are resolved. If you set
System.baseURL = "https://example.com/scripts/"thenimport "jquery"maps to the URL"https://example.com/scripts/jquery.js".A few lines: you can use
System.ondemand()to set the URL for each module you use. If you callSystem.ondemand({"https://example.com/jquery-1.9.1.js": "jquery"})thenimport "jquery"maps to the URL you specified (imports for modules that aren't in the table will fall back on the loader's baseURL).Beyond that: write a resolve hook and locate modules however you want.
Option 0 only applies if a specific design choice for the syntax of logical names is already hardwired into the default resolver logic. You assume that syntax to be plain path names, but as I tried to point out in my previous replies, interpreting plain "jquery" as a logical name is simply not an option in the current set-up, since it fundamentally violates URL semantics -- allowing URLs in imports, and using plain paths as global logical names, are mutually exclusive design choices. So you have to distinguish logical names syntactically somehow, and map that syntax in the resolver. I'd prefer the language not to prescribe a specific syntax for that, since actual package managers might want to make their own choices.
All other options additionally require staging. That is, it will be more than just one line, and you need to run it in a separate set-up script. At that point you arguably have implemented a rudimentary manager, i.e., it's not just built in -- which, to be clear, I find perfectly fine.
/Andreas
On Sat, May 4, 2013 at 12:46 AM, Jason Orendorff <jason.orendorff at gmail.com> wrote:
- No effort: modules are loaded relative to the document base url, with ".js" appended. So
import "jquery"maps to the relative URL "jquery.js".
Isn't that weird? If I have a script at /resources/test which does a bunch of imports and I use /resources/test from resources /a/b and /c/d the document base URL will be different for both. (Document base URL does not work for workers either.)
Appending ".js" also seems very counter to any other API in the platform and kinda counter web architecture.
One line of code: you can set the root URL from which all imports are resolved. If you set
System.baseURL = "https://example.com/scripts/"thenimport "jquery"maps to the URL "example.com/scripts/jquery.js".A few lines: you can use System.ondemand() to set the URL for each module you use. If you call
System.ondemand({"https://example.com/jquery-1.9.1.js": "jquery"})thenimport "jquery"maps to the URL you specified (imports for modules that aren't in the table will fall back on the loader's baseURL).
I thought the import statements had to be first. Does this effectively require two script elements?
On Sat, May 4, 2013 at 3:12 AM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Sat, May 4, 2013 at 12:46 AM, Jason Orendorff <jason.orendorff at gmail.com> wrote:
- No effort: modules are loaded relative to the document base url, with ".js" appended. So
import "jquery"maps to the relative URL "jquery.js".Isn't that weird? If I have a script at /resources/test which does a bunch of imports and I use /resources/test from resources /a/b and /c/d the document base URL will be different for both.
It's a little weird, but it would also be weird to treat module names as URLs relative to the containing script by default, and then when the user goes to Option 1, suddenly treat them as relative to one central URL. I'm sensitive to that because Option 1 is the sweet spot for small-to-medium-sized sites that don't have so much JS that they feel the need for a package manager. (I believe that package managers shouldn't be obligatory for everyone in practice.)
In practice, when a user realizes Option 0 relative URLs are biting them, they can go to Option 1, which is one line of code
<script>System.baseURL = "https://mysite.example.com/static/js/";</script>
...and now they can live comfortably for weeks or years as their project grows, before deciding they need System.ondemand(), or a package manager, or whatever.
And this "weird" behavior is hardly without precedent: aren't all other URLs a script deals with, including the existing APIs for loading other scripts (adding a <script> tag or loading the code with
an XHR), resolved relative to the document base URL?
(Document base URL does not work for workers either.)
Good point. It can do what XHR does in workers: use the script's base url. (If the site happens to have a single directory for serving static js, and the worker script happens to be in it, that's especially convenient.)
I thought the import statements had to be first. Does this effectively require two script elements?
Yes.
On Fri, May 3, 2013 at 9:22 PM, Kevin Smith <zenparsing at gmail.com> wrote:
Do we? It seems to me that "filenames as import-specifiers" is a design option that has been available to, and rejected by, all of the JS module systems and all recent programming languages, except PHP.* And the reason seems obvious enough: hard-coding a library's location into every import site is bad.
First, I assume that you're just referring to inter-package dependencies. Relative paths are fine for a library's internal modules.
Well, yes and no. Relative paths are fine for development; Sam and Andreas have been discussing (in this thread) what happens when you go to production and you want to put all that in one file. It seems better not to be embedding URLs in code, even relative ones, just because it makes concatenation more complex. I don't think we want it to be considered normal for your production code to look like it has a bunch of relative URLs in it, when in fact the loader is rewriting or ignoring them all. Too misleading. (An ahead-of-time-compiled system like Dart might not care about this; for the Web I think we should care.)
Earlier I think you suggested making everything to do with inter-package dependencies out of scope for ES6. That doesn't make sense to me. Code using a package manager should be able to use import syntax to get packages. So package managers will need to slot into the system we're designing.
I want it to be easy to start out hacking on a web page with
import "jquery" as $, "underscore" as _;
and add a package manager next week. Adding a package manager shouldn't be required, but it shouldn't be disruptive either if you've done the usual stuff. In particular the default should be that if I write a package that uses underscore, you should be able to import my package without switching to the package manager I use—and ideally without using a package manager at all.
Second, I'm well aware of the issues you point out with using URLs or file paths for inter-package dependencies. I think URLs can perhaps do some interesting things for us here which we haven't seen yet, but I'm not arguing against using a flat namespace.
I would like to hear more about that when you're ready -- I think the default relative-ness of URLs makes them unsuitable when what you want is a flat namespace.
But (and this is the key) such a flat namespace must be layered on top of URLs. It cannot overload URLs as the current design attempts to do. Why not? Because such an overloaded semantics would conflict with the resolution semantics as defined by the platform in HTML, CSS, etc. Truth, beauty conceptual integrity, and all that good stuff. : )
I can totally understand not wanting module names to be ambiguous with URLs or look like relative URLs but behave totally differently. As Dave said, that's a surface syntax issue (which is not to say it's not important).
But I still don't see why we should layer something on top of URLs when we already know users strongly prefer that it not be coupled to locations.
On Mon, May 6, 2013 at 4:39 PM, Jason Orendorff <jason.orendorff at gmail.com> wrote:
And this "weird" behavior is hardly without precedent: aren't all other URLs a script deals with, including the existing APIs for loading other scripts (adding a <script> tag or loading the code with an XHR), resolved relative to the document base URL?
Typically the entry script's base URL, which can be different in cross-document scenarios. And for XMLHttpRequest it's the document associated with its constructor object (which annoyingly is somewhat different from the rest).
Good point. It can do what XHR does in workers: use the script's base url. (If the site happens to have a single directory for serving static js, and the worker script happens to be in it, that's especially convenient.)
So now you have two conventions...
Anne van Kesteren wrote:
On Mon, May 6, 2013 at 4:39 PM, Jason Orendorff <jason.orendorff at gmail.com> wrote:
And this "weird" behavior is hardly without precedent: aren't all other URLs a script deals with, including the existing APIs for loading other scripts (adding a<script> tag or loading the code with an XHR), resolved relative to the document base URL?
Typically the entry script's base URL, which can be different in cross-document scenarios. And for XMLHttpRequest it's the document associated with its constructor object (which annoyingly is somewhat different from the rest).
Noted!
Good point. It can do what XHR does in workers: use the script's base url. (If the site happens to have a single directory for serving static js, and the worker script happens to be in it, that's especially convenient.)
So now you have two conventions...
If the newer one is better, that's progress. Building on a broken ("annoyingly") convention to keep some unity-of-conventions ideal at the expense of usability is not the obvious winner.
With the Web, you never can hope to have "one way to do it". The goal is "better over time".
Well, yes and no. Relative paths are fine for development; Sam and Andreas have been discussing (in this thread) what happens when you go to production and you want to put all that in one file.
That's when you want to bundle sub-modules together into a single module. That's one reason why we need lexical modules - so that we can bundle up a library's internal modules without dumping a bunch of useless entries into the "global" module table.
Really, we need lexical modules.
Earlier I think you suggested making everything to do with inter-package dependencies out of scope for ES6. That doesn't make sense to me. Code using a package manager should be able to use import syntax to get packages. So package managers will need to slot into the system we're designing.
I suppose that's a reasonable position to take. I could argue that a module loader API provides enough of a slot. I mean, you should be able to implement a custom "package:" scheme in just one line:
<script>
System.resolve = url => /^package:/.test(url) ? `${ this.baseURL
}/packages/${ url.replace(/^package:/, "") }.js` : url; </script>
<script async>
import $ from "package:jquery";
</script>
In particular the default should be that if I
write a package that uses underscore, you should be able to import my package without switching to the package manager I use—and ideally without using a package manager at all.
Sure - but that's tricky, opinionated business in general and it deserves an entire debate of it's own. Regardless, we shouldn't even be having that debate until we have a solid module system in place. Again, inter-package dependency resolution should be layered on top of the core module system. In the current design, it is conflated.
I can totally understand not wanting module names to be ambiguous with
URLs or look like relative URLs but behave totally differently. As Dave said, that's a surface syntax issue (which is not to say it's not important).
I don't think so. I think the current design is entirely wrapped up in modeling modules as "logical" names in some flat namespace. It is central to the design rather than layered on top, as it should be.
But I still don't see why we should layer something on top of URLs when we already know users strongly prefer that it not be coupled to locations.
A custom scheme would layer on top of URIs just fine and would only take a handful of lines of code to implement:
import $ from "package:jquery";
Are you opposed to such URIs?
On Mon, May 6, 2013 at 7:10 PM, Brendan Eich <brendan at mozilla.com> wrote:
Anne van Kesteren wrote:
So now you have two conventions...
If the newer one is better, that's progress. Building on a broken ("annoyingly") convention to keep some unity-of-conventions ideal at the expense of usability is not the obvious winner.
With the Web, you never can hope to have "one way to do it". The goal is "better over time".
I meant doing something different for worker scripts and document scripts. Document scripts would all share a base URL whereas worker scripts would each have their own. CSS @import also behaves different from JavaScript import then.
That URLs set at runtime have a different base URLs seems kinda logical as there's no obvious way to relate them to the script, but for static URLs it feels wrong. (As does the appending ".js" and such. Too much magic.)
On Mon, May 6, 2013 at 7:40 PM, Anne van Kesteren wrote:
On Mon, May 6, 2013 at 4:39 PM, Jason Orendorff wrote:
[...] aren't all other URLs a script deals with, including the existing APIs for loading other scripts (adding a <script> tag or loading the code with an XHR), resolved relative to the document base URL?
Typically the entry script's base URL, which can be different in cross-document scenarios. And for XMLHttpRequest it's the document associated with its constructor object (which annoyingly is somewhat different from the rest).
Oh, interesting.
...reads a lot of spec text...
OK, check my work here: For resolve hooks, there's no entry script. The loader calls the resolve hook asynchronously, from an empty stack. Or, empty except for the loader implementation. Since we're talking about the browser's default resolve hook, that's not a script either. No user code is executing.
Tentatively I still think what XHR does is right for loaders. Each Loader, like a Worker or DOM node, should have its own base URL and use it consistently. It seems unhelpful for a cross-document call elsewhere on the stack to affect System.load(), causing modules to be loaded from the wrong place. Note that loaded modules are cached, so the oddness would persist.
I dunno, Anne. I'm sure I'm not the first n00b to say this, but caring about what code is at the bottom of the stack is creepy already. The "entry script's base URL" convention seems bad, even if we could follow it.
Jason Orendorff wrote:
I dunno, Anne. I'm sure I'm not the first n00b to say this, but caring about what code is at the bottom of the stack is creepy already.
This is red-alert material for Mark, too. Lots of security follies in olden Java-in-browser days based on the stack.
Uh oh. I'd been ignoring this thread hoping I didn't need to worry about it. I see that it has 56 messages already. Could someone summarize the red alert issue? If there is any possibility that the issue might indicate a vulnerability that is not already obvious from public posts, please email me privately. Thanks.
On Tue, May 7, 2013 at 10:18 AM, Jason Orendorff <jason.orendorff at gmail.com> wrote:
OK, check my work here: For resolve hooks, there's no entry script. The loader calls the resolve hook asynchronously, from an empty stack. Or, empty except for the loader implementation. Since we're talking about the browser's default resolve hook, that's not a script either. No user code is executing.
Tentatively I still think what XHR does is right for loaders. Each Loader, like a Worker or DOM node, should have its own base URL and use it consistently. It seems unhelpful for a cross-document call elsewhere on the stack to affect System.load(), causing modules to be loaded from the wrong place. Note that loaded modules are cached, so the oddness would persist.
My point was that you'd want neither the entry script's URL or the document's base URL, but rather the URL of the script resource itself. Since these are static links similar to <img src> in an HTML document
or url() in a CSS resource, I thought you'd want them to work the same way.
The associated document could work, but that means you'd always be required to use links such as /path/to/script as otherwise the link breaks if the script that does the importing is included from both /doc1 and /example/doc2.
I dunno, Anne. I'm sure I'm not the first n00b to say this, but caring about what code is at the bottom of the stack is creepy already. The "entry script's base URL" convention seems bad, even if we could follow it.
I don't have insight as to why browsers have this model. I'm pretty sure we cannot change it though. (As you probably know, the script execution model defined by HTML was in existence for a long time before that. Browsers simply reverse engineered it from each other. Now it's defined, but as with the same-origin policy, it doesn't mean it was a great idea to begin with.)
On Tue, May 7, 2013 at 12:29 PM, Mark S. Miller <erights at google.com> wrote:
Uh oh. I'd been ignoring this thread hoping I didn't need to worry about it. I see that it has 56 messages already. Could someone summarize the red alert issue?
I don't think you need to follow it. I was surprised to learn that window.open()'s URL resolution behavior (and history.pushState(), location.href=, and more) depends on what script is at the bottom of the stack. If that's no surprise to you, you're all set.
On Tue, May 7, 2013 at 12:53 PM, Anne van Kesteren <annevk at annevk.nl>wrote:
My point was that you'd want neither the entry script's URL or the document's base URL, but rather the URL of the script resource itself. Since these are static links similar to <img src> in an HTML document or url() in a CSS resource, I thought you'd want them to work the same way.
If module names were URLs, that would definitely be the right thing.
Module names aren't URLs, though. These aren't static links to static locations (for reasons discussed in this thread; e.g., it'd be counterproductive for backbone.js to have a static URL for underscore embedded in it).
The associated document could work, but that means you'd always be
required to use links such as /path/to/script as otherwise the link breaks if the script that does the importing is included from both /doc1 and /example/doc2.
You pointed out this issue earlier, and I replied (on Monday, May 6; search for "sweet spot"). What do you think?
I don't have insight as to why browsers have this model. I'm pretty
sure we cannot change it though. [...]
Oh, we're on the same page then.
On Tue, May 7, 2013 at 11:18 AM, Jason Orendorff <jason.orendorff at gmail.com> wrote:
If module names were URLs, that would definitely be the right thing.
Module names aren't URLs, though. These aren't static links to static locations (for reasons discussed in this thread; e.g., it'd be counterproductive for backbone.js to have a static URL for underscore embedded in it).
But you are treating them as URLs by default (with a small dose of magic).
The associated document could work, but that means you'd always be required to use links such as /path/to/script as otherwise the link breaks if the script that does the importing is included from both /doc1 and /example/doc2.
You pointed out this issue earlier, and I replied (on Monday, May 6; search for "sweet spot"). What do you think?
I think that because of this you cannot really use the default loader semantics in most scenarios. Lots of sites reuse a script on several pages (i.e. documents) with disparate URLs, similar to how they reuse CSS.
On Tue, May 7, 2013 at 2:35 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Tue, May 7, 2013 at 11:18 AM, Jason Orendorff <jason.orendorff at gmail.com> wrote:
If module names were URLs, that would definitely be the right thing.
Module names aren't URLs, though. These aren't static links to static locations (for reasons discussed in this thread; e.g., it'd be counterproductive for backbone.js to have a static URL for underscore embedded in it).
But you are treating them as URLs by default (with a small dose of magic).
No. This is not what's happening at all.
Modules are given logical names, such a "backbone" or "jquery/ui". Module loaders map logical names to URLs and then map the URLs to JavaScript source [1].
The default behavior of the initial module loader for the browser is to convert a logical name to a URL in a certain way, and then to fetch that URL with http in the usual way to get JS source.
The way we think that conversion should happen is to use the loader baseURL, which defaults to the document base URL, as a prefix, and ".js" as a suffix. Certainly we could choose other prefixes instead, but it doesn't sound like you think there's a clear better choice. We could also choose a different suffix, or no suffix, but it would be odd to force everyone to remove the extensions on their files.
The associated document could work, but that means you'd always be required to use links such as /path/to/script as otherwise the link breaks if the script that does the importing is included from both /doc1 and /example/doc2.
You pointed out this issue earlier, and I replied (on Monday, May 6; search for "sweet spot"). What do you think?
I think that because of this you cannot really use the default loader semantics in most scenarios. Lots of sites reuse a script on several pages (i.e. documents) with disparate URLs, similar to how they reuse CSS.
In that case, such a script probably won't use relative module names which are expected to be fetched by the default mapping I describe above. That doesn't mean they need an new loader, though. It would be just as easy to use another script tag, another module in the same script tag, or a use of System.ondemand().
Sam
[1] In fact, "URL" is here just an arbitrary string, but usually it will be a URL.
If module names were URLs, that would definitely be the right thing.
Module names aren't URLs, though. These aren't static links to static locations (for reasons discussed in this thread; e.g., it'd be counterproductive for backbone.js to have a static URL for underscore embedded in it).
Well, they're not URLs except when they are URLs - that's the problem.
And calling them "not URLs" is highly dubious in all cases. Under the smoke and mirrors, they are effectively locators just like any URL. Just locators that don't play nice with the rest of the web platform.
If you really want a flat namespace for inter-package dependency resolution, then I think you'll want a custom URI scheme.
On 7 May 2013 20:49, Kevin Smith <zenparsing at gmail.com> wrote:
If module names were URLs, that would definitely be the right thing.
Module names aren't URLs, though. These aren't static links to static locations (for reasons discussed in this thread; e.g., it'd be counterproductive for backbone.js to have a static URL for underscore embedded in it).
Well, they're not URLs except when they are URLs - that's the problem.
Right.
Sam, Jason, I'm sorry, but I really can't follow your line of reasoning on this. Aren't you just papering over a problem by defining it away with convenient wording?
[coming back to this a few days later]
On Thu, May 2, 2013 at 10:47 AM, Andreas Rossberg <rossberg at google.com> wrote:
On 1 May 2013 00:01, David Herman <dherman at mozilla.com> wrote:
On Apr 29, 2013, at 6:34 AM, Andreas Rossberg <rossberg at google.com> wrote:
...you are generally assuming the use of some package manager. This is a fairly significant assumption, that I wished I had heard being made explicit in any of the previous discussions. AFAICT, it is key, and without it, your proposal cannot fly. Not in a world with tens of thousands of module releases, anyway.
Not only is this wrong -- it'll be just fine to use ES6 without package managers -- but your proposal does not make any material difference in the relative dependence on package managers.
I don't see how logical names can possibly make sense without at least a rudimentary manager that maps them. Otherwise they are just physical names.
AMD works today in the absence of a package manager, and the logical names are not physical names there either.
My point on the topic of external naming is that the language (1) should not prescribe any specific naming scheme; (2) should not willfully violate URI semantics; (3) should properly separate it from internal naming.
(1) No "naming scheme" is prescribed here, unless you mean the semantics of relative paths
(2) As I pointed out in the mail I just wrote in reply to Anne, nothing violates URI semantics here. Logical names are not URIs, and thus treating them differently than URIs doesn't violate URI semantics.
(3) Internal naming in the sense you mean is primarily a concern for organizing code within a single lexical scope. This is a real issue, but it's not nearly as pressing as the other issues here, and I think we can safely defer it.
Package managers are great, and I agree that they will (and should) become part of the eco system. But leave design decisions to them, and don't limit their options a priori. I'm not sure why we even disagree on that.
I think we agree on this.
More specifically, everything should still play nice with standard web mechanisms. For me, the web platform implies that I should be able to address modules as remote resources (it's a choice not to use that mechanism). That requires that the system allows proper URLs as module identifiers.
Fully agreed, and if it wasn't clear, the current system does completely support this. You can use a URL as a module name. In the default browser loader, you can choose to import a module from an explicit URL:
import { $ } from "http://my.cdn.com/jquery/1.9";and it will use that URL both as the logical name and as the place to go and fetch the bits.
OK, great! Can I also use such URLs for module definitions then?
I don't know what you're asking here.
If you don't use an absolute URL, it treats it as a path, which defaults to loading as a source file from a relative URL derived from the module name.
And here we get to the core of the problem. Namely, how do you interpret relative and absolute here? URLs have a very clear semantics: everything that isn't absolute is relative to the referrer. So far, everything I have seen in any of the recent module specs or presentations is a blunt violation of these semantics.
In particular, as I mentioned before, you cannot make "a" mean something different than "./a" without violating URLs [1,2]. Yet that is not only what you envision, it is what you de facto prescribe with your proposal. I think that's simply a total no-go.
[1] tools.ietf.org/html/rfc3986#section-4.2 [2] tools.ietf.org/html/rfc3986#section-5.2.2
No, this is not correct. Neither "a" not "./a" are URLs, and thus treating them differently does not violate the semantics of URLs.
That does not mean that logical names have to become unreadable or awful to type. Just that they are slightly more explicit. A schema prefix -- say, jsp: for a random strawman -- is good enough and should not offend anybody in terms of verbosity. (If it did, then I'd have little hope for the evolution of the web platform. :) )
First, let's be clear: this is literally an argument over the surface syntax of public module names, nothing more. You're arguing for the universal addition of a few characters of boilerplate to all registered module names. That's just bad ergonomics for no gain. Not only that, it hurts interoperability with existing JavaScript module systems.
I don't see how it hurts interop. It just means that you have to find a convenient and proper way to embed a logical name space into URLs. I proposed using a schema, but anything that is compatible with URL semantics is OK. Just taking AMD-style strings verbatim is not.
It hurts interop because AMD-style modules are named as "a/b" and not "jsp:a/b". A required prefix would therefore require impedance matching somewhere.
It is not something which you can justify with AMD/Node precedence either, since AFAICT these systems do not have the notational overloading between logical paths and URLs that you propose.
Both AMD and Node automatically convert module names like "a/b" into URLs (in the AMD case) and file paths (in the Node case).
There's a reason why the existing systems favor lightweight name systems. They're nicely designed to make the common case syntactically lighter-weight. And they absolutely can co-exist with naming modules directly with URL's.
Sure they can coexist. But please not by bastardizing URL semantics.
Again, these are not URLs. You can't on the one hand complain that we aren't using URLs for names, and then on the other hand complain that our treatment of logical names doesn't follow the rules for URLs.
On Thu, May 2, 2013 at 2:13 PM, Andreas Rossberg <rossberg at google.com> wrote:
On 1 May 2013 01:15, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Mon, Apr 29, 2013 at 9:34 AM, Andreas Rossberg <rossberg at google.com> wrote:
I brought up [removing module declarations] as a "min-max" sort of compromise at the March meeting, but it did not get much love. And I agree that it isn't pretty, although it actually has some technical advantages, e.g. enabling parallel parsing.
No, that isn't correct.
What isn't correct?
Sorry, I think I quoted the wrong thing here.
There are several reasons why module declarations are important.
Right now, every JS construct can be written at the REPL, in a script tag, in a bunch of files in a Node package, or anywhere else. This is a very important property for the language, and one we should maintain. Without module declarations except as separate files, we would lose this uniformity.
Being able to define multiple modules concisely in a single file is simply more convenient than having to separate them into multiple files. Even clunky old Java allows this simple convenience!
Serving modules as script tags, and defining modules in inline script tags, is likely to be important for a wide variety of use cases.
Concatenation.
I generally sympathise with (1). However, modules arguably are a borderline case. They fall into the same category as multiple scripts -- a very module-ish "construct" that cannot accurately be expressed in pure JavaScript either.
That multiple scripts can't be expressed in source is no reason to add more constructs like it.
As for the others, shrug. All valid points, and I agree that that it's suboptimal if you cannot do that. On the other hand, nothing will break, nothing essential would be missing. That's the nature of a min-max path.
But my broader point, of course, is that lexical modules address all these cases just fine.
My point was not to say that concatenation is a weak argument and that we need more. You agree that these are valuable features. However, concatenation support is a required feature. The world in which multiple http requests are free is not here, and not on the horizon.
This would be a coherent addition to the system, but is not a necessary one. Below, you point out some issues that would be potentially addressed by adding them. However, multiple JS module systems are in use today, and with the partial exception of the TypeScript namespaces feature (which is a significantly different design), none of them has lexically-named modules in the sense you advocate.
I do not think this observation is particularly relevant. As I said already, emulating a module system within JS necessarily has much more severe limitations than designing it as a language feature. In particular, I'm not sure you even could build a lexical module system under those constraints. There is no reason to apply those limitations to ES6 modules, though.
It's certainly possible to build a system like this in a compile-to-JS language like Typescript, and I don't believe it's happened.
Also, the incoherent programs you refer to are ruled out by a suggestion (of yours!) that we adopted at the last meeting: declarative module forms that define a module that already exists are a static error.
The suggestion you mention rather deals with the next case (clashes between different internally defined modules). It does not generally help when embedded resources clash with externally defined resources.
When the second resource loaded is a declarative form, then this addresses this case as well.
Yes, but not the other way round.
True, but I contend that these accidental clashes will be much more likely with the declarative form -- using loader.set() is likely to be less common and used more knowledgeably.
- "Local" references are globally overridable.
This is only true in the sense that executing first lets you define the meaning of particular modules. But this is a feature. This is just another way to describing configuration. It's not reasonable to think that we can decide for everyone what will need configuration and what won't.
Er, I'm not saying that we should decide it. I'm saying that the implementer of a module should be able to make that choice. In the current proposal, he doesn't have a choice at all, he is forced to make each and every module definition essentially "mutable" by the rest of the world.
There are two possible scenarios here. If your module is loaded inside a loader that you don't trust, you have zero semantic guarantees. The translate and link hooks give the loader arbitrary power, lexical modules or no.
Sure.
The initial loader in the JS environment is mutable and thus if you are in a mutually-untrusting situation, you can't rely on anything.
Well, in practice it's not all-or-nothing. In this scenario, I worry about accident more than malice. Once modular JS applications grow in scale, and we get more interesting towers of libraries, I'm sure this will become a problem.
In the case of accident, just putting the two modules in the same bit of code that's compiled together (such as a file) will prevent such accidents. Across files, lexical scope won't be sufficient unless you're using a build tool.
Needing to explicitly register a module will certainly be common in a node-like system, where small modules are the norm. Which is the common case in general isn't a question we can answer today. But I don't think that needing to avoid registering will be common.
Pro JS code today certainly goes to great length to avoid polluting the global name space, by using anonymous functions. I think the situation is much more similar than you'd like it to be.
Mostly, this is managed by using a single namespace that everything hangs of, as with jQuery. This strategy is easy to apply to the module name space as well.
But ultimately, it's simply a question of the right default. Making everything externally visible, let alone overridable, is, plain and simple, the wrong default. I'm still puzzled why you'd argue otherwise, since for very good reasons, you made the opposite choice one level down, for the contents of a module. I fail to see the qualitative difference.
These are designed for different purposes. Modules are intended to be abstractions, module names are designed to help manage collections of modules.
While it's sad that everyone doesn't use the same filesystem ;), this is inevitable unless we force everyone to write both internal and external names explicitly for every module; that's clearly more user-hostile than the alternative.
But you do not want to do that for every module! In fact, you rarely need to explicitly define a module with an external name at all, at least as I envision it. You only want to do that for a module that you want to "export" from your current script, so to speak. Such "export" (i.e., globally registering a module you defined textually) should be a very rare operation to write manually (as mentioned, you don't usually do the equivalent in AMD, for example).
Anonymous module defintions are often used in AMD for modules that are implicitly named by the file they're in. In ES6, that simply doesn't require a wrapper.
Yes, my point? Thing is, in AMD experience, named module definitions are neither common, nor recommended. You use separate files. Why expect something different for ES6?
Right, you use separate files and that gives you a bunch of string-named modules registered in the global modules registry. That's then compiled to explicitly named modules all in one file. That this is similar to our design should be unsurprising, since they're working on the same problem, and both systems care about concatenation.
The reason to use concatenation is to avoid consuming excessive client resources -- in this setting of course you won't want to run translation on the client side. Translation hooks are important (a) in less perf-sensitive settings like development and (b) for isolating remotely-loaded code, neither of which require concatenation.
I don't believe it's as clear cut, and I can imagine a number of cool use cases for translation that do not necessarily fall into this simple pattern.
Can you say more about these use cases?
For example, imagine statically generating (the definition of) a non-trivial JS data structure from a domain-specific description language and some dynamic information. Or things like FFTW. In a similar vein, you could implement a semi-dynamic macro system on top of JS.
These are great use cases, but if you're using them you'll have to make a choice between separate files and string literals. We can't make design decisions based on assuming a packaging format for multiple files that will exist in the hypothetical future. If such a system existed and was shipping already, things would be different.
On Tue, May 7, 2013 at 11:48 AM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Tue, May 7, 2013 at 2:35 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
But you are treating them as URLs by default (with a small dose of magic).
No. This is not what's happening at all.
Modules are given logical names, such a "backbone" or "jquery/ui". Module loaders map logical names to URLs and then map the URLs to JavaScript source.
The default behavior of the initial module loader for the browser is to convert a logical name to a URL in a certain way, and then to fetch that URL with http in the usual way to get JS source.
How is that not treating it as a URL with a dose of magic by default? E.g. import "code.jquery.com/jquery-1.9.1.min" will work given the way things are defined now. Similarly if I put jquery in root import "/jquery-1.9.1.min" will always work, regardless of location. This might not be how things are intended to be used, but I'd expect to see this kind of usage.
The way we think that conversion should happen is to use the loader baseURL, which defaults to the document base URL, as a prefix, and ".js" as a suffix. Certainly we could choose other prefixes instead, but it doesn't sound like you think there's a clear better choice. We could also choose a different suffix, or no suffix, but it would be odd to force everyone to remove the extensions on their files.
I think making them URLs (i.e. requiring import "/js/jquery.js") and allowing people who want them to be something different to override that with hooks is better. (Or with a new URL scheme as Kevin suggests.)
On Tue, May 7, 2013 at 4:01 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Tue, May 7, 2013 at 11:48 AM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Tue, May 7, 2013 at 2:35 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
But you are treating them as URLs by default (with a small dose of magic).
No. This is not what's happening at all.
Modules are given logical names, such a "backbone" or "jquery/ui". Module loaders map logical names to URLs and then map the URLs to JavaScript source.
The default behavior of the initial module loader for the browser is to convert a logical name to a URL in a certain way, and then to fetch that URL with http in the usual way to get JS source.
How is that not treating it as a URL with a dose of magic by default? E.g. import "code.jquery.com/jquery-1.9.1.min" will work given the way things are defined now. Similarly if I put jquery in root import "/jquery-1.9.1.min" will always work, regardless of location. This might not be how things are intended to be used, but I'd expect to see this kind of usage.
Using an absolute URL (as in your first example) opts out of any processing. Otherwise it wouldn't work, of course -- adding the baseURL would break that URL.
In the second case, what is the complain? That's a reasonable module name.
The way we think that conversion should happen is to use the loader baseURL, which defaults to the document base URL, as a prefix, and ".js" as a suffix. Certainly we could choose other prefixes instead, but it doesn't sound like you think there's a clear better choice. We could also choose a different suffix, or no suffix, but it would be odd to force everyone to remove the extensions on their files.
I think making them URLs (i.e. requiring import "/js/jquery.js") and allowing people who want them to be something different to override that with hooks is better. (Or with a new URL scheme as Kevin suggests.)
The whole point of my original email to Andreas is that this doesn't
work. These names are not intended to specify where to find the
resource. They are intended to be names that different modules can
coordinate with. Should backbone put import "/js/jquery.js" in
their source file?
The whole point of my original email to Andreas is that this doesn't work. These names are not intended to specify where to find the resource. They are intended to be names that different modules can coordinate with. Should backbone put
import "/js/jquery.js"in their source file?
I think we all understand the problem that you're trying to solve here, but aren't you putting the cart before the horse? Can we not design the module system first, and then layer packages on top? Do you not agree that that would be a more robust approach?
On Tue, May 7, 2013 at 1:22 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Tue, May 7, 2013 at 4:01 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
How is that not treating it as a URL with a dose of magic by default? E.g. import "code.jquery.com/jquery-1.9.1.min" will work given the way things are defined now. Similarly if I put jquery in root import "/jquery-1.9.1.min" will always work, regardless of location. This might not be how things are intended to be used, but I'd expect to see this kind of usage.
Using an absolute URL (as in your first example) opts out of any processing. Otherwise it wouldn't work, of course -- adding the baseURL would break that URL.
"Adding"? What's the actual processing model for the default loader here? I thought it was:
- Append ".js" to module name.
- Use the URL parser to parse module name relative to base URL.
- Fetch result of that operation.
In the second case, what is the complain? That's a reasonable module name.
If that's reasonable you might as well not add ".js" by default and require that to be specified if people want that. (Tying file extensions to content is dubious in a web context in general.)
The whole point of my original email to Andreas is that this doesn't work. These names are not intended to specify where to find the resource. They are intended to be names that different modules can coordinate with. Should backbone put
import "/js/jquery.js"in their source file?
If backbone has its own loader and everything it doesn't really matter what the default semantics are.
On Tue, May 7, 2013 at 4:31 PM, Kevin Smith <zenparsing at gmail.com> wrote:
The whole point of my original email to Andreas is that this doesn't work. These names are not intended to specify where to find the resource. They are intended to be names that different modules can coordinate with. Should backbone put
import "/js/jquery.js"in their source file?I think we all understand the problem that you're trying to solve here, but aren't you putting the cart before the horse? Can we not design the module system first, and then layer packages on top? Do you not agree that that would be a more robust approach?
No, I do not agree. If our module system does not address the use cases that people have already built JS module systems to solve then we have failed as stewards of the language.
On Tue, May 7, 2013 at 4:33 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Tue, May 7, 2013 at 1:22 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Tue, May 7, 2013 at 4:01 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
How is that not treating it as a URL with a dose of magic by default? E.g. import "code.jquery.com/jquery-1.9.1.min" will work given the way things are defined now. Similarly if I put jquery in root import "/jquery-1.9.1.min" will always work, regardless of location. This might not be how things are intended to be used, but I'd expect to see this kind of usage.
Using an absolute URL (as in your first example) opts out of any processing. Otherwise it wouldn't work, of course -- adding the baseURL would break that URL.
"Adding"? What's the actual processing model for the default loader here? I thought it was:
- Append ".js" to module name.
- Use the URL parser to parse module name relative to base URL.
- Fetch result of that operation.
The actual processing model is:
- Produce a string that is, roughly, System.baseURL + moduleName + ".js".
- Pass that to the fetch hook (which by default fetches that URL).
In the second case, what is the complain? That's a reasonable module name.
If that's reasonable you might as well not add ".js" by default and require that to be specified if people want that. (Tying file extensions to content is dubious in a web context in general.)
It's reasonable in the sense that we aren't disallowing it. It's a bad idea because it's bad for coordination.
The whole point of my original email to Andreas is that this doesn't work. These names are not intended to specify where to find the resource. They are intended to be names that different modules can coordinate with. Should backbone put
import "/js/jquery.js"in their source file?If backbone has its own loader and everything it doesn't really matter what the default semantics are.
I no longer know what you're talking about. Why would backbone have it's own loader? If you're using a page with multiple libraries that depend on jquery, you need a way for them all to use the same jquery. That's the point of giving it a well-known name, like "jquery". This is similar to the way you'd use AMD for this. Please go back and read the email I mentioned.
On Tue, May 7, 2013 at 3:31 PM, Kevin Smith <zenparsing at gmail.com> wrote:
I think we all understand the problem that you're trying to solve here, but aren't you putting the cart before the horse? Can we not design the module system first, and then layer packages on top? Do you not agree that that would be a more robust approach?
I don't think you get better design by declaring the most important part of the problem off-limits while designing the first half of the solution, no. That sounds like a surefire recipe for designing something that's pretty, but fails to address the problems developers actually face.
Considerations of whether people will actually be able use this to share code, and how it will support projects of various sizes, are very much on topic. Nothing else in this conversation is as important.
On Tue, May 7, 2013 at 1:47 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Tue, May 7, 2013 at 4:33 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
"Adding"? What's the actual processing model for the default loader here? I thought it was:
- Append ".js" to module name.
- Use the URL parser to parse module name relative to base URL.
- Fetch result of that operation.
The actual processing model is:
- Produce a string that is, roughly, System.baseURL + moduleName + ".js".
- Pass that to the fetch hook (which by default fetches that URL).
Could you be more specific? That seems broken. E.g. often a document's base URL will be something like "example.org/file.html".
On Tue, May 7, 2013 at 4:54 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Tue, May 7, 2013 at 1:47 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Tue, May 7, 2013 at 4:33 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
"Adding"? What's the actual processing model for the default loader here? I thought it was:
- Append ".js" to module name.
- Use the URL parser to parse module name relative to base URL.
- Fetch result of that operation.
The actual processing model is:
- Produce a string that is, roughly, System.baseURL + moduleName + ".js".
- Pass that to the fetch hook (which by default fetches that URL).
Could you be more specific? That seems broken. E.g. often a document's base URL will be something like "example.org/file.html".
Sorry, this was overly quick of me. The correct steps are:
- If we have an absolute URL, skip steps 1-3.
- Split the module name on "/", url-encode, and rejoin.
- Append ".js" to module name.
- Use the URL parser with the base URL to produce an absolute URL.
- Pass the absolute URL to the fetch hook.
On Tue, May 7, 2013 at 2:00 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
- If we have an absolute URL, skip steps 1-3.
How do you define this? We currently do not have this concept really. You could parse and compare if the serialization is equivalent I suppose, but even that would fail to detect certain cases, such as test/test\test ... Or http:test which is an absolute URL in theory, but treated identical to "test" if the base URL's scheme is http.
- Split the module name on "/", url-encode, and rejoin.
- Append ".js" to module name.
- Use the URL parser with the base URL to produce an absolute URL.
- Pass the absolute URL to the fetch hook.
On Tue, May 7, 2013 at 5:12 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Tue, May 7, 2013 at 2:00 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
- If we have an absolute URL, skip steps 1-3.
How do you define this? We currently do not have this concept really. You could parse and compare if the serialization is equivalent I suppose, but even that would fail to detect certain cases, such as test/test\test ... Or http:test which is an absolute URL in theory, but treated identical to "test" if the base URL's scheme is http.
If it's a URI, then it's treated this way. That is, if it starts 'scheme:' for some scheme.
On Tue, May 7, 2013 at 2:15 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Tue, May 7, 2013 at 5:12 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Tue, May 7, 2013 at 2:00 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
- If we have an absolute URL, skip steps 1-3.
How do you define this? We currently do not have this concept really. You could parse and compare if the serialization is equivalent I suppose, but even that would fail to detect certain cases, such as test/test\test ... Or http:test which is an absolute URL in theory, but treated identical to "test" if the base URL's scheme is http.
If it's a URI, then it's treated this way. That is, if it starts 'scheme:' for some scheme.
I think what you're failing to understand is that there's no concept as "if it's a URI" within the context of the web platform, so I'm asking how you'd thought about going about defining that new concept.
Maybe seeing what I meant in action will help: software.hixie.ch/utilities/js/live-dom-viewer/?<!DOCTYPE html><img src%3Dhttp%3Aimage>
On Tue, May 7, 2013 at 5:18 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Tue, May 7, 2013 at 2:15 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Tue, May 7, 2013 at 5:12 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Tue, May 7, 2013 at 2:00 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
- If we have an absolute URL, skip steps 1-3.
How do you define this? We currently do not have this concept really. You could parse and compare if the serialization is equivalent I suppose, but even that would fail to detect certain cases, such as test/test\test ... Or http:test which is an absolute URL in theory, but treated identical to "test" if the base URL's scheme is http.
If it's a URI, then it's treated this way. That is, if it starts 'scheme:' for some scheme.
I think what you're failing to understand is that there's no concept as "if it's a URI" within the context of the web platform, so I'm asking how you'd thought about going about defining that new concept.
I understood what you meant about the treatment of 'http:' URLs of that form. I think I gave a reasonable definition for URI recognition in the part you quote above; I could rewrite it as a JS program if you'd like.
Maybe seeing what I meant in action will help: software.hixie.ch/utilities/js/live-dom-viewer/?<!DOCTYPE html><img src%3Dhttp%3Aimage>
This seems relevant for what will happen if you import from 'http:foo' with the default semantics, but I don't see the relevance otherwise.
On Tue, May 7, 2013 at 10:18 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Tue, May 7, 2013 at 2:15 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Tue, May 7, 2013 at 5:12 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Tue, May 7, 2013 at 2:00 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
- If we have an absolute URL, skip steps 1-3.
How do you define this? We currently do not have this concept really. You could parse and compare if the serialization is equivalent I suppose, but even that would fail to detect certain cases, such as test/test\test ... Or http:test which is an absolute URL in theory, but treated identical to "test" if the base URL's scheme is http.
If it's a URI, then it's treated this way. That is, if it starts 'scheme:' for some scheme.
I think what you're failing to understand is that there's no concept as "if it's a URI" within the context of the web platform, so I'm asking how you'd thought about going about defining that new concept.
What URLs remain the same after relative resolution against any other URL?
On 7 May 2013 21:17, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Thu, May 2, 2013 at 10:47 AM, Andreas Rossberg <rossberg at google.com> wrote:
In particular, as I mentioned before, you cannot make "a" mean something different than "./a" without violating URLs [1,2]. Yet that is not only what you envision, it is what you de facto prescribe with your proposal. I think that's simply a total no-go.
[1] tools.ietf.org/html/rfc3986#section-4.2 [2] tools.ietf.org/html/rfc3986#section-5.2.2
No, this is not correct. Neither "a" not "./a" are URLs, and thus treating them differently does not violate the semantics of URLs.
[Quick reply only, will address the rest of your mail tomorrow.]
Come on Sam, now you're really splitting hairs. Fine, technically it's called a "URI reference" [3] -- and in practically all places on the web where you reference a URL you are doing so using such a beast. You seriously want to divorce yourself from the rest of the web? (Except, of course, that you still want to allow those URI references as URI references that e.g. start with a "." -- really, this whole idea is highly schizophrenic.)
On Tue, May 7, 2013 at 5:45 PM, Andreas Rossberg <rossberg at google.com> wrote:
On 7 May 2013 21:17, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Thu, May 2, 2013 at 10:47 AM, Andreas Rossberg <rossberg at google.com> wrote:
In particular, as I mentioned before, you cannot make "a" mean something different than "./a" without violating URLs [1,2]. Yet that is not only what you envision, it is what you de facto prescribe with your proposal. I think that's simply a total no-go.
[1] tools.ietf.org/html/rfc3986#section-4.2 [2] tools.ietf.org/html/rfc3986#section-5.2.2
No, this is not correct. Neither "a" not "./a" are URLs, and thus treating them differently does not violate the semantics of URLs.
[Quick reply only, will address the rest of your mail tomorrow.]
Come on Sam, now you're really splitting hairs. Fine, technically it's called a "URI reference" [3] -- and in practically all places on the web where you reference a URL you are doing so using such a beast. You seriously want to divorce yourself from the rest of the web? (Except, of course, that you still want to allow those URI references as URI references that e.g. start with a "." -- really, this whole idea is highly schizophrenic.)
Sorry if I was unclear, I didn't mean to be making the hair-splitting point you're responding to.
The point I'm making is that logical names aren't URLs, they aren't specified to be URLs, and thus treating them differently isn't a violation of the meaning of URLs.
Is your contention that AMD and Node are also violating the semantics of URLs?
On 7 May 2013 23:50, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Tue, May 7, 2013 at 5:45 PM, Andreas Rossberg <rossberg at google.com> wrote:
On 7 May 2013 21:17, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Thu, May 2, 2013 at 10:47 AM, Andreas Rossberg <rossberg at google.com> wrote:
In particular, as I mentioned before, you cannot make "a" mean something different than "./a" without violating URLs [1,2]. Yet that is not only what you envision, it is what you de facto prescribe with your proposal. I think that's simply a total no-go.
[1] tools.ietf.org/html/rfc3986#section-4.2 [2] tools.ietf.org/html/rfc3986#section-5.2.2
No, this is not correct. Neither "a" not "./a" are URLs, and thus treating them differently does not violate the semantics of URLs.
[Quick reply only, will address the rest of your mail tomorrow.]
Come on Sam, now you're really splitting hairs. Fine, technically it's called a "URI reference" [3] -- and in practically all places on the web where you reference a URL you are doing so using such a beast. You seriously want to divorce yourself from the rest of the web? (Except, of course, that you still want to allow those URI references as URI references that e.g. start with a "." -- really, this whole idea is highly schizophrenic.)
Sorry if I was unclear, I didn't mean to be making the hair-splitting point you're responding to.
The point I'm making is that logical names aren't URLs, they aren't specified to be URLs, and thus treating them differently isn't a violation of the meaning of URLs.
It is, if logical names (1) syntactically overlap with URI references, (2) are used in places where you also allow URI references, and (3) have a meaning different from (and incompatible with) the respective sublanguage of URI references. Unfortunately, your design is doing exactly that.
That's why I have been saying all along that you need to use a proper syntactic embedding. Or a disjoint syntax. I really see no way around it.
Is your contention that AMD and Node are also violating the semantics of URLs?
No, because AFAIK, they don't overload path syntax with URL syntax in the same way.
Is your contention that AMD and Node are also violating the semantics of URLs?
Historical side note: Both Node and AMD's usage of "module IDs" derives from CommonJS modules, which were designed primarily for the nascent server-side JS scene in 2009. Module ID semantics were intentionally shell-like because of this focus on the server. Later on, those same semantics were adapted for the browser by require.js. The module format for require.js eventually became codified as AMD.
On Tue, May 7, 2013 at 1:49 PM, Kevin Smith <zenparsing at gmail.com> wrote:
Well, they're not URLs except when they are URLs - that's the problem.
Yeah, I can't entirely disagree. Permitting absolute URLs in imports seems to be misleading, from what we've seen so far.
Most imports will be pretty boring, import "underscore" as _; and such.
If that's all we allowed, I don't think anyone would even think to call
them URLs.
And calling them "not URLs" is highly dubious in all cases. Under the smoke and mirrors, they are effectively locators just like any URL.
Hmm. It's funny, you know, Python programs sometimes say
import os
and I don't think anyone has ever claimed that the word "os" there is really just a URL or really just a filename. Sure, there's an algorithm [1] that maps that "os" to a filename?loading some code is after all the whole point?but the mapping is configurable, and very often configured in some non-default way. Same as in Java. Same as in Ruby.
Set aside absolute-url imports. Suppose we just dropped them. Would you still think that module names are URLs? If so, do you think about other languages in the same way?
So why does this proposal draw this particular response, "module names are
URLs"? I think it's the syntax. Python uses a dot in import email.parser;
the proposal uses a slash, which is more URL-like: import "email/parser" as emailparser;. Python supports relative imports with a leading-dot
syntax, like import .email.parser, meaning "<my_parent_package>.email.parser"; I think the proposal supports the same
thing using .., as in import "../email/parser". All these elements, in
this context, seem to suggest "this is a URL" very strongly. Unfortunate,
as none of these examples are meant to be URLs at all.
On Tue, May 7, 2013 at 4:39 PM, Jason Orendorff <jason.orendorff at gmail.com> wrote:
Set aside absolute-url imports. Suppose we just dropped them. Would you still think that module names are URLs? If so, do you think about other languages in the same way?
I think it's weird to try to equate JavaScript with those languages. They operate on multiple platforms that do not share a universal addressing system and therefore a layer of abstraction had to be invented to make it easier to work those languages across multiple platforms. JavaScript within a browser context operates on a single platform that has a universal addressing system. And within that platform there are multiple non-programming languages, such as HTML and CSS, none of which support remapping URLs at the moment in a manner as it is proposed here, although slightlyoff/NavigationController will give that to us soon(ish).
I guess that's another point I had not really thought of, the web platform will get a way to execute some script at "fetch", which is basically a low-level version of the module loader.
Anne van Kesteren wrote:
On Tue, May 7, 2013 at 4:39 PM, Jason Orendorff <jason.orendorff at gmail.com> wrote:
Set aside absolute-url imports. Suppose we just dropped them. Would you still think that module names are URLs? If so, do you think about other languages in the same way?
I think it's weird to try to equate JavaScript with those languages. They operate on multiple platforms that do not share a universal addressing system and therefore a layer of abstraction had to be invented to make it easier to work those languages across multiple platforms.
Again, that's not the issue.
The question is how should libraries and apps coordinate. For example, how does Ember express a dependency on jQuery without tying itself and its consumers to a particular version at a given URL, and without requiring custom loaders for everyone?
The idea of coordination names is not outr?. Condemning such names for not being URLs is bad religion. Objecting to mixing URLs and lexically indistinguishable non-URLs in the same syntactic context -- there I think you are on more solid ground!
JavaScript within a browser context operates on a single platform that has a universal addressing system. And within that platform there are multiple non-programming languages, such as HTML and CSS, none of which support remapping URLs at the moment in a manner as it is proposed here, although slightlyoff/NavigationController will give that to us soon(ish).
(Speaking of misnomers... :-P)
I guess that's another point I had not really thought of, the web platform will get a way to execute some script at "fetch", which is basically a low-level version of the module loader.
Forcing coordination names to be URLs and then magically remapping them with another subsystem smells. Why shouldn't the system work, batteries included, in any JS embedding (Node, e.g., where there's no NavigationController)?
In our private mail exchange, the idea of syntactic disjunction:
import "name";
vs.
import url "http://foo.com/bar/baz.js";
with the contextual url keyword after import, came up. I'm not
endorsing it, just publicizing it. If we are almost in agreement, but
for the import syntax using a quoted string that's sometimes a URL and
sometimes not, then it ought to be considered.
I think one point that's being hinted at, but not explicitly called out, is the confusing nature of import "foo" in the proposed scheme. Notably, it shares this confusion with AMD, but not with Node.js.
The problem is that import "foo" can mean either "import the main module for package with name foo" or "import foo.js resolved relative to the base URL".
In AMD, this problem has been a constant headache on projects that I've worked on at my jobs. In Node.js, however, import "foo" always means the former, and never the latter—Node.js has no concept of "base URL." Instead it has the option of doing import "./foo", which has different semantics: "import foo.js relative to the module in which this code is found."
One can summarize the Node.js scheme as "if it starts with a dot, use relative resolution; otherwise, delegate to the package manager." This solves the problem nicely of preventing anyone from confusing module IDs with URLs, since although technically the start-with-dot subset could be interpreted as URLs, it's not common to see URLs like that.
I'm not sure the Node.js scheme is the best idea for the web, but I would like to emphasize that the AMD scheme is not a good one and causes exactly the confusion we're discussing here between URLs and module IDs.
On Tue, May 7, 2013 at 5:21 PM, Domenic Denicola <domenic at domenicdenicola.com> wrote:
I'm not sure the Node.js scheme is the best idea for the web, but I would like to emphasize that the AMD scheme is not a good one and causes exactly the confusion we're discussing here between URLs and module IDs.
I believe this is mixing the argument of allowing URLs in IDs with how logical IDs might be resolved in node vs AMD. Not all AMD loaders allow URLs as dependency string IDs, but requirejs does. That choice is separate from how logical IDs are resolved.
If normal logical ID notation is used with an AMD loader (like, 'some/thing', as in Node/CommonJS), then it is similar to node/CJS. It just has a different resolution semantic than node, which is probably an adjustment for a node developer. But the resolution is different for good reason. Multiple IO scans to find a matching package are a no-go over a network.
I think the main stumbling block in this thread is just "should module IDs allow both URLs and logical IDs". While I find the full URL a nice convenience (explained more below), I think it would be fine to just limit the IDs to logical IDs if this was too difficult to agree on. That choice is still compatible with AMD loaders though, and some AMD loaders make the choice to only support logical IDs already.
Perhaps the confusing choice in requirejs was treating IDs ending in '.js' to be URLs, and not a logical ID. I did this originally to make it easy for a user to just copy paste their existing script src urls and dump them into a require() call. However, in practice this was not really useful, and I believe the main source of confusion with URL support in requirejs. I would probably not do that if I were to do it over again.
However, in that do-over case, I would still consider allowing full URLs, using the other URL detection logic in requirejs: if it starts with a / or has a protocol, like http:, before the first /, it is an URL and not a logical ID.
That has been useful for one-off dependencies, like third party analytics code that lived on another domain (so referenced either with an explicit protocol, some.domain, or protocol-less, //some.domain.com), which is only "imported" once, in the main app module, just to get some code on the page. Needing to set up a logical ID mapping for it just seemed overkill in those cases.
James
Hmm. It's funny, you know, Python programs sometimes say
import osand I don't think anyone has ever claimed that the word "os" there is really just a URL or really just a filename.
There are at least two issues underlying this difference:
-
Other languages are generally free to define their own semantics for referencing external "things", whereas JavaScript (as embedded in the browser) already has such semantics in-place. Adopting different semantics will introduce cognitive dissonance.
-
Many other languages (compile-to-js aside) can assume access to a file system. They can assume a PATH variable and they can search for an external "things" in a multitude of places. There is a higher level of indirection between the logical path and the physical path.
On 7 May 2013 21:17, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Thu, May 2, 2013 at 10:47 AM, Andreas Rossberg <rossberg at google.com> wrote:
My point on the topic of external naming is that the language (1) should not prescribe any specific naming scheme; (2) should not willfully violate URI semantics; (3) should properly separate it from internal naming.
(1) No "naming scheme" is prescribed here, unless you mean the semantics of relative paths
Well, of course it is. You are prescribing the syntax of logical names in the form of "module IDs". It's a very specific (and kind of arbitrary) choice. I'd prefer if we did not bake that into the language beyond saying "the syntax for external module names is URIs, a loader can interpret that in whatever way it sees fit".
(2) As I pointed out in the mail I just wrote in reply to Anne, nothing violates URI semantics here. Logical names are not URIs, and thus treating them differently than URIs doesn't violate URI semantics.
I disagree, for the reasons I explained.
(3) Internal naming in the sense you mean is primarily a concern for organizing code within a single lexical scope. This is a real issue, but it's not nearly as pressing as the other issues here, and I think we can safely defer it.
I think it is pressing. Furthermore, it will make parts of the current design redundant. More on that in another reply.
Package managers are great, and I agree that they will (and should) become part of the eco system. But leave design decisions to them, and don't limit their options a priori. I'm not sure why we even disagree on that.
I think we agree on this.
I think we still disagree, see above.
More specifically, everything should still play nice with standard web mechanisms. For me, the web platform implies that I should be able to address modules as remote resources (it's a choice not to use that mechanism). That requires that the system allows proper URLs as module identifiers.
Fully agreed, and if it wasn't clear, the current system does completely support this. You can use a URL as a module name. In the default browser loader, you can choose to import a module from an explicit URL:
import { $ } from "http://my.cdn.com/jquery/1.9";and it will use that URL both as the logical name and as the place to go and fetch the bits.
OK, great! Can I also use such URLs for module definitions then?
I don't know what you're asking here.
In line with what I said above, string-named module declarations (if we need them at all) should allow arbitrary URI syntax, and not prescribe some specific form of logical name. It wasn't clear from Dave's response (or any of the design notes) whether you intend that to be the case or not. For symmetry, I hope you do. :)
I don't see how it hurts interop. It just means that you have to find a convenient and proper way to embed a logical name space into URLs. I proposed using a schema, but anything that is compatible with URL semantics is OK. Just taking AMD-style strings verbatim is not.
It hurts interop because AMD-style modules are named as "a/b" and not "jsp:a/b". A required prefix would therefore require impedance matching somewhere.
No, why? You are just embedding the AMD namespace into the URI namespace. It's clear where you use one or the other. The loader plug-in for AMD would perform the (trivial) bijection.
It is not something which you can justify with AMD/Node precedence either, since AFAICT these systems do not have the notational overloading between logical paths and URLs that you propose.
Both AMD and Node automatically convert module names like "a/b" into URLs (in the AMD case) and file paths (in the Node case).
That wasn't my point. As explained elsewhere, Node does not allow general URLs in the same syntactic context as module IDs, so does not have the same issue of syntactic overloading (I still have an issue with the distinction between "a" and "./a", but it's less serious outside a URL context). I thought the same was true for AMD, but I overlooked some of its API corner cases. Seriously, though, that does not make the idea any better, and AMD's rules are not something I'd ever want to put into a standard.
There's a reason why the existing systems favor lightweight name systems. They're nicely designed to make the common case syntactically lighter-weight. And they absolutely can co-exist with naming modules directly with URL's.
Sure they can coexist. But please not by bastardizing URL semantics.
Again, these are not URLs. You can't on the one hand complain that we aren't using URLs for names, and then on the other hand complain that our treatment of logical names doesn't follow the rules for URLs.
Is my complaint clearer now?
On 7 May 2013 21:48, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Thu, May 2, 2013 at 2:13 PM, Andreas Rossberg <rossberg at google.com> wrote:
On 1 May 2013 01:15, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
There are several reasons why module declarations are important.
Right now, every JS construct can be written at the REPL, in a script tag, in a bunch of files in a Node package, or anywhere else. This is a very important property for the language, and one we should maintain. Without module declarations except as separate files, we would lose this uniformity.
Being able to define multiple modules concisely in a single file is simply more convenient than having to separate them into multiple files. Even clunky old Java allows this simple convenience!
Serving modules as script tags, and defining modules in inline script tags, is likely to be important for a wide variety of use cases.
Concatenation.
[...]
But my broader point, of course, is that lexical modules address all these cases just fine.
My point was not to say that concatenation is a weak argument and that we need more. You agree that these are valuable features. However, concatenation support is a required feature. The world in which multiple http requests are free is not here, and not on the horizon.
You seem to believe otherwise, but I think you still need to explain how any of the above cases is not sufficiently (or even superiorly) supported by lexical modules + the loader API.
You keep asserting that lexical modules didn't work, but I think your reasoning why that necessitated the new design has been conflating two separate issues all along, and the conclusion is largely a non-sequitur.
I believe what you really mean is that using lexical names for coordination does not work well in general (at least not too well, cf. global scope). That is unsurprising. Rather, I was surprised to learn that you ever made this assumption. I always assumed that you were intending to use some form of higher-level URL mechanism for that. And the loader API had the necessary hooks for it.
In any case, I don't see how this observation necessarily implies anything for the form of module declarations. I suspect that your thinking had very much to do with (naive) concatenation, but concatenation without a tool will never be possible for modules anyway. So it really is a totally moot point.
OK, let's get concrete and see how this works. Assume you are writing some subsystem or library. There are three phases.
- Development. Most modules will typically be in individual files, that you import through external names. These names would either be relative paths or logical names, maybe depending on whether they cross library boundaries (I see both in e.g. the Node world). Logical names would most likely require some minimal loader configuration of the base URL.
Module declarations don't play a big role here, except if you want to define some local auxiliary modules somewhere. In that case, you are clearly better off with lexically scoped declarations, which (1) provide cheap, reliable references, (2) don't pollute the global namespace, (3) can be defined locally in other modules, and (4) don't require a separate import declaration.
- Deployment. Once you're ready to release, you want to bundle everything into one file ("concatenation"). Either way, that requires a tool.
With path-named module declarations, like in your design, the most basic version of such a tool needs to be given a set of JS module file paths and a base path. The resulting file contains the contents of all individual files, wrapped into module declarations whose names presumably are the tail fragments of the respective file paths relative to the base path -- or the logical names you were already using. That file is your deployed "package".
With lexical module declarations, the tool will be similar, but distinguishes the set of module paths that you intend to export from the package. The resulting file will again contain all individual files wrapped into module declarations, but this time with lexical names (generated by some means -- the details don't matter since there is no intention of making them accessible; you could even wrap everything into an anonymous module). All import file names referring to non-exported modules are replaced by their lexical references. In addition, the tool generates a couple of suitable calls to loader.set for registering the exported modules. The names for those are picked in the same manner as in the other approach.
The advantage of the latter approach is again that all intra-bundle references can be (1) cheap and reliable and (2) don't pollute the global namespace. At the same time, you are free to export as many of the modules as you like. So this approach is strictly more powerful, the user has control.
- Usage. In both approaches, the bundled file you have created contains a number of modules with external (logical) names. There are two ways to fetch them in another program: (a) either through a script tag, or (b) by calling loader.ondemand with the list of provided names. Both ways work equally for either of the designs!
So assuming this is the concatenation scenario you have in mind, where in this setup are path-named module declarations relevant? Where would the programmer prefer them over lexical declarations? (Sure you can come up with simple use cases where they provide some minor convenience, but none of these cases would scale to anything interesting, AFAICS.)
I hope your argument isn't that the bundling tool for the lexical approach is more complicated. Because it is only marginally so, and certainly far simpler than existing tools like e.g. AMD's optimizer. I probably could write you a simple version in half a day.
Some individual responses below.
This would be a coherent addition to the system, but is not a necessary one. Below, you point out some issues that would be potentially addressed by adding them. However, multiple JS module systems are in use today, and with the partial exception of the TypeScript namespaces feature (which is a significantly different design), none of them has lexically-named modules in the sense you advocate.
I do not think this observation is particularly relevant. As I said already, emulating a module system within JS necessarily has much more severe limitations than designing it as a language feature. In particular, I'm not sure you even could build a lexical module system under those constraints. There is no reason to apply those limitations to ES6 modules, though.
It's certainly possible to build a system like this in a compile-to-JS language like Typescript, and I don't believe it's happened.
Yeah, OK, clearly compilation is in a totally different ballpark.
Needing to explicitly register a module will certainly be common in a node-like system, where small modules are the norm. Which is the common case in general isn't a question we can answer today. But I don't think that needing to avoid registering will be common.
Pro JS code today certainly goes to great length to avoid polluting the global name space, by using anonymous functions. I think the situation is much more similar than you'd like it to be.
Mostly, this is managed by using a single namespace that everything hangs of, as with jQuery. This strategy is easy to apply to the module name space as well.
"Mostly" only if you have no intention of also replacing the common "module pattern" using IIFEs with real modules. I'd view the module system as a failure if it couldn't replace that.
But ultimately, it's simply a question of the right default. Making everything externally visible, let alone overridable, is, plain and simple, the wrong default. I'm still puzzled why you'd argue otherwise, since for very good reasons, you made the opposite choice one level down, for the contents of a module. I fail to see the qualitative difference.
These are designed for different purposes. Modules are intended to be abstractions, module names are designed to help manage collections of modules.
Sorry, but no. It's the same thing. You bundle together a number of definitions, and some of those you want to make externally accessible and others not. Collections of modules in a package are the same in that regard as collections of values in a module, just on a larger scale.
While it's sad that everyone doesn't use the same filesystem ;), this is inevitable unless we force everyone to write both internal and external names explicitly for every module; that's clearly more user-hostile than the alternative.
But you do not want to do that for every module! In fact, you rarely need to explicitly define a module with an external name at all, at least as I envision it. You only want to do that for a module that you want to "export" from your current script, so to speak. Such "export" (i.e., globally registering a module you defined textually) should be a very rare operation to write manually (as mentioned, you don't usually do the equivalent in AMD, for example).
Anonymous module defintions are often used in AMD for modules that are implicitly named by the file they're in. In ES6, that simply doesn't require a wrapper.
Yes, my point? Thing is, in AMD experience, named module definitions are neither common, nor recommended. You use separate files. Why expect something different for ES6?
Right, you use separate files and that gives you a bunch of string-named modules registered in the global modules registry. That's then compiled to explicitly named modules all in one file. That this is similar to our design should be unsurprising, since they're working on the same problem, and both systems care about concatenation.
My point was that those explicitly named module declarations are all generated by a tool. So there is no need to support it by path-named declaration forms. The tool can just as well generate calls into the loader API, and nobody would care about the difference. See above.
On Tue, May 7, 2013 at 7:21 PM, Domenic Denicola < domenic at domenicdenicola.com> wrote:
I think one point that's being hinted at, but not explicitly called out, is the confusing nature of
import "foo"in the proposed scheme. Notably, it shares this confusion with AMD, but not with Node.js.The problem is that
import "foo"can mean either "import the main module for package with namefoo" or "importfoo.jsresolved relative to the base URL".In AMD, this problem has been a constant headache on projects that I've worked on at my jobs. In Node.js, however,
import "foo"always means the former, and never the latter—Node.js has no concept of "base URL." Instead it has the option of doingimport "./foo", which has different semantics: "importfoo.jsrelative to the module in which this code is found."
Here's what you would do under the proposal:
// import a module in the same package/project
import "./controllers" as controllers;
// import some other package
import "backbone" as backbone;
The surface syntax deliberately follows Node. The first import is relative and the second is absolute, within the tree of module names (not URLs; neither of those module names is a URL).
From: Jason Orendorff [jason.orendorff at gmail.com]
Here's what you would do under the proposal:
// import a module in the same package/project import "./controllers" as controllers; // import some other package import "backbone" as backbone;The surface syntax deliberately follows Node. The first import is relative and the second is absolute, within the tree of module names (not URLs; neither of those module names is a URL).
That is not my understanding of Sam's message at esdiscuss/2013-May/030553. Following those steps for "backbone" would, according to that message,
- Split the module name on "/", url-encode, and re-join (yielding
"backbone"). - Append ".js" to the module name (yielding
"backbone.js"). - Use the URL parser with the base URL to produce an absolute URL (yielding
"http://example.com/path/to/base/backbone.js"). - Pass the absolute URL to the fetch hook.
On Wed, May 8, 2013 at 1:03 PM, Domenic Denicola <domenic at domenicdenicola.com> wrote:
From: Jason Orendorff [jason.orendorff at gmail.com]
Here's what you would do under the proposal:
// import a module in the same package/project import "./controllers" as controllers; // import some other package import "backbone" as backbone;The surface syntax deliberately follows Node. The first import is relative and the second is absolute, within the tree of module names (not URLs; neither of those module names is a URL).
That is not my understanding of Sam's message at esdiscuss/2013-May/030553. Following those steps for
"backbone"would, according to that message,
- Split the module name on "/", url-encode, and re-join (yielding
"backbone").- Append ".js" to the module name (yielding
"backbone.js").- Use the URL parser with the base URL to produce an absolute URL (yielding
"http://example.com/path/to/base/backbone.js").- Pass the absolute URL to the fetch hook.
How is this in disagreement with what Jason said? His point is that if you're in the module "a/b/c", "./controllers" refers to "a/b/controllers", and "backbone" refers to "backbone". Once you have a module name, there's a default resolution semantics to produce a URL for the fetch hook, which you describe accurately.
From: samth0 at gmail.com [samth0 at gmail.com] on behalf of Sam Tobin-Hochstadt [samth at ccs.neu.edu]
How is this in disagreement with what Jason said? His point is that if you're in the module "a/b/c", "./controllers" refers to "a/b/controllers", and "backbone" refers to "backbone".
Ah, I see, there are two levels of translation! First from "non-canonical module IDs" to "canonical module IDs", which in this case means from "./controllers" to "a/b/controllers", and then another from "canonical module IDs" to URLs. It's confusing since there are two concepts of "base" in play: the current module's "canonical module ID" is used as a "base" when resolving "non-canonical module IDs", and the base URL is used when resolving "canonical module IDs" to URLs.
(Sorry for the heavy use of scare-quotes, but I wanted to make it clear I don't know the right names for things and am open to correction.)
But, even then, that seems at least somewhat at odds with what Jason said. He implied that "backbone" would resolve to the backbone package, and not to the URL "http://example.com/path/to/base/backbone.js". In particular, he contrasted "some other package" (Backbone) with "a module in the same package/project," which to me would be modules under "http://example.com/path/to/base/". I look forward to finding out which part I misunderstood :).
On Wed, May 8, 2013 at 2:08 PM, Domenic Denicola <domenic at domenicdenicola.com> wrote:
From: samth0 at gmail.com [samth0 at gmail.com] on behalf of Sam Tobin-Hochstadt [samth at ccs.neu.edu]
How is this in disagreement with what Jason said? His point is that if you're in the module "a/b/c", "./controllers" refers to "a/b/controllers", and "backbone" refers to "backbone".
Ah, I see, there are two levels of translation! First from "non-canonical module IDs" to "canonical module IDs", which in this case means from
"./controllers"to"a/b/controllers", and then another from "canonical module IDs" to URLs. It's confusing since there are two concepts of "base" in play: the current module's "canonical module ID" is used as a "base" when resolving "non-canonical module IDs", and the base URL is used when resolving "canonical module IDs" to URLs.
I would say "module name" and "relative module name" for the two terms you're looking for here.
But, even then, that seems at least somewhat at odds with what Jason said. He implied that
"backbone"would resolve to the backbone package, and not to the URL"http://example.com/path/to/base/backbone.js". In particular, he contrasted "some other package" (Backbone) with "a module in the same package/project," which to me would be modules under"http://example.com/path/to/base/". I look forward to finding out which part I misunderstood :).
I think what Jason meant by "a module in the same package/project" was just that it's something that you expect to be a part of your package, and thus would refer to with a relative name. It's important to use relative names for that so that your code can be dropped in anywhere and have the module references continue to work.
In contrast, usually you want to be using that global version of "backbone", not something specific to your library. Of course, you can bundle backbone, and refer to it with "./backbone" if that's what you want, but that's probably a less-common case.
From: samth0 at gmail.com [samth0 at gmail.com] on behalf of Sam Tobin-Hochstadt [samth at ccs.neu.edu]
In contrast, usually you want to be using that global version of "backbone", not something specific to your library. Of course, you can bundle backbone, and refer to it with "./backbone" if that's what you want, but that's probably a less-common case.
OK! So, this is the confusion. Because the semantics you gave resolve "backbone" to a specific URL, "http://example.com/path/to/base/backbone.js". To me that doesn't correspond at all to "the global version of Backbone". Unless I guess you are assuming projects are set up such that their root directory contains a bunch of main module files for all the packages they use? So a web dev's workflow is something like
index.html
backbone.js
chai.js
rsvp.js
lib/
entry.js
otherModule.js
packages/
backbone/
backbone.js
README.md
package.json
chai/
index.js
... lots of other JS files
package.json
rsvp/
index.js
promise.js
... lots of other JS files
package.json
And the root rsvp.js contains
import { resolve, Promise, ... } from "packages/rsvp/index"; // or "./packages/rsvp/index"
export { resolve, Promise, ... }
I guess a tool would be needed to generate all these delegating files that live in your root directory?
Was that the intent of the way your algorithm resolves "backbone", to move web devs toward such a structure?
No, we're not trying to prescribe a specific structure.
There's a default place to fetch files from, because there has to be some default. However, I expect that most developers will do one of the following (Jason listed these options earlier):
- Load a script tag with
module "backbone" { ... }in it. - Call System.load(some_url) to fetch some source code with
module "backbone" { ... }in it. - Use System.ondemand() to specify where to fetch "backbone" from.
- Use a more complex resolve hook, either written by them or from a loader like YUI, to map "backbone" to some file somewhere.
Of course, there's a sensible default, but for production sites that will likely not be the right choice.
From: samth0 at gmail.com [samth0 at gmail.com] on behalf of Sam Tobin-Hochstadt [samth at ccs.neu.edu]
There's a default place to fetch files from, because there has to be some default.
Why?
This is the core of my problem with AMD, at least as I have used it in the real world with RequireJS. You have no idea what require("string") means---is "string" a package or a URL relative to the base URL? It can be either in RequireJS, and it sounds like that would be the idea here. Super-confusing!
I think it would be better if import "string" simply did not work by default (threw an error at the resolution stage, for example), unless a named module was already loaded with that name. Then the only behavior that would work by default, when not using named modules, is relative module names, and maybe absolute URLs (URIs?).
This would make it much clearer that import "string" is the purview of named modules (= concatenated builds) or custom resolve hooks. When reading code that does import "backbone", you would know the string "backbone" has meaning only from some external agency, and is not something you should go hunt around on disk for next to your index.html.
On Wed, May 8, 2013 at 10:44 AM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
How is this in disagreement with what Jason said? His point is that if you're in the module "a/b/c", "./controllers" refers to "a/b/controllers", and "backbone" refers to "backbone". Once you have a module name, there's a default resolution semantics to produce a URL for the fetch hook, which you describe accurately.
For a developer coming from Node, this may be slightly new to them, and I think when Jason mentions "package", it may not all fit together with how they understand Node to work. Here is a shot at trying to bridge that gap:
Node resolves relative IDs relative to the path for the reference module, and not relative to the reference module's ID. This is a subtle distinction, but one where node and AMD systems differ. AMD resolves relative IDs relative to reference module ID, then translates that to a path, similar to what Sam describes above.
I believe Node's behavior mainly falls out from Node using the path to store the module export instead of the ID, and it makes it easier to support the nested node_modules case, and the package.json "main" introspection.
However, that approach is not a good one for the browser, where concatenation should preserve logical IDs not use paths for IDs. This allows disparate concatenated bundles and CDN-loaded resources to coordinate.
For ES modules and Node's directory+package.json main property resolution, I expect it would work something like this:
Node would supply a Module Loader instance with some normalize and resolve hooks such that 'backbone' is normalized to the module ID 'backbone/backbone' after reading the backbone/package.json file's main property that points to 'backbone.js'. The custom resolver maps 'backbone'/backbone' to the node_modules/backbone/backbone.js file. For nested node_modules case, Node could decide to either make new Module Loader instances seeded with the parent instance's data, or just expand the IDs to be unique to include the nested "node_modules" directory in the normalized logical ID name.
If 'backbone', expanded to 'backbone/backbone' after directory/package.json scanning, asked for an import via a relative ID, './other', that could still be resolved to 'backbone/other', which would be found inside the "package" folder. So I think it works out.
James
On Wed, May 8, 2013 at 11:35 AM, Domenic Denicola <domenic at domenicdenicola.com> wrote:
From: samth0 at gmail.com [samth0 at gmail.com] on behalf of Sam Tobin-Hochstadt [samth at ccs.neu.edu]
There's a default place to fetch files from, because there has to be some default.
Why?
This is the core of my problem with AMD, at least as I have used it in the real world with RequireJS. You have no idea what
require("string")means---is"string"a package or a URL relative to the base URL? It can be either in RequireJS, and it sounds like that would be the idea here. Super-confusing!
What part is confusing? Logical IDs are found at baseURL + ID + '.js', and if it is not there, then look at the require.config call to find where it came from.
By not having a default, it would mean always needing to set up configuration or specialized module loader bootstrap script to start a project, and still requires the developer to introspect a config or understand the loader bootstrap script to find things.
Why always force a config step and/or a specialized module loader bootstrap? There are simple cases that can get by fine without any configuration or loader bootstrap.
James
From: James Burke [jrburke at gmail.com]
On Wed, May 8, 2013 at 11:35 AM, Domenic Denicola <domenic at domenicdenicola.com> wrote:
This is the core of my problem with AMD, at least as I have used it in the real world with RequireJS. You have no idea what
require("string")means---is"string"a package or a URL relative to the base URL? It can be either in RequireJS, and it sounds like that would be the idea here. Super-confusing!
What part is confusing? Logical IDs are found at baseURL + ID + '.js', and if it is not there, then look at the require.config call to find where it came from.
This dual behavior is exactly what is confusing.
On Wed, May 8, 2013 at 3:59 PM, Domenic Denicola <domenic at domenicdenicola.com> wrote:
From: James Burke [jrburke at gmail.com]
On Wed, May 8, 2013 at 11:35 AM, Domenic Denicola <domenic at domenicdenicola.com> wrote:
This is the core of my problem with AMD, at least as I have used it in the real world with RequireJS. You have no idea what
require("string")means---is"string"a package or a URL relative to the base URL? It can be either in RequireJS, and it sounds like that would be the idea here. Super-confusing!What part is confusing? Logical IDs are found at baseURL + ID + '.js', and if it is not there, then look at the require.config call to find where it came from.
This dual behavior is exactly what is confusing.
This dual behavior is there in lots of systems. For example, Java looks first locally, then in the CLASSPATH, and all of that is mediated by the class loader.
On Wed, May 8, 2013 at 8:22 AM, Kevin Smith <zenparsing at gmail.com> wrote:
Other languages are generally free to define their own semantics for referencing external "things", whereas JavaScript (as embedded in the browser) already has such semantics in-place. Adopting different semantics will introduce cognitive dissonance.
You're saying we have no choice but to make users hard-code locations into every import site, because HTML. I disagree. We do have a choice. Require.js had a choice, and it went for abstract names.
Because in practice you need them. The real use cases and problems that motivate module names in all these other systems don't just vanish when you move to an environment with URLs.
What you've proposed is to have package loaders:
- invent new URL schemes for URLs that aren't actually locators; and/or
- treat URLs as names, XML-namespaces-style; and/or
- take module names that are URLs and process them contrary to URL semantics.
I don't think any of those options is a better fit for the Web than module names.
Many other languages (compile-to-js aside) can assume access to a file system. They can assume a PATH variable and they can search for an external "things" in a multitude of places. There is a higher level of indirection between the logical path and the physical path.
The lack of a fast, reliable filesystem makes some things (like CLASSPATH) less attractive, and other things (preloading, on-demand server-side concatenation, client-side caching, CDNs, etc.) more attractive. Internet access makes new things possible. I think the options that package loaders will want to explore for JS, vs. other languages, are more diverse, not less.
You're saying we have no choice but to make users hard-code locations into every import site, because HTML. I disagree.
That is not my position. My position has always been that if you want "logical names", then a reasonable way to do that is via a scheme:
import $ from "package:jquery";
There are others (e.g. a syntax-based solution like Brendan mentioned), but the important thing is that we don't overload the semantics that are already in place for URIs.
What you've proposed is to have package loaders:
- invent new URL schemes for URLs that aren't actually locators; and/or
- treat URLs as names, XML-namespaces-style; and/or
- take module names that are URLs and process them contrary to URL semantics.
I don't think that your list adequately represents my position. Let me try to restate.
I've proposed that module loaders have the freedom to specify whatever semantics they choose for URIs that appear in module specifiers, even if those semantics are questionable. For consistency, the default loader should not choose semantics that conflict with standard URLs.
And please do not represent my position by using the term "XML". Thanks : )
The lack of a fast, reliable filesystem makes some things (like CLASSPATH) less attractive, and other things (preloading, on-demand server-side concatenation, client-side caching, CDNs, etc.) more attractive. Internet access makes new things possible. I think the options that package loaders will want to explore for JS, vs. other languages, are more diverse, not less.
I agree with you 100%, of course. That is why, personally, I would rather us provide the ingredients for experimentation than try to bake something in : )
On Tue, May 7, 2013 at 7:01 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Tue, May 7, 2013 at 4:39 PM, Jason Orendorff <jason.orendorff at gmail.com> wrote:
Set aside absolute-url imports. Suppose we just dropped them. Would you still think that module names are URLs? If so, do you think about other languages in the same way?
I think it's weird to try to equate JavaScript with those languages.
Not weirder than trying to equate it to HTML and CSS, surely!
They operate on multiple platforms that do not share a universal
addressing system and therefore a layer of abstraction had to be invented to make it easier to work those languages across multiple platforms.
I don't think that's the reason those systems have abstract names for
packages/modules. All of these systems were designed for use on systems
with hierarchical filesystems supporting absolute and relative paths. Any
one of them could all have chosen to implement import by mapping the
package/module name directly to a filename in a dead-simple, one-to-one
way; but none did.
(Even C, a language designed 40+ years ago by people whose key technical virtue was their willingness to eliminate unnecessary abstractions, lets you configure #include paths at compile time.)
Are the different environments where people will deploy JS code really more uniform than where people deploy Java? I don't know. But surely not so much so that third-party packages should just go ahead and embed the locations of their dependencies.
I guess that's another point I had not really thought of, the web
platform will get a way to execute some script at "fetch", which is basically a low-level version of the module loader.
Yeah, these two efforts should meet and talk before long. Related ideas going around, for sure.
That is not my position. My position has always been that if you want "logical names", then a reasonable way to do that is via a scheme:
import $ from "package:jquery";
A possible alternative would be to switch the defaults
Kevin Smith wrote:
I agree with you 100%, of course.
Great!
That is why, personally, I would rather us provide the ingredients for experimentation than try to bake something in : )
D'oh!
By saying "batteries not included", you are telling developers they have to write a bootstrap <script> element at least. This is a deal-killer
for most people. It's severe "stop energy". It is a mistake.
Can we try to agree on having a sane built-in default, even if we split syntax inside the string or outside to avoid violating the most holy URL hand-grenade?
[sorry if you saw an earlier empty message - unknown keycombo!-(]
That is not my position. My position has always been that if you want "logical names", then a reasonable way to do that is via a scheme:
import $ from "package:jquery";
A possible alternative might be to switch defaults, using generic relative syntax (<scheme>:<relative>) to keep the two uses apart
while avoiding having to introduce a new scheme
import $ from "http:jquery"; // it's a URL, don't mess with it
import $ from "jquery"; // it's a logical name, do your thing
The default loader could still cache URL-based resources (permitting bundling), but should not impose non-URL semantics.
Claus
By saying "batteries not included", you are telling developers they have to write a bootstrap <script> element at least. This is a deal-killer for most people. It's severe "stop energy". It is a mistake.
Can we try to agree on having a sane built-in default, even if we split syntax inside the string or outside to avoid violating the most holy URL hand-grenade?
As I mentioned somewhere upthread, personal preferences aside, I think this is a reasonable position to take.
I think it would help, before proceeding down this path, if we could define explicitly what "batteries included" actually means. What is the list of requirements that we need to account for?
On May 8, 2013, at 7:39 AM, Andreas Rossberg <rossberg at google.com> wrote:
On 7 May 2013 21:17, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Thu, May 2, 2013 at 10:47 AM, Andreas Rossberg <rossberg at google.com> wrote:
My point on the topic of external naming is that the language (1) should not prescribe any specific naming scheme; (2) should not willfully violate URI semantics; (3) should properly separate it from internal naming.
(1) No "naming scheme" is prescribed here, unless you mean the semantics of relative paths
Well, of course it is. You are prescribing the syntax of logical names in the form of "module IDs". It's a very specific (and kind of arbitrary) choice. I'd prefer if we did not bake that into the language beyond saying "the syntax for external module names is URIs, a loader can interpret that in whatever way it sees fit".
There's something I fear may be getting missed in this conversation. Maybe you were aware but I think it's important to clarify.
The naming policy we're talking about here is purely for the web's default System loader, not baked into the core language. In the core language, the semantics of module names is that they are strings, nothing more. The core semantics doesn't care what's in those strings. IOW: "the syntax for module names is strings, a loader can interpret that in whatever way it sees fit."
Note that requiring them to be a URI would be strictly more of a policy.
The point I'm making is that logical names aren't URLs, they aren't specified to be URLs, and thus treating them differently isn't a violation of the meaning of URLs.
It is, if logical names (1) syntactically overlap with URI references, (2) are used in places where you also allow URI references, and (3) have a meaning different from (and incompatible with) the respective sublanguage of URI references. Unfortunately, your design is doing exactly that.
I think this is the crispest statement of your objection to the browser's module name syntax. I feel you are being absolutist: the syntax is simply the (disjoint!) union of absolute URI's and a separate production (one that happens to overlap with URI references, but they are simply not part of this language). This is perfectly well-defined. It's simply a different grammar than that of general URI's. It just happens to overlap in some ways.
Now, you can argue that that design is confusing, because the presence of absolute URL's might lead a user to conclude incorrectly that something that isn't an absolute URL has the semantics of a URI reference. I think that's a fair objection, as far as it goes (although I personally disagree). You are overreaching, however, to say that it's a "violation" or suggest that it's some sort of fundamental unsoundness.
But let's look at the options here. We're talking about a syntax that allows for a user to specify either a logical name or a URL. So the syntax needs to accommodate those two cases. I see three basic categories of alternatives:
Option 1: distinguish the cases explicitly by injecting them into a common super-language e.g.: "{ 'path': 'a/b/c' }" and "{ 'url': 'example.com' }" e.g.: "jsp:a/b/c" and "example.com"
Option 2: distinguish the cases implicitly by exploiting their syntactic disjointness e.g.: "a/b/c" and "example.com"
Option 3: distinguish the cases explicitly by only injecting one case into a syntactically disjoint wrapper e.g.: "a/b/c" and "url(example.com)"
My issue with Option 1 is that it taxes the common case of logical names with extra boilerplate. Littering entire programs with jsp: at the beginning of every module name is just a non-starter. However, I actually find something like the example in Option 3 pretty appealing, since it doesn't tax the common case, and it uses a familiar syntax from CSS. It also makes it possible to admit relative URL's, which seems like a win.
In line with what I said above, string-named module declarations (if we need them at all) should allow arbitrary URI syntax, and not prescribe some specific form of logical name. It wasn't clear from Dave's response (or any of the design notes) whether you intend that to be the case or not. For symmetry, I hope you do. :)
They allow arbitrary string syntax, so yes, of course, you can define a module declaratively with a URI as its name.
(More replies to your next message in a few minutes...)
On May 8, 2013, at 8:05 AM, Andreas Rossberg <rossberg at google.com> wrote:
You seem to believe otherwise, but I think you still need to explain how any of the above cases is not sufficiently (or even superiorly) supported by lexical modules + the loader API.
The most important flaw of this is staging. The loader API lets you dynamically modify the registry, but those changes cannot be used by code compiled at the same time as the code that does the modification.
In any case, I don't see how this observation necessarily implies anything for the form of module declarations. I suspect that your thinking had very much to do with (naive) concatenation, but concatenation without a tool will never be possible for modules anyway. So it really is a totally moot point.
I think it's important not just to be able to write bundlers, but to be able to do the transformation in a fine-grained way. It should be possible to move module declarations around by hand, it should be possible for servers to make on-the-fly decisions about where to place module declarations, etc. And it should be possible to do this without greatly perturbing the shape of the resulting code.
Put differently, controlling where and when code is bundled together is something that sophisticated web applications do often and sometimes at fine granularity, and it's done by experts at web development who shouldn't have to be experts at writing compilers or reading their output. Consider Eric Ferraiuolo's examples of servers making decisions to speculatively bundle additional modules in a response to a request. These are decisions that are about network efficiency, and they shouldn't have to deal with code transformations at the same time.
OK, let's get concrete and see how this works. Assume you are writing some subsystem or library. There are three phases.
Development. Most modules will typically be in individual files, that you import through external names...
Deployment. Once you're ready to release, you want to bundle everything into one file ("concatenation"). Either way, that requires a tool.
You don't necessarily bundle everything into one file. Large-scale apps may bundle things in various ways.
With lexical module declarations, the tool will be similar, but distinguishes the set of module paths that you intend to export from the package. The resulting file will again contain all individual files wrapped into module declarations, but this time with lexical names (generated by some means -- the details don't matter since there is no intention of making them accessible; you could even wrap everything into an anonymous module). All import file names referring to non-exported modules are replaced by their lexical references. In addition, the tool generates a couple of suitable calls to loader.set for registering the exported modules. The names for those are picked in the same manner as in the other approach.
As I said above, this is broken. If we don't provide a declarative way to register modules, then they have to be added to the registry in a different stage from any code that uses them. This forces you to sequentialize the loading of different packages of your application, which is a non-starter.
What's more, when you start from your assumption #1 (and I do) that most modules will be in separate files, then lexical modules aren't buying the user much at all. Except for the cases where it makes sense to have small sub-modules that fit within a single file. As I've said before, I can imagine this being occasionally useful, but it's simply not important enough for ES6.
The advantage of the latter approach is again that all intra-bundle references can be (1) cheap and reliable
Sounds like you're misinterpreting the semantics of static linking. I have clarified this several times already. References between ES6 modules are not dynamically indirected through a dynamic table. They are statically resolved at link time. Linked modules are every bit as cheap and reliable as lexical references.
and (2) don't pollute the global namespace. At the same time, you are free to export as many of the modules as you like. So this approach is strictly more powerful, the user has control.
Except that as I explained above, without a way to register a module name declaratively, this forces you to load dependencies sequentially, which is unacceptable.
It also couples structural containment ("module X is conceptually part of package Y") with textual containment, which is strictly less flexible for controlling the delivery of the source over the wire.
- Usage. In both approaches, the bundled file you have created contains a number of modules with external (logical) names. There are two ways to fetch them in another program: (a) either through a script tag, or (b) by calling loader.ondemand with the list of provided names. Both ways work equally for either of the designs!
Doing it through a script tag is sequentializing. I don't understand how loader.ondemand works if the contents of the file has to do a dynamic loader.set. Once again, this sounds broken from a staging perspective.
A possible alternative might be to switch defaults, using generic relative syntax (<scheme>:<relative>) to keep the two uses apart while avoiding having to introduce a new scheme
import $ from "http:jquery"; // it's a URL, don't mess with it import $ from "jquery"; // it's a logical name, do your thing
Actually, that has a serious flaw for in-browser delivery, in that it would force naming a single scheme for relative URLs, whereas the same browser code can be delivered via several base schemes.
So, this option seems out, and I agree that burdening the common case of logical names with a new scheme (jsp:) isn't nice, either.
Currently, tagging the location refs as URLs (url(<url>)) appeals
most (unless there are conflicts hiding there, too).
Claus
My biggest concern is that people will assume a certain structure of the module path. Then they will assume that relative paths and logical names can be used interchangeably.
E.g. if I'm inside of package "a/b.js" I can access a top level package using either "../backbone.js" or "backbone". I may even use both patterns in the same package.
Most of the time it'll work because that file structure is common enough. That will eventually force everyone to use this file structure if they want to reuse modules that mix these patterns.
Java doesn't have this problem since there's no reasonable way to guess the CLASS_PATH nor use a relative path.
On 9 May 2013 08:15, David Herman <dherman at mozilla.com> wrote:
On May 8, 2013, at 7:39 AM, Andreas Rossberg <rossberg at google.com> wrote:
On 7 May 2013 21:17, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Thu, May 2, 2013 at 10:47 AM, Andreas Rossberg <rossberg at google.com> wrote:
My point on the topic of external naming is that the language (1) should not prescribe any specific naming scheme; (2) should not willfully violate URI semantics; (3) should properly separate it from internal naming.
(1) No "naming scheme" is prescribed here, unless you mean the semantics of relative paths
Well, of course it is. You are prescribing the syntax of logical names in the form of "module IDs". It's a very specific (and kind of arbitrary) choice. I'd prefer if we did not bake that into the language beyond saying "the syntax for external module names is URIs, a loader can interpret that in whatever way it sees fit".
There's something I fear may be getting missed in this conversation. Maybe you were aware but I think it's important to clarify.
The naming policy we're talking about here is purely for the web's default System loader, not baked into the core language. In the core language, the semantics of module names is that they are strings, nothing more. The core semantics doesn't care what's in those strings. IOW: "the syntax for module names is strings, a loader can interpret that in whatever way it sees fit."
Note that requiring them to be a URI would be strictly more of a policy.
Thanks for clarifying. I am aware of the distinction, but I wasn't actually sure whether we are on the same page. Because at some point, e.g. the module notes prescribed names of the form "a/b/c" in module declarations.
So yes, we are talking about the loader. And yes, the loader should use URIs. They are the natural choice, the most general mechanism, well established, and well-understood. In particular, the loader table should be indexed by fully resolved, absolute URIs. Everything else will just create a semantic mess.
The point I'm making is that logical names aren't URLs, they aren't specified to be URLs, and thus treating them differently isn't a violation of the meaning of URLs.
It is, if logical names (1) syntactically overlap with URI references, (2) are used in places where you also allow URI references, and (3) have a meaning different from (and incompatible with) the respective sublanguage of URI references. Unfortunately, your design is doing exactly that.
I think this is the crispest statement of your objection to the browser's module name syntax. I feel you are being absolutist: the syntax is simply the (disjoint!) union of absolute URI's and a separate production (one that happens to overlap with URI references, but they are simply not part of this language). This is perfectly well-defined. It's simply a different grammar than that of general URI's. It just happens to overlap in some ways.
I don't know your definition of "disjoint", but it doesn't seem to match mine. ;)
Now, you can argue that that design is confusing, because the presence of absolute URL's might lead a user to conclude incorrectly that something that isn't an absolute URL has the semantics of a URI reference. I think that's a fair objection, as far as it goes (although I personally disagree). You are overreaching, however, to say that it's a "violation" or suggest that it's some sort of fundamental unsoundness.
Look, consider the following names:
"a/b/c" "a/b/./c" "a/./b/c" "./a/b/c "a/b//c" "a/b/../b/c" "a/b/c.d" "a/b.d/c" "/a/b/c "http:a/b" "http:./a/b" "http:/a/b"
Elsewhere on the web, or (ignoring the last ones) in a file system, they would all be interpreted in a consistent manner (and moreover, the first 7 would all be deemed equivalent).
In your scheme, I honestly cannot tell. Which ones are absolute logical module names, which ones are relative logical module names, and which ones are relative URLs? Where does a path starting with "a" denote a relative URL reference and where doesn't it? When does it make a difference to add a "./" to a path? When is a path starting in "./a" relative to a referrer's module name and when is it relative to a referrer's URL? How can I write a relative URL when I need one? Or short, WAT?
Note that this is not only confusing for human readers, it can also be a pitfall for libraries or tools that rewrite module references, because the non-overlapping sublanguage of URLs won't be closed under some common transformations.
Also, why reinvent something URLs already do? AFAICT, your scheme introduces two completely separate notions of relative path, that are interpreted on different levels, and apply based on subtle syntactic differences. If you embed logical names into URIs, you get all that for free, and consistently so.
But let's look at the options here. We're talking about a syntax that allows for a user to specify either a logical name or a URL. So the syntax needs to accommodate those two cases. I see three basic categories of alternatives:
Option 1: distinguish the cases explicitly by injecting them into a common super-language e.g.: "{ 'path': 'a/b/c' }" and "{ 'url': 'example.com' }" e.g.: "jsp:a/b/c" and "example.com"
No new "super language", URIs are that language. They where designed to be!
Option 2: distinguish the cases implicitly by exploiting their syntactic disjointness e.g.: "a/b/c" and "example.com"
Option 3: distinguish the cases explicitly by only injecting one case into a syntactically disjoint wrapper e.g.: "a/b/c" and "url(example.com)"
My issue with Option 1 is that it taxes the common case of logical names with extra boilerplate. Littering entire programs with jsp: at the beginning of every module name is just a non-starter.
Every non-relative module import only. Calling that a non-starter seems to be your subjective opinion. To me, it adds clarity and prevents ambiguity.
By the same argument, I could complain that your scheme litters every relative import with ./ at the beginning. In some Node code I've seen, for example, that is the more common case.
However, I actually find something like the example in Option 3 pretty appealing, since it doesn't tax the common case, and it uses a familiar syntax from CSS. It also makes it possible to admit relative URL's, which seems like a win.
This option is certainly better than 2, but the CSS thing never struck me as elegant. Nor does it seem to be necessary or preferable here.
In line with what I said above, string-named module declarations (if we need them at all) should allow arbitrary URI syntax, and not prescribe some specific form of logical name. It wasn't clear from Dave's response (or any of the design notes) whether you intend that to be the case or not. For symmetry, I hope you do. :)
They allow arbitrary string syntax, so yes, of course, you can define a module declaratively with a URI as its name.
(More replies to your next message in a few minutes...)
(Have to reply to that one tomorrow.)
Put differently, controlling where and when code is bundled together is something that sophisticated web applications do often and sometimes at fine granularity, and it's done by experts at web development who shouldn't have to be experts at writing compilers or reading their output. Consider Eric Ferraiuolo's examples of servers making decisions to speculatively bundle additional modules in a response to a request. These are decisions that are about network efficiency, and they shouldn't have to deal with code transformations at the same time.
While we wait for Andreas' response, I would like to simply point out (without judgement, for now) that this amounts to inventing a declarative programming construct to solve network efficiency issues.
My biggest concern is that people will assume a certain structure of the module path. Then they will assume that relative paths and logical names can be used interchangeably.
E.g. if I'm inside of package "a/b.js" I can access a top level package using either "../backbone.js" or "backbone". I may even use both patterns in the same package.
Most of the time it'll work because that file structure is common enough. That will eventually force everyone to use this file structure if they want to reuse modules that mix these patterns.
If I understand your concern correctly, that is just another variation of the external reference/internal name overlap under discussion:
1 internal names, locally configured to map to external references, are the recommended practice (ie, avoiding the path structure dependencies you are concerned about is an education task)
2 some of us would like to ensure that both namespaces can be identified unambiguously (ie, from looking at the module name, it should be clear whether we're looking for an URL-referenced file on disk or on the web, or for a locally configured/registered name)
One advantage of not separating internal and external names is that you can take an existing project and reconfigure its module names, to compensate for changes in directory structure or web locations (even if the project imports directly from urls, you can rewrite and redirect the import references).
An advantage of separating internal and external names is that you can spell out best practices: avoid importing directly from external references such as "url!path/to/jquery" or "url!file:path/to/jquery"; instead import from internal names and configure those to point to external references. As long as the default loader passes through "url!<some url>" as "<some url>", no confusion is possible about
whether an import uses an internal name or an external one. And if best practices about keeping external references configurable are followed, getting two external projects to match is a question of matching their configuration data.
From a readability and maintenance perspective, I prefer the latter.
It would also help with explaining the configuration idea and best practices to JS coders. However, our module system champions currently prefer the former (everything is an internal name and can be configured).
Claus
On 9 May 2013 08:47, David Herman <dherman at mozilla.com> wrote:
On May 8, 2013, at 8:05 AM, Andreas Rossberg <rossberg at google.com> wrote:
You seem to believe otherwise, but I think you still need to explain how any of the above cases is not sufficiently (or even superiorly) supported by lexical modules + the loader API.
The most important flaw of this is staging. The loader API lets you dynamically modify the registry, but those changes cannot be used by code compiled at the same time as the code that does the modification.
I know. It just seems that it isn't actually a problem, because all interesting use cases require staging anyway. For example, you cannot use loader.ondemand without staging, and that's what you'll have to use for bundles. If loader.ondemand has the semantics I expected it to have (see below), that is all the staging you need to do manually.
In any case, I don't see how this observation necessarily implies anything for the form of module declarations. I suspect that your thinking had very much to do with (naive) concatenation, but concatenation without a tool will never be possible for modules anyway. So it really is a totally moot point.
I think it's important not just to be able to write bundlers, but to be able to do the transformation in a fine-grained way. It should be possible to move module declarations around by hand, it should be possible for servers to make on-the-fly decisions about where to place module declarations, etc. And it should be possible to do this without greatly perturbing the shape of the resulting code.
Can you explain how one form of module declaration is easier to "move around"? In a single script there surely is no difference.
If you are talking about moving modules across scripts/files, or even turning files into module declarations, then I don't see how that can possibly 'just work' in general, even in your approach. It seems you will need some support from your bundling/optimization/configuration tool, and/or you will need to adjust the set-up of your loader appropriately (i.e., set up the appropriate .ondemand entries). And once you have to go there, how does it make a difference?
Put differently, controlling where and when code is bundled together is something that sophisticated web applications do often and sometimes at fine granularity, and it's done by experts at web development who shouldn't have to be experts at writing compilers or reading their output. Consider Eric Ferraiuolo's examples of servers making decisions to speculatively bundle additional modules in a response to a request. These are decisions that are about network efficiency, and they shouldn't have to deal with code transformations at the same time.
Again, I don't see how any of that would be affected. Can you be more specific about what you think would not work? Even when done dynamically, the framework will need to know how to create bundles, and will have to implement a certain amount of rewriting. Moreover, nothing prevents you from creating a fully dynamic bundle, even in the approach I suggest -- it just doesn't force you to (and in most cases, you probably don't want it.)
On a more general note, I think you are trying to solve problems on the language level here for which that simply is the wrong attack vector. JavaScript cannot address web latency on its own. As I mentioned before, real applications like Gmail, which very definitely need to deal with these problems, won't be helped by a solution that can handle only JS resources.
OK, let's get concrete and see how this works. Assume you are writing some subsystem or library. There are three phases.
Development. Most modules will typically be in individual files, that you import through external names...
Deployment. Once you're ready to release, you want to bundle everything into one file ("concatenation"). Either way, that requires a tool.
You don't necessarily bundle everything into one file. Large-scale apps may bundle things in various ways.
Sure. For that reason, I was specifically talking about a library or subsystem. A bundle can still contain external references to modules outside that bundle.
With lexical module declarations, the tool will be similar, but distinguishes the set of module paths that you intend to export from the package. The resulting file will again contain all individual files wrapped into module declarations, but this time with lexical names (generated by some means -- the details don't matter since there is no intention of making them accessible; you could even wrap everything into an anonymous module). All import file names referring to non-exported modules are replaced by their lexical references. In addition, the tool generates a couple of suitable calls to loader.set for registering the exported modules. The names for those are picked in the same manner as in the other approach.
As I said above, this is broken. If we don't provide a declarative way to register modules, then they have to be added to the registry in a different stage from any code that uses them. This forces you to sequentialize the loading of different packages of your application, which is a non-starter.
I'm still not sure what your argument is. Yes, you have to use staging. But no, you cannot avoid it, declarative registration or not. In any use case involving partial bundling, you will always need staging. You have to use either separate script tags, which are staged by definition, or invoke loader.ondemand in a prior stage.
(Except in the edge case where you actually bundle everything in one file. In that case, obviously, you don't need staging either way.)
What's more, when you start from your assumption #1 (and I do) that most modules will be in separate files, then lexical modules aren't buying the user much at all. Except for the cases where it makes sense to have small sub-modules that fit within a single file. As I've said before, I can imagine this being occasionally useful, but it's simply not important enough for ES6.
I think I've explained the reasons at great lengths already at the beginning of this thread, and at various points in the middle. I'm happy to reiterate, but these posts are already getting long...
The advantage of the latter approach is again that all intra-bundle references can be (1) cheap and reliable
Sounds like you're misinterpreting the semantics of static linking. I have clarified this several times already. References between ES6 modules are not dynamically indirected through a dynamic table. They are statically resolved at link time. Linked modules are every bit as cheap and reliable as lexical references.
I don't think I'm misinterpreting. You are talking about linked modules. The problem is, in your approach, all modules start out as unlinked, and I can only link them through unreliable dynamic references. As I've pointed out in my OP, there is no way in which your proposal allows me to choose that the thing I link is the one I've actually defined on the previous line. That is bad for all the well-known reasons, and the global object story should make us wary. (So far, the only counter argument I have heard was "but a loader can mess with your source code anyway". But that's reductio ad absurdum, and could be used to argue against sane language semantics for anything.)
and (2) don't pollute the global namespace. At the same time, you are free to export as many of the modules as you like. So this approach is strictly more powerful, the user has control.
Except that as I explained above, without a way to register a module name declaratively, this forces you to load dependencies sequentially, which is unacceptable.
See above and below.
It also couples structural containment ("module X is conceptually part of package Y") with textual containment, which is strictly less flexible for controlling the delivery of the source over the wire.
Why should that cause any such coupling? You'd still be able to do the same sort of exports as before. You just don't have to.
- Usage. In both approaches, the bundled file you have created contains a number of modules with external (logical) names. There are two ways to fetch them in another program: (a) either through a script tag, or (b) by calling loader.ondemand with the list of provided names. Both ways work equally for either of the designs!
Doing it through a script tag is sequentializing. I don't understand how loader.ondemand works if the contents of the file has to do a dynamic loader.set. Once again, this sounds broken from a staging perspective.
OK, perhaps I am under wrong assumptions about the semantics of ondemand. I assumed that when you first import a module that has been registered via ondemand, the respective script will be executed, recursively, and then linking of the outer script continues, taking the updated loader environment into account.
If that assumption is true, then all the staging you need for my suggestion to work is already inherent in ondemand.
If my assumption is not true, then I have a couple of questions:
(1) What is the semantics of ondemand? When is the script body executed? (2) Why is that semantics needed? Or IOW, what would be the practical problem with the semantics I assumed? (3) Would you agree that, with the semantics I assumed, my suggestion would work? (4) Wouldn't the semantics I assumed be more consistent with the model that we have been discussing for imports from AMD-like module systems at the end of the last meeting?
I can think of one possible answer to (2), and that's cross-package mutual recursion. But I believe you agreed in March that that does not seem to be common practice, and there's no need to support it.
On Thu, May 9, 2013 at 4:42 AM, Sebastian Markbåge <sebastian at calyptus.eu>wrote:
My biggest concern is that people will assume a certain structure of the module path. Then they will assume that relative paths and logical names can be used interchangeably.
E.g. if I'm inside of package "a/b.js" I can access a top level package using either "../backbone.js" or "backbone". I may even use both patterns in the same package.
You could import "../backbone", a relative module name (note that it's not a relative URL; the proposed default loader never treats module names as relative URLs). It's just extra typing, though—the normalize hook will convert that to "backbone" right at the beginning of the import process, and from then on the two styles really are exactly identical. I don't think it could cause any portability problems.
Most of the time it'll work because that file structure is common enough.
That will eventually force everyone to use this file structure if they want to reuse modules that mix these patterns.
I don't think this could possibly happen under the proposal, because relative module names aren't URLs.
The pipeline is normalize -> resolve -> fetch -> translate -> link
and relative module names are processed at the first step, normalize. This is before the resolve step, which figures out the URL.
On May 9, 2013, at 11:33 AM, Kevin Smith <zenparsing at gmail.com> wrote:
Put differently, controlling where and when code is bundled together is something that sophisticated web applications do often and sometimes at fine granularity, and it's done by experts at web development who shouldn't have to be experts at writing compilers or reading their output. Consider Eric Ferraiuolo's examples of servers making decisions to speculatively bundle additional modules in a response to a request. These are decisions that are about network efficiency, and they shouldn't have to deal with code transformations at the same time.
While we wait for Andreas' response, I would like to simply point out (without judgement, for now) that this amounts to inventing a declarative programming construct to solve network efficiency issues.
That's just an outlandish statement, Kevin. I'm talking making a common refactoring semantically equivalent -- a refactoring that's well-motivated by a common use case: programmers solving their network efficiency issues by controlling which modules are loaded when. It's not some magical construct that automatically solves network issues.
On May 10, 2013, at 7:18 AM, Andreas Rossberg <rossberg at google.com> wrote:
Can you explain how one form of module declaration is easier to "move around"? In a single script there surely is no difference.
Clients of a module can write:
import { f } from "foo";
and regardless of how the module "foo" is shipped -- in a separate file or with an explicit module declaration in a bundled file -- the client code is unperturbed. This means that a single package can easily be deployed with any number of files without affecting client code.
OK, perhaps I am under wrong assumptions about the semantics of ondemand. I assumed that when you first import a module that has been registered via ondemand, the respective script will be executed, recursively, and then linking of the outer script continues, taking the updated loader environment into account.
OK, this is a big difference between your imagined system and the current one. The current system does not execute modules until dependencies have been fetched and linked -- I explained this in the March meeting. (The .ondemand method is sort of a distraction; it's nothing more than an API convenience layered upon the underlying resolution hook.) This means that cyclic dependencies work, and dependencies are resolved statically across multiple files, causing a full static dependency graph to be concurrently fetched.
What you describe has a sharp distinction between packages and modules, where packages cannot have cyclic dependencies, must be dynamically loaded and registered, and are composed only of lexically scoped modules. What this seems to mean is, to implement a package (such as a library, application, or component of an application), you have to choose from one of three options:
- implement the package in one big file.
- implement the package in multiple files via some extension to ECMAScript (e.g., include) that requires a tool to assemble it back together in a single file with only lexical modules.
- split the package into smaller packages, each comprising only one or at least very few modules, forgo any cyclic dependencies, and effectively get little to no benefit from lexical modules.
Please do tell me if I'm missing something, because all of the above scenarios seem obviously impractical. In particular, it's a necessity that the system should work well out of the box, with no additional tools. Obviously large and sophisticated applications will be willing to buy into additional infrastructure, but you should certainly be able to put a package's modules in separate files without build tools.
That's just an outlandish statement, Kevin. I'm talking making a common refactoring semantically equivalent -- a refactoring that's well-motivated by a common use case: programmers solving their network efficiency issues by controlling which modules are loaded when. It's not some magical construct that automatically solves network issues.
Sorry.
As far as I can tell, the only use case for module registrations that isn't adequately covered by lexical module declarations is multiplexing. Thanks for pointing that out. Of course, the general solution for multiplexing of resources on the web is at the network protocol layer. That is the end-game and from my point of view other techniques (i.e. image spriting) are stop-gaps.
It is possible to fill that gap without registration syntax using a small module loader plugin and a simple multiplexing tool, but I understand the desire to make such tooling unnecessary.
On Tue, May 7, 2013 at 2:12 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Tue, May 7, 2013 at 2:00 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
- If we have an absolute URL, skip steps 1-3.
How do you define this? We currently do not have this concept really.
I found a standard that defines this!
"An absolute URL is a URL url.spec.whatwg.org/#concept-url with a
scheme url.spec.whatwg.org/#concept-url-scheme." [1]
[1] URL living standard url.spec.whatwg.org. Anne van Kesteren,
editor.
On Tue, May 14, 2013 at 2:03 PM, Jason Orendorff <jason.orendorff at gmail.com> wrote:
I found a standard that defines this!
"An absolute URL is a URL with a scheme."
That doesn't mean we use this concept in the platform anywhere. (And also, what I pointed out earlier about "scheme:bits", base -> parser -> serializer -> something else, base is also still true for relative
schemes.)
On Tue, May 14, 2013 at 2:16 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Tue, May 14, 2013 at 2:03 PM, Jason Orendorff <jason.orendorff at gmail.com> wrote:
I found a standard that defines this!
"An absolute URL is a URL with a scheme."
That doesn't mean we use this concept in the platform anywhere. (And also, what I pointed out earlier about "scheme:bits", base -> parser -> serializer -> something else, base is also still true for relative schemes.)
Right, I understand and agree on both points, I just had to smile when I ran across that today and realized you wrote it.
Missing smiley in the post, sorry. Here it is... :-D
Better late than never, -j :
On Tue, May 14, 2013 at 10:16 PM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Tue, May 14, 2013 at 2:03 PM, Jason Orendorff <jason.orendorff at gmail.com> wrote:
I found a standard that defines this!
"An absolute URL is a URL with a scheme."
That doesn't mean we use this concept in the platform anywhere.
Dear Anne,
Though this does not directly bear on the module discussion at hand, I have tired of your continued spreading of misinformation regarding URIs. Surely you must be mistaken in your repeated assertions against the fixed notion of absolute URL? Perhaps you mean no decision procedure exists for relative-scheme URL absoluteness? I believe such a procedure does exist and you have already attempted to specify it.
In your own document, you define a "base URL" as an absolute URL url.spec.whatwg.org/#concept-base-url.
By your own example esdiscuss/2013-May/030557,
there are URLs with schemes that are not absolute. This contradicts your WHATWG document which purports to reflect browser implementations.
Though it esdiscuss/2013-May/030559
is not a mathematically constructive definition, a very concrete definition of absolute URI as universal fixpoint exists. I am certain that a constructive definition also exists. In fact, you have already specified one.
The primary use of "absolute URLs" is unambiguity (aside: many specs refer to URIs as if they are all absolute and use the term "relative URI" or "relative reference" for non-absolute URIs). They are widely used and their properties relied upon in many documents, programs, and interfaces. Your own parsing/normalizing/resolving procedure in your URL document only defines a representation for absolute URLs: "Parsing (provided it does not return failure) and serializing a URL will turn it into an absolute URL."
There are many standards that do not define relative resolution of URL references and instead require an absolute URL.
Many of these standards seem to get by with a reference to STD66's absolute URI ABNF production.
To wit:
WHATWGML defines www.whatwg.org/specs/web-apps/current-work/multipage/the-input-element.html#attr-input-type-keywords
the "url" input element type attribute as producing "An absolute URL" www.whatwg.org/specs/web-apps/current-work/multipage/states-of-the-type-attribute.html#url-state-(type=url).
Applications will rely on this type when processing.
WHATWGML mandates www.whatwg.org/specs/web-apps/current-work/multipage/links.html#other-link-types
that other link types having a COLON ":" must be absolute URLs.
WHATWGML's @itemprop tokens www.whatwg.org/specs/web-apps/current-work/#names:-the-itemprop-attribute
for typed items may be absolute URLs but not relative URL references.
RFC 4395 "Guidelines and Registration Procedures for New URI Schemes" requires tools.ietf.org/html/rfc4395#section-2.2 new schemes
define absolute syntaxes and avoid notation used by relative schemes for relative references.
XML namespaces, XPath, etc also use absolute URLs lists.w3.org/Archives/Public/xml-uri/2000Sep/0083.html.
RFC 3404 "DDDS" only tools.ietf.org/html/rfc3404#section-4.1
resolves absolute URLs.
RFC 2326 "RTSP" requests use tools.ietf.org/html/rfc2326#page-21 absolute URLs only.
RFC 5842 "WebDAV Binding Extensions" use absolute URIs in URI Mappings tools.ietf.org/html/rfc5842#section-1.1.
Maximally interoperable server implementations of the HTTP 1.1 (RFC 2616) Location header tools.ietf.org/html/rfc2616#section-14.30 emit only absolute
URLs.
RFC 4468 "Message Submission BURL Extension" defines a new SMTP command to retrieve data from an absolute IMAP URL tools.ietf.org/html/rfc4468#section-3.5.
RFC 6749 "OAuth 2.0 Authorization Framework", though troubled, requires absolute URLs for endpoints tools.ietf.org/html/rfc6749#section-3.1.2 as well as
extension grant types tools.ietf.org/html/rfc6749#section-4.5
and access token types tools.ietf.org/html/rfc6749#section-8.1.
RFC 6066 (proposed) "Transport Layer Security (TLS) Extensions: Extension Definitions" defines tools.ietf.org/html/rfc6066#section-5 client certificate URLs
that must be absolute.
RFC 6134 (proposed) "Sieve Extension: Externally Stored Lists" for email filtering defines the name of externally stored lists tools.ietf.org/html/rfc6134#section-2.5 as always an absolute
URI.
This is only a partial list from a short bout of research. I'm sure if you exert yourself, you will discover that you were mistaken.
Sincerely,
On 14 May 2013 02:37, David Herman <dherman at mozilla.com> wrote:
On May 10, 2013, at 7:18 AM, Andreas Rossberg <rossberg at google.com> wrote:
Can you explain how one form of module declaration is easier to "move around"? In a single script there surely is no difference.
Clients of a module can write:
import { f } from "foo";and regardless of how the module "foo" is shipped -- in a separate file or with an explicit module declaration in a bundled file -- the client code is unperturbed. This means that a single package can easily be deployed with any number of files without affecting client code.
I don't believe that is true. If the module is moved either from or to a bundle file (and what else should such a move consist of?) then you typically will have to do at least one of the following:
(1) Change the set-up of .ondemand calls. (2) Change the invocation of your bundling tool.
As soon as you have to go there, you've lost almost all advantages of the ID-based declaration form. Its assumed convenience doesn't scale to non-trivial scenarios.
OK, perhaps I am under wrong assumptions about the semantics of ondemand. I assumed that when you first import a module that has been registered via ondemand, the respective script will be executed, recursively, and then linking of the outer script continues, taking the updated loader environment into account.
OK, this is a big difference between your imagined system and the current one. The current system does not execute modules until dependencies have been fetched and linked -- I explained this in the March meeting. (The .ondemand method is sort of a distraction; it's nothing more than an API convenience layered upon the underlying resolution hook.)
Wait, I talked about the script registered for .ondemand itself, i.e. its toplevel, not the modules that it contains. The script itself is not a module.
And no, I don't think .ondemand is a distraction at all! It is the core mechanism to make bundling work (other than separate script tags). And that has considerable implications on module declarations themselves, which I'm trying to pin down.
This means that cyclic dependencies work, and dependencies are resolved statically across multiple files, causing a full static dependency graph to be concurrently fetched.
I agree about (cross-bundle) cyclic dependencies, but no package system I'm aware of in any language or system supports cross-package cycles. We seemed to agree that such recursion isn't relevant.
On the other hand, I don't see how parallel fetching is affected, nor how your statement about fetching the full dependency graph in parallel is ever true. Correct me if I'm wrong, but either way, you can fetch the ondemand script itself in parallel with other (already known) imports. And either way, any import that (used) modules defined in that script need will require another round of fetching.
The only real difference is at what point you execute the script itself, before that other round of fetching or later. A bundle script shouldn't usually have much toplevel code in it, so its execution shouldn't matter too much for the relevant use case, and I could see it happen in the middle. Also, we had been discussing interleaved execution for legacy imports already, so this does not seem different.
I do realise now, however, that it gets uglier when an import triggers multiple ondemand scripts at once, because then their execution would have to be sequentialised.
In any case, I'm afraid I still don't know what exact semantics you intend for ondemand. Can you please clarify? When is the script body executed? And what effect do updates to the loader during execution of that script have?
Fortunately, I think there is a fairly easy solution that obsoletes string-based declarations, while avoiding sequentialised execution (or execution at all) during fetches. More on that later.
What you describe has a sharp distinction between packages and modules,
Yes it has! And that's a feature. ;) Because:
- intra-package module references should be internal and fixed,
- inter-package module references need to be external and configurable,
- packaging should mean constructing a larger package from a set of smaller ones by turning (a subset of) its external references into internal ones.
(You are not tied to this scheme with what I propose, but that's what you usually want.)
where packages cannot have cyclic dependencies, must be dynamically loaded and registered, and are composed only of lexically scoped modules. What this seems to mean is, to implement a package (such as a library, application, or component of an application), you have to choose from one of three options:
- implement the package in one big file.
- implement the package in multiple files via some extension to ECMAScript (e.g., include) that requires a tool to assemble it back together in a single file with only lexical modules.
Why would that require an extension? Import from URL is just fine for that purpose. And when you deploy, you run the tool, see above.
- split the package into smaller packages, each comprising only one or at least very few modules, forgo any cyclic dependencies, and effectively get little to no benefit from lexical modules.
Please do tell me if I'm missing something, because all of the above scenarios seem obviously impractical. In particular, it's a necessity that the system should work well out of the box, with no additional tools. Obviously large and sophisticated applications will be willing to buy into additional infrastructure, but you should certainly be able to put a package's modules in separate files without build tools.
Counter question: what other options are available in the current design, under your no-tool and no-staging assumption? AFAICS, you can do either of the following:
-
Write script files with module declarations. Then you can concatenate naively, but you cannot import as is without staging (setting up .ondemand for each file).
-
Write module files. Then you can import without staging, but you cannot concatenate naively.
I don't envision many people would want ever to use the first option, it's neither natural nor convenient. Nor does it provide a significant advantage over what you do lexically. And the other option requires tool support. And that's what I have tried to argue in the last couple of posts: that your system does not really work without tools either. You cannot "concatenate" module files without a tool.
Once you commit to using a tool, whether that does marginally more rewriting or not is immaterial. At the same time, lexical modules make the result more robust (plus they provide the other advantages I've mentioned).
(Okay - never using i.e./e.g. again because I can't seem to keep it straight...)
It is possible to fill that gap without registration syntax using a small
module loader plugin and a simple multiplexing tool, but I understand the desire to make such tooling unnecessary.
FWIW, this is what I had in mind:
gist.github.com/zenparsing/5586837
Basically, the tool outputs lexical modules for each input file and then encodes a data structure at the end of the file which maps absolute URLs to character offsets, which the translate hook uses to populate a prefetch cache. The prefetch cache is consulted in the fetch hook.
Just a quick proof of concept that arbitrary multiplexing is possible (strictly speaking) without module registration syntax.
I'm still trying to make sense of the conflicting module naming design goals and usage consequences.
You seem to believe otherwise, but I think you still need to explain how any of the above cases is not sufficiently (or even superiorly) supported by lexical modules + the loader API.
The most important flaw of this is staging. The loader API lets you dynamically modify the registry, but those changes cannot be used by code compiled at the same time as the code that does the modification.
If loader/registry manipulation and module declaration happen in different stages, and we have sub-projects that both provide and consume common libraries, then we have a staging problem.
This happens when the loader configuration stage cannot refer to lexical names in the project loading stage.
As I said above, this is broken. If we don't provide a declarative way to register modules, then they have to be added to the registry in a different stage from any code that uses them. This forces you to sequentialize the loading of different packages of your application, which is a non-starter.
Replacing module name declarations with strings that are registered in-stage works around that problem, at the price of replacing scoped declaration with (compilation-time single-)assignment and storing/ referencing all local modules in the same (loader-)global registry.
Lexical module names and string-like module registry names fulfill different purposes, and trouble comes from trying to make one serve the uses of the other: the earlier modules design only had lexical names, which is awkward wrt configuration; the current design has registry entries, which is awkward wrt local references.
The way to avoid such awkward mismatches of provided concepts and use cases, then, seems to require both lexical modules and registry entries, as separate concepts, each with their own uses.
Since we also need to get external modules from somewhere, this leaves us with three levels of references:
-
module declarations and module references for import/export use lexical naming
module m { ... } // local declaration import { ... } from m // local reference
-
registry entries for module registration and reference
(a) use string-like naming
module m as "jquery" // non-local/loader registry entry module m from "jquery" // non-local/loader registry lookup
(b) use property-like naming
module m as Registry.jquery // non-local/loader registry entry module m from Registry.jquery // non-local/loader registry lookup
(c ) use modules of modules
export { jquery: m } to Registry // non-local/loader registry entry import { jquery: m } from Registry // non-local/loader registry lookup
-
external references use urls, here marked via plugin-style prefix, to separate from registry references
module m from "url!<jquery url>" // external resource reference
The main points of registry manipulation are that it happens before runtime and is single-assignment, so that it can affect the loader that is currently active.
I'm not sure that I'd call this declarative (to begin with, it seems order-dependent), and string names (2a) do not seem to be necessary, either - they just make it easy to embed URLs in names.
String names (2a) make registry entries look like external references, or rather, they put what looks like external references under control of the loader. There could be a convention (3) that "url!<url>" refers
to an external reference, via <url>, but -by design- all string-named
module references are configurable. If lexical module names (1) are not included, all module references are configurable.
Property names (2b) make registry entries look like lexical references, the only indication of (load-time) configurability being the "Registry." prefix. That is even more apparent in the import/export variation (2c).
No matter which of the three variations of (2) is used, the part about register-a-module is a little odd, and my variations are meant to highlight this oddity
module m as "jquery" // (2a)
module m as Registry.jquery // (2b)
export { jquery: m } to Registry // (2c)
Other variations would obscure the oddity, e.g., mixing definition and registration in a form that suggests (local) naming
module "jquery" { ... }
To illustrate the oddity a little further: if we consider a project SPa with sub-projects SP1 and SP2, whose modules need to use some common library like JQuery, we end up with two phases for SP:
phase 1: configure and register jquery (versions/locations) phase 2: load SP1 and SP2, do the SPa things
The proposed spec makes it possible to load configuration script and sub-projects in one go, because the configuration script modifies the loader that is used by the sub-projects. Which means that the phases have to be loaded in this order, and that re-configuration has to be an error to preserve single-assignment.
However, this only works because neither SP1 nor SP2 do their own configuration: without lexical names, there are no local modules, so the sub-projects would conflict over the common registry name; also, if the sub-projects weren't open wrt common library references, the configuration phase wouldn't affect them.
Now, what if we want to use SPa as a whole, perhaps together with another sub-project SPb? It seems we have to split SPa into its configuration and load phases again, so that
- we can re-configure common library uses between SPa and SPb
- we do not get conflicts wrt configurations in SPa and SPb
One common solution to this problem is to make the open-ness/ import-configurability explicit in the module system, by using module-level functions (standard example SML). We know how such systems can be made to work. I'm not sure what the suggested use pattern for ES6 modules with module path configuration is.
Claus
On May 15, 2013, at 10:42 AM, Andreas Rossberg <rossberg at google.com> wrote:
(1) Change the set-up of .ondemand calls. (2) Change the invocation of your bundling tool.
As soon as you have to go there, you've lost almost all advantages of the ID-based declaration form. Its assumed convenience doesn't scale to non-trivial scenarios.
No. You've missed the point. Configuration code is not the problem; there's at most a 1:1 relationship between modules and their configuration.
There is a 1:many relationship between a module and import declarations that refer to it. This is what matters: even if you reconfigure the app to change the way the module is shipped, every client module's import declaration should remain the same. This may include repackaging, file renaming, switching between CDN's, switching between versions, etc.
I do realise now, however, that it gets uglier when an import triggers multiple ondemand scripts at once, because then their execution would have to be sequentialized.
That's exactly what I meant. If you have to sequentialize the execution of the transitive package dependency graph, then you over-sequentialize the fetching of the transitive module dependency graph.
When is the script body executed? And what effect do updates to the loader during execution of that script have?
The execution semantics goes roughly like this: once all the module dependencies are computed and fetched, the linking process begins. If there are no module-factories (i.e., the AMD-style modules that require their dependencies to have been executed before they can compute their exports), linking is observably atomic. Then the bodies (script bodies and any as-yet unexecuted module bodies that have any clients importing from them) are executed sequentially. If, however, there are module-factories, the process is more interleaved: the system atomically links all declarative modules transitively required by each module factory, then executes those declarative modules, then executes the module factory, rinse, repeat.
When linking is atomic, loader updates don't matter. When the interleaving happens, loader updates can affect future linkage. It's not a good idea to be mucking the loader in the middle of initializing modules. It's better to place it at the very beginning of an application, before any importing starts happening.
- intra-package module references should be internal and fixed,
You keep making this claim as if there's some black-and-white distinction. When we pointed out that your intra-package references are not tamper-proof given the translate hook, you said "well but in practice..." So, that argument cuts both ways. In practice, intra-package references will not be messed with externally. They are only hookable at compile-time; once runtime starts they are completely hard-bound. And if there are any collisions, there will be a compile-time error anyway. This is a tempest in a teapot.
- implement the package in multiple files via some extension to ECMAScript (e.g., include) that requires a tool to assemble it back together in a single file with only lexical modules.
Why would that require an extension? Import from URL is just fine for that purpose. And when you deploy, you run the tool, see above.
It requires non-standard semantics if you want to allow cyclic module dependencies, and probably also if you want to preserve the same execution semantics as lexical modules.
Counter question: what other options are available in the current design, under your no-tool and no-staging assumption? AFAICS, you can do either of the following:
Write script files with module declarations...
Write module files...
I never said staging is bad, I said the staging in your solution is broken: it over-sequentialize the fetching of resources.
The requirement I'm talking about -- which is absolutely critical for the practicality and adoption of ES6 modules -- is that the system has to work well out of the box without tools. And the current system does. The only thing you have to change if you decide to repackage modules into scripts is a single piece of code staged before application loading that configures the locations of the modules. Note that this is how require.js works.
I don't envision many people would want ever to use the first option... And the other option requires tool support.
That's simply not true. Look at require.js:
http://requirejs.org/docs/api.html#config
With even just a reasonably convenient ondemand, you can easily move some or all of your modules into scripts by hand. Even without ondemand, the resolve hook is straightforward. (BTW I hate the name ondemand; it'll get renamed to something more declarative-sounding, like AMD's paths.)
Your suggestions would result in a system that is unusable without tools. That's not acceptable and it's not going to happen.
The requirement I'm talking about -- which is absolutely critical for the practicality and adoption of ES6 modules -- is that the system has to work well out of the box without tools. And the current system does. The only thing you have to change if you decide to repackage modules into scripts is a single piece of code staged before application loading that configures the locations of the modules. Note that this is how require.js works.
Does it really work so well out-of-the-box, though? Let's say that I want to develop an app which depends on some module M. Let's say that in the module dependency graph reachable from M, there are 15 other modules. So in order to get things working at all, I'll have to go out and find the source code for all 16 modules in this graph and place them (with the correct names) into my module base URL.
It seems to me that when the module graph scales to a certain size, a package manager (and name registry) is going to be essential to this design.
On the other hand, I think it is possible with URLs to create a system which truly does work out-of-the-box.
Let's imagine a world where publicly available modules are located at sites specifically designed for naming and serving JS modules. Call it a web registry. Intra-package dependencies are bundled together using lexical modules - the package is the smallest unit that can be referenced in such a registry. The registry operates using SPDY, with a fallback on HTTPS, so for modern browsers multiplexing is not a critical issue. In such a world, I can create a page:
<!doctype html>
<html>
<head>
<script async>
import Foo from "https://webregistry.org/foo";
</script>
<body></body>
</html>
And no matter how large "foo"'s dependency graph is, it just works without any configuration or copying of files.
Of course, long URLs aren't any fun to type and authors may want to localize external dependencies to avoid such typing:
// foo.js, in the current project
export * from "https://webregistry.org/foo";
// Referencing the localized dependency
import Foo from "foo.js";
By making the module specifier an absolute URL in this manner, we can statically analyze or load entire module graphs by looking at the source code alone. That's pretty neat.
(I've thought about, but did not include any versioning considerations in the above.)
On Mon, May 20, 2013 at 12:07 PM, Kevin Smith <zenparsing at gmail.com> wrote:
The requirement I'm talking about -- which is absolutely critical for the practicality and adoption of ES6 modules -- is that the system has to work well out of the box without tools. And the current system does. The only thing you have to change if you decide to repackage modules into scripts is a single piece of code staged before application loading that configures the locations of the modules. Note that this is how require.js works.
Does it really work so well out-of-the-box, though? Let's say that I want to develop an app which depends on some module M. Let's say that in the module dependency graph reachable from M, there are 15 other modules. So in order to get things working at all, I'll have to go out and find the source code for all 16 modules in this graph and place them (with the correct names) into my module base URL.
Or add a few <script> tags.
It seems to me that when the module graph scales to a certain size, a package manager (and name registry) is going to be essential to this design.
Here, you're saying the design needs a package manager to work well at scale, and calling it a flaw.
On the other hand, I think it is possible with URLs to create a system which truly does work out-of-the-box.
Let's imagine a world where publicly available modules are located at sites specifically designed for naming and serving JS modules. Call it a web registry. Intra-package dependencies are bundled together using lexical modules - the package is the smallest unit that can be referenced in such a registry.
Here, you're assuming a "web registry" and claiming that it makes your suggestion "work out-of-the-box".
The registry operates using SPDY, with a fallback on HTTPS, so for modern browsers multiplexing is not a critical issue.
Here, you're confusing SPDY with "a magic technique that makes HTTP requests free".
It seems to me that when the module graph scales to a certain size, a package manager (and name registry) is going to be essential to this design.
Here, you're saying the design needs a package manager to work well at scale, and calling it a flaw.
I wouldn't call it a flaw. But a requirement that has repeatedly surfaced in this discussion is that the system must work out-of-the-box, and therefore I think it's fair game to analyze just how well the "logical names" design works out-of-the-box.
Let's imagine a world where publicly available modules are located at sites
specifically designed for naming and serving JS modules. Call it a web registry. Intra-package dependencies are bundled together using lexical modules - the package is the smallest unit that can be referenced in such a registry.
Here, you're assuming a "web registry" and claiming that it makes your suggestion "work out-of-the-box".
Sure - and why not? The internet is the included battery. Obviously, I understand that there exists no such registry today, but everything is in flux.
On Mon, May 20, 2013 at 12:07 PM, Kevin Smith <zenparsing at gmail.com> wrote:
On the other hand, I think it is possible with URLs to create a system which truly does work out-of-the-box.
Let's imagine a world where publicly available modules are located at sites specifically designed for naming and serving JS modules. Call it a web registry. Intra-package dependencies are bundled together using lexical modules - the package is the smallest unit that can be referenced in such a registry. The registry operates using SPDY, with a fallback on HTTPS, so for modern browsers multiplexing is not a critical issue. In such a world,
There are lots of problems with this kind of URL-based IDs with a web registry, which I will not enumerate because they basically boil down to the problems with using URLs: URLs, particularly when version information gets involved, is too restrictive. The IDs need to have some fuzziness to make library code sharing easier. I have given some real world examples previously.
That fuzziness needs to be resolved, but it should be done once, at dependency install time, not for every run of the module code. "Dependency install time" can just mean, "create a file at this location", does not mandate tools.
At this point, I would like to see "only URLs as default IDs" tabled unless someone actually builds a system that used them and that system got some level of adoption.
If it was a great idea and it solved problems better than other solutions, I would expect it to get good adoption. However all the data points so far, ones from other languages, and ones from systems implemented in JS, indicate the URL choice is not desirable.
Note that this problem domain is different from something that needs new language capabilities, like the design around mutable slots for "import". This is just basic code referencing and code layout. It does not require any new magic from the language, it is something that could be built in code now.
Side note: existing HTML script tag use of URLs is not a demonstration of the success of URLs for a module system since they are decoupled from the references of code in JavaScript, and requires the developer to manually code the dependency graph without much help from the actual code.
Another side note: if someone wanted to use a web registry for library dependencies, it could just set the baseURL to the web registry location and have one config call to set the location for app-local code. It would end up with less string and config overhead than "only URLs as IDs". There has even been a prototype done to this extent: jspm.io -- it is backed by the module ID approach used for AMD modules/requirejs.
But again, these are all side notes. The weight of implementations, and real world use cases, indicate "only URLs as default IDs" are not the way to go.
James
Kevin Smith wrote:
Sure - and why not? The internet is the included battery. Obviously, I understand that there exists no such registry today, but everything is in flux.
Please! There is no magic pixie dust whereby the Internet solves the configuration problem for us.
Also, the configuration problem must be solvable locally (to the page), without hosted registries, and without writing custom loaders.
On May 9, 2013, at 6:30 AM, Andreas Rossberg <rossberg at google.com> wrote:
In your scheme, I honestly cannot tell. Which ones are absolute logical module names, which ones are relative logical module names, and which ones are relative URLs?
I realized I left this sub-thread hanging. While I think you've overstated your argument in several places, I do recognize that combining URL's and module names that look like paths into one syntactic space is confusing.
But really, there was no real need for loading directly from a URL in the first place, since it's better practice to use an abstract name and configure it to the URL you want anyway. (If people really want the additional convenience they can configure the loader to accept URL's.)
So the right resolution for this question is: the browser loader recognizes logical modules names only. No URI's, no URL's, just logical module name paths. If a particular module name needs to be loaded from a remote URL, you can use the ondemand configuration to map the logical name ("jquery") to the URL ("code.jquery.com/jquery-1.9.1.min.js").
Please! There is no magic pixie dust whereby the Internet solves the configuration problem for us.
No pixie dust was involved - just vision. If you would like to define exactly what you mean by "configuration problem", I would be happy to get specific.
I'm not entirely sure where the knee-jerk reaction is coming from, but this is what I'd like for anyone to take away from my previous post:
- Any claims that the "logical names" design works out-of-the-box is false advertising.
- A URL-based system using today's tech can provide superior out-of-the-box usability.
I've provided demonstrations for these claims above. If you have a concise counter argument, let's see it! As always, I am happy to be proven wrong.
On Mon, May 20, 2013 at 6:51 PM, Kevin Smith <zenparsing at gmail.com> wrote:
I've provided demonstrations for these claims above. If you have a concise counter argument, let's see it! As always, I am happy to be proven wrong.
No, what you provided was a "demonstration" where you complained about a registry for one system, and assumed it for the other. This is just bad-faith argumentation.
Kevin Smith wrote:
Please! There is no magic pixie dust whereby the Internet solves the configuration problem for us.No pixie dust was involved - just vision. If you would like to define exactly what you mean by "configuration problem", I would be happy to get specific.
Upthread, Sam's first real reply to Andreas:
- As a way for two separately developed components to coordinate about which module they mean.
"Coordination problem" would be another phrase for the same thing, but the problem is solved in any proposal (including yours) by configuration of a registry.
I'm not entirely sure where the knee-jerk reaction is coming from,
A short reply is not a knee-jerk reply. An appeal to unspecified, yet-to-be-created services on "the Internet" as the batteries in a batteries-included solution is an oxymoron. That's all.
but this is what I'd like for anyone to take away from my previous post:
- Any claims that the "logical names" design works out-of-the-box is false advertising.
Where did you demonstrate this?
- A URL-based system using today's tech can provide superior out-of-the-box usability.
URLs are locations. They're versioned, explicitly so in general (e.g. ajax.googleapis.com/ajax/libs/jquery/2.0.0/jquery.min.js).
URLs are great when you need them but they're not superior out of the box for solving the "coordination problem".
No, what you provided was a "demonstration" where you complained about a registry for one system, and assumed it for the other. This is just bad-faith argumentation.
First, I've shown that a package manager is necessary for non-trivial module graphs, and therefore we should not admit claims that the current design works out-of-the-box for non-trivial scenarios. That's all.
Second, I've attempted to show that:
-
Given the right infrastructure, with "logical names" the user must install a package manager, and then invoke the package manager to copy modules into the correct folder, ala NPM. This must happen before any code works.
-
Given the right infrastructure, with URLs the user can just reference a URL and everything works. Later in the development process, the user can invoke a package manager to copy everything to a local server and set up the necessary loader hooks.
I don't think that this line of reasoning constitutes bad faith, but I apologize if it seems that way. I would like to provide more details for my thoughts behind #2, but I'll do so through a separate link.
Thanks for your time,
On 18 May 2013 09:12, David Herman <dherman at mozilla.com> wrote:
On May 15, 2013, at 10:42 AM, Andreas Rossberg <rossberg at google.com> wrote:
(1) Change the set-up of .ondemand calls. (2) Change the invocation of your bundling tool.
As soon as you have to go there, you've lost almost all advantages of the ID-based declaration form. Its assumed convenience doesn't scale to non-trivial scenarios.
No. You've missed the point. Configuration code is not the problem; there's at most a 1:1 relationship between modules and their configuration.
There is a 1:many relationship between a module and import declarations that refer to it. This is what matters: even if you reconfigure the app to change the way the module is shipped, every client module's import declaration should remain the same. This may include repackaging, file renaming, switching between CDN's, switching between versions, etc.
Agreed to all of this. But it doesn't contradict what I said. I did not state that you shouldn't be using some form of logical name (at the package boundary). I said that "moving stuff around" is not free with them either.
If all you meant earlier is that the logical name as such doesn't need to change when you move the underlying definition then we're on the same page. But that is independent of how you actually register logical names.
The execution semantics goes roughly like this: once all the module dependencies are computed and fetched, the linking process begins. If there are no module-factories (i.e., the AMD-style modules that require their dependencies to have been executed before they can compute their exports), linking is observably atomic. Then the bodies (script bodies and any as-yet unexecuted module bodies that have any clients importing from them) are executed sequentially. If, however, there are module-factories, the process is more interleaved: the system atomically links all declarative modules transitively required by each module factory, then executes those declarative modules, then executes the module factory, rinse, repeat.
I see. I was thinking of ondemand scripts as "module factories". (Is that a new term? I haven't seen it used before.)
- intra-package module references should be internal and fixed,
You keep making this claim as if there's some black-and-white distinction. When we pointed out that your intra-package references are not tamper-proof given the translate hook, you said "well but in practice..." So, that argument cuts both ways.
Hold on. You and Sam are the ones who keep bringing up the black-and-white "but the loader can do anything anyway" argument.
My view is more differentiated: there is a spectrum of "tamper-proofness" or robustness or integrity. On that scale, lexical scoping > physical external names > logical names in a sandbox >
logical names in a world with free write access to the loader. At the high end, tampering is only possible for somebody with high-trust capabilities (who defined the loader), whereas at the low end, any dork can mess with everything.
It's good practice to place as much of your code as high in this spectrum as possible. Your proposal, however, puts everything at the bottom end. That's not only the default, it's the only possibility! Don't you think that's strictly worse?
In practice, intra-package references will not be messed with externally. They are only hookable at compile-time; once runtime starts they are completely hard-bound. And if there are any collisions, there will be a compile-time error anyway. This is a tempest in a teapot.
There is no generally useful distinction between compile time and run time in JavaScript. Because of staging, eval, and all that. That makes that a very weak argument.
As for the "in practice" part, we can just hope you are right. I, for one, see a lot of potential for accident and malice there.
- implement the package in multiple files via some extension to ECMAScript (e.g., include) that requires a tool to assemble it back together in a single file with only lexical modules.
Why would that require an extension? Import from URL is just fine for that purpose. And when you deploy, you run the tool, see above.
It requires non-standard semantics if you want to allow cyclic module dependencies, and probably also if you want to preserve the same execution semantics as lexical modules.
Sorry, I don't follow. What should be the problem with cyclic import through URLs?
Counter question: what other options are available in the current design, under your no-tool and no-staging assumption? AFAICS, you can do either of the following:
Write script files with module declarations...
Write module files...
I never said staging is bad, I said the staging in your solution is broken: it over-sequentialize the fetching of resources.
The requirement I'm talking about -- which is absolutely critical for the practicality and adoption of ES6 modules -- is that the system has to work well out of the box without tools. And the current system does. The only thing you have to change if you decide to repackage modules into scripts is a single piece of code staged before application loading that configures the locations of the modules.
If that single piece of code isn't entirely trivial then you don't want to maintain it by hand. You want to generate it, in the same way you want to, say, generate Makefile dependencies. Sure you can write them down manually, but not for an application with many modules in many packages.
All I'm saying is that "no tools" may work for small projects, but does not scale to anything more serious.
Note that this is how require.js works.
Require.js provides its "optimizer" for packaging. And AFAICT, that's what people use in practice. I don't see them writing bundles by hand much.
I don't envision many people would want ever to use the first option... And the other option requires tool support.
That's simply not true. Look at require.js:
http://requirejs.org/docs/api.html#configWith even just a reasonably convenient ondemand, you can easily move some or all of your modules into scripts by hand. Even without ondemand, the resolve hook is straightforward. (BTW I hate the name
ondemand; it'll get renamed to something more declarative-sounding, like AMD'spaths.)Your suggestions would result in a system that is unusable without tools. That's not acceptable and it's not going to happen.
I don't know what specific suggestion you are referring to. I have no intention of making the system less usable without tools. Rather, my point is that for anything of a certain size you want tools anyway. Certainly, you do not want to turn more than a couple of module files into a script manually. Why would you?
On 21 May 2013 03:41, David Herman <dherman at mozilla.com> wrote:
On May 9, 2013, at 6:30 AM, Andreas Rossberg <rossberg at google.com> wrote:
In your scheme, I honestly cannot tell. Which ones are absolute logical module names, which ones are relative logical module names, and which ones are relative URLs?
I realized I left this sub-thread hanging. While I think you've overstated your argument in several places, I do recognize that combining URL's and module names that look like paths into one syntactic space is confusing.
But really, there was no real need for loading directly from a URL in the first place, since it's better practice to use an abstract name and configure it to the URL you want anyway. (If people really want the additional convenience they can configure the loader to accept URL's.)
So the right resolution for this question is: the browser loader recognizes logical modules names only. No URI's, no URL's, just logical module name paths. If a particular module name needs to be loaded from a remote URL, you can use the ondemand configuration to map the logical name ("jquery") to the URL ("code.jquery.com/jquery-1.9.1.min.js").
Of course, that is not the "right" resolution in my mind, but the wrong one entirely. ;) Moreover, haven't you just pushed the problem to the ondemand API then? Or to "configured" loaders?
On Wed, May 22, 2013 at 11:27 AM, Andreas Rossberg <rossberg at google.com> wrote:
On 21 May 2013 03:41, David Herman <dherman at mozilla.com> wrote:
On May 9, 2013, at 6:30 AM, Andreas Rossberg <rossberg at google.com> wrote:
In your scheme, I honestly cannot tell. Which ones are absolute logical module names, which ones are relative logical module names, and which ones are relative URLs?
I realized I left this sub-thread hanging. While I think you've overstated your argument in several places, I do recognize that combining URL's and module names that look like paths into one syntactic space is confusing.
But really, there was no real need for loading directly from a URL in the first place, since it's better practice to use an abstract name and configure it to the URL you want anyway. (If people really want the additional convenience they can configure the loader to accept URL's.)
So the right resolution for this question is: the browser loader recognizes logical modules names only. No URI's, no URL's, just logical module name paths. If a particular module name needs to be loaded from a remote URL, you can use the ondemand configuration to map the logical name ("jquery") to the URL ("code.jquery.com/jquery-1.9.1.min.js").
Of course, that is not the "right" resolution in my mind, but the wrong one entirely. ;) Moreover, haven't you just pushed the problem to the ondemand API then? Or to "configured" loaders?
I recognize that it isn't the solution you want, but it is clearly a
solution, since it means there's no confusion between logical names
and URLs. They appear on different sides in ondemand.
On 22 May 2013 12:31, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Wed, May 22, 2013 at 11:27 AM, Andreas Rossberg <rossberg at google.com> wrote:
On 21 May 2013 03:41, David Herman <dherman at mozilla.com> wrote:
But really, there was no real need for loading directly from a URL in the first place, since it's better practice to use an abstract name and configure it to the URL you want anyway. (If people really want the additional convenience they can configure the loader to accept URL's.)
So the right resolution for this question is: the browser loader recognizes logical modules names only. No URI's, no URL's, just logical module name paths. If a particular module name needs to be loaded from a remote URL, you can use the ondemand configuration to map the logical name ("jquery") to the URL ("code.jquery.com/jquery-1.9.1.min.js").
Of course, that is not the "right" resolution in my mind, but the wrong one entirely. ;) Moreover, haven't you just pushed the problem to the ondemand API then? Or to "configured" loaders?
I recognize that it isn't the solution you want, but it is clearly a solution, since it means there's no confusion between logical names and URLs. They appear on different sides in
ondemand.
I suppose you are right for ondemand, but it doesn't apply to the "configuring the loader to accept URL's" case Dave was alluding to, right?
Andreas Rossberg wrote:
I suppose you are right for ondemand, but it doesn't apply to the "configuring the loader to accept URL's" case Dave was alluding to, right?
Finally I believe I've caught up and can jump in and answer: right! (Bracing for impact!)
However, the other way than ondemand to use a URL remains: <script src=...>.
On 22 May 2013 12:55, Brendan Eich <brendan at mozilla.com> wrote:
Andreas Rossberg wrote:
I suppose you are right for ondemand, but it doesn't apply to the "configuring the loader to accept URL's" case Dave was alluding to, right?
Finally I believe I've caught up and can jump in and answer: right! (Bracing for impact!)
However, the other way than ondemand to use a URL remains: <script src=...>.
Doesn't really help, since you can't load a module file through a script tag. Also, script tags introduce staging, which Dave considers essential avoiding.
On Wed, May 22, 2013 at 12:05 PM, Andreas Rossberg <rossberg at google.com> wrote:
On 22 May 2013 12:55, Brendan Eich <brendan at mozilla.com> wrote:
Andreas Rossberg wrote:
I suppose you are right for ondemand, but it doesn't apply to the "configuring the loader to accept URL's" case Dave was alluding to, right?
Finally I believe I've caught up and can jump in and answer: right! (Bracing for impact!)
However, the other way than ondemand to use a URL remains: <script src=...>.
Doesn't really help, since you can't load a module file through a script tag.
I don't understand what you mean here. A <script> tag can certainly
contain a module declaration. That's a big advantage of having syntactic support for module declarations.
Also, script tags introduce staging, which Dave considers essential avoiding.
No, Dave's point is that requiring staging is a bad idea. Having multiple script tags is a common practice, not a bad idea to be avoided.
On 22 May 2013 13:11, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:
On Wed, May 22, 2013 at 12:05 PM, Andreas Rossberg <rossberg at google.com> wrote:
On 22 May 2013 12:55, Brendan Eich <brendan at mozilla.com> wrote:
Andreas Rossberg wrote:
I suppose you are right for ondemand, but it doesn't apply to the "configuring the loader to accept URL's" case Dave was alluding to, right?
Finally I believe I've caught up and can jump in and answer: right! (Bracing for impact!)
However, the other way than ondemand to use a URL remains: <script src=...>.
Doesn't really help, since you can't load a module file through a script tag.
I don't understand what you mean here. A <script> tag can certainly contain a module declaration. That's a big advantage of having syntactic support for module declarations.
A <script src=...> cannot refer to a module, only a script
defining a module. The point of using URLs would be that you do the former, no?
Also, script tags introduce staging, which Dave considers essential avoiding.
No, Dave's point is that requiring staging is a bad idea. Having multiple script tags is a common practice, not a bad idea to be avoided.
Sure, I was referring to the context of the other subdiscussion about linking. In that context you want to avoid extra staging.
The module proposal has made good progress, thanks to the hard work by Dave & Sam. I'm glad to see it close to the home stretch for the ES6 race (some healthy minor controversies on the side notwithstanding :) ).
However, there is one central part of the design over which we still do not have agreement: the naming and declaration mechanism for modules. We did not yet have a serious discussion about it -- and I fully admit this being my fault as well, since despite my repeated criticism, I hadn't found the time to put that in a coherent form. I try to make up for that with this post. :) Summary is that I still (strongly) believe that adopting what's currently on the table would be a severe mistake, and that it's best to take a step back and discuss the motivation as well as alternatives.
My sincere apologies for the excessively long post...
** The problem **
In the original module proposal, modules were declared via ordinary lexical identifiers, but could also be imported from external sources denoted by strings. In November, the proposal was changed to use string-valued "module ids" for both. The motivation was to simplify the language, and to provide better support for configuration and concatenation.
Unfortunately, the new scheme leads to a significant conflation of concerns. In particular, it tries to obscure the fact that there are fairly fundamental differences between internal names and external ones:
As an analogy, take strings. A string may exist inside the language, as a value bound to a variable. Or it may be stored in some file that you can access via a file name. Both the variable and the file name ultimately reference "a string", but in completely different worlds. Nobody would suggest to use file paths in place of variable identifiers internally. Yet, that is almost exactly what the proposal does for modules!
Conflating the two notions may seem tempting and convenient at first, but it's bad. In the case of the current ES module proposal in particular, the attempt to do so with the "module id" scheme has all kinds of awkward consequences:
As an internal naming mechanisms, it lacks standard scoping semantics.
As various discussions show, people want and expect scope chain behaviour, and for very good reasons: e.g. nesting modules, confining modules to local scope, convenient local names, etc. The module id approach cannot sanely support that (which is why e.g. nested modules got pulled from the proposal).
As an external naming mechanisms, it violates standard relative path/URL semantics.
When using paths to actually reference external names, one might likewise expect certain semantics, e.g., that "a" and "./a" refer to the same thing, like they usually do on the web or in a file system. The current mechanism intentionally breaks with this via a non-standard interpretation of "paths". It implies that a set of module files, by default, is not relocatable within a project tree, or across project trees, as the standard idiom for module names actually denotes semi-absolute paths within a project tree. The main reason for it is that paths are overloaded to serve both internal and external naming. (The path semantics is inherited from legacy module frameworks for JS, such as AMD. It is a fine solution under the constraints that these frameworks have to operate in -- in particular, the inability to extend syntax or add new primitives. However, for ES6 most of these constraints don't apply, and there is no particular reason to limit the design to making the same compromises.)
The shared name space between internal and external modules can lead to incoherent programs.
Internal and external module definitions can resolve to the same path. For example. there might be a definition, somewhere, for module "a/b", but also a file "a/b.js". Which one takes precedence generally depends on the execution order of a (staged) program (e.g., when other imports are performed), and can, in fact, differ at different points in time. It is worth noting that, presumably for this reason, AMD strongly discourages the use of named module declarations, except by optimization tools (and Node does not support it at all, AFAICT). With the ES proposal, however, nifty syntax strongly conveys the impression that named module declarations are a good and recommended feature to use manually, instead of discouraging their day-to-day use.
Likewise, a single global name space for all internally defined modules can lead to incoherent programs.
Several internally defined modules can clash arbitrarily, there is nothing preventing two completely unrelated modules in completely unrelated files from clobbering the same name and stepping on each other's feet. Worse, there is no way to confine a module to a local scope, all definitions have to be globally visible and compete within the same global name space. (And unlike URLs, this name space is not particularly structured.) Of course, conventions can help to work around the problem in practice, but clearly, a well-designed language mechanism is preferable and more reliable.
"Local" references are globally overridable.
Since every reference goes through the loader, there is no way to have a definite (i.e., static, stable) module reference, even within a single script or scope. Any other script can come by and modify the meaning of any seemingly local module reference, either accidentally or intentionally (unless laboriously sandboxed). This clearly is bad for abstraction, encapsulation, integrity, security, and all related notions. And to add insult to injury, the loader also induces non-trivial runtime cost for a mechanism that isn't even desirable in these cases. Plain and simple, we repeat the mistake of the JS global object, but worse, because there is no other scope for modules to escape to if you care about integrity.
Internal module definition is coupled with external module registration, and modules cannot be renamed or re-registered.
It is not possible to define a module without registering it globally, and dually, it is not possible to register a module without defining a new one. In particular, that prevents renaming a module, and more importantly, registering a module under a different (external) name than the one under which it was defined/imported. The latter, however, is needed for some configuration use cases. (In the current proposal, it has to be simulated awkwardly by "eta-expanding" the renamed module, i.e., declaring "module 'A' { export * from 'B'; }", which creates a separate module.)
Language-level naming semantics interferes with file system/URL naming semantics.
As other communities have rediscovered many times before, it is a problem to naively map between internal and external names, because the meaning of internal definitions or references may then depend on idiosyncrasies of the hosting environment. For example, it may affect the program behaviour whether names are case-sensitive in the OS (not all JS hosts are browsers), or what makes a well-formed file name. What if you define a module "M" and also have a file "m.js"? Something else in node.js on Windows than on Linux, potentially. It is best to minimise the problem by limiting the use of external names to actual external references.
Bundling ("concatenation") generally would require embedding arbitrary string resources, not syntactic modules.
One powerful feature of loaders is the translation hook. When importing a module from an external source, there is no restriction on its syntactic content, since a translation hook can transform it freely. But if one were to bundle ("concatenate") an application that actually makes use of this liberty, then the current proposal could not actually support that consistently. Consequently, if one primary goal the current proposal is to make module declarations a mechanism for bundling modules, then they are an incomplete solution. In general, you'd need to be able to embed arbitrary resources (as strings), and be able to run translate hooks on those.
Module "declarations" are not a declarative mechanism, but an operational one.
Because module declaration is coupled with loader registration, it has a non-trivial operational effect, and its semantics is in turn subject to interference from other operational effects from inside and outside the program (as described above). Yet, it is disguised in a seemingly innocent declarative syntax. That is likely to create pitfalls and wrong assumptions (again, like with the global object).
In summary, not only do path-named module declarations lack desirable expressiveness, regularity, and integrity, they also do not support more interesting configuration and concatenation use cases, which is what they were intended for.
Most of the above problems cannot be fixed without adding lexically scoped modules to the language. It seems very clear that we need those, rather sooner than later. Also, I think we want a more structured approach to the global name space for external modules. At that point, rethinking the proposed approach may be the best idea.
** Proposal **
I think it is highly advisable to follow a simple strategy for the whole naming business: avoid ad-hoc inventions, stick to well-established, standard mechanisms.
Specifically:
Let me point out again that both lexical scope and URLs are backed by decades of experience and have proved superior over and over again, in many, many different languages and environments. We can only lose if we try to do "better".
More concretely, I envision using lexical module declarations as the primary means for defining modules (you guessed that from the start :) ). Modules can nest. (Supporting local modules, like Mark suggested, is more difficult, because of unclear interactions with other constructs like classes or eval. Certainly not ES6.)
There may (or may not, see below) be a (separate?) pseudo-declarative form for registering modules as resources with the loader -- however, if so, it should ideally scale to handle non-ES sources.
Module resources are identified by URLs. Those can be absolute or relative; relative URLs are interpreted relative to the importing file (just like HTML links). The loader table only contains absolute URLs. Likewise, every script is associated with its absolute URL. Any relative import is first normalised to absolute using the absolute URL of the importer (plus obvious steps for normalising occurrences of "." and "..").
A custom loader can, in principle, perform arbitrary interpretation or rewriting of URLs. In particular, this could be used to implement interop to absolute repository paths a la AMD or Node, e.g. by interpreting an "amd:" schema for importing AMD modules that are relative to a separately configured base URL. In other words, you'd write
import M1 from "a/b"; // native ES6 import, relative path import M2 from "amd:c/d"; // import of AMD module, relative to AMD base URL
Should we have a declarative form for registering resources, then its URL would be resolved in the same manner, relative to the path of the containing file. However, the programmer is free to use an absolute URL.
At that point, the only remaining purpose for path-named module declarations would be registering external references. However, registration is already possible through the loader API. Doing so requires additional staging (a script setting up the loader before executing the actual script). But staging is necessary anyway, for every slightly more interesting configuration case -- e.g. any scenario that involves loader.ondemand, translation, etc. It is not clear to me that the remaining cases justify an additional semantic short cut, and I think there is a case to be made that path-named module declarations are neither sufficient nor necessary. But that point is mostly independent from the rest.
Clearly, there are more details to be discussed and worked out. I also know that Dave & Sam have been over lots of it before. Still, I'm positive that it is a fairly solvable problem, and will yield a more well-behaved and more scalable solution. And yes, essentially it means reverting this part of the module proposal to its earlier stage -- but the good news is that not much else in the proposal is affected. :)
The module proposal has made good progress, thanks to the hard work by Dave & Sam. I'm glad to see it close to the home stretch for the ES6 race (some healthy minor controversies on the side notwithstanding :) ). However, there is one central part of the design over which we still do not have agreement: the naming and declaration mechanism for modules. We did not yet have a serious discussion about it -- and I fully admit this being my fault as well, since despite my repeated criticism, I hadn't found the time to put that in a coherent form. I try to make up for that with this post. :) Summary is that I still (strongly) believe that adopting what's currently on the table would be a severe mistake, and that it's best to take a step back and discuss the motivation as well as alternatives. My sincere apologies for the excessively long post... ** The problem ** In the original module proposal, modules were declared via ordinary lexical identifiers, but could also be imported from external sources denoted by strings. In November, the proposal was changed to use string-valued "module ids" for both. The motivation was to simplify the language, and to provide better support for configuration and concatenation. Unfortunately, the new scheme leads to a significant conflation of concerns. In particular, it tries to obscure the fact that there are fairly fundamental differences between _internal_ names and _external_ ones: - internal names reference _language entities_, external names reference _resources_; - internal references are under control of the language, while external references are delegated to a platform mechanism (with potentially platform-dependent interpretations and behaviour); - internal references are definite, stable, and high-integrity, while external references are indefinite, late-bound, and can fail or be clobbered. As an analogy, take strings. A string may exist inside the language, as a value bound to a variable. Or it may be stored in some file that you can access via a file name. Both the variable and the file name ultimately reference "a string", but in completely different worlds. Nobody would suggest to use file paths in place of variable identifiers internally. Yet, that is almost exactly what the proposal does for modules! Conflating the two notions may seem tempting and convenient at first, but it's bad. In the case of the current ES module proposal in particular, the attempt to do so with the "module id" scheme has all kinds of awkward consequences: * As an internal naming mechanisms, it lacks standard scoping semantics. As various discussions show, people want and expect scope chain behaviour, and for very good reasons: e.g. nesting modules, confining modules to local scope, convenient local names, etc. The module id approach cannot sanely support that (which is why e.g. nested modules got pulled from the proposal). * As an external naming mechanisms, it violates standard relative path/URL semantics. When using paths to actually reference external names, one might likewise expect certain semantics, e.g., that "a" and "./a" refer to the same thing, like they usually do on the web or in a file system. The current mechanism intentionally breaks with this via a non-standard interpretation of "paths". It implies that a set of module files, by default, is not relocatable within a project tree, or across project trees, as the standard idiom for module names actually denotes semi-absolute paths within a project tree. The main reason for it is that paths are overloaded to serve both internal and external naming. (The path semantics is inherited from legacy module frameworks for JS, such as AMD. It is a fine solution under the constraints that these frameworks have to operate in -- in particular, the inability to extend syntax or add new primitives. However, for ES6 most of these constraints don't apply, and there is no particular reason to limit the design to making the same compromises.) * The shared name space between internal and external modules can lead to incoherent programs. Internal and external module definitions can resolve to the same path. For example. there might be a definition, somewhere, for module "a/b", but also a file "a/b.js". Which one takes precedence generally depends on the execution order of a (staged) program (e.g., when _other_ imports are performed), and can, in fact, differ at different points in time. It is worth noting that, presumably for this reason, AMD strongly discourages the use of named module declarations, except by optimization tools (and Node does not support it at all, AFAICT). With the ES proposal, however, nifty syntax strongly conveys the impression that named module declarations are a good and recommended feature to use manually, instead of discouraging their day-to-day use. * Likewise, a single global name space for all internally defined modules can lead to incoherent programs. Several internally defined modules can clash arbitrarily, there is nothing preventing two completely unrelated modules in completely unrelated files from clobbering the same name and stepping on each other's feet. Worse, there is no way to confine a module to a local scope, all definitions _have to be_ globally visible and compete within the same global name space. (And unlike URLs, this name space is not particularly structured.) Of course, conventions can help to work around the problem in practice, but clearly, a well-designed language mechanism is preferable and more reliable. * "Local" references are globally overridable. Since every reference goes through the loader, there is no way to have a definite (i.e., static, stable) module reference, even within a single script or scope. Any other script can come by and _modify the meaning_ of any seemingly local module reference, either accidentally or intentionally (unless laboriously sandboxed). This clearly is bad for abstraction, encapsulation, integrity, security, and all related notions. And to add insult to injury, the loader also induces non-trivial runtime cost for a mechanism that isn't even desirable in these cases. Plain and simple, we repeat the mistake of the JS global object, but worse, because there is no other scope for modules to escape to if you care about integrity. * Internal module definition is coupled with external module registration, and modules cannot be renamed or re-registered. It is not possible to define a module without registering it globally, and dually, it is not possible to register a module without defining a new one. In particular, that prevents renaming a module, and more importantly, registering a module under a different (external) name than the one under which it was defined/imported. The latter, however, is needed for some configuration use cases. (In the current proposal, it has to be simulated awkwardly by "eta-expanding" the renamed module, i.e., declaring "module 'A' { export * from 'B'; }", which creates a separate module.) * Language-level naming semantics interferes with file system/URL naming semantics. As other communities have rediscovered many times before, it is a problem to naively map between internal and external names, because the meaning of internal definitions or references may then depend on idiosyncrasies of the hosting environment. For example, it may affect the program behaviour whether names are case-sensitive in the OS (not all JS hosts are browsers), or what makes a well-formed file name. What if you define a module "M" and also have a file "m.js"? Something else in node.js on Windows than on Linux, potentially. It is best to minimise the problem by limiting the use of external names to actual external references. * Bundling ("concatenation") generally would require embedding arbitrary string resources, not syntactic modules. One powerful feature of loaders is the translation hook. When importing a module from an external source, there is no restriction on its syntactic content, since a translation hook can transform it freely. But if one were to bundle ("concatenate") an application that actually makes use of this liberty, then the current proposal could not actually support that consistently. Consequently, if one primary goal the current proposal is to make module declarations a mechanism for bundling modules, then they are an incomplete solution. In general, you'd need to be able to embed arbitrary _resources_ (as strings), and be able to run translate hooks on those. * Module "declarations" are not a declarative mechanism, but an operational one. Because module declaration is coupled with loader registration, it has a non-trivial operational effect, and its semantics is in turn subject to interference from other operational effects from inside and outside the program (as described above). Yet, it is disguised in a seemingly innocent declarative syntax. That is likely to create pitfalls and wrong assumptions (again, like with the global object). In summary, not only do path-named module declarations lack desirable expressiveness, regularity, and integrity, they also do not support more interesting configuration and concatenation use cases, which is what they were intended for. Most of the above problems cannot be fixed without adding lexically scoped modules to the language. It seems very clear that we need those, rather sooner than later. Also, I think we want a more structured approach to the global name space for external modules. At that point, rethinking the proposed approach may be the best idea. ** Proposal ** I think it is highly advisable to follow a simple strategy for the whole naming business: avoid ad-hoc inventions, stick to well-established, standard mechanisms. Specifically: 1. Have a clean separation of concerns between internal and external names. 2. For internal names, use the standard language mechanism: lexical scope. 3. For external names, use the standard web mechanism: URLs. 4. Have a clean separation of concerns between the declarative _definition_ of a module, and the operational notion of _registering_ it with a loader. Let me point out again that both lexical scope and URLs are backed by decades of experience and have proved superior over and over again, in many, many different languages and environments. _We can only lose if we try to do "better"_. More concretely, I envision using lexical module declarations as the primary means for defining modules (you guessed that from the start :) ). Modules can nest. (Supporting local modules, like Mark suggested, is more difficult, because of unclear interactions with other constructs like classes or eval. Certainly not ES6.) There may (or may not, see below) be a (separate?) pseudo-declarative form for registering modules as resources with the loader -- however, if so, it should ideally scale to handle non-ES sources. Module resources are identified by URLs. Those can be absolute or relative; relative URLs are interpreted relative to the importing file (just like HTML links). The loader table only contains absolute URLs. Likewise, every script is associated with its absolute URL. Any relative import is first normalised to absolute using the absolute URL of the importer (plus obvious steps for normalising occurrences of "." and ".."). A custom loader can, in principle, perform arbitrary interpretation or rewriting of URLs. In particular, this could be used to implement interop to absolute repository paths a la AMD or Node, e.g. by interpreting an "amd:" schema for importing AMD modules that are relative to a separately configured base URL. In other words, you'd write import M1 from "a/b"; // native ES6 import, relative path import M2 from "amd:c/d"; // import of AMD module, relative to AMD base URL Should we have a declarative form for registering resources, then its URL would be resolved in the same manner, relative to the path of the containing file. However, the programmer is free to use an absolute URL. At that point, the only remaining purpose for path-named module declarations would be registering external references. However, registration is already possible through the loader API. Doing so requires additional staging (a script setting up the loader before executing the actual script). But staging is necessary anyway, for every slightly more interesting configuration case -- e.g. any scenario that involves loader.ondemand, translation, etc. It is not clear to me that the remaining cases justify an additional semantic short cut, and I think there is a case to be made that path-named module declarations are neither sufficient _nor_ necessary. But that point is mostly independent from the rest. Clearly, there are more details to be discussed and worked out. I also know that Dave & Sam have been over lots of it before. Still, I'm positive that it is a fairly solvable problem, and will yield a more well-behaved and more scalable solution. And yes, essentially it means reverting this part of the module proposal to its earlier stage -- but the good news is that not much else in the proposal is affected. :) /Andreas