Sam Tobin-Hochstadt (2013-04-25T22:27:54.000Z)
First, I appreciate you setting your thoughts down in detail.  I think
this will help us move forward in the discussion.

You write in a later message:

> Having said that, interoperability with existing module systems was
> not the main motivation for the change in the proposal, as far as I
> can tell. It rather was simplicity (which is debatable, as I hope I've
> made clear), and convenient support for common configuration and
> concatenation use cases.

I don't think this is right, and I think this is the heart of the
issue. Let me back up a bit to explain this.

Module names play a role in three processes, in general:

1. As a way to identify local components.
2. As a way to find the physical resource that is the source code (or
object code) of the module.
3. As a way for two separately developed components to coordinate
about which module they mean.

In the current design, the internal names (eg, "jquery") serve role 1,
and URLs (as generated by the loader hooks) serve role 2. The
coordination role is played by internal names in a shared registry.

To pick another example (at random :), take SML with CM [1].  Here,
lexically-bound module name serve role 1, and CM serves role 2.
Coordination is managed by the ML toplevel, and files are typically
open in the sense that they refer to modules that will ultimately be
bound at the top level during the compilation process.

The system you outline at the end of your message, and the similar
system that Dave and I originally proposed, doesn't have anything that
really fulfills role 3.  That's why we changed the design.

This is perhaps easiest to understand with an example.  Imagine you're
developing a library, which will depend on jQuery, but you're not
shipping jQuery with your library (this is how Ember works, as do
a lot of other libraries).  Then you have to write something like
this:

    // in ember.js
    import "jquery" as $;

That's what you'd write in the current design.

In the design you're proposing, there are a few choices.  You can give
up on allowing people choose where they load jQuery from, and hardcode
a dependency on a specific URL in the Ember source code.  This is
obviously not ok.

You could decide that instead there's a single well-known URL that
everyone is going to identify with jQuery.  This is basically using
the URL namespace as keys in the module registry of the current
design, except without separating logical names from URLs. Then your
program looks like this:

   // in ember.js
   module jquery = "http://code.jquery.com/jquery-1.9.1.js";
   import jquery as $;

Of course, even though that looks like a URL, it isn't being used to
fetch anything, and everyone has to use some form of loader hook to
map it to where they actually keep jQuery.  Moreover, not everyone
will have the resources or want to take the trouble to register a
domain to publish their library.

Or you could settle on a local URL to use in the same way, again as
basically a registry key.

   // in ember.js
   module jquery = "scripts/jquery.js";
   import jquery as $;

Again, this thing that looks like a URL isn't being fetched, instead
it will have to be rewritten to some other URL.  And again, we have
the same division into internal names that are in a registry, and
external URLs that are actually for fetching the resource, except now
because your proposal tries to abolish this distinction, it's much
harder to manage.

In other words, we are now manually simulating the registry, and it's
more painful and problematic for exactly the reasons that manually
simulating things on top of abstractions that don't support them
always is.

In contrast, the current design factors these two concerns, so that
this configuration is about the mapping from internal names (like
"jquery") to external names (like "http://....").

#### The key takeaway

The key takeaway here is this: in the current design, different
modules from different developers can coordinate based on the
*internal* names of their dependencies. In your lexical proposal, it's
not possible to coordinate globally on internal names, because they're
lexical.  So instead developers would have to coordinate on *external*
names.  This is fundamentally flawed, because external names are about
where to get bits, not which abstraction a name represents.

Supporting this use case properly is what led us to realize that the
earlier lexically-named proposal was flawed.

Note that none of this is about concatenation.  We've made some
particular design decisions where concatenation played a role, but it
wasn't a part of the reason we moved away from lexical modules.

#### Some smaller points

> * As an external naming mechanisms, it violates standard relative
> path/URL semantics.

This isn't done particularly to be compatible, and node or AMD could
have required "/" in front of absolute paths.  But since that's an
important use case, we don't think it's a good idea to tax it with
extra typing.  This convention has become popular in JS for a reason.

> * The shared name space between internal and external modules can lead
> to incoherent programs.

I've already pointed out above how these namespaces aren't shared, and
in fact their separation is an important reason for the current
design.  Also, the incoherent programs you refer to are ruled out by a
suggestion (of yours!) that we adopted at the last meeting:
declarative module forms that define a module that already exists are
a static error.

> * Likewise, a single global name space for all internally defined
> modules can lead to incoherent programs.

I would be more worried about this if (a) we didn't provide convenient
ways to structure this name space and (b) it wasn't already an
existing successful approach in real-world JS systems.

> * "Local" references are globally overridable.

This is only true in the sense that executing first lets you define
the meaning of particular modules.  But this is a *feature*.  This is
just another way to describing configuration.  It's not reasonable to
think that we can decide for everyone what will need configuration and
what won't.

> * Internal module definition is coupled with external module
> registration, and modules cannot be renamed or re-registered.

These are not "external" names, but it is true that declaring a module
registers it.  It's possible to take it out of the table afterward, or
give it a new name, or have it share multiple names, all by using the
loader API in very easy ways.  I don't think these will be common
operations, and this seems like a reasonable tradeoff.

> * Language-level naming semantics interferes with file system/URL
> naming semantics.

While it's sad that everyone doesn't use the same filesystem ;), this
is inevitable unless we force everyone to write *both* internal and
external names explicitly for every module; that's clearly more
user-hostile than the alternative.  Lots of languages (Java, Node,
...) manage a default mapping to the file system while making Windows
and Mac filesystems work sensibly.

> * Bundling ("concatenation") generally would require embedding
> arbitrary string resources, not syntactic modules.

The reason to use concatenation is to avoid consuming excessive client
resources -- in this setting of course you won't want to run
translation on the client side.  Translation hooks are important (a)
in less perf-sensitive settings like development and (b) for isolating
remotely-loaded code, neither of which require concatenation.

> * Module "declarations" are not a declarative mechanism, but an operational one.

This comes back to my original point.  Registration of module names is
about coordination, and thus this is an import feature, not a problem.

This is again long, but I hope it's clarified why the current design
is the way it is, and why the alternative you propose doesn't solve
the use cases we need to address.

Sam

[1] http://www.smlnj.org/doc/CM/ for those following along at home.
github at esdiscuss.org (2013-07-12T02:27:00.318Z)
First, I appreciate you setting your thoughts down in detail.  I think
this will help us move forward in the discussion.

You write in a later message:

> Having said that, interoperability with existing module systems was
> not the main motivation for the change in the proposal, as far as I
> can tell. It rather was simplicity (which is debatable, as I hope I've
> made clear), and convenient support for common configuration and
> concatenation use cases.

I don't think this is right, and I think this is the heart of the
issue. Let me back up a bit to explain this.

Module names play a role in three processes, in general:

1. As a way to identify local components.
2. As a way to find the physical resource that is the source code (or
object code) of the module.
3. As a way for two separately developed components to coordinate
about which module they mean.

In the current design, the internal names (eg, "jquery") serve role 1,
and URLs (as generated by the loader hooks) serve role 2. The
coordination role is played by internal names in a shared registry.

To pick another example (at random :), take SML with CM [1].  Here,
lexically-bound module name serve role 1, and CM serves role 2.
Coordination is managed by the ML toplevel, and files are typically
open in the sense that they refer to modules that will ultimately be
bound at the top level during the compilation process.

The system you outline at the end of your message, and the similar
system that Dave and I originally proposed, doesn't have anything that
really fulfills role 3.  That's why we changed the design.

This is perhaps easiest to understand with an example.  Imagine you're
developing a library, which will depend on jQuery, but you're not
shipping jQuery with your library (this is how Ember works, as do
a lot of other libraries).  Then you have to write something like
this:

```js
// in ember.js
import "jquery" as $;
```

That's what you'd write in the current design.

In the design you're proposing, there are a few choices.  You can give
up on allowing people choose where they load jQuery from, and hardcode
a dependency on a specific URL in the Ember source code.  This is
obviously not ok.

You could decide that instead there's a single well-known URL that
everyone is going to identify with jQuery.  This is basically using
the URL namespace as keys in the module registry of the current
design, except without separating logical names from URLs. Then your
program looks like this:

```js
// in ember.js
module jquery = "http://code.jquery.com/jquery-1.9.1.js";
import jquery as $;
```

Of course, even though that looks like a URL, it isn't being used to
fetch anything, and everyone has to use some form of loader hook to
map it to where they actually keep jQuery.  Moreover, not everyone
will have the resources or want to take the trouble to register a
domain to publish their library.

Or you could settle on a local URL to use in the same way, again as
basically a registry key.

```js
// in ember.js
module jquery = "scripts/jquery.js";
import jquery as $;
```

Again, this thing that looks like a URL isn't being fetched, instead
it will have to be rewritten to some other URL.  And again, we have
the same division into internal names that are in a registry, and
external URLs that are actually for fetching the resource, except now
because your proposal tries to abolish this distinction, it's much
harder to manage.

In other words, we are now manually simulating the registry, and it's
more painful and problematic for exactly the reasons that manually
simulating things on top of abstractions that don't support them
always is.

In contrast, the current design factors these two concerns, so that
this configuration is about the mapping from internal names (like
"jquery") to external names (like "http://....").

#### The key takeaway

The key takeaway here is this: in the current design, different
modules from different developers can coordinate based on the
*internal* names of their dependencies. In your lexical proposal, it's
not possible to coordinate globally on internal names, because they're
lexical.  So instead developers would have to coordinate on *external*
names.  This is fundamentally flawed, because external names are about
where to get bits, not which abstraction a name represents.

Supporting this use case properly is what led us to realize that the
earlier lexically-named proposal was flawed.

Note that none of this is about concatenation.  We've made some
particular design decisions where concatenation played a role, but it
wasn't a part of the reason we moved away from lexical modules.

#### Some smaller points

> * As an external naming mechanisms, it violates standard relative
> path/URL semantics.

This isn't done particularly to be compatible, and node or AMD could
have required "/" in front of absolute paths.  But since that's an
important use case, we don't think it's a good idea to tax it with
extra typing.  This convention has become popular in JS for a reason.

> * The shared name space between internal and external modules can lead
> to incoherent programs.

I've already pointed out above how these namespaces aren't shared, and
in fact their separation is an important reason for the current
design.  Also, the incoherent programs you refer to are ruled out by a
suggestion (of yours!) that we adopted at the last meeting:
declarative module forms that define a module that already exists are
a static error.

> * Likewise, a single global name space for all internally defined
> modules can lead to incoherent programs.

I would be more worried about this if (a) we didn't provide convenient
ways to structure this name space and (b) it wasn't already an
existing successful approach in real-world JS systems.

> * "Local" references are globally overridable.

This is only true in the sense that executing first lets you define
the meaning of particular modules.  But this is a *feature*.  This is
just another way to describing configuration.  It's not reasonable to
think that we can decide for everyone what will need configuration and
what won't.

> * Internal module definition is coupled with external module
> registration, and modules cannot be renamed or re-registered.

These are not "external" names, but it is true that declaring a module
registers it.  It's possible to take it out of the table afterward, or
give it a new name, or have it share multiple names, all by using the
loader API in very easy ways.  I don't think these will be common
operations, and this seems like a reasonable tradeoff.

> * Language-level naming semantics interferes with file system/URL
> naming semantics.

While it's sad that everyone doesn't use the same filesystem ;), this
is inevitable unless we force everyone to write *both* internal and
external names explicitly for every module; that's clearly more
user-hostile than the alternative.  Lots of languages (Java, Node,
...) manage a default mapping to the file system while making Windows
and Mac filesystems work sensibly.

> * Bundling ("concatenation") generally would require embedding
> arbitrary string resources, not syntactic modules.

The reason to use concatenation is to avoid consuming excessive client
resources -- in this setting of course you won't want to run
translation on the client side.  Translation hooks are important (a)
in less perf-sensitive settings like development and (b) for isolating
remotely-loaded code, neither of which require concatenation.

> * Module "declarations" are not a declarative mechanism, but an operational one.

This comes back to my original point.  Registration of module names is
about coordination, and thus this is an import feature, not a problem.

This is again long, but I hope it's clarified why the current design
is the way it is, and why the alternative you propose doesn't solve
the use cases we need to address.

Sam

[1] http://www.smlnj.org/doc/CM/ for those following along at home.