Integrating the Webs' dependency systems
I would therefore like to propose that we integrate these so that we end up with just one system that manages all three.
What do people think?
Absolutely. I've been meaning to start a similar discussion on the w3c lists, but I don't think I have quite the breadth of knowledge required. Thank you for bringing this up.
On Fri, 23 May 2014, Ryosuke Niwa wrote:
On May 23, 2014, at 3:14 PM, Ian Hickson <ian at hixie.ch> wrote:
- <script needs=""> in HTML
Could someone give me a pointer about this? This is the first time I've heard of this feature.
It's a work in progress, but the last public post from me about it was:
lists.w3.org/Archives/Public/public-whatwg-archive/2013Aug/0277.html
I'm currently going through feedback sent since then to work out how to proceed in its design; that's what led me to looking into the HTML Imports and ES6 Modules features.
The basic theme of <script needs=""> is making it possible for resources (probably more than just scripts) depend on each other and delay loads until they are needed. For example, being able to say "this script shouldn't download yet. When it is needed, though, it should also download and apply this style sheet, that image, that HTML Import, and that other script, and then once all of those are downloaded and applied, it should itself be applied". There's also related work into how to control the relative priority of downloads (e.g. needed now, needed after things that are needed now, not yet needed, not yet needed but may be precached).
I think this is a great discussion and it would be cool if it could get mirrored to the other working groups (specifically esdiscuss).
It would be great if WebComponents used the Loader and thus you could easily import a component into an es6 module and you could build a component that is entirely an es6 module that simply exports it's relevent bits. This would require something like <link rel="imports" type="module">
but I almost.
I'd love to be able to import css from an es6 module or import a template etc.
- Matthew Robb
On 5/23/14, Ian Hickson <ian at hixie.ch> wrote:
On Fri, 23 May 2014, Ryosuke Niwa wrote:
On May 23, 2014, at 3:14 PM, Ian Hickson <ian at hixie.ch> wrote:
Hi guys -
- <script needs=""> in HTML
Could someone give me a pointer about this? This is the first time I've heard of this feature.
It's a work in progress, but the last public post from me about it was:
lists.w3.org/Archives/Public/public-whatwg-archive/2013Aug/0277.html
CoR meets browser resources.
That looks familiar! Especially Use-case U. See: "defer on style, depends". lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2009-February/018439.html
The examples that appear linked to my prior site at dhtmlkitchen.com are not available anymore, unfortunately. That was when I was using Java, before it fell out of style. And incidentally, the CoR pattern proposal was inspired by Apache Ant. (credit where credit is due).
And can we change "needs=" back to "depends="?
The basic theme of <script needs=""> is making it possible for resources (probably more than just scripts) depend on each other and delay loads until they are needed. For example, being able to say "this script shouldn't download yet.
Shouldn't download or shouldn't be evaluated?
When it is needed, though, it should also download
and apply this style sheet,
Stylesheets, too? :-)
On Fri, 23 May 2014, Matthew Robb wrote:
I think this is a great discussion and it would be cool if it could get mirrored to the other working groups (specifically esdiscuss).
Assuming you mean this thread, it is. :-) (If you meant the HTML spec discussion regarding needs="" and company, that's more about HTML than scripting specifically, which is why it's happening on the WHATWG list.)
It would be great if WebComponents used the Loader and thus you could easily import a component into an es6 module and you could build a component that is entirely an es6 module that simply exports it's relevent bits. This would require something like <link rel="imports" type="module"> but I almost.
I'd love to be able to import css from an es6 module or import a template etc.
Could you elaborate on what it would mean for a module to import a template or style sheet? I'm not super-familiar with ES6 modules yet so I'm not sure exactly how this would manifest syntactically.
Abstractly, as far as I can tell, we can imagine the superset of the dependency parts of ES6 modules, HTML imports, and HTML in general as described by a model where some central processor has a list of items, along with a mechanism for adding new items to the list, and each item:
- has some means of identification so it can be deduped if it is added to the list multiple times
- has the ability to fetch its contents if not already available
- has the ability to be executed or applied
- has a list of other items that it depends on
- has a list of promises that it must wait for before being applied
- has a current state, probably one of: idle, downloading, waiting for dependencies, ready, applying/executing, loaded, error
- has some sort of execution policy, probably one of: block scripts until run, run asap, run when needed (i.e. when its ability to be executed is invoked); maybe some more esoteric policies like running only once anything that depends on it is about to be run itself
- has some sort of download policy, e.g.: download asap, download once the "asap" things are downloaded, download only when needed, download when needed but can be precached if there is free available bandwidth.
So for example, a style sheet imported by <link> today would be identified
by its URL, fetching its contents would use the "Fetch" specification (e.g. fetching over HTTP), applying it would mean adding it to the style sheet set of the current Document, it wouldn't depend on anything initially but if any @imports were found while it was being parsed then those would probably be dependencies, and it would have the policies of "run asap" and "download asap". (Once we have this model in place, it might make sense to add a feature to HTML to control these policies for each resource in the list; e.g., for <link rel=stylesheet>, an attribute.)
Similarly, an ES6 module would presumably map to this as follows: the identifier would be the module name, the ability to fetch the contents would be defined by the Loader (if I understand the spec right), what it depends on would be based on the "import" statements, and it would presumably have the policies of "run asap" and "download asap".
A <script src defer> element found during parsing could, in this model, be
described as an item that is waiting for the promise that describes the Document being parsed, and it would probably have the policies of "run asap" and "download once the 'asap' things are downloaded".
One could also imagine having <img whenneeded ...>, an image that would be
represented in this system as an item identified by the HTMLImageElement object, which can be told to download via some method e.g. img.fetch(), and whose policies would be "run when needed", "download when needed but can be precached".
A script could then tell the system to add this <img> element /
HTMLImageElement object as one of the dependencies for a script it was injecting, so that that script doesn't run until the image is downloaded. (Doing this, assuming the script is then told to run immediately, would be another way of telling the image it was needed. That's one reason to have the dependency system be abstract like this: it lets different parts of the platform participate by just exposing a generic interface, so that Web authors think we designed this all as one coherent whole.)
Does all that make sense? Did I miss some feature or behaviour that ES6 modules have that can't be described in the above?
What/where would be the best place to define such a system, and how would we hook all the specs together to use it?
A script could then tell the system to add this <img> element / HTMLImageElement object as one of the dependencies for a script it was injecting, so that that script doesn't run until the image is downloaded. (Doing this, assuming the script is then told to run immediately, would be another way of telling the image it was needed. That's one reason to have the dependency system be abstract like this: it lets different parts of the platform participate by just exposing a generic interface, so that Web authors think we designed this all as one coherent whole.)
Do you think that, in principle, a resource of a given type should be able to declaratively specify a dependency upon a resource of a different type?
So, for example, an ES module might hypothetically declare a dependency upon a CSS stylesheet and vice versa?
I think the critical attribute that such a dependency system must have is a universal syntax and semantics for identifying resources, across all resource types. In other words, all resources must be identified by URL/URI.
On Fri, May 23, 2014 at 5:04 PM, Ian Hickson <ian at hixie.ch> wrote
What/where would be the best place to define such a system,
Github.
and how would we hook all the specs together to use it?
Define a "System" object for ES6 that implements the ES6 Loader API and extends it to dynamically load CSS and HTML.
jjb
On Tue, May 27, 2014 at 8:09 AM, John Barton <johnjbarton at google.com> wrote:
On Fri, May 23, 2014 at 5:04 PM, Ian Hickson <ian at hixie.ch> wrote
What/where would be the best place to define such a system,
Github.
and how would we hook all the specs together to use it?
Define a "System" object for ES6 that implements the ES6 Loader API and extends it to dynamically load CSS and HTML.
jjb
This is basically what I was trying to say. Additionally I was suggesting that you also specify HTML Imports as using the System loader and thus the shared registry and dependency tree.
The most important problem to iron out with HTML Import + ES6 Loader is the timing confusion. Both systems use a combination of asynchronous and blocking-for-dependents loading. The issues could be explored with any of the existing ES6 pre-standard Loader implementations. (If the developer uses an ES6 Loader in an HTML Import, the registry will be automatically shared).
This is a great observation, often shared. I recall a related conversation about a year ago, that echoed a proposal from Yehuda a year prior yet (to which I hope Yeuhuda can exhume a link).
twitter.com/kriskowal/status/400703578605486080
And James Burke brought up some ideas on the topic.
gist.github.com/jrburke/7455354
It would be lovely if HTML could be trained to resolve URL’s through the module system. The module system may provide a level of indirection, so "x/y.css" might follow "x" into another package. Resolving ultimately to URL’s allows, as Alex Russel insists, keeps the issues of resolution, packaging, caching, bundling, web components, and service workers all orthogonal. The issue of how to implement packaging and module identifier resolution can, as John J. Barton points out, be left as a community exercise in custom implementations of System, e.g., one compatible with npm.
Kris Kowal
On Fri, 23 May 2014, Garrett Smith wrote:
And can we change "needs=" back to "depends="?
I haven't gotten as far as figuring out what the API should look like, so it's probably too early to bikeshed specific attribute names. :-)
The basic theme of <script needs=""> is making it possible for resources (probably more than just scripts) depend on each other and delay loads until they are needed. For example, being able to say "this script shouldn't download yet.
Shouldn't download or shouldn't be evaluated?
Both.
When it is needed, though, it should also download and apply this style sheet,
Stylesheets, too? :-)
Anything, ideally.
On Fri, 23 May 2014, Matthew Robb wrote:
On Fri, May 23, 2014 at 5:04 PM, Ian Hickson <ian at hixie.ch> wrote:
Could you elaborate on what it would mean for a module to import a template or style sheet? I'm not super-familiar with ES6 modules yet so I'm not sure exactly how this would manifest syntactically.
So my (albeit limited) understanding of ES6 Loader is that it is up to the environment to create a relevant implementation. This has been referred to as the System global loader. What I am proposing is that in the browser, dependencies for things like modules, components, templates, stylesheets, are all important for different reasons and in each of those contexts.
So the browser's System loader should have a method of loading any of these common browser dependencies. The only thing left to decide is what the exports of non-modules would be. For components it seems obvious by default they would export their components by global name.
document.register("x-foo", ...); and import { XFoo } from "path/to/component";
Ah so the ES6 "import" statement can import anything, even things that aren't ES6 modules?
It would certainly be cool if that was possible.
But you could go further and give components the ability to represent modules in script explicitly. You might also want to export all linked components from within a given component by default.
<link rel="imports" href="x-foo-button.html"/> <module> import { XFooButton } from "x-foo-button.html"; </module>
In the above case you would either implicitly or explicitly be able to do
<module> export XFooButton as "x-foo-button.html"; </module>
I hope any of what I just typed is understandable, if not I will try to put it in better words when I have more time.
Well presumably we wouldn't want to require that people write script to add things to the dependency system, right? I mean, we presumably would want the markup to also do the "export" step implicitly.
On Fri, 23 May 2014, Kevin Smith wrote:
A script could then tell the system to add this <img> element / HTMLImageElement object as one of the dependencies for a script it was injecting, so that that script doesn't run until the image is downloaded. (Doing this, assuming the script is then told to run immediately, would be another way of telling the image it was needed. That's one reason to have the dependency system be abstract like this: it lets different parts of the platform participate by just exposing a generic interface, so that Web authors think we designed this all as one coherent whole.)
Do you think that, in principle, a resource of a given type should be able to declaratively specify a dependency upon a resource of a different type?
I don't think this is a general principle, e.g. I don't think it makes much sense for us to extend the PNG spec to integrate with this.
But there are certainly multiple parts of the platform that want to declare dependencies on and from different resource types.
So, for example, an ES module might hypothetically declare a dependency upon a CSS stylesheet and vice versa?
If that is something that ES wants to be able to declare, I would be happy for us to support it.
I think the critical attribute that such a dependency system must have is a universal syntax and semantics for identifying resources, across all resource types. In other words, all resources must be identified by URL/URI.
I don't think a URL is the right way to identify everything. Many things in the Web platform that you could legitimately want to depend on don't have a URL. For example, a promise, or an inline <script> block, or an
HTML <video> element (the latter in particular would have several URLs,
the whole point would be to depend on the element so that things only happen once that element has figured out which file it wants to play and is ready to do so).
On Tue, 27 May 2014, John Barton wrote:
On Fri, May 23, 2014 at 5:04 PM, Ian Hickson <ian at hixie.ch> wrote
What/where would be the best place to define such a system,
Github.
I really meant which specification. As I see it we have four obvious places where we could concretely define the Web's dependency system:
- the ES spec
- the HTML spec
- the Fetch spec
- some new spec
(HTML Imports will eventually become merged with the HTML spec, so I haven't listed that separately.)
Putting it in ES would be fine by me (less work for me!). (I don't think it's a problem for HTML to depend on ES for something like this, even if not all browsers implement ES -- for example, the HTML spec already defines things in terms of the DOM and CSS specs, even for browsers that don't implement the DOM API or CSS. It just means the end-result should be as if the underlying spec was implemented, even if the rest of that underlying spec isn't implemented.)
So, what should we do here? Do people agree that the model I described here is the model that covers everything we need here?:
esdiscuss.org/topic/integrating-the-webs-dependency-systems#content-5
What would be the best spec to put this in? How should we ensure that all the specs that are to use this actually do use this? What interfaces should we have between the specifications to do this?
and how would we hook all the specs together to use it?
Define a "System" object for ES6 that implements the ES6 Loader API and extends it to dynamically load CSS and HTML.
Can you elaborate on this? I don't really follow.
On Tue, 27 May 2014, John Barton wrote:
The most important problem to iron out with HTML Import + ES6 Loader is the timing confusion. Both systems use a combination of asynchronous and blocking-for-dependents loading.
Right, that's why I think we should just have one processing model which both specs hook into. That way there'd only be one system using a combination of asynchronous and blocking-for-dependents loading.
On Tue, 27 May 2014, Kris Kowal wrote:
It would be lovely if HTML could be trained to resolve URL's through the module system.
By "HTML" here I presume you mean the underlying Fetch mechanism. Could you elaborate on exactly how this would work? Would this be in the layer before or after Service Workers?
@Ian, It seems like the first real question is, based on what will eventually be in the ES6 Spec for the Loader API, what is the System Loader as it pertains to the web/browser environment and is there potentially a need for a specification of it here (or at least outside of tc39).
- Matthew Robb
On Tue, May 27, 2014 at 3:04 PM, Ian Hickson <ian at hixie.ch> wrote:
On Tue, 27 May 2014, Kris Kowal wrote:
It would be lovely if HTML could be trained to resolve URL's through the module system. By "HTML" here I presume you mean the underlying Fetch mechanism. Could you elaborate on exactly how this would work? Would this be in the layer before or after Service Workers?
Supposing that a page has both a service worker and a custom loader, I would expect all requests for URL’s in the page (href, src, etc) to
- pass through the page’s loader
- pass through the browser's fetch mechanism
- pass through the service worker
- pass through the browser's fetch mechanism again
- possibly arrive at the true service 6… proxies all the way down?
Kris Kowal
I don't think a URL is the right way to identify everything. Many things in the Web platform that you could legitimately want to depend on don't have a URL. For example, a promise, or an inline <script> block, or an HTML <video> element (the latter in particular would have several URLs, the whole point would be to depend on the element so that things only happen once that element has figured out which file it wants to play and is ready to do so).
How would you identify such dependencies, within this system? (The identification must somehow allow de-duplication.)
On Tue, May 27, 2014 at 3:04 PM, Ian Hickson <ian at hixie.ch> wrote:
On Tue, 27 May 2014, John Barton wrote:
On Fri, May 23, 2014 at 5:04 PM, Ian Hickson <ian at hixie.ch> wrote
...
and how would we hook all the specs together to use it?
Define a "System" object for ES6 that implements the ES6 Loader API and extends it to dynamically load CSS and HTML.
Can you elaborate on this? I don't really follow.
TC39, the ES6 folks, have a partial spec for module loading. They have defined the language elements of a module system and part of the runtime system, a class called Loader. The instance of Loader used to actually load modules is not defined except that it is given a name, System. Loading configuration and customization will happen through the instance. (I believe the Loader is written up in the ES6 document but not official on the ES6 roadmap, whatever that means).
I believe the logic here is that each platform must define normalize/locate/fetch/translate, so the standard specification should also be done at the platform layer.
I think the Loader nicely isolates these particular functions: I don't see any urgency in standardizing them. However the delegation of the specification of System leaves us in the weird place of having everything except the actually external API spec-ed. Web folks could almost just say "Browsers shall have a window.System instance of Loader". There is a missing part of the Loader spec essential for Web and not very important for node: bundling for network transmission see systemjs/systemjs.
But to your original question about non-JS loading: you can extend the Loader class to add methods for CSS and HTML. These would be a cache in front of xhr for the most part, hence my suggestion that implementation (GitHub) is a good place to start. The next step, and one quite critical to practice, is integration with bundling. Here practical expertise is essential so again implementation would be a critical guide. There exist solutions in the pre-ES6 space to consult as well.
HTH, jjb
On Tue, 27 May 2014, Matthew Robb wrote:
@Ian, It seems like the first real question is, based on what will eventually be in the ES6 Spec for the Loader API, what is the System Loader as it pertains to the web/browser environment and is there potentially a need for a specification of it here (or at least outside of tc39).
I'm not sure I understand this question, but I'm happy for us to first answer this question, sure. :-)
How do we determine the answer to this question?
On Tue, 27 May 2014, John Barton wrote:
I think the Loader nicely isolates these particular functions: I don't see any urgency in standardizing them. However the delegation of the specification of System leaves us in the weird place of having everything except the actually external API spec-ed. Web folks could almost just say "Browsers shall have a window.System instance of Loader". There is a missing part of the Loader spec essential for Web and not very important for node: bundling for network transmission see systemjs/systemjs.
I don't understand this paragraph. What does it mean for us to not standardise Loader's functions? Is System something that we are expecting some non-ES spec, e.g. Fetch or HTML, to define?
But to your original question about non-JS loading: you can extend the Loader class to add methods for CSS and HTML. These would be a cache in front of xhr for the most part, hence my suggestion that implementation (GitHub) is a good place to start. The next step, and one quite critical to practice, is integration with bundling. Here practical expertise is essential so again implementation would be a critical guide. There exist solutions in the pre-ES6 space to consult as well.
This seems to imply that to do anything with this mechanism, someone (the page author?) has to provide some script code. However, it's very much my goal to be able to address use cases that don't involve any script at all (and that work even when the browser has scripting disabled), for example, I'd like whatever solution we develop here to be able to handle prioritising the loading and applying of different style sheets and images, e.g. so that images that are not currently rendered aren't downloaded yet, even though they are mentioned later in the page. One could imagine having markup like:
<link href="foo.css" rel=stylesheet load-policy=on-demand id="foo-styles"> ... <img src="foo.png" alt="..." load-policy=on-demand depends-on="foo-styles">
...where an <img> element is defined as being needed when it's rendered,
and where the browser would therefore only fetch the image and its stylesheet when the image is scrolled into view.
This would need to work without any script involved at all, but should the author later add script with its own dependencies, we would want all the script dependency stuff to use the same underlying system so that if the script needs foo-styles or the foo.png image, it would all work as one coherent whole.
On Tue, 27 May 2014, Kris Kowal wrote:
Supposing that a page has both a service worker and a custom loader, I would expect all requests for URL’s in the page (href, src, etc) to
- pass through the page’s loader
- pass through the browser's fetch mechanism
- pass through the service worker
- pass through the browser's fetch mechanism again
- possibly arrive at the true service [...]
This sounds like feedback for the Fetch spec. Anne?
On Tue, 27 May 2014, Kevin Smith wrote:
I don't think a URL is the right way to identify everything. Many things in the Web platform that you could legitimately want to depend on don't have a URL. For example, a promise, or an inline <script> block, or an HTML <video> element (the latter in particular would have several URLs, the whole point would be to depend on the element so that things only happen once that element has figured out which file it wants to play and is ready to do so).
How would you identify such dependencies, within this system? (The identification must somehow allow de-duplication.)
Well for anything that is handed to the dependency system by calling some API and passing it an object, e.g. a promise or DOM Node, the object's very identity would presumably be sufficient. For declarative mechanisms that refer to elements by ID, the ID would presumably be a way to look up the element and then the element's inherent identity would suffice. In fact in the strawman I had proposed last year, the URL for a script wasn't actually used as the identity, what the proposal did was use the URL to look up the <script> element and then used the <script> element as the
target of any deduping.
On Tue, May 27, 2014 at 3:57 PM, Ian Hickson <ian at hixie.ch> wrote:
On Tue, 27 May 2014, John Barton wrote:
I think the Loader nicely isolates these particular functions: I don't see any urgency in standardizing them. However the delegation of the specification of System leaves us in the weird place of having everything except the actually external API spec-ed. Web folks could almost just say "Browsers shall have a window.System instance of Loader". There is a missing part of the Loader spec essential for Web and not very important for node: bundling for network transmission see systemjs/systemjs.
I don't understand this paragraph. What does it mean for us to not standardise Loader's functions?
It means that browsers implement them within the constraints already imposed the rest of the (large and quite specific) Loader specification, common sense, and feedback from users. Among the many important things to solve in the module space, specing these functions ranks very low. (BTW the Loader also has poorly specified compiler hooks that need work, but that clearly is TC39 business).
Is System something that we are expecting some non-ES spec, e.g. Fetch or HTML, to define?
TC39 members have more than once explained that they expect some non-ES spec to define System.
But to your original question about non-JS loading: you can extend the Loader class to add methods for CSS and HTML. These would be a cache in front of xhr for the most part, hence my suggestion that implementation (GitHub) is a good place to start. The next step, and one quite critical to practice, is integration with bundling. Here practical expertise is essential so again implementation would be a critical guide. There exist solutions in the pre-ES6 space to consult as well.
This seems to imply that to do anything with this mechanism, someone (the page author?) has to provide some script code.
I believe that such an extension of the ES6 Loader has a chance to provide value to the community. I guess it won't solve all problems ;-)
However, it's very much my goal to be able to address use cases that don't involve any script at all (and that work even when the browser has scripting disabled), for example, I'd like whatever solution we develop here to be able to handle prioritising the loading and applying of different style sheets and images, e.g. so that images that are not currently rendered aren't downloaded yet, even though they are mentioned later in the page. One could imagine having markup like:
<link href="foo.css" rel=stylesheet load-policy=on-demand id="foo-styles"> ... <img src="foo.png" alt="..." load-policy=on-demand depends-on="foo-styles">
...where an <img> element is defined as being needed when it's rendered, and where the browser would therefore only fetch the image and its stylesheet when the image is scrolled into view.
This would need to work without any script involved at all, but should the author later add script with its own dependencies, we would want all the script dependency stuff to use the same underlying system so that if the script needs foo-styles or the foo.png image, it would all work as one coherent whole.
I think that developers with sufficient background to master load policies and dependency graphs will also be able to master a small script to achieve a similar result. On the other hand, experienced developers would like to avoid these kinds of complex declarative solutions if possible, since they are difficult to debug and understand.
HTML Imports offers a middle ground: while it is declarative it is also modular so the complexity can be divided and conquered. I don't think it requires script so it is an alternative starting point. My uninformed concern with HTML Imports lies in the limitations it imposes in order to enforce performance goals.
Overall I think we should experiment with the tools we almost have before we try to couple them together or develop another overlapping alternative.
jjb
On Tue, 27 May 2014, John Barton wrote:
Is System something that we are expecting some non-ES spec, e.g. Fetch or HTML, to define?
TC39 members have more than once explained that they expect some non-ES spec to define System.
Ah, great. (Sorry if I sound dumb here, I'm very new to the whole module thing. I tried reading the spec but it wasn't very clear to me.)
Is there a description of what the non-ES spec should say? That is, what is the interface that System exposes that needs to be "implemented" by this non-ES spec? What are the spec hooks that this non-ES spec would need to invoke?
But to your original question about non-JS loading: you can extend the Loader class to add methods for CSS and HTML. These would be a cache in front of xhr for the most part, hence my suggestion that implementation (GitHub) is a good place to start. The next step, and one quite critical to practice, is integration with bundling. Here practical expertise is essential so again implementation would be a critical guide. There exist solutions in the pre-ES6 space to consult as well.
This seems to imply that to do anything with this mechanism, someone (the page author?) has to provide some script code.
I believe that such an extension of the ES6 Loader has a chance to provide value to the community. I guess it won't solve all problems ;-)
Would this be a replacement to the aforementioned "System"?
However, it's very much my goal to be able to address use cases that don't involve any script at all (and that work even when the browser has scripting disabled), for example, I'd like whatever solution we develop here to be able to handle prioritising the loading and applying of different style sheets and images, e.g. so that images that are not currently rendered aren't downloaded yet, even though they are mentioned later in the page. One could imagine having markup like:
<link href="foo.css" rel=stylesheet load-policy=on-demand id="foo-styles"> ... <img src="foo.png" alt="..." load-policy=on-demand depends-on="foo-styles">
...where an <img> element is defined as being needed when it's rendered, and where the browser would therefore only fetch the image and its stylesheet when the image is scrolled into view.
This would need to work without any script involved at all, but should the author later add script with its own dependencies, we would want all the script dependency stuff to use the same underlying system so that if the script needs foo-styles or the foo.png image, it would all work as one coherent whole.
I think that developers with sufficient background to master load policies and dependency graphs will also be able to master a small script to achieve a similar result.
That's quite possible. However, scripting is sometimes disabled. The idea is that this mechanism would keep functioning even in such an environment.
Also, note that many Web developers will use these facilities without mastering them first. It would be unfortunate, IMHO, if anyone who wanted to say "only load this image once it's in view" had to also paste in some boilerplate script that they didn't understand.
On the other hand, experienced developers would like to avoid these kinds of complex declarative solutions if possible, since they are difficult to debug and understand.
Yes, whatever solution we provide here should definitely also provide an API that authors can hook into.
HTML Imports offers a middle ground: while it is declarative it is also modular so the complexity can be divided and conquered. I don't think it requires script so it is an alternative starting point.
My goal here is for us to make sure that ES6 modules and HTML Imports end up specified using the same dependency mechanism.
Overall I think we should experiment with the tools we almost have before we try to couple them together or develop another overlapping alternative.
My goal isn't to define anything overlapping; my goal here is very much to avoid defining anything overlapping but for us to make sure that the one system we have actually addresses the needs of the three ongoing projects here - ES6 modules, HTML Imports, and the features being added to HTML to handle such things as not loading images until they're needed.
On Tue, May 27, 2014 at 4:51 PM, Ian Hickson <ian at hixie.ch> wrote:
On Tue, 27 May 2014, John Barton wrote:
Is System something that we are expecting some non-ES spec, e.g. Fetch or HTML, to define?
TC39 members have more than once explained that they expect some non-ES spec to define System.
Ah, great. (Sorry if I sound dumb here, I'm very new to the whole module thing. I tried reading the spec but it wasn't very clear to me.)
Is there a description of what the non-ES spec should say? That is, what is the interface that System exposes that needs to be "implemented" by this non-ES spec? What are the spec hooks that this non-ES spec would need to invoke?
Each implementation of System has to supply the functions described in people.mozilla.org/~jorendorff/es6-draft.html, people.mozilla.org/~jorendorff/es6-draft.html
literally under the term 'hooks'.
As I mentioned, the more important issue for browsers is bundling. Unless that is solved, the uptake of ES6 module loading will rely on non-standard mechanisms. Unlike the 'hooks', a bundling API is not defined by ES6 and it's not useful to node. It has superficial overlap with HTML Import, but the focus is on load performance rather than modularity.
But to your original question about non-JS loading: you can extend the Loader class to add methods for CSS and HTML. These would be a cache in front of xhr for the most part, hence my suggestion that implementation (GitHub) is a good place to start. The next step, and one quite critical to practice, is integration with bundling. Here practical expertise is essential so again implementation would be a critical guide. There exist solutions in the pre-ES6 space to consult as well.
This seems to imply that to do anything with this mechanism, someone (the page author?) has to provide some script code.
I believe that such an extension of the ES6 Loader has a chance to provide value to the community. I guess it won't solve all problems ;-)
Would this be a replacement to the aforementioned "System"?
No, not replace. I meant 'extension' in the es6 sense: System would remain an instance of Loader, it just would have additional functions, functions only relevant to code dedicated to working on CSS/HTML.
HTML Imports offers a middle ground: while it is declarative it is also
modular so the complexity can be divided and conquered. I don't think it requires script so it is an alternative starting point.
My goal here is for us to make sure that ES6 modules and HTML Imports end up specified using the same dependency mechanism.
I thought this would just work. HTML Imports will have <script> tags and
those <script> tags will invoke System.import() and thus use the same
dependency mechanism.
The reverse however, where System could have a dynamic effect similar to static HTML Imports and use the same dependency mechanism for the CSS/HTML is not as far as I know possible short of adding Import tags to the document. This would be nice to have, but requires coordination with HTML Import.
jjb
On Wed, May 28, 2014 at 12:29 AM, Kris Kowal <kris.kowal at cixar.com> wrote:
On Tue, May 27, 2014 at 3:04 PM, Ian Hickson <ian at hixie.ch> wrote:
On Tue, 27 May 2014, Kris Kowal wrote:
It would be lovely if HTML could be trained to resolve URL's through the module system.
By "HTML" here I presume you mean the underlying Fetch mechanism. Could you elaborate on exactly how this would work? Would this be in the layer before or after Service Workers?
Supposing that a page has both a service worker and a custom loader, I would expect all requests for URL’s in the page (href, src, etc) to
- pass through the page’s loader
- pass through the browser's fetch mechanism
- pass through the service worker
Only http/https URL fetches enter the service worker.
- pass through the browser's fetch mechanism again
This depends on what the service worker does. It might instead perform its own fetches (using fetch()) which presumably would go through the service worker's loader if we were to define that as such (it having its own global environment). The service worker might also pop back up to fetch and let it handle the fetch (not handling the fetch event) or might indeed pass it through fetch (using event.default()). It might also return a generated response or some such.
- possibly arrive at the true service
The HTTP layer you mean I guess.
6… proxies all the way down?
I suppose you could further divide the HTTP layer.
On Tuesday, May 27, 2014 8:51 PM, Ian Hickson <ian at hixie.ch> wrote:
Is there a description of what the non-ES spec should say? That is, what is the interface that System exposes that needs to be "implemented" by this non-ES spec? What are the spec hooks that this non-ES spec would need to invoke?
The W3C System would have to provide the following hooks in a way that matches what the web platform is supposed to do: * normalize hook: take a module ID string (like from import 'some-id' and normalize it to a unique ID through out the system * locate hook: based on the module ID, locate the asset to be fetched somehow (find its URL) * fetch hook: make the necessary network request * translate hook: probably not something the web platform will care about for now since its intended to translate other languages into EcmaScript * instantiate hook: if the fetched source is an ES6 module, then everything is covered. But if it's any other asset or a dynamic module (read AMD or similar) the instantiate hook is in charge of creating the necessary Module instance. For example, it would deal with things like importing CSS files by returning an empty module. Ah so the ES6 "import" statement can import anything, even things that
aren't ES6 modules?
It would certainly be cool if that was possible.
It could. You can avoid importing anything into the ES scope by doing import "foo"without specifying what exactly you would be importing. Based on that, you could implement the previous hooks so that the fetch hook knows how to fetch assets other than scripts and the instantiate hook knows how to insert the asset into the page.
I believe that such an extension of the ES6 Loader has a chance to
provide value to the community. I guess it won't solve all problems ;-)
Would this be a replacement to the aforementioned "System"?
It would probably be an extension to System or any other ES Loader instance. See systemjs/systemjs for an example.
Given all this and the previously mentioned <module> tag, <script needs=""> probably doesn't make much sense, does it?
Juan Dopazo
On Tue, 27 May 2014, John Barton wrote:
On Tue, May 27, 2014 at 4:51 PM, Ian Hickson <ian at hixie.ch> wrote:
On Tue, 27 May 2014, John Barton wrote:
Is System something that we are expecting some non-ES spec, e.g. Fetch or HTML, to define?
TC39 members have more than once explained that they expect some non-ES spec to define System.
Ah, great. (Sorry if I sound dumb here, I'm very new to the whole module thing. I tried reading the spec but it wasn't very clear to me.)
Is there a description of what the non-ES spec should say? That is, what is the interface that System exposes that needs to be "implemented" by this non-ES spec? What are the spec hooks that this non-ES spec would need to invoke?
Each implementation of System has to supply the functions described in people.mozilla.org/~jorendorff/es6-draft.html literally under the term 'hooks'.
I hate to be "that guy", but can you be more specific? I did a search for the term "hook" and there were 85 hits; "hooks" had 6 which mentioned various things but none of them seemed to be near a list of specific hooks, their interface, and their meaning. Is the list that Juan described (see below) the list you had in mind?
As I mentioned, the more important issue for browsers is bundling. Unless that is solved, the uptake of ES6 module loading will rely on non-standard mechanisms. Unlike the 'hooks', a bundling API is not defined by ES6 and it's not useful to node. It has superficial overlap with HTML Import, but the focus is on load performance rather than modularity.
I agree that a bundling system would also be useful. It's not what I happen to be working on at the moment, however.
But to your original question about non-JS loading: you can extend the Loader class to add methods for CSS and HTML. These would be a cache in front of xhr for the most part, hence my suggestion that implementation (GitHub) is a good place to start. The next step, and one quite critical to practice, is integration with bundling. Here practical expertise is essential so again implementation would be a critical guide. There exist solutions in the pre-ES6 space to consult as well.
This seems to imply that to do anything with this mechanism, someone (the page author?) has to provide some script code.
I believe that such an extension of the ES6 Loader has a chance to provide value to the community. I guess it won't solve all problems ;-)
Would this be a replacement to the aforementioned "System"?
No, not replace. I meant 'extension' in the es6 sense: System would remain an instance of Loader, it just would have additional functions, functions only relevant to code dedicated to working on CSS/HTML.
Ah, I see.
My goal here is to define the System part so that that part is well-defined and, more specifically, so that other aspects of the Web platform that do similar things, such as HTML imports and various other new HTML features, can all be defined in terms of the same underlying mechanism. Hopefully having such a system be well defined and interoperable will enable the kinds of scripted extensions you mention.
HTML Imports offers a middle ground: while it is declarative it is also modular so the complexity can be divided and conquered. I don't think it requires script so it is an alternative starting point.
My goal here is for us to make sure that ES6 modules and HTML Imports end up specified using the same dependency mechanism.
I thought this would just work. HTML Imports will have <script> tags and those <script> tags will invoke System.import() and thus use the same dependency mechanism.
Well, something in the HTML spec has to "invoke System.import()" and all that jazz, right? That's what I'm trying to make sure we spec here.
The reverse however, where System could have a dynamic effect similar to static HTML Imports and use the same dependency mechanism for the CSS/HTML is not as far as I know possible short of adding Import tags to the document. This would be nice to have, but requires coordination with HTML Import.
Right, that's what I'm trying to do.
On Wed, 28 May 2014, Juan Ignacio Dopazo wrote:
The W3C System would have to provide the following hooks in a way that matches what the web platform is supposed to do:
normalize hook: take a module ID string (like from import 'some-id' and normalize it to a unique ID through out the system
locate hook: based on the module ID, locate the asset to be fetched somehow (find its URL)
fetch hook: make the necessary network request
translate hook: probably not something the web platform will care about for now since its intended to translate other languages into EcmaScript
instantiate hook: if the fetched source is an ES6 module, then everything is covered. But if it's any other asset or a dynamic module (read AMD or similar) the instantiate hook is in charge of creating the necessary Module instance. For example, it would deal with things like importing CSS files by returning an empty module.
Aha, thanks. Having this made looking up the hooks in the spec easier.
Are these to be provided by a host object that implements these hooks?
Is there an IDL block that describes the System object that the browser has to implement? That would make my job of speccing the default System object (assuming that's my job) significantly easier.
One thing I don't see in the hooks above is anything to do with actually processing dependencies. How would I (at the spec level) tell the ES6 module system that it should not evaluate a particular module until some non-script resource, e.g. a style sheet, is available?
One thing about this is it seems very script-centric. It would be weird if every time there was a <link rel=stylesheet> in the document, we
implicitly went through an ES6 module's instantiate hook just to fire up the style sheet. Is there some way we can make this more generic?
For example, would it make sense to have the logic that handles Load Records be moved to another spec?
Alternatively, if ES6 is the spec that's going to manage the load records for all the loads for a Web page, is there some way we can augment this mechanism to handle the features that we need for other parts of the Web platform, like lazy loading, deferring, and so on?
Given all this and the previously mentioned <module> tag, <script needs=""> probably doesn't make much sense, does it?
Well, not everything is going to use modules (e.g. nothing today does). Also, it's not only scripts that are the problem here. For example, having images depend on style sheets (with both loaded just when the image comes into view) is one of the use cases that people have brought up that would be good to support.
On Wed, May 28, 2014 at 3:26 PM, Ian Hickson <ian at hixie.ch> wrote:
On Tue, 27 May 2014, John Barton wrote:
On Tue, May 27, 2014 at 4:51 PM, Ian Hickson <ian at hixie.ch> wrote:
On Tue, 27 May 2014, John Barton wrote:
Is System something that we are expecting some non-ES spec, e.g. Fetch or HTML, to define?
TC39 members have more than once explained that they expect some non-ES spec to define System.
Ah, great. (Sorry if I sound dumb here, I'm very new to the whole module thing. I tried reading the spec but it wasn't very clear to me.)
Is there a description of what the non-ES spec should say? That is, what is the interface that System exposes that needs to be "implemented" by this non-ES spec? What are the spec hooks that this non-ES spec would need to invoke?
Each implementation of System has to supply the functions described in people.mozilla.org/~jorendorff/es6-draft.html literally under the term 'hooks'.
I hate to be "that guy", but can you be more specific? I did a search for the term "hook" and there were 85 hits; "hooks" had 6 which mentioned various things but none of them seemed to be near a list of specific hooks, their interface, and their meaning. Is the list that Juan described (see below) the list you had in mind?
Yes. You can also look at Guy Bedford's implementation of System: ModuleLoader/es6-module-loader/blob/master/lib/system.js or Traceur's implementation of LoaderHooks: google/traceur-compiler/blob/master/src/runtime/LoaderHooks.js (The traceur implementation includes compiler-ish hooks not relevant to this discussion).
...
HTML Imports offers a middle ground: while it is declarative it is also modular so the complexity can be divided and conquered. I don't think it requires script so it is an alternative starting point.
My goal here is for us to make sure that ES6 modules and HTML Imports end up specified using the same dependency mechanism.
I thought this would just work. HTML Imports will have <script> tags and those <script> tags will invoke System.import() and thus use the same dependency mechanism.
Well, something in the HTML spec has to "invoke System.import()" and all that jazz, right? That's what I'm trying to make sure we spec here.
The document imported by HTML Import would depend on some ES6 modules and it would contain some CSS and HTML. Knitting these together would require some JS execution which would compute references to the CSS/HTML and pass them to ES6 code for processing. That bit of glue code would probably live in the Imported document. We can have examples but I don't see how to spec it unless there are Import-specific functions for obtaining the references to the content from the Import.
The reverse however, where System could have a dynamic effect similar to static HTML Imports and use the same dependency mechanism for the CSS/HTML is not as far as I know possible short of adding Import tags to the document. This would be nice to have, but requires coordination with HTML Import.
Right, that's what I'm trying to do.
This would seem to require Web-specific extensions to System as I mentioned before.
jjb
Perhaps it's too cumbersome to use the System loader to specify the business logic for the browsers dependency loading/registry etc. Maybe a low-level browser spec needs to exist that a light System object could eventually defer to. This almost seems easier and more straight forward (requiring less cross-spec coordination) to make progress on.
- Matthew Robb
On Wed, May 28, 2014 at 4:20 PM, Ian Hickson <ian at hixie.ch> wrote:
On Wed, 28 May 2014, John Barton wrote:
Is the list that Juan described (see below) the list you had in mind?
Yes.
Cool, thanks. (It would be great if this list could be more explicitly in the spec, for ease of reference.)
HTML Imports offers a middle ground: while it is declarative it is also modular so the complexity can be divided and conquered. I don't think it requires script so it is an alternative starting point.
My goal here is for us to make sure that ES6 modules and HTML Imports end up specified using the same dependency mechanism.
I thought this would just work. HTML Imports will have <script> tags and those <script> tags will invoke System.import() and thus use the same dependency mechanism.
Well, something in the HTML spec has to "invoke System.import()" and all that jazz, right? That's what I'm trying to make sure we spec here.
The document imported by HTML Import would depend on some ES6 modules and it would contain some CSS and HTML. Knitting these together would require some JS execution which would compute references to the CSS/HTML and pass them to ES6 code for processing. That bit of glue code would probably live in the Imported document. We can have examples but I don't see how to spec it unless there are Import-specific functions for obtaining the references to the content from the Import.
To import an HTML Import doesn't require any script, it's just:
<link rel=import href="...">
The browser's implementation of this would presumably go through the machinery that System.import() goes through, right? I mean, we don't want to require script just to use HTML imports. We certainly don't want to require that the author provide a script that extends Loader just to be able to use <img>, but we presumably want <img> to eventually go through the same machinery.
Or maybe not? I'm not sure I really understand whether there is agreement that the ES6 module system should be able to work with the rest of the platform's loading and dependency system.
Perhaps my explanation was unclear; I have nothing to say about browser implementation. I'll just try again. Recall that I am talking about the case where an HTML Import uses ES6 modules.
The HTML Import has CSS, HTML (including IMG) and JS parts. How will the JS parts relate to ES6 modules? The simple answer is that the JS <script> tags
in the Import will be the root of a dependency tree. The modules in the tree will be generic code independent of the particular CSS/HTML in the Import. This root code will be responsible for passing references bound to the CSS/HTML into the generic code in the tree.
Will the JS part be an ES6 module? Could be an anonymous module but it need not be a named module imported by other JS. This root JS would only make sense with the CSS and HTML it comes with. The unit of modularity is Import here.
Is there any special API for ES6 modules in HTML Imports? Not needed to get started.
I'm trying to say that ES6 modules and the ES6 module system will just work in Imports. I'm not saying anything about other combinations or issues.
The reverse however, where System could have a dynamic effect similar to static HTML Imports and use the same dependency mechanism for the CSS/HTML is not as far as I know possible short of adding Import tags to the document. This would be nice to have, but requires coordination with HTML Import.
Right, that's what I'm trying to do.
This would seem to require Web-specific extensions to System as I mentioned before.
I was suggesting a completely different and more pedestrian feature, the
ability to imperatively load HTML Imports. (Maybe that already exists?) But I'll guess at some answers anyway:
Ok. I'm not really sure how to extend the ES6 module system in a way that won't stomp on this working group. How do I (at the spec level) tell the ES6 module system that it should not evaluate a particular module until some non-script resource, e.g. a style sheet, is available?
The ES6 Loader is driven, it's not the driver. Typically the Loader is driven by a <script> tag that roots the dependency tree.
If the browser has a way to express that a stylesheet depends on an ES6 module, it can invoke the loader once the stylesheet arrives. But without a way to tell the JS how to refer to its special stylesheet, I don't see what we gain. To push modularity up to the next level we need JS bindings to DOM that are 'modular'. That is, the ES6 modules would not use global.foo or window.foo but mySpecialStyleSheet.foo and mySpecialStyleSheet would be only known to the module. Today we accomplish this with global functions on DOM.
It seems like it needs a closer integration with load records than the APIs would enable. For example, would we be able to hook the CSS @import loading mechanism into load records?
The Loader is very generic, it might be fun to see whether it could load something other than JS. The mechanism isn't really focused on load records. Rather, the Loader is a dependency-tree walker with some callbacks. Some callbacks are clearly defined and nicely generic (normalize, locate, fetch, translate). Some callbacks are vague, 'parse this and tell me what it imports'. Current these latter callbacks are very JS-centric. These would need work to eg parse CSS and return imports.
How about marking a load as low-priority, is that something we can do with load records?
No, because it makes no sense in a dependency-driven loader. The root of a tree could be delayed (by the browser for example) but once we start loading, priority does not matter.
jjb
Ok. I'm not really sure how to extend the ES6 module system in a way that won't stomp on this working group. How do I (at the spec level) tell the ES6 module system that it should not evaluate a particular module until some non-script resource, e.g. a style sheet, is available? It seems like it needs a closer integration with load records than the APIs would enable. For example, would we be able to hook the CSS @import loading mechanism into load records? How about marking a load as low-priority, is that something we can do with load records?
So, what we currently have is ES6 defining a dependency loading framework and a requirement that the host environment satisfy the contract represented by that framework.
An alternative might be to assume that host environment defines a loading framework available through some global, promise-returning primatives, and defining ES module loading in terms of those primatives. I think this would entail moving much of the loader pipeline into HTML, and allowing the ES spec to focus specifically on linkage.
My intuition says that this would be a better overall design, since it would bring all resource loading concerns under a common subsystem using a common resource model.
Are you thinking along these same general lines?
On Thu, May 29, 2014 at 3:09 AM, John Barton <johnjbarton at google.com> wrote:
Perhaps my explanation was unclear; I have nothing to say about browser implementation.
That's the only part Ian is interested in. He helps authoring the standards for the various features he mentioned in the initial post. He wants to reconcile them with ES6 Modules. I get the impression you are talking about something else.
My intuition is that any such plan would be vigorously opposed by the JS community. Or perhaps vigorously ignored: browsers are falling behind current technology and are no longer in a position to dictate what JS means.
On Thu, May 29, 2014 at 4:40 PM, John Barton <johnjbarton at google.com> wrote:
My intuition is that any such plan would be vigorously opposed by the JS community. Or perhaps vigorously ignored: browsers are falling behind current technology and are no longer in a position to dictate what JS means.
I don't even know what this means. You are aware it's mostly browsers that are
TC39 includes members (e.g., Paypal, Yahoo) that are heavy users of server side JS. Samsung expressed strong interest in JS for embedded devices. Microsoft uses JS as a Windows scripting and application development language. And in any case, TC39 is committed to growing JS as a language not specific to browsers.
That said, I'm not really sure what this argument is about. Should we grow the ES6 module loading mechanism so that it can better handle the loading of other resources beside JS code that is relevant to JS? Probably. Do we know precisely where we want to draw the line of how much loading logic is to be provided by JS and specced by TC39, vs provided by a hosting environment and specced by those that spec that specific hosting environment? Probably not. But if the issue is of general interest across many hosting environments, then it should probably be handled by JS and TC39, rather than duplicating work among the hosting environment specs.
On Wednesday, May 28, 2014 7:27 PM, Ian Hickson <ian at hixie.ch> wrote:
One thing I don't see in the hooks above is anything to do with actually
processing dependencies. How would I (at the spec level) tell the ES6 module system that it should not evaluate a particular module until some non-script resource, e.g. a style sheet, is available?
When you create a dynamic module in the instantiate hook you can specify a list of dependencies like this:
System.instantiate = function (loadRecord) { return { deps: ['some', 'list', 'of', 'dependencies'], execute: function () { // return a new Module instance } }; };
The Loader will know to wait for those dependencies.
One thing about this is it seems very script-centric. It would be weird if
every time there was a <link rel=stylesheet> in the document, we implicitly went through an ES6 module's instantiate hook just to fire up the style sheet. Is there some way we can make this more generic?
Yes, the System loader would be an API for imperatively loading stuff from a script. The Loader hooks are the way of exposing to the JS environment how the browser loads stuff.
This is interesting because users could actually modify the System loader. For example, they could set it up to load modules conditionally according to the browser's features. It could also be interesting to leave that door open for non-script imports also being subject to user rules.
Juan
On 5/29/14, 11:12 AM, Mark S. Miller wrote:
But if the issue is of general interest across many hosting environments, then it should probably be handled by JS and TC39, rather than duplicating work among the hosting environment specs.
Assuming the requirements of the different hosting environments are similar enough that it's possible to factor out some sort of common thing (presumably with hookable bits for the parts where hosting environments want different behavior), of course. If the hookable bits start being the whole thing, that would suggest that the requirements are not similar enough.... Hard to say until we try.
On Thu, May 29, 2014 at 5:12 PM, Mark S. Miller <erights at google.com> wrote:
That said, I'm not really sure what this argument is about. Should we grow the ES6 module loading mechanism so that it can better handle the loading of other resources beside JS code that is relevant to JS? Probably. Do we know precisely where we want to draw the line of how much loading logic is to be provided by JS and specced by TC39, vs provided by a hosting environment and specced by those that spec that specific hosting environment? Probably not.
Right, that is the issue. If you look at the web platform there's many different types of resources. Images, style sheets, media, scripts, ... There's a couple of solutions being thought of for dependency management, stipulated by Ian. It would be nice if we got to something more coherent for web developers.
But if the issue is of general interest across many hosting environments, then it should probably be handled by JS and TC39, rather than duplicating work among the hosting environment specs.
What other hosting environment has similar characteristics to loading of web pages?
On Thu, May 29, 2014 at 7:45 AM, Anne van Kesteren <annevk at annevk.nl> wrote:
On Thu, May 29, 2014 at 4:40 PM, John Barton <johnjbarton at google.com> wrote:
My intuition is that any such plan would be vigorously opposed by the JS community. Or perhaps vigorously ignored: browsers are falling behind current technology and are no longer in a position to dictate what JS means.
I don't even know what this means. You are aware it's mostly browsers that are on TC39 right and drive JS implementation?
I mean that the nodejs community accounts for a significant and growing fraction of the JS community. Its module capability has proven very effective and that module capability sits at the center of its community. Browsers, on the other hand, are in decline, having failed to respond to the innovation demanded by the rise of mobile-first and generally mired in efforts to create better designs rather than ship innovations. In my opinion, JS developers, including those involve with or influencing TC39, understand this situation and will not agree to yet another redesign of the ES module system. Some features could be cut; some could be added, but overall its up to the browser to deal with the Loader not the other way around.
I think the ES6 work plus module loading is exceptional engineering by world class experts. But it will only count if it ships.
Just one opinion of course. Ask around. jjb
On Thu, May 29, 2014 at 8:20 AM, Boris Zbarsky <bzbarsky at mit.edu> wrote:
On 5/29/14, 11:12 AM, Mark S. Miller wrote:
But if the issue is of general interest across many hosting environments, then it should probably be handled by JS and TC39, rather than duplicating work among the hosting environment specs.
Assuming the requirements of the different hosting environments are similar enough that it's possible to factor out some sort of common thing (presumably with hookable bits for the parts where hosting environments want different behavior), of course. If the hookable bits start being the whole thing, that would suggest that the requirements are not similar enough.... Hard to say until we try.
I wholeheartedly agree. Let's try.
Assuming the requirements of the different hosting environments are similar enough that it's possible to factor out some sort of common thing (presumably with hookable bits for the parts where hosting environments want different behavior), of course. If the hookable bits start being the whole thing, that would suggest that the requirements are not similar enough.... Hard to say until we try.
I wholeheartedly agree. Let's try.
I think a better approach (perhaps parallel) would be to imagine a scenario where the host environment defines the loading framework, and seeing how that changes the ES spec. My hunch (untested of course), is that, as long as the host provides some elegant promise-returning primatives, the ES spec would be much simplified and streamlined by the change.
Parallel experiments would be great.
On Thu, 29 May 2014, Kevin Smith wrote:
Ok. I'm not really sure how to extend the ES6 module system in a way that won't stomp on this working group. How do I (at the spec level) tell the ES6 module system that it should not evaluate a particular module until some non-script resource, e.g. a style sheet, is available? It seems like it needs a closer integration with load records than the APIs would enable. For example, would we be able to hook the CSS @import loading mechanism into load records? How about marking a load as low-priority, is that something we can do with load records?
So, what we currently have is ES6 defining a dependency loading framework and a requirement that the host environment satisfy the contract represented by that framework.
An alternative might be to assume that host environment defines a loading framework available through some global, promise-returning primatives, and defining ES module loading in terms of those primatives.
I think this would entail moving much of the loader pipeline into HTML, and allowing the ES spec to focus specifically on linkage.My intuition says that this would be a better overall design, since it would bring all resource loading concerns under a common subsystem using a common resource model.
Are you thinking along these same general lines?
That's an option. I don't have an opinion on whether this is the better way to go or whether ES6 should be extended to provide what is needed to define the plumbing for other features, nor do I mind who writes the spec text. My only opinion here really is that I think we should make sure the Web doesn't end up with multiple redundant dependency systems.
On Thu, 29 May 2014, John Barton wrote:
My intuition is that any such plan would be vigorously opposed by the JS community.
Then it's probably not a good plan.
This suggests that the plan should be to instead make sure that whatever ES6 defines can be used by other specs to define the Web's loading model. Is extending the ES6 model (not the API, but the underlying model with load records and so on) something that could be on the table?
On Thu, May 29, 2014 at 9:50 AM, Ian Hickson <ian at hixie.ch> wrote: [...]
This suggests that the plan should be to instead make sure that whatever ES6 defines can be used by other specs to define the Web's loading model. Is extending the ES6 model (not the API, but the underlying model with load records and so on) something that could be on the table?
Compatibly extending the ES6 model after ES6 is certainly on the table. Regarding ES6 itself, not unless it's critical, and perhaps not even then, depending. That ship is sailing, airplane doors have closed, or whatever metaphor you like ;).
What's a "load record"?
On Wed, 28 May 2014, Matthew Robb wrote:
Perhaps it's too cumbersome to use the System loader to specify the business logic for the browsers dependency loading/registry etc. Maybe a low-level browser spec needs to exist that a light System object could eventually defer to. This almost seems easier and more straight forward (requiring less cross-spec coordination) to make progress on.
If this is something that a future ES spec would defer to, I'd be happy to work on that. Is that feasible?
On Wed, 28 May 2014, John Barton wrote:
The Loader is very generic, it might be fun to see whether it could load something other than JS. The mechanism isn't really focused on load records. Rather, the Loader is a dependency-tree walker with some callbacks. Some callbacks are clearly defined and nicely generic (normalize, locate, fetch, translate). Some callbacks are vague, 'parse this and tell me what it imports'. Current these latter callbacks are very JS-centric. These would need work to eg parse CSS and return imports.
Right.
How about marking a load as low-priority, is that something we can do with load records?
No, because it makes no sense in a dependency-driven loader. The root of a tree could be delayed (by the browser for example) but once we start loading, priority does not matter.
The contexts in which I am interested about this kind of thing is:
-
some people want to predeclare a bunch of scripts and their dependencies, without actually downloading any of them ahead of time, and then later, when they're needed, have the entire tree downloaded all at once. For example, you could have a lot of code to back video playback widgets, but not want to download them (or the videos!) until the user indicates a that they want to watch a video.
-
sometimes, even a file that is not downloaded immediately upon being referenced by another file might still want to be precached, so that when it is needed, it's already locally available. These loads, however, have to happen later than any loads for scripts or other resources that are needed immediately.
-
some resources are needed to load the page, but should be deferred until after the page is rendered. For example, Google+ tries to render as fast as possible a static non-interactive version of itself, and then after that is done, it downloads scripts and goes back and wires up all the logic so that the page is interactive.
Many of these apply to modules, or interact with modules, which is why I'm interested in how we can adapt the ES6 spec to support prioritisation, predeclaring dependencies, dependencies on non-script resources, and so forth.
On Thu, 29 May 2014, Juan Ignacio Dopazo wrote:
When you create a dynamic module in the instantiate hook you can specify a list of dependencies like this:
System.instantiate = function (loadRecord) { return { deps: ['some', 'list', 'of', 'dependencies'], execute: function () { // return a new Module instance } }; };
The Loader will know to wait for those dependencies.
I'm having difficulty understanding the spec for the instantiate hook. Does it get called before the fetch hook? Is there some way to dynamically add dependencies later?
On Thu, 29 May 2014, Mark Miller wrote:
Compatibly extending the ES6 model after ES6 is certainly on the table. Regarding ES6 itself, not unless it's critical, and perhaps not even then, depending. That ship is sailing, airplane doors have closed, or whatever metaphor you like ;).
I just mean extending the ES model in general, I'm not trying to imply any particular version or whatever. (I forget that some specs are still using versioned snapshots, sorry. HTML is just on a rolling "living standard" model now, so new features can be added in one area at the same time as other areas are being implemented.)
What's a "load record"?
Unless I misunderstood the spec, it's how the ES6 spec respresents a specific resource in the dependency tree.
On Thursday, May 29, 2014 4:34 PM, Ian Hickson <ian at hixie.ch> wrote:
The contexts in which I am interested about this kind of thing is:
- some people want to predeclare a bunch of scripts and their
dependencies, without actually downloading any of them ahead of time, and then later, when they're needed, have the entire tree downloaded all at once.
At the moment we're doing this by running static analysis tools on the ES6 modules and using the information from the dependency tree to know how to download all of them at once.
- sometimes, even a file that is not downloaded immediately upon being
referenced by another file might still want to be precached, so that when it is needed, it's already locally available
The Loader object is actually a dictionary, so it acts as its own cache. You can call loader.fetch() yourself without the module being executed in order to cache it.
On Thu, 29 May 2014, Juan Ignacio Dopazo wrote:
The Loader will know to wait for those dependencies.
I'm having difficulty understanding the spec for the instantiate hook. Does it get called before the fetch hook? Is there some way to dynamically add dependencies later?
The lifecycle of a module request is normalize -> locate -> fetch -> translate -> instantiate. Instantiate doesn't actually execute anything until all of the dependencies have gone through the same 5 steps.
Juan
On Thu, 29 May 2014, Juan Ignacio Dopazo wrote:
- some people want to predeclare a bunch of scripts and their dependencies, without actually downloading any of them ahead of time, and then later, when they're needed, have the entire tree downloaded all at once.
At the moment we're doing this by running static analysis tools on the ES6 modules and using the information from the dependency tree to know how to download all of them at once.
Right, it's relatively common for people to do this. Not just for scripts, indeed, but also for style sheets and other resources. Ideally the browser would take care of this logic for the author. This becomes especially difficult when there's multiple leaves in the dependency tree, with a lot of shared roots, and you just want the files you need (and in particular, you don't want to wait for the files to have been downloaded to discover what the next level of dependencies is, since that would add one round-trip time per dependency level).
- sometimes, even a file that is not downloaded immediately upon being referenced by another file might still want to be precached, so that when it is needed, it's already locally available
The Loader object is actually a dictionary, so it acts as its own cache. You can call loader.fetch() yourself without the module being executed in order to cache it.
The idea here is that the browser would be able to determine when it's best to preload the file; the author would just want to tell the UA that the file is something worth precaching, not actually do it.
I'm having difficulty understanding the spec for the instantiate hook. Does it get called before the fetch hook? Is there some way to dynamically add dependencies later?
The lifecycle of a module request is normalize -> locate -> fetch -> translate -> instantiate. Instantiate doesn't actually execute anything until all of the dependencies have gone through the same 5 steps.
Aha, ok.
Is there a way to add dependencies before the "fetch" hook is called?
When is the "fetch" hook called in the current ES6 logic? As soon as "locate" provides a URL?
On Thu, May 29, 2014 at 2:31 PM, Ian Hickson <ian at hixie.ch> wrote:
On Thu, 29 May 2014, Juan Ignacio Dopazo wrote:
- some people want to predeclare a bunch of scripts and their dependencies, without actually downloading any of them ahead of time, and then later, when they're needed, have the entire tree downloaded all at once.
At the moment we're doing this by running static analysis tools on the ES6 modules and using the information from the dependency tree to know how to download all of them at once.
Right, it's relatively common for people to do this. Not just for scripts, indeed, but also for style sheets and other resources.
FWIW, we call this "bundling". I mentioned it earlier as a problem that ES6 has not addressed. It is specific to browser needs.
Ideally the browser would take care of this logic for the author. This becomes especially difficult when there's multiple leaves in the dependency tree, with a lot of shared roots, and you just want the files you need (and in particular, you don't want to wait for the files to have been downloaded to discover what the next level of dependencies is, since that would add one round-trip time per dependency level).
At least for ES this has to be done in the server: the dependencies are declared in the source.
...
I'm having difficulty understanding the spec for the instantiate hook. Does it get called before the fetch hook? Is there some way to dynamically add dependencies later?
The lifecycle of a module request is normalize -> locate -> fetch -> translate -> instantiate. Instantiate doesn't actually execute anything until all of the dependencies have gone through the same 5 steps.
Aha, ok.
Is there a way to add dependencies before the "fetch" hook is called?
At least the draft of the Loader spec we followed did not allow a "loader hook" to add dependencies. Honestly the newer draft has so much detail I cannot be sure what the API is, but there were a few calls that amount to parsing the file and extracting the dependencies. If these call were API, then tools could add dependencies at the right time in the pipeline. I've advocated for this simply because it makes the system easier to understand.
When is the "fetch" hook called in the current ES6 logic? As soon as "locate" provides a URL?
Yes, unless the request already has full URL and locate() is not called.
Here is a working example of dependency tree caching with an ES6 loader extension - systemjs/systemjs/blob/master/lib/extension-depCache.js
This allows dependencies to be declared upfront, and then all deep dependencies are requested in parallel with the module request (removing the round trip latency problem). It works out quite elegantly within the loader hooks, and avoids problems with duplication etc as long as the current spec bug relating to load promise caching is addressed.
In a similar way bundling scenarios can also be fully catered for through the hooks.
Surely these sorts of scenarios should be the job of Link subresource hinting, and HTTP/2 push in future? If so having them solved through the hooks and not the spec itself makes a lot of sense.
On Thursday, May 29, 2014 6:31 PM, Ian Hickson <ian at hixie.ch> wrote:
On Thu, 29 May 2014, Juan Ignacio Dopazo wrote:
- some people want to predeclare a bunch of scripts and their dependencies, without actually downloading any of them ahead of time, and then later, when they're needed, have the entire tree downloaded all at once.
At the moment we're doing this by running static analysis tools on the ES6 modules and using the information from the dependency tree to know how to download all of them at once.
Right, it's relatively common for people to do this. Not just for scripts, indeed, but also for style sheets and other resources. Ideally the browser would take care of this logic for the author.
Yes. The author would still have to provide the dependency tree upfront (I wrote this for exactly this use case www.npmjs.org/package/module-graph) , but it's something that would be nice if the browser Loader did.
- sometimes, even a file that is not downloaded immediately upon being
referenced by another file might still want to be precached, so that when it is needed, it's already locally available
The Loader object is actually a dictionary, so it acts as its own cache. You can call loader.fetch() yourself without the module being executed in order to cache it.
The idea here is that the browser would be able to determine when it's best to preload the file; the author would just want to tell the UA that the file is something worth precaching, not actually do it.
I think this would get along well with the <module> tag. If the browser had an attribute that said if any remote resource (images, stylesheets, modules) may be preloaded, then it would just have to hook into loader.fetch() to tell the ES Loader that it already started preloading the module.
I'm having difficulty understanding the spec for the instantiate hook.
Does it get called before the fetch hook? Is there some way to dynamically add dependencies later?
The lifecycle of a module request is normalize -> locate -> fetch -> translate -> instantiate. Instantiate doesn't actually execute anything until all of the dependencies have gone through the same 5 steps.
Aha, ok.
Is there a way to add dependencies before the "fetch" hook is called?
Not without jumping through some hoops. But the dependencies aren't really what matter at the "fetch" stage. You can fetch other resources, which will generate internal LoadRecords for them, without specifying they're dependencies.
When is the "fetch" hook called in the current ES6 logic? As soon as
"locate" provides a URL?
Yes. It's promise based though. "locate" may even take time (we're using this to do the opposite of prefetching: generate a single HTTP request from multiple imports).
Juan
On Thu, 29 May 2014, John Barton wrote:
On Thu, May 29, 2014 at 2:31 PM, Ian Hickson <ian at hixie.ch> wrote:
On Thu, 29 May 2014, Juan Ignacio Dopazo wrote:
- some people want to predeclare a bunch of scripts and their dependencies, without actually downloading any of them ahead of time, and then later, when they're needed, have the entire tree downloaded all at once.
At the moment we're doing this by running static analysis tools on the ES6 modules and using the information from the dependency tree to know how to download all of them at once.
Right, it's relatively common for people to do this. Not just for scripts, indeed, but also for style sheets and other resources.
FWIW, we call this "bundling". I mentioned it earlier as a problem that ES6 has not addressed. It is specific to browser needs.
Ok.
Deferring loads and pre-declaring dependencies to enable browsers to bundle requests is part of what I would like to address.
Ideally the browser would take care of this logic for the author. This becomes especially difficult when there's multiple leaves in the dependency tree, with a lot of shared roots, and you just want the files you need (and in particular, you don't want to wait for the files to have been downloaded to discover what the next level of dependencies is, since that would add one round-trip time per dependency level).
At least for ES this has to be done in the server: the dependencies are declared in the source.
Right. CSS has them inline too. For other kinds of resources there's no way to declare them at all right now. Ideally I'd like to add something to browsers (probably something closely related to, augmenting, or subsuming the ES6 module loading infrastructure) that enables authors to declaratively pre-declare the relationships. This would also help with the problem of how to bundle requests for modules and CSS.
I'm having difficulty understanding the spec for the instantiate hook. Does it get called before the fetch hook? Is there some way to dynamically add dependencies later?
The lifecycle of a module request is normalize -> locate -> fetch -> translate -> instantiate. Instantiate doesn't actually execute anything until all of the dependencies have gone through the same 5 steps.
Aha, ok.
Is there a way to add dependencies before the "fetch" hook is called?
At least the draft of the Loader spec we followed did not allow a "loader hook" to add dependencies. Honestly the newer draft has so much detail I cannot be sure what the API is, but there were a few calls that amount to parsing the file and extracting the dependencies. If these call were API, then tools could add dependencies at the right time in the pipeline. I've advocated for this simply because it makes the system easier to understand.
The current spec seems to require that [[Dependencies]] be empty while [[Status]] is 'loading', so I guess we'd have to change that if we were to extend this. Can anyone explain what the implications of changing this would be?
When is the "fetch" hook called in the current ES6 logic? As soon as "locate" provides a URL?
Yes, unless the request already has full URL and locate() is not called.
I guess to make it possible to defer loads, we'd have to add some sort of logic in ProceedToLocate()?
On Thu, May 29, 2014 at 4:05 PM, Ian Hickson <ian at hixie.ch> wrote:
On Thu, 29 May 2014, John Barton wrote:
On Thu, May 29, 2014 at 2:31 PM, Ian Hickson <ian at hixie.ch> wrote:
On Thu, 29 May 2014, Juan Ignacio Dopazo wrote:
- some people want to predeclare a bunch of scripts and their dependencies, without actually downloading any of them ahead of time,
and then later, when they're needed, have the entire tree downloaded
all at once.
At the moment we're doing this by running static analysis tools on the ES6 modules and using the information from the dependency tree to know how to download all of them at once.
Right, it's relatively common for people to do this. Not just for scripts, indeed, but also for style sheets and other resources.
FWIW, we call this "bundling". I mentioned it earlier as a problem that ES6 has not addressed. It is specific to browser needs.
Ok.
Deferring loads and pre-declaring dependencies to enable browsers to bundle requests is part of what I would like to address.
The important dependency analysis problem that only a browser can solve is the resources needed to paint the visible screen. For example, if the footer is not going to be visible, then we can delay loading the components that build the footer. While I know that such an analysis would be difficult, this is the missing feature that deferring, async etc are poor proxies for. We need a component model that supports load-by-visible order or some such.
Deferring loads isn't a dependency problem in the ES6 sense, it should operate at the root level or on HTML Imports.
Building a list of dependencies to then have the browser send that list back to the server to request the resources is extra work: we can just have the server send all of these files. The server knows the dependencies, the browser does not. This is an optimization problem and the optimal place to perform it is the server (or the build plus server).
Ideally the browser would take care of this logic for the author. This
becomes especially difficult when there's multiple leaves in the dependency tree, with a lot of shared roots, and you just want the files you need (and in particular, you don't want to wait for the files to have been downloaded to discover what the next level of dependencies is, since that would add one round-trip time per dependency level).
At least for ES this has to be done in the server: the dependencies are declared in the source.
Right. CSS has them inline too. For other kinds of resources there's no way to declare them at all right now. Ideally I'd like to add something to browsers (probably something closely related to, augmenting, or subsuming the ES6 module loading infrastructure) that enables authors to declaratively pre-declare the relationships. This would also help with the problem of how to bundle requests for modules and CSS.
Rather than bundling requests we can request bundles. The server already knows the dependency relationship because it has all the files.
jjb
Right now there appear to be three separate efforts to develop dependency systems for the Web platform:
At the moment they are mostly not overlapping, but it seems likely that all will grow over time and that could get messy.
I would therefore like to propose that we integrate these so that we end up with just one system that manages all three.
What do people think?