Bundling vs sending serialized dependency graph
On Wed, Aug 20, 2014 at 4:06 PM, John Barton <johnjbarton at google.com> wrote:
On Mon, Aug 18, 2014 at 10:43 AM, Ian Hickson <ian at hixie.ch> wrote:
This just doens't work.
Suppose the dependency graph looks like this:
Feature A --> Dependency A1 \__\ Dependency \ Feature B --> Dependency B1 / / AB >--> Dependency D Feature C --> Dependency C1 ---> Dependency C2 /All of A, B, and C are to be fetched on-demand-only, to avoid using up too much bandwidth. All the files here are non-trivial in size.
How do you package this?
If you make a package for A, a package for B, and a package for C, then you'll have redundant content in the packages, and when the client asks for B after already having asked for A, the amount of content sent back will be greater than necessary and therefore it'll be slower than necessary. If you create multiple packages such that you group as much as possible into each package as possible without overlap, then you still end up with multiple resources to download when you need any of A, B, or C. Basically, it boils down to:
Package A \__\ Package \ Package B / / AB >--> Package D Package C -------------> /...and then you're back to the problem I asked about. If you don't have server-side support, then to avoid round-trips the client needs to know about the dependencies before it makes the first request. It can't wait til it receives the packages to discover the dependencies because if you do that then you're serialising your RTTs instead of pipelining them.
I assume you are imagining a densely connected graph with random access to any of the roots.
I'm not quite sure what that means.
I mean a world where different otherwise unrelated leaf modules or resources depend on common shared dependencies.
I expect that real life pages have well partitioned graphs (widgets) that share some dense parts (utilities) and simple access patterns -- main page, utilities, a set of widgets.
I doubt that the Web is that convenient. On some well-designed sites it might work out that way.
Consider a site like Google+, though. However well-designed it is, it fundamentally has a lot of common files used by lots of intermediate shared files used by lots of leaf modules (and a lot more depth while we're at it). What exactly is needed depends on what posts are displayed, the user's preferences with respect to features like Hangouts, etc. It's a complicated graph.
But sure, it would be great to have a complete solution if it's not a lot more complex.
I don't think it should be particularly complex. It only requires some minor changes. One is that we need to be able to declare dependencies ahead of the "instantiate" hook. Another (a subset, really) is that we need to be able to declare dependencies for ES6 modules as well as letting the ES6 infrastructure discover them automatically. Finally, it would be ideal if we could also adjust those dependencies on the fly, since if we're reflecting dependencies described in the mutable DOM structure, it might be mutated.
I guess you're proposing the send the dependency graph to the browser, then when a new root is needed, the stored graph is compared with the currently-loaded modules. The additional modules needed are then requested as a group. Up to this point we can just use build tools and browsers.
Actually, modulo the changes described above, the ES6 loader already does all this. It just doesn't quite handle it at the level of pre-emptive declaration of dependencies. But suppose you had two modules A, B, and C. A and B depend on C. With ES6 today, when A is loaded, it loads C. If late you load B, B doesn't reload C; it just links into it. So this is all already supported. All that's needed is a way to tell the ES6 system to get C before it has even received A.
How will we tell the server "send me this list of files all in one response?"
HTTP pipelining does this today, that's a solved problem.
On Wed, Aug 20, 2014 at 4:53 PM, Ian Hickson <ian at hixie.ch> wrote:
On Wed, Aug 20, 2014 at 4:06 PM, John Barton <johnjbarton at google.com> wrote:
On Mon, Aug 18, 2014 at 10:43 AM, Ian Hickson <ian at hixie.ch> wrote:
This just doens't work.
Suppose the dependency graph looks like this:
Feature A --> Dependency A1 \__\ Dependency \ Feature B --> Dependency B1 / / AB >--> Dependency D Feature C --> Dependency C1 ---> Dependency C2 /All of A, B, and C are to be fetched on-demand-only, to avoid using up too much bandwidth. All the files here are non-trivial in size.
How do you package this?
If you make a package for A, a package for B, and a package for C, then you'll have redundant content in the packages, and when the client asks for B after already having asked for A, the amount of content sent back will be greater than necessary and therefore it'll be slower than necessary. If you create multiple packages such that you group as much as
possible into each package as possible without overlap, then you still end
up with multiple resources to download when you need any of A, B, or C. Basically, it boils down to:
Package A \__\ Package \ Package B / / AB >--> Package D Package C -------------> /...and then you're back to the problem I asked about. If you don't have server-side support, then to avoid round-trips the client needs to know about the dependencies before it makes the first request. It can't wait til it receives the packages to discover the dependencies because if you do that then you're serialising your RTTs instead of pipelining them.
I assume you are imagining a densely connected graph with random access to any of the roots.
I'm not quite sure what that means.
It means that the graph of all possible modules for a page cannot be partitioned into packages without sending much of the shared graph over the wire multiple times.
I mean a world where different otherwise unrelated leaf modules or resources depend on common shared dependencies.
Leaf modules don't depend on anything. That's what a leaf means.
I expect that real life pages have well partitioned graphs (widgets) that share some dense parts (utilities) and simple access patterns -- main page, utilities, a set of widgets.
I doubt that the Web is that convenient. On some well-designed sites it might work out that way.
Consider a site like Google+, though. However well-designed it is, it fundamentally has a lot of common files used by lots of intermediate shared files used by lots of leaf modules (and a lot more depth while we're at it).
I think your graph is upside down from mine ;-) As I learned it, leaf nodes were the ones at the ends of branches and hence were not dependent on any other nodes; no node depended on a root node.
What exactly is needed depends on what posts are displayed, the user's preferences with respect to features like Hangouts, etc. It's a complicated graph.
The only issue that matters for the efficiency of bundle loading is how many nodes are shared between commonly used dynamically loaded root modules. If the module is needed always it will be loaded always. If the module is only used by a single dynamically loaded feature, then there is no penalty for bundle loading. Even in the case where two or more dynamic loads use the same large number of modules we can simply put those modules in a shared bundle. So the case where bundles lose is very rare.
Here is an attempt and a graph:
A B C D E | / \ | / \ / R X Y
R is always loaded so its bundle loads A and B. Optional feature X loads C and D but not B its already loaded. Optional feature Y loads E and possibly DE if X did not already load.
Only D is extra work and thus this scenario is relatively rare. Typically optional modules will share dependencies with the default loaded page modules or have unique modules.
But sure, it would be great to have a complete solution if it's not a lot more complex.
I don't think it should be particularly complex. It only requires some minor changes. One is that we need to be able to declare dependencies ahead of the "instantiate" hook.
By the way I recently discovered that the deplist returned by the instantiate hook does not enter the dependency graph analysis. These aren't dependencies in a list rather a list of things needed to be loaded.
Another (a subset, really) is that we need to be able to declare dependencies for ES6 modules as well as letting the ES6 infrastructure discover them automatically.
In theory this should be straight-forward. In practice, well good luck.
Finally, it would be ideal if we could also adjust those dependencies on the fly, since if we're reflecting dependencies described in the mutable DOM structure, it might be mutated.
I think this one is technically difficult.
I guess you're proposing the send the dependency graph to the browser, then when a new root is needed, the stored graph is compared with the currently-loaded modules. The additional modules needed are then requested as a group. Up to this point we can just use build tools and browsers.
Actually, modulo the changes described above, the ES6 loader already does all this.
Huh? How do you plan to parse the modules to obtain dependencies without sending them to the browser?
It just doesn't quite handle it at the level of
pre-emptive declaration of dependencies. But suppose you had two modules A, B, and C. A and B depend on C. With ES6 today, when A is loaded, it loads C. If late you load B, B doesn't reload C; it just links into it. So this is all already supported. All that's needed is a way to tell the ES6 system to get C before it has even received A.
You've really lost me now. I thought your goal was to avoid sending C over the network. Now you want to send it without even seeing A?
jjb
On Thu, Aug 21, 2014 at 11:00 AM, John Barton <johnjbarton at google.com> wrote:
Finally, it would be ideal if we could also adjust those dependencies on the fly, since if we're reflecting dependencies described in the mutable DOM structure, it might be mutated.
I think this one is technically difficult.
I don't think this is actually all that hard. We have a dependency graph, along with a list of things that are (a) required, (b) already loaded, and (optionally) (c) in-flight. We mutate the graph arbitrarily, recompute the list of things that we need to load given the new (a), compare that to (b) and (c), and then start new/cancel old loads as necessary. I don't think it's worthwhile to specify incremental algorithms here, just specify the stop-the-world computation over the complete graph. Optimization left to the implementation, if needed.
Huh? How do you plan to parse the modules to obtain dependencies without sending them to the browser? [...] You've really lost me now. I thought your goal was to avoid sending C over the network. Now you want to send it without even seeing A?
I think Ian has explained this multiple times already. The HTML file contains a declarative specification of all resource dependencies, via attributes on script tags or some such. This is either generated manually or via some future authoring tool.
This doesn't seem so crazy to me. And, since presumably we can already deal with mutation of the dependency graph (see above), we can always adjust those declarations to match "reality" after we parse the module file (although probably only to add new edges, not remove any declared dependencies). Thus the base case with no explicit declarations in the HTML matches the on-demand behavior given by current ES6.
Ian, FWIW, I've been staying out of this thread mostly because you seem to be on top of it -- and because frankly I've already been exhausted by the module wars. But IMO you're doing excellent work.
On Thu, Aug 21, 2014 at 8:37 AM, C. Scott Ananian <ecmascript at cscott.net>
wrote:
On Thu, Aug 21, 2014 at 11:00 AM, John Barton <johnjbarton at google.com> wrote:
Finally, it would be ideal if we could also adjust those dependencies on the fly, since if we're reflecting dependencies described in the mutable DOM structure, it might be mutated.
I think this one is technically difficult.
I don't think this is actually all that hard. We have a dependency graph,
Where? The Load Request records imply a dependency graph. Are these maintained though out the life of the page? I don't see any existing reason to expect these are maintained.
along with a list of things that are (a) required, (b) already loaded, and (optionally) (c) in-flight. We mutate the graph arbitrarily, recompute the list of things that we need to load given the new (a), compare that to (b) and (c), and then start new/cancel old loads as necessary. I don't think it's worthwhile to specify incremental algorithms here, just specify the stop-the-world computation over the complete graph. Optimization left to the implementation, if needed.
Huh? How do you plan to parse the modules to obtain dependencies without sending them to the browser? [...] You've really lost me now. I thought your goal was to avoid sending C over the network. Now you want to send it without even seeing A?
I think Ian has explained this multiple times already. The HTML file contains a declarative specification of all resource dependencies, via attributes on script tags or some such. This is either generated manually or via some future authoring tool.
You deleted the context of my comment. I asserted that a build tool was needed. Ian claimed the Loader could do it. Now you are claiming I'm wrong because a build tool is used.
This doesn't seem so crazy to me.
That's because you left out the crazy part ;-)
And, since presumably we can already deal with mutation of the dependency graph (see above), we can always adjust those declarations to match "reality" after we parse the module file (although probably only to add new edges, not remove any declared dependencies). Thus the base case with no explicit declarations in the HTML matches the on-demand behavior given by current ES6.
Ian, FWIW, I've been staying out of this thread mostly because you seem to be on top of it -- and because frankly I've already been exhausted by the module wars. But IMO you're doing excellent work.
I agree that the part of Ian's story about sending over the deps list is beginning to sound very attractive. In fact I think it is a stronger story than the current spec altogether.
jjb
On Thu, Aug 21, 2014 at 11:54 AM, John Barton <johnjbarton at google.com> wrote:
On Thu, Aug 21, 2014 at 8:37 AM, C. Scott Ananian <ecmascript at cscott.net> wrote: Where? The Load Request records imply a dependency graph. Are these maintained though out the life of the page? I don't see any existing reason to expect these are maintained.
Ian's proposal (well, mutation in general) certainly implies that these (or an appropriate summary) should be maintained. Again, possibly you can drop parts of the graph from memory once all requests have been fulfilled... but I think that's an implementation optimization. The simplest spec would just stipulate a persistent dependency graph. I'd guess that you'd want to restrict arbitrary reads of the graph in order to allow the optimization. Write-only access would be best.
You deleted the context of my comment. I asserted that a build tool was needed. Ian claimed the Loader could do it. Now you are claiming I'm wrong because a build tool is used.
I apologize if I misconstrued the crux of your disagreement.
On Thu, 21 Aug 2014, C. Scott Ananian wrote:
On Thu, Aug 21, 2014 at 11:54 AM, John Barton <johnjbarton at google.com> wrote:
Where? The Load Request records imply a dependency graph. Are these maintained though out the life of the page? I don't see any existing reason to expect these are maintained.
Ian's proposal (well, mutation in general) certainly implies that these (or an appropriate summary) should be maintained. Again, possibly you can drop parts of the graph from memory once all requests have been fulfilled... but I think that's an implementation optimization. The simplest spec would just stipulate a persistent dependency graph. I'd guess that you'd want to restrict arbitrary reads of the graph in order to allow the optimization. Write-only access would be best.
I don't think we need to maintain the dependency graph beyond the link stage. Once something is linked, it doesn't really matter if the dependency is still true or not, since it doesn't make any difference
Not sure if my real world use case would be super helpful here, but just in case, here it is. The app I work on is a very large single page app - over 150,000 lines of JS across more than 2000 files. Uncompressed, unminified, and concatenated together, it weighs in at close to 10MB. We've been using an in-house module solution (written before most modern tools), and it does smart bundling using a dependency graph. We use import statements (in comments) that are read using a build tool which builds the graph. The full graph is broken into a hierarchy of code bundles that rely on load order expectations. The hierarchy can go many levels deep if needed. Common dependencies are moved up to the closest shared ancestor bundle, so there is no code in multiple code bundles.
Right now, this solution works pretty well for us, but is certainly not ideal. One nice thing is that there is a relatively small number (~40) of independent bundles and they can all be created upfront and served from a CDN with far future expiration. The downside is that there are definitely common dependencies that are bundled into ancestors higher up than they ideally should be.
With the current spec of modules and loaders, there is no way that we could switch and rely on the behavior of ES6 modules for the reasons specified here. If I have some new code loading point, I want all of the required code to start loading at once. I can't wait for the dependency graph to be figured out as files are getting loaded. The proposal for being able to preload a dependency graph upfront would be tremendously useful for making the switch. My preference would be some kind of json format through a JavaScript API and not some weird script tag thing.
On Thu, 21 Aug 2014, John Barton wrote:
I think your graph is upside down from mine ;-) As I learned it, leaf nodes were the ones at the ends of branches and hence were not dependent on any other nodes; no node depended on a root node.
I don't really mind which way we view the graph. To put it your way: I mean a world where different otherwise unrelated root modules or resources depend on common shared dependencies.
What exactly is needed depends on what posts are displayed, the user's preferences with respect to features like Hangouts, etc. It's a complicated graph.
The only issue that matters for the efficiency of bundle loading is how many nodes are shared between commonly used dynamically loaded root modules. If the module is needed always it will be loaded always. If the module is only used by a single dynamically loaded feature, then there is no penalty for bundle loading. Even in the case where two or more dynamic loads use the same large number of modules we can simply put those modules in a shared bundle. So the case where bundles lose is very rare.
I think bundles are clearly always a win, since at the extreme (one resource per bundle) they just boil down to the same as no bundles. They're never worse than not bundling.
But they don't solve the dependency problem.
Here is an attempt and a graph:
A B C D E | / \ | / \ / R X Y
R is always loaded so its bundle loads A and B. Optional feature X loads C and D but not B its already loaded. Optional feature Y loads E and possibly DE if X did not already load.
Only D is extra work and thus this scenario is relatively rare.
Rare in this case. In practice, large sites have thousands of modules with many layers of depth in the graph. Modules just change the scale of the problem, they don't remove the problem.
Typically optional modules will share dependencies with the default loaded page modules or have unique modules.
Can you elaborate on how you determine that this is the typical case?
Consider G+, or Facebook, or other sites of that nature, where there's hundreds of "leaf" modules (different dialogs, post types, views, etc), and there's modules for each widget (search toolbar, app drawer, notification bell, share box, nav menu, tab strip, chat widget, post box, post box photo view, photo picker, ACL entry box, photo uploader, button, checkbox, radio button, link picker, video picker, event picker, theme picker, drop down button, date picker, time picker, link widget, time zone picker, location picker, map, multiline text box, people browser, list picker, hovercard, the list goes on and on and I've barely scratched the surface of what G+ has here), each of which depends on a number of submodules and so on. I don't think that having multiple levels of dependency is rare at all.
But sure, it would be great to have a complete solution if it's not a lot more complex.
I don't think it should be particularly complex. It only requires some minor changes. One is that we need to be able to declare dependencies ahead of the "instantiate" hook.
By the way I recently discovered that the deplist returned by the instantiate hook does not enter the dependency graph analysis. These aren't dependencies in a list rather a list of things needed to be loaded.
I'm not sure what you mean here. The list returned from "instantiate" is treated the exact same way as the list auto-discovered from "import" statements when "instantiate" returns undefined: it's passed to ProcessLoadDependencies(), which calls RequestLoad() and AddDependencyLoad(), which, if necessary, updates [[Dependencies]].
That's all I'm talking about. I want to be able to update [[Dependencies]] earlier than "instantiate", and I want to be able to mutate [[Dependencies]] to remove nodes that are no longer dependencies before the load is complete.
Another (a subset, really) is that we need to be able to declare dependencies for ES6 modules as well as letting the ES6 infrastructure discover them automatically.
In theory this should be straight-forward. In practice, well good luck.
Good luck with what?
Finally, it would be ideal if we could also adjust those dependencies on the fly, since if we're reflecting dependencies described in the mutable DOM structure, it might be mutated.
I think this one is technically difficult.
I don't think anyone here is going to shy away from technically difficult problems, it's kind of our bailiwick. :-)
The idea is, in fact, to move as many of the technically difficult problems from things authors have to keep reinventing to things that browsers just support natively.
I guess you're proposing the send the dependency graph to the browser, then when a new root is needed, the stored graph is compared with the currently-loaded modules. The additional modules needed are then requested as a group. Up to this point we can just use build tools and browsers.
Actually, modulo the changes described above, the ES6 loader already does all this.
Huh? How do you plan to parse the modules to obtain dependencies without sending them to the browser?
You send them to the browser, just not in the module itself.
It just doesn't quite handle it at the level of pre-emptive declaration of dependencies. But suppose you had two modules A, B, and C. A and B depend on C. With ES6 today, when A is loaded, it loads C. If late you load B, B doesn't reload C; it just links into it. So this is all already supported. All that's needed is a way to tell the ES6 system to get C before it has even received A.
You've really lost me now. I thought your goal was to avoid sending C over the network. Now you want to send it without even seeing A?
Not sending C over the network at all wouldn't work, since it would mean A doesn't have its dependencies available. I don't follow.
The idea is that authors be able to predeclare (relevant parts of) the dependency tree such that when a node in that tree is needed, e.g. A in the example above, all the relevant nodes can be fetched in parallel, e.g. A and C in the example above, rather than in a serialised manner, e.g. first fetching A, then parsing it, then fetching C.
One way to do this would be to predeclare the modules, as in:
<script type=module src=a.js id=a needs=c load-policy=when-needed> </script> <script type=module src=b.js id=b needs=c load-policy=when-needed> </script> <script type=module src=c.js id=c load-policy=when-needed> </script>
...
// in some script or event handler or some such document.scripts.a.load(); // a is now needed, a and c get fetched in // parallel with only one RTT.
On Thu, Aug 21, 2014 at 1:55 PM, Ian Hickson <ian at hixie.ch> wrote:
On Thu, 21 Aug 2014, John Barton wrote: ... more misunderstanding about bundles skipped...
Let's give upon discussing bundles and pursue your dependency list scheme.
I don't think it should be particularly complex. It only requires some minor changes. One is that we need to be able to declare dependencies ahead of the "instantiate" hook.
By the way I recently discovered that the deplist returned by the instantiate hook does not enter the dependency graph analysis. These aren't dependencies in a list rather a list of things needed to be loaded.
I'm not sure what you mean here. The list returned from "instantiate" is treated the exact same way as the list auto-discovered from "import" statements when "instantiate" returns undefined: it's passed to ProcessLoadDependencies(), which calls RequestLoad() and AddDependencyLoad(), which, if necessary, updates [[Dependencies]].
I'll defer to Guy Bedford's expertise: ModuleLoader/es6-module-loader#204
That's all I'm talking about. I want to be able to update [[Dependencies]] earlier than "instantiate", and I want to be able to mutate [[Dependencies]] to remove nodes that are no longer dependencies before the load is complete.
Another (a subset, really) is that we need to be able to declare dependencies for ES6 modules as well as letting the ES6 infrastructure discover them automatically.
In theory this should be straight-forward. In practice, well good luck.
Good luck with what?
The JS module space has had so many rounds of discussions that many participants are no longer willing to consider alternatives to their current point of view.
Finally, it would be ideal if we could also adjust those dependencies on the fly, since if we're reflecting dependencies described in the mutable DOM structure, it might be mutated.
I think this one is technically difficult.
I don't think anyone here is going to shy away from technically difficult problems, it's kind of our bailiwick. :-)
Sometimes it is better to have a simpler and less powerful tool which can be well understood than a tool with many difficult-to-master features. Mutating dependencies opens the door to many new kinds of bugs, not only in the implementation of the Loader but, more important, in the use of the Loader by developers.
The idea is, in fact, to move as many of the technically difficult problems from things authors have to keep reinventing to things that browsers just support natively.
I guess you're proposing the send the dependency graph to the browser, then when a new root is needed, the stored graph is compared with the currently-loaded modules. The additional modules needed are then requested as a group. Up to this point we can just use build tools and browsers.
Actually, modulo the changes described above, the ES6 loader already does all this.
Huh? How do you plan to parse the modules to obtain dependencies without sending them to the browser?
You send them to the browser, just not in the module itself.
Where do you get them from? For ES6 it has to be from a build tool or from the server because no one is going to type these in twice.
Such a build tool is just not very difficult and it can be built in to dev servers so it has no impact on developers work flow.
It just doesn't quite handle it at the level of pre-emptive declaration of dependencies. But suppose you had two modules A, B, and C. A and B depend on C. With ES6 today, when A is loaded, it loads C. If late you load B, B doesn't reload C; it just links into it. So this is all already supported. All that's needed is a way to tell the ES6 system to get C before it has even received A.
You've really lost me now. I thought your goal was to avoid sending C over the network. Now you want to send it without even seeing A?
Not sending C over the network at all wouldn't work, since it would mean A doesn't have its dependencies available. I don't follow.
The idea is that authors be able to predeclare (relevant parts of) the dependency tree such that when a node in that tree is needed, e.g. A in the example above, all the relevant nodes can be fetched in parallel, e.g. A and C in the example above, rather than in a serialised manner, e.g. first fetching A, then parsing it, then fetching C.
Authors have already predeclared the dependency relationships. Now you want them to re-declare them. Try it and let me know how many devs you convince.
One way to do this would be to predeclare the modules, as in:
<script type=module src=a.js id=a needs=c load-policy=when-needed> </script> <script type=module src=b.js id=b needs=c load-policy=when-needed> </script> <script type=module src=c.js id=c load-policy=when-needed> </script>
Go back to G+ and ask them if they plan to type thousands of such lines into their .html files. Talk about a scaling problem!
jjb
On Thu, 21 Aug 2014, John Barton wrote:
I'm not sure what you mean here. The list returned from "instantiate" is treated the exact same way as the list auto-discovered from "import" statements when "instantiate" returns undefined: it's passed to ProcessLoadDependencies(), which calls RequestLoad() and AddDependencyLoad(), which, if necessary, updates [[Dependencies]].
I'll defer to Guy Bedford's expertise: ModuleLoader/es6-module-loader#204
No need to defer to Guy. You can just look at the spec.
But FWIW, what Guy is saying there does not contradict what I said above, as far as I can tell.
The JS module space has had so many rounds of discussions that many participants are no longer willing to consider alternatives to their current point of view.
It's not clear to me what to make of this.
Is feedback on the module system not welcome any more? The ES process is very opaque to me; it's not clear to me how to send feedback.
Finally, it would be ideal if we could also adjust those dependencies on the fly, since if we're reflecting dependencies described in the mutable DOM structure, it might be mutated.
I think this one is technically difficult.
I don't think anyone here is going to shy away from technically difficult problems, it's kind of our bailiwick. :-)
Sometimes it is better to have a simpler and less powerful tool which can be well understood than a tool with many difficult-to-master features.
That is orthogonal to how difficult it is to provide those features. Indeed simpler features are usually more difficult to provide than the difficult-to-master features.
Mutating dependencies opens the door to many new kinds of bugs, not only in the implementation of the Loader but, more important, in the use of the Loader by developers.
I do not disagree.
The number of bugs that might be present if we have two separate dependency systems seems yet higher still, though. That's the alternative here, as far as I can tell, since HTML imports allow you to mutate dependencies already. What I'm trying to do is make HTML imports reuse the ES6 model as much as possible to avoid us having to have two dependency systems.
I guess you're proposing the send the dependency graph to the browser, then when a new root is needed, the stored graph is compared with the currently-loaded modules. The additional modules needed are then requested as a group. Up to this point we can just use build tools and browsers.
Actually, modulo the changes described above, the ES6 loader already does all this.
Huh? How do you plan to parse the modules to obtain dependencies without sending them to the browser?
You send them to the browser, just not in the module itself.
Where do you get them from?
Same place you get the import statements you put in the script modules.
For ES6 it has to be from a build tool or from the server because no one is going to type these in twice.
A tool would certainly make things simpler, but I think you underestimate the motivations authors face when it comes to squeezing performance out of their Web pages.
Such a build tool is just not very difficult and it can be built in to dev servers so it has no impact on developers work flow.
Great! Such tools could trivially output the dependencies in a fashion interpretable by HTML processors.
The idea is that authors be able to predeclare (relevant parts of) the dependency tree such that when a node in that tree is needed, e.g. A in the example above, all the relevant nodes can be fetched in parallel, e.g. A and C in the example above, rather than in a serialised manner, e.g. first fetching A, then parsing it, then fetching C.
Authors have already predeclared the dependency relationships. Now you want them to re-declare them. Try it and let me know how many devs you convince.
I'm not trying to convince them, they're trying to convince me. I've received voluminous feedback over the years to the effect that they want to be able to do this.
One way to do this would be to predeclare the modules, as in:
<script type=module src=a.js id=a needs=c load-policy=when-needed> </script> <script type=module src=b.js id=b needs=c load-policy=when-needed> </script> <script type=module src=c.js id=c load-policy=when-needed> </script>
Go back to G+ and ask them if they plan to type thousands of such lines into their .html files. Talk about a scaling problem!
Yup, this wouldn't work when there's thousands of modules. It does when it's just a few dozen, though, which is the more common case. Luckily for us, sites with the complexity of G+ are the extreme, and they are also the cases where the developers are most likely to have the resources to produce their own custom solutions.
Having said that, my understanding is that G+ does actually ship this information to the client currently, in a custom format. So it would not be that much of a stretch to imagine them putting it in an HTML import, say. Something that would allow them to have the dependency chain, without blocking the initial page load.
One way to do this would be to predeclare the modules, as in:
<script type=module src=a.js id=a needs=c load-policy=when-needed> </script> <script type=module src=b.js id=b needs=c load-policy=when-needed> </script> <script type=module src=c.js id=c load-policy=when-needed> </script>
All of these scripts would need to be empedded in every page, which would significantly increase the size of a document and the complexity in creating a document. Not everyone is making a Single Page Application, and needing to maintain many lists of modules used on many pages and downloading that list on every page load seems wasteful.
Pre-fetching dependencies seems like it could most easily be implemented using ServiceWorkers1. A simple build tool, given a set of modules, can generate a JSON file with each module id (URI) and its dependencies (list of URIs). This JSON file would contain the entire dependency forrest from all roots to all leaves, and could be made available on the webserver for the ServiceWorker to download. The ServiceWorker now has the entire dependency graph, and can take the correct action when any one of the modules in the graph is requested. For example, when module A, which depends on B, is requested, it can start a request for B as well. Both of these modules will then be added to the cache, so by the time the Loader discovers that A depends on B, it will already have been downloaded from the server. The next time module A is requested (which can be another session, or in a few minutes), it would know that module B is in the cache, and nothing needs to be done.
There are other ways implement this, without having to send the entire dependency graph as a JSON file to the client (for example using bloom filters and a slightly smarter server), and I'm sure the community will find the best pre-fetching strategy with the tools already available to them.
Marius Gundersen
On Fri, 22 Aug 2014, Marius Gundersen wrote:
One way to do this would be to predeclare the modules, as in:
<script type=module src=a.js id=a needs=c load-policy=when-needed> </script> <script type=module src=b.js id=b needs=c load-policy=when-needed> </script> <script type=module src=c.js id=c load-policy=when-needed> </script>
All of these scripts would need to be empedded in every page
Only the ones that are needed. But actually it's not that bad. These dependencies can all be listed in an HTML import, which is then marked as heavily cachable. This ends up being just as cheap to pass to the server as a JSON file of the dependencies.
Pre-fetching dependencies seems like it could most easily be implemented using ServiceWorkers[1]. A simple build tool, given a set of modules, can generate a JSON file with each module id (URI) and its dependencies (list of URIs). This JSON file would contain the entire dependency forrest from all roots to all leaves, and could be made available on the webserver for the ServiceWorker to download.
That wouldn't let you do preparsing, which is a big part of the reason to want to prefetch resources.
(ServiceWorkers are an important part of the upcoming revolution in resource loading on the Web, but dependency handling is at a different level, namely, the ES6 loader level.)
On Mon, Aug 18, 2014 at 10:43 AM, Ian Hickson <ian at hixie.ch> wrote:
I assume you are imagining a densely connected graph with random access to any of the roots. I expect that real life pages have well partitioned graphs (widgets) that share some dense parts (utilities) and simple access patterns -- main page, utilities, a set of widgets.
But sure, it would be great to have a complete solution if it's not a lot more complex.
I guess you're proposing the send the dependency graph to the browser, then when a new root is needed, the stored graph is compared with the currently-loaded modules. The additional modules needed are then requested as a group. Up to this point we can just use build tools and browsers. How will we tell the server "send me this list of files all in one response?"
jjb