Discuss
  • A few module ideas

A few module ideas

# James Browning (7 years ago)

These are just some ideas I've had for improving es modules based on my experiences with them. The syntax and stuff with them isn't too important, the main point is the problem I'm trying to solve with each of them, feedback and criticism is welcome.

Dynamic modules

One of the biggest issues I've had with ES modules is not being able to load classic scripts as part of the dependency graph, one of the solutions I've used but am not particularly happy with is having an async loader function e.g.:

function loadScript(url) {
    return new Promise(resolve => {
        const scriptElem = document.create('script')
        scriptElem.src = url
        scriptElem.onload = resolve
    })
}

// some other file

async function computeSpline() {
    await loadScript('./mathjs.js')
    // use math here
}

And while this approach works somewhat it's a bit of a pain for a couple reasons:

  • If something gains a script dependency it necessarily breaks all consumers by becoming asynchronous even if the original operations were synchronous
  • It's not generic for other types of resources e.g. I can't load an image without creating another loader function or so on

My proposed solution, dynamic (but static) export:

// math.mjs
import loadScript from ".../loadScript.js"

loadScript('./math.js').then(_ => {
    const math = window.math
    delete window.math
    export({
        math as default
    })
})

This solution is also generic so it can be used for loading any type of resource:

// highPerformanceMath.mjs
fetch('.../math.wasm').then(response => response.arrayBuffer())
    .then(buffer => WebAssembly.instantiate(buffer, {}))
    .then(({ instance }) => {
        export({
            instance.exports as default
        })
        // or potentially named exports
        export({
            instance.exports.fastFourierTransform as fastFourierTransform
            ...
    })

Now this solution would be nice because it's generic and allows for loading any (even asynchronous) object as part of the module graph and doesn't cause explosions where because one part becomes asynchronous everything becomes asynchronous.

However there is a deficiency in that it can be quite verbose for similar tasks e.g. loading WebAssembly modules which is why I thought of idea 2:

Module Arguments

Effectively module arguments would allow passing data to a module (statically) during loading e.g.:

// some-file.js
import dict from "./dictionary.js" where { lang = "en-US" }

// dictionary.js
fetch(`./dictionaries/${ import.arguments.lang }.txt`)
    .then(response => response.text())
    .then(text => export({
        JSON.parse(text) as default
    })

This solves the previous problem of very similar dynamic modules for similar types by allowing details like that to be passed in as arguments e.g.:

import math from "./loadScript.mjs" where {
    script = './math.js',
    globalName = 'math'
}

Lazy Export-From

One of the nice things about named exports is you can minimize the amount of mostly similar import declarations e.g.:

import map from "./lodash/map.js"
import filter from "./lodash/filter.js"
import flatMap from "./lodash/flatMap.js"
...

// can become
import { map, filter, flatMap, ... } from "./lodash/lodash.js"

However it has a major downside of massively increasing the amount of fetch/parse/execute time for all those additional things exported by the combined module.

My idea is to allow modules to declare that parts need to not be fetched parsed or executed if they're not actually imported e.g.:

// my-operators-library.js
static export { map } from "./map.js"
static export { filter } from "./filter.js"
static export { reduce } from "./reduce.js"

Effectively all my idea adds is the static export (syntax not important) form that effectively says these names should only be resolved if they're actually imported and can be safely ignored if they're not used. This way you get both the benefits of collection modules (easier to import and reduces duplication) and the benefits of individual imports (lesser loading sizes).

Summary

Basically the ideas suggested here are to solve these particular problems I've had with ES modules:

  • Unable to load classic scripts (and other types of resources statically e.g. conditional modules) as part of the module graph
  • Unable to specify more specific behavior for a module to prevent duplication
  • Either have to have lots of almost duplicate import declarations or have to load unnecessary files

The solutions I proposed aimed to keep the constraint that module exports should remain statically parsable which is why export({ ... }) shares the syntactic form.

I refrained from specifying the semantics of the specific operations as there's details that'd need to be sorted out for all of them if there is any interest whatsoever in implementing them.

# kai zhu (7 years ago)

for the forseeable future, the only practical solution i see to these frontend performance issues is to avoid using es modules, and revert to the simple, old-fashioned method of assigning libraries to the window namespace.

# Caridy Patino (7 years ago)

export values are dynamic today! you can do:

let lib = null; export default lib; loadScript("foo").then(o => lib = o);

but what cannot be dynamic, is the set of export names, which shapes the module.

# dante federici (7 years ago)
  • Unable to load classic scripts (and other types of resources statically e.g. conditional modules) as part of the module graph

How are conditional imports static? In both examples I see the module as being async, and therefore every dependent module is async. Your "dynamic but static" is explicitly using "then" -- or are you implying a module exporting async resources is a better solution than an async module?

  • Unable to specify more specific behavior for a module to prevent duplication

By passing arguments in, what do you expect to occur? Do you expect the module itself to be run with those arguments, exporting a set of things? How is that any better than just importing a constructor function from the module/library? This problem sounds like designing the library in a better way would make more sense than affording config to be passed into import, which would mean each import would re-run the module, so no caching.

  • Either have to have lots of almost duplicate import declarations or have to load unnecessary files

I can see a benefit for reducing files in the static export -- that suggestion has been a good example of existing problems with tree shaking d3, to which the response has been "design better exports". As for the multiple fetch part of the problem, HTTP/2 spec addresses the performance hit for that, and it's effectively what you're asking the "static" prefix to assert. Out of curiosity, how would you expect static to work in the first place? Who would do the assertion that it doesn't depend on any other symbol in the module it is a part of?

I feel like out of these, the solution is much closer to "Better library design". I'm still not 100% on how your dynamic example addresses "turns my code async". Static export is an interesting one -- effectively asking for pure symbols. Maybe identify an entire file as "load only these symbols, ignore other source"?

# Andrea Giammarchi (7 years ago)

I've solved this (for my needs) long time ago, the pattern is the following:

export default new Promise(async $export => {
  // await anything that needs to be imported
  // await anything that asynchronous
  // finally export the module resolving the Promise
  // as object, function, class, ... anything
  $export(
    {module: 'object'} ||
    function () {}     ||
    class Anything {}
  );
});

You can do pretty much everything you need as both consumer or exporter.

// ES2017 Asynchronous Export
// module.js
export default new Promise(async $export => {
  const module = await Promise.resolve(
    {my: 'module'}
  );
  $export(module);
});

// - - - - - - - - - - - - - - - - - - - - - - - - - - - -

// ES2015 consumer
import module from './module.js';

module.then(exports => {
  // will log "module"
  console.log(exports.my);
});

// - - - - - - - - - - - - - - - - - - - - - - - - - - - -

// ES2017 consumer
(async () => {
  const module = await (
    await import('./module.js')
  ).default;
})();


// - - - - - - - - - - - - - - - - - - - - - - - - - - - -

// ES2017 consumer and exporter
export default new Promise(async $export => {
  const module = await (
    await import('./module.js')
  ).default;
  $export({module, method(){}});
});

The pattern easily inter-operate with CommonJS

// CommonJS consumer and/or importer
module.exports = new Promise(async $export => {
  const module = await require('./module');
  $export({module, method(){}});
});

I still don't understand why it's difficult to imagine asynchronous exports when it's apparently normal to imagine asynchronous imports .... but that's another story.

Best

# James Browning (7 years ago)

export values are dynamic today! you can do:

let lib = null; export default lib; loadScript("foo").then(o => lib = o);

but what cannot be dynamic, is the set of export names, which shapes the module.

Cariday, while interesting the main problem with this approach is it doesn't guarantee that the desired module is actually usable at any particular point e.g. this might not work:

// math.mjs
let math
export { math as default }
loadScript('./math.js').then(_ => {
    math = window.math
    delete window.math
})

// cardinalSpline.mjs
import math from "./math.mjs"

export default function cardinalSpline() {
   // use math to compute the spline here
}

// import that function
import cardinalSpline from "./cardinalSpline.mjs"

// I can't reliably use cardinalSpline here yet as math might still be undefined
// even after asynchronous work I can't reliably use it as the script
// might still be fetching

Dante, I didn't really clarify what I meant by conditional exports, I don't mean things that sometime export but sometimes don't, the names exported should remain static but rather what is exported depends on (possibly asynchronous) things e.g.:

if (typeof self !== 'undefined') {
    // Browser like
    loadScript('./math.js')
        .then(_ => {
             const math = window.math
             export({ math })
             delete window.math
        })
} else {
    // Try node like
    // we'll ignore other environments for now for simplicity
    export({ require('mathjs') as math })
}

I'd expect it to be a syntax error if the exported names in one export(...) were different to any other export(...) to preserve the staticness of names in ES modules.

Concerning the other point modules in the browser are already async, it's just not observable that they're async and that's the whole point of why I want dynamic modules is that they can do asynchronous work before completion, note that the imported module is available synchronously as the module won't be executed until the export resolves e.g.:

// math.mjs
loadScript('./math.js').then(_ => {
    const math = window.math
    export({ math })
    delete window.math
})

// other file
import math from "./math.mjs"

// We can reliably use math here as this file will not
// be executed until export({ ... }) is reached
// in my idea export(...) is similar to Promise's resolve

Foreign module types is nothing new the spec is specifically designed for them, this is how CommonJS will work with import commonJS from "commonJSmodule". My idea is simply to add a way to add those dynamic module types as a part of the language instead of part of the loader.

By passing arguments in, what do you expect to occur? Do you expect the module itself to be run with those arguments, exporting a set of things? How is that any better than just importing a constructor function from the module/library? This problem sounds like designing the library in a better way would make more sense than affording config to be passed into import, which would mean each import would re-run the module, so no caching.

Yes I'd expect it to evaluate multiple times (but fetch/parse only once) which saves round trips, I mostly only thought of it because of the way I suggested how dynamic export could work, without dynamic export it's not particularly useful, it's mostly for reducing the amount of those script/wasm -> es module modules.

Admittedly I hadn't really thought module arguments through that much (would same arguments result in the same module object, etc etc), the whole idea might be rubbish, but the main problem I was trying to solve with them was automatic creation of dynamic modules so that you wouldn't need a module like:

// highPerformanceMath.mjs
fetch('.../math.wasm').then(response => response.arrayBuffer())
    .then(buffer => WebAssembly.instantiate(buffer, {}))
    .then(({ instance }) => {
        export({
            instance.exports as default
        })
        // or potentially named exports
        export({
            instance.exports.fastFourierTransform as fastFourierTransform
            ...
    })

for every single WebAssembly module you wanted to import and use synchronously, although in retrospect you probably would still need to anyway if you want to name the exports as the export({...}) syntax is still a static declaration of names (it's not an object you can just populate with names).

I can see a benefit for reducing files in the static export -- that suggestion has been a good example of existing problems with tree shaking d3, to which the response has been "design better exports". As for the multiple fetch part of the problem, HTTP/2 spec addresses the performance hit for that, and it's effectively what you're asking the "static" prefix to assert. Out of curiosity, how would you expect static to work in the first place? Who would do the assertion that it doesn't depend on any other symbol in the module it is a part of?

HTTP/2 is orthogonal to the goal of static export ... from, basically HTTP/2 allows for serving all dependencies faster when the dependency graph is known. My idea of static export ... from is basically built-in tree shaking, if a name isn't imported then that part of the module graph is simply not fetched/parsed/evaluated for example:

// Note that my suggestion *only* works with export-from
// it does not work with plain `export` as that is already
// fetched and parsed

// operators.mjs
static export { map } from "./operators/map.js"
static export { filter } from "./operators/filter.js"
static export { reduce } from "./operators/reduce.js"
static export { flatMap } from "./operators/flatMap.js"

// other file
import { map, flatMap } from "./operators.mjs"
// only ./operators/map.js and ./operators/flatMap.js
// will be fetched parsed and executed (assuming no other modules)

// another example
import * as operators from "./operators.mjs"
// we can't reason that some of the things might not be used
// so files are fetched/parsed/evaluated

I feel like out of these, the solution is much closer to "Better library design". I'm still not 100% on how your dynamic example addresses "turns my code async". Static export is an interesting one -- effectively asking for pure symbols. Maybe identify an entire file as "load only these symbols, ignore other source"?

The problem is while it's easy to design within your own code a good API, if you include a classic script as part of the dependency graph currently then that forces things to become async for example this simple example:

// classic script loaded to access functions
// as math.mjs
export default loadScript('./math.js')
    .then(_ => {
        const math = window.math
        delete window.math
        return math
    })

// cardinalSpline.mjs
import math from "./math.mjs"

// This function is needlessly async, if math.js were an ES module
// this function would easily be synchronous, only the
// fact that I had to load a classic script is this async
async function cardinalSpline(points, divisions) {
    const m = await math
    // compute cardinal spline points here
}

The worst part about this is if any module needs to load a classic script it potentially explodes throughout the code base converting many previously synchronous operations into needlessly asynchronous ones.

The whole point of my dynamic module idea was so that a classic script can be added as a dependency which is part of the module graph, but doesn't cause an explosion where previously synchronous functions become asynchronous just because of a classic script.

Now I've never actually let the explosion thing happen because instead of turning all the codebase into async functions for otherwise synchronous things I tend to just take the code of the library, convert it to a module myself and then use it. But this is time costly, converting all these classic scripts (or pulling out parts of them I need) into ES modules is just cumbersome.

The fact that dynamic modules allow for potentially any fetch/parse/evaluation desired (e.g. WebAssembly, HTML modules, anything you want really) is just a nice consequence of the problem I was trying to solve, the fact it allows for so many generic use cases is why I suggested it should be part of the language itself.

If there's no interest in implementing dynamic modules then I might just suggest the idea of import math from "script:./math.js" as part of the HTML spec for loading classic scripts as part of the module graph, but I think dynamic modules would be more powerful and useful.

# James Browning (7 years ago)

Just as a follow up to these ideas, the first proposal I suggested can be entirely resolved with the top level await proposal: MylesBorins/proposal-top-level-await.

As a consequence the Module Arguments idea is acceptably resolvable within Node/Browsers anyway by using url parameters (as unique url parameters will trigger multiple module instances) so the dictionary example could be written as:

// dictionary.js
...

const language = new URL(import.meta.url).searchParams.get("lang") ||
"en-US";
const languageData = await loadData(new URL(`../langs/${ language }.js`,
import.meta.url)

export default makeDictionary(languageData)

// main.js
import englishDict from "./dictionary.js"
import spanishDict from "./dictionary.js?lang=es-ES"

The only downside to this approach is that there might be redundancy in presence of multiple such args as import foo from "./foo.js?bar=12&cats=13" would be different from ... "./foo.js?cats=13&bar=12" but I think it's relatively unimportant as I can't actually think of a concrete example of multiple arguments.

Although I actually think in the presence of top-level-await that the Lazy-Export from proposal would be even better, as even in the presence of tree-shaking build tools those tools may not be able to determine where a side-effect is actually important or not to the exported values. e.g. Consider the above code, can a tree-shaker viably detect that the above file is (pseudo) side-effect free? With complex usage this may become in general too difficult.

Whereas given a static form a tree-shaking tool need not even consider static export { someExport } from "./someModule.js" if someExport is never actually imported in the receiving code. For example consider the following code:

// audioTools.js

export { default as audioContext } from "./audioContext.js"
export { default as simpleBeat } from "./simpleBeat.js"
...

// audioContext.js
export default new AudioContext()

// main.js
import { simpleBeat } from "./audioTools.js"

/* Create own OfflineAudioContext */
const ctx = new OfflineAudioContext()

...

simpleBeat.connect(ctx,destination)
ctx.startRendering().then(...)

In this example audioContext.js has real side-effects (as there can only be up to 6 or so live audio contexts in most browsers and because the module lifetime persists with the page it can't be garbage collected). But yet audioContext is never actually used so it's a pointless side-effect. Can a tree-shaking tool detect this? Maybe? But I'm skeptical that tree-shaking tools can detect all of these what I'm referring to as pseudo side-effects.

Note that while I'm arguing in the context of build tools, this also applies outside of build tools as it still potentially saves many loads on small sites where a build tool may not be necessary (e.g. dynamic blogs, or so on). Especially in the case of hundreds of exports (e.g. import { util1, util2 } from "./lodash.js" could just work only loading the three files lodash.js, lodash/util1.js, lodash/util2.js) as it doesn't actually need to construct the whole module record.

If there's any TC39 members that would actually be interested in championing something like this I'd be happy to write some material and draft spec-text to clarify the exact problems and solution I'm proposing in this idea.