Maven like dependency management for browsers
Have you had a look at JSPM and SystemJS? They cover similar grounds in userspace, with the intent for SystemJS to be the upcoming standard Module Loader.
JSPM and SystemJS, IMHO, are not as elegant as the way Maven (Java) or Bundler (Ruby) can manage dependencies.
Apologies if the following explanation is redundant and you already know how Maven works, but just for the sake of wider audience I will briefly explain it here.
In Maven, each library is resolved by its coordinates: group id, artifact id, and version. For example, the Google Guava library is made of the following coordinates:
Group ID: com.google.guava mvnrepository.com/artifact/com.google.guava
Artifact ID: guava mvnrepository.com/artifact/com.google.guava/guava
Version: 18.0 mvnrepository.com/artifact/com.google.guava/guava/18.0
The short form notation is com.google.guava:guava:18.0.
Guava itself might depend on other dependencies, but you don't have to explicitly require them in your project. As long as you specify you require com.google.guava:guava:18.0, all its dependencies are added to your project automatically.
Most open source libraries are available in the Maven Central repository which is enabled by default. Users can additionally specify other repositories that Maven should query in order to find libraries with specific coordinates.
In SystemJS, baseURL looks similar to the notion of a Maven repository, but it is not as elegant: you can't have multiple baseURLs (if I am not wrong), and dependencies are specified relative to the baseURL, not as absolute coordinates and not versioned.
I am looking for something similar to SystemJS that can be used like this:
System.config({ repositories: [ 'cdn.w3c.org', 'cdn.google.com', 'cdn.example.com' ] });
System.import('com.modernizr:modernizr:2.8.3'); System.import('com.jquery:jquery-ui:1.11.4'); System.import('com.lodash:lodash:3.10.0');
The call to System.import('com.modernizr:modernizr:2.8.3') will first query cdn.w3c.org for com.modernizr:modernizr:2.8.3. If found, it will use that, if not it queries cdn.google.com and then cdn.example.com until it finds it or fails.
Similarly the call to System.import('com.jquery:jquery-ui:1.11.4') will do the same thing, but as jquery-ui depends on jquery, it will also download that.
One benefit of this compared to what is already out there is that at the moment if Site A uses CDN 1 for loading jQuery v1 and Site B uses CDN 2 for the same version of jQuery, the browser has to download both versions. But with this approach, as long as both sites depend on com.jquery:jquery-ui:1.11.4 it only has to be downloaded once.
On Thu, Jul 30, 2015 at 7:14 PM Sébastien Cevey <seb.cevey at guardian.co.uk>
wrote:
Hi Behrang,
Have you had a look at JSPM and SystemJS? They cover similar grounds in userspace, with the intent for SystemJS to be the upcoming standard Module Loader.
jspm/jspm-cli, systemjs/systemjs
On Thu, 30 Jul 2015 at 09:59 Behrang Saeedzadeh <behrangsa at gmail.com> wrote:
Hi,
Has this been discussed before? In a nutshell this diagram illustrates the idea:
- Rather than referencing a url in <script> or <style> tags, we require the browser to load name-spaced, versioned libraries. For example: com.jquery:jquery:1.0.
- This can either be done using the <script>/<style> tags similar to <script module="com.jquery:jquery:1.0"/> or <style module="com.jquery:jquery:1.0" /> or probably programmatically too: (in a JS file: require "com.jquery:jquery:1.0")
- There will be a central placed governed by W3C, for example, that can host libraries. Third-party repositories could also exist and be used (e.g. <script module="com.jquery:jquery:1.0" repository=" cdn.jquery.com"/>
- One benefit of this is dependency management. If jquery, for example, depends on lodash, it also gets downloaded and loaded automatically.
This is just the core of the idea. Nothing new (already similar systems are available for Java, Node, .NET, Ruby, etc.) but this will bring it to the web.
What do you think?
Best , Behrang Saeedzadeh
es-discuss mailing list es-discuss at mozilla.org, mail.mozilla.org/listinfo/es-discuss
Visit theguardian.com. On your mobile and tablet, download the Guardian iPhone and Android apps theguardian.com/guardianapp and our tablet editions theguardian.com/editions. Save up to 57% by subscribing to the Guardian and Observer - choose the papers you want and get full digital access. Visit subscribe.theguardian.com
This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software.
Guardian News & Media Limited is a member of Guardian Media Group plc. Registered Office: PO Box 68164, Kings Place, 90 York Way, London, N1P 2AP. Registered in England Number 908396
--
Best , Behrang Saeedzadeh
But Maven has many more facilities like the 'Bill of Materials'(BOM) that doesn't need a version. Moreover the dependencies are handled based on the project lifecycles like 'compile','test' etc.
Thanks, Mohan
On Fri, 31 Jul 2015 at 11:50 Behrang Saeedzadeh <behrangsa at gmail.com> wrote:
Hello Behrang,
JSPM and SystemJS, IMHO, are not as elegant as the way Maven (Java) or
Bundler (Ruby) can manage dependencies.
I am not contesting that, but it's worth describing what qualities of Maven you're asking for.
Group ID: com.google.guava mvnrepository.com/artifact/com.google.guava Artifact ID: guava mvnrepository.com/artifact/com.google.guava/guava Version: 18.0 mvnrepository.com/artifact/com.google.guava/guava/18.0
The short form notation is com.google.guava:guava:18.0.
With JSPM, package descriptors are of the form:
github:theefer/theseus at 0.5.0 npm:theseus at 0.5.0 etc.
The descriptor contains the registry resolver (e.g. github, npm, or any custom one). The package name can contain namespacing, as in the github example, if the registry supports it. The version is explicited and supports semver.
Guava itself might depend on other dependencies, but you don't have to
explicitly require them in your project. As long as you specify you require com.google.guava:guava:18.0, all its dependencies are added to your project automatically.
JSPM/SystemJS also resolve, install and automatically import a package's dependencies automatically.
Most open source libraries are available in the Maven Central repository which is enabled by default. Users can additionally specify other repositories that Maven should query in order to find libraries with specific coordinates.
This is similar to what registries give you in JSPM.
In SystemJS, baseURL looks similar to the notion of a Maven repository, but it is not as elegant: you can't have multiple baseURLs (if I am not wrong)
I'm not sure what you mean by that. If you're talking about your example where you have a list of registries and fallback to the next if a package isn't found in one, that seems rather like an anti-pattern on the web (esp if these are going to be calls from your browser).
If it's just about not having to specify which repository contains a given package, you're right, JSPM requires you to specify that, but it's really not that much of a burden, and it also makes the resolution more deterministic and robust given the variety of ways people package even the same project to different repositories (e.g. AMD/CJS/UMD, build/sources, etc).
and dependencies are specified relative to the baseURL, not as absolute coordinates and not versioned.
Not sure what you mean by that but everything is versioned in JSPM and you can lock the entire dependency tree (something that's less common in Maven AFAIK).
Versions can be specified either at the top-level config, or even at the import level, e.g.:
System.import('npm:lodash at 3.10.0').then(function() { console.log(.max([1, 2, 3, 4])); });
The call to System.import('com.modernizr:modernizr:2.8.3') will first query cdn.w3c.org for com.modernizr:modernizr:2.8.3. If found, it will use that, if not it queries cdn.google.com and then cdn.example.com until it finds it or fails.
You may be able to write a custom registry resolver that does something like that, if you really think that's a good idea.
Similarly the call to System.import('com.jquery:jquery-ui:1.11.4') will do the same thing, but as jquery-ui depends on jquery, it will also download that.
That'd already be the case.
One benefit of this compared to what is already out there is that at the moment if Site A uses CDN 1 for loading jQuery v1 and Site B uses CDN 2 for the same version of jQuery, the browser has to download both versions. But with this approach, as long as both sites depend on com.jquery:jquery-ui:1.11.4 it only has to be downloaded once.
That's only true if they both happen to provide the list of repositories in the same order. If not, they would each download jquery-ui from the first in the list and hence download it twice. The only way to avoid that would be to lookup if jquery-ui is in the browser cache for any of the repositories in the list -- obviously this is not allowed by the browser for security reasons.
Best,
PS: this discussion doesn't seem particularly relevant for es-discuss (?), so if you want to reply it may be best to do so in private to me to avoid unnecessary noise.
Looks like JSPM/SystemJS can do all the things I was looking for. Have to spend some time to read more about time.
Anyway, thanks for your help and feedback.
On Sun, Aug 2, 2015 at 10:38 PM Sébastien Cevey <seb.cevey at guardian.co.uk>
wrote:
On Fri, 31 Jul 2015 at 11:50 Behrang Saeedzadeh <behrangsa at gmail.com> wrote:
Hello Behrang,
JSPM and SystemJS, IMHO, are not as elegant as the way Maven (Java) or
Bundler (Ruby) can manage dependencies.
I am not contesting that, but it's worth describing what qualities of Maven you're asking for.
Group ID: com.google.guava mvnrepository.com/artifact/com.google.guava Artifact ID: guava mvnrepository.com/artifact/com.google.guava/guava Version: 18.0 mvnrepository.com/artifact/com.google.guava/guava/18.0
The short form notation is com.google.guava:guava:18.0.
With JSPM, package descriptors are of the form:
github:theefer/theseus at 0.5.0 npm:theseus at 0.5.0 etc.
The descriptor contains the registry resolver (e.g. github, npm, or any custom one). The package name can contain namespacing, as in the github example, if the registry supports it. The version is explicited and supports semver.
Guava itself might depend on other dependencies, but you don't have to
explicitly require them in your project. As long as you specify you require com.google.guava:guava:18.0, all its dependencies are added to your project automatically.
JSPM/SystemJS also resolve, install and automatically import a package's dependencies automatically.
Most open source libraries are available in the Maven Central repository which is enabled by default. Users can additionally specify other repositories that Maven should query in order to find libraries with specific coordinates.
This is similar to what registries give you in JSPM.
In SystemJS, baseURL looks similar to the notion of a Maven repository, but it is not as elegant: you can't have multiple baseURLs (if I am not wrong)
I'm not sure what you mean by that. If you're talking about your example where you have a list of registries and fallback to the next if a package isn't found in one, that seems rather like an anti-pattern on the web (esp if these are going to be calls from your browser).
If it's just about not having to specify which repository contains a given package, you're right, JSPM requires you to specify that, but it's really not that much of a burden, and it also makes the resolution more deterministic and robust given the variety of ways people package even the same project to different repositories (e.g. AMD/CJS/UMD, build/sources, etc).
and dependencies are specified relative to the baseURL, not as absolute coordinates and not versioned.
Not sure what you mean by that but everything is versioned in JSPM and you can lock the entire dependency tree (something that's less common in Maven AFAIK).
Versions can be specified either at the top-level config, or even at the import level, e.g.:
System.import('npm:lodash at 3.10.0').then(function() { console.log(.max([1, 2, 3, 4])); });
The call to System.import('com.modernizr:modernizr:2.8.3') will first query cdn.w3c.org for com.modernizr:modernizr:2.8.3. If found, it will use that, if not it queries cdn.google.com and then cdn.example.com until it finds it or fails.
You may be able to write a custom registry resolver that does something like that, if you really think that's a good idea.
Similarly the call to System.import('com.jquery:jquery-ui:1.11.4') will do the same thing, but as jquery-ui depends on jquery, it will also download that.
That'd already be the case.
One benefit of this compared to what is already out there is that at the moment if Site A uses CDN 1 for loading jQuery v1 and Site B uses CDN 2 for the same version of jQuery, the browser has to download both versions. But with this approach, as long as both sites depend on com.jquery:jquery-ui:1.11.4 it only has to be downloaded once.
That's only true if they both happen to provide the list of repositories in the same order. If not, they would each download jquery-ui from the first in the list and hence download it twice. The only way to avoid that would be to lookup if jquery-ui is in the browser cache for any of the repositories in the list -- obviously this is not allowed by the browser for security reasons.
Best,
PS: this discussion doesn't seem particularly relevant for es-discuss (?), so if you want to reply it may be best to do so in private to me to avoid unnecessary noise.
Visit theguardian.com. On your mobile and tablet, download the Guardian iPhone and Android apps theguardian.com/guardianapp and our tablet editions theguardian.com/editions. Save up to 57% by subscribing to the Guardian and Observer - choose the papers you want and get full digital access. Visit subscribe.theguardian.com
This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software.
Guardian News & Media Limited is a member of Guardian Media Group plc. Registered Office: PO Box 68164, Kings Place, 90 York Way, London, N1P 2AP. Registered in England Number 908396
--
Best , Behrang Saeedzadeh
On 8/7/15, Behrang Saeedzadeh <behrangsa at gmail.com> wrote:
Hi Sebastien,
Guys —
On Sun, Aug 2, 2015 at 10:38 PM Sébastien Cevey <seb.cevey at guardian.co.uk> wrote:
[...]
Versions can be specified either at the top-level config, or even at the import level, e.g.:
System.import('npm:lodash at 3.10.0').then(function() { console.log(.max([1, 2, 3, 4])); });
That looks like the client (browser) would need to fetch the dependency list and mapping?
Declarative chain of responsibility pattern was proposed years ago. The idea is different: It works more like ANT <target>s, where
dependencies are declared right there in the document itself.
The thread got some noise and then it died.
<script src="fabric.js" async id="fabric"> <script depends="fabric"> alert("loaded!"); </script>
I still haven't looked at JSPM/SystemJS thoroughly, but regarding your comment, I personally prefer the dependencies get resolved automatically. For example, we can have www.example.com/app-deps.json where app-deps.json contains something like this:
{ dependencies: [ { name: 'backbone', src: 'www.example.org/backbone.js', dependencies: [ { name: 'underscore', src: 'www.example.org/underscore.js' } ] }, { name: 'jquery-ui', src: 'www.example.org/jquery-ui.js', dependencies: [ { name: 'jquery', src: 'www.example.org/jquery.js' } ] } ] };
Then we can use a loader system to load the dependencies:
Loader.load('www.example.com/app-deps.json');
Now this only works efficiently if we have a central repository similar to Maven, otherwise there will be a request hell for each app/page/site to download these dependencies. But if they all point to the same CDN/Repository or use IDs+MD5/SHA1 hashes rather than URLs (e.g. org.backbone:backbone:1.0), it will help the browser to avoid downloading the same version of the same library more than once just because one is fetched from foo.com/backbone-1.0.js and the other one from bar.com/backbone-2.0.js.
Has this been discussed before? In a nutshell this diagram illustrates the idea:
i.imgur.com/X1n57iQ.png
This is just the core of the idea. Nothing new (already similar systems are available for Java, Node, .NET, Ruby, etc.) but this will bring it to the web.
What do you think?