Should we move our test suites to ecmascript.org?
There didn't seem to be any follow-up discussion on this, that I could see. Looking at the (top-level) change logs for es5conform and sputniktests, you can see there has been some working going on since Mark's post.
Any status on the individual suites, a combination of the suites, or re-hosting them?
On Sep 12, 2009, at 2:38 PM, Mark S. Miller wrote:
Currently, there are two open source EcmaScript test suites and one open source JavaScript test suite.
- es5conform.codeplex.com for testing the differences between ES5 and ES3
- code.google.com/p/sputniktests for testing ES3
- www.mozilla.org/js/tests/library.html for testing Mozilla's JavaScript.
In committee, we all agreed that we like the structure of the first two, as they follow the structure of the spec. We also agreed that we'd like the results of developing and possibly merging these test suites to be hosted at ecmascript.org, and to eventually earn some kind of status as the official conformance test suites. The open question is where to do the development. Having the development be split among three, or even two, code development sites that some participants might regard as partisan, and that operate in somewhat different ways, is doable but not optimal.
Current resources at ecmascript.org that one might start from for joint development are hg.ecmascript.org and bugs.ecmascript.org . Are these adequate starting points? Are there other suitable resources hosted by Ecma? Could there be? Is this realistic? What about other alternatives like sourceforge or github? Or are we better off just sticking with the current split development that, frankly, is working rather well?
Patrick Mueller - muellerware.org
Sorry, I missed Mark's original message.
Who maintains *.ecmascript.org? If we can expect it to be reliable I would have no problem moving development of the sputnik test suite there. Actually I would be prepared to move as soon as we can agree on what structure we want. One small issue is how to review code changes. Sputnik already uses a "third-party" tool (as in it's developed by google but not an integrated part of code.google.com) and for now we could continue to use that, just from a different base repository.
As for merging sputnik with es5conform and the subset of the mozilla tests that reflect the spec I see no reason why we couldn't do that, other than the work it would take to merge the heterogeneous frameworks.
On Dec 8, 2009, at 2:03 PM, Christian Plesner Hansen wrote:
Sorry, I missed Mark's original message.
Who maintains *.ecmascript.org?
I do with help from others (David-Sarah has helped fix up the trac,
for example).
If we can expect it to be reliable I would have no problem moving development of the sputnik test suite there. Actually I would be prepared to move as soon as we can agree on what structure we want. One small issue is how to review code changes. Sputnik already uses a "third-party" tool (as in it's developed by google but not an integrated part of code.google.com) and for now we could continue to use that, just from a different base repository.
Can you mail me about what is required to host Sputnik and this tool,
in detail? Thanks.
As for merging sputnik with es5conform and the subset of the mozilla tests that reflect the spec I see no reason why we couldn't do that, other than the work it would take to merge the heterogeneous frameworks.
This is a lot of work, so the best way to do it is according to a plan
we all agree with, but almost ceratinly not with any "big bang"
integration. Just lots of patches. The separate tests will have their
own integrity and greater self-consistency for a while. If we get a
single unified suite, we'll know when it is "ready".
I believe that there are still IPR policy issues that need to be worked through before any test suite development that is affiliated with ECMA/T39 could accept contributions from organizations or individuals who do not have an ECMA membership affiliation. An advantage of the current google/codeplex/mozilla projects, is that they don't have this restriction. While ecmascript.org is not exactly officially associated with ECMA/TC-39 it is close enough that I don't think we should try to host a test suite project there until we resolve the IPR issues.
I don't particularly see why TC39 would want to go to the trouble of building and supporting a "foundry". It's enough trouble to do what is necessary to setup and manage individual project without having to be responsible for all the underlying infrastructure. I personally think that both the google code and codeplex hosting environments are fine and future work could be done using either foundry (or another that isn't tainted by a vender affiliation, if this continues to be a concern).
My sense is that both Google and Microsoft has taken the necessary steps to ensure that the projects we initiated can be very open. However, I also think that they both may suffer from a perception that they aren't. Note that I'm a "committer" on the sputnik project and Christian Plesner Hansen of Google and Rob Sayre of Mozilla are "coordinators" on the ES5conform codeplex project. However, I don't think any of us have exercised any of the associated privileges other than for our "own" projects.
I think the biggest issues that we need to resolve to move forward are about project planning, coordination, and process. What needs to be done. How should it be done. Who is going to commit to doing to the individual tasks? I've informally suggested a couple time that maybe we need to have an open planning "summit" do work these things out. I still think it would be a good idea.
On Dec 8, 2009, at 3:43 PM, Allen Wirfs-Brock wrote:
I believe that there are still IPR policy issues that need to be
worked through before any test suite development that is affiliated
with ECMA/T39 could accept contributions from organizations or
individuals who do not have an ECMA membership affiliation. An
advantage of the current google/codeplex/mozilla projects, is that
they don't have this restriction. While ecmascript.org is not
exactly officially associated with ECMA/TC-39 it is close enough
that I don't think we should try to host a test suite project there
until we resolve the IPR issues.
My main concern about these projects is not hosting or control, but
the fact that there are several of them. I think it would be more
useful to the community to have a single ECMAScript conformance test
suite, covering all of ES5, including the bits inherited from ES3. If
all parties are willing to use one of the existing sites for hosting,
that is not a problem as far as I'm concerned.
Note: the WebKit project has a number of ECMAScript tests at <trac.webkit.org/browser/trunk/LayoutTests/fast/js
, though we have not sorted over them to determine which are for
ECMAScript proper rather than extensions, and which would be suitable
as conforamance tests. We would consider donating tests to a shared
ECMAScript conformance test suite if there were a single canonical
test suite. Many of our tests cover edge cases that may not otherwise
be covered by a conformance suite.
, Maciej
I believe that there are still IPR policy issues that need to be worked through before any test suite development that is affiliated with ECMA/T39 could accept contributions from organizations or individuals who do not have an ECMA membership affiliation. An advantage of the current google/codeplex/mozilla projects, is that they don't have this restriction. While ecmascript.org is not exactly officially associated with ECMA/TC-39 it is close enough that I don't think we should try to host a test suite project there until we resolve the IPR issues.
I agree that for ECMA to give its stamp of approval, whatever form that might take, causes extra complications. But I don't know why ecmascript.org should be so closely linked with ECMA that we can't use it for this. Isn't it up to us, Brendan really, to decide how to use it?
My main concern about these projects is not hosting or control, but the fact that there are several of them. I think it would be more useful to the community to have a single ECMAScript conformance test suite, covering all of ES5, including the bits inherited from ES3. If all parties are willing to use one of the existing sites for hosting, that is not a problem as far as I'm concerned.
I think it's important that this is and is perceived as, now and in the long run, a vendor neutral project. If we host it with vendor A, for any choice of A including google, it's much too easy for people to consider it A's project, and in particularly for other vendors to not consider it theirs.
Note: the WebKit project has a number of ECMAScript tests at trac.webkit.org/browser/trunk/LayoutTests/fast/js, though we have not sorted over them to determine which are for ECMAScript proper rather than extensions, and which would be suitable as conforamance tests. We would consider donating tests to a shared ECMAScript conformance test suite if there were a single canonical test suite. Many of our tests cover edge cases that may not otherwise be covered by a conformance suite.
I've been using the ES5 layout tests myself and they are indeed very thorough. Having them be part of the common test suite would be excellent.
Currently, there are two open source EcmaScript test suites and one open source JavaScript test suite.
In committee, we all agreed that we like the structure of the first two, as they follow the structure of the spec. We also agreed that we'd like the results of developing and possibly merging these test suites to be hosted at ecmascript.org, and to eventually earn some kind of status as the official conformance test suites. The open question is where to do the development. Having the development be split among three, or even two, code development sites that some participants might regard as partisan, and that operate in somewhat different ways, is doable but not optimal.
Current resources at ecmascript.org that one might start from for joint development are hg.ecmascript.org and bugs.ecmascript.org . Are these adequate starting points? Are there other suitable resources hosted by Ecma? Could there be? Is this realistic? What about other alternatives like sourceforge or github? Or are we better off just sticking with the current split development that, frankly, is working rather well?