Announcing TC39 ECMAScript Test Suite Project - Test262
Allen,
is the Test262 test suite more similar to W3C suites or might there
be an acidic element?
ie something the public might understand?
Jonathan Chetwynd
The intent is a suite that tests an implementation against the actual requirement of the Ecma-262 standard. It will be based upon and similar to what you can already see in the Sputnik and ES5conform suites. Those tests generally identify a specific requirement of a specific section of the spec. and then try to prove that the requirement is/isn't met by an implementation.
Give that the standard specification itself is not particularly understandable to the general public, it probably isn't reasonable to expect that all the tests of the specification would be understandable to the same public.
However, reasonable understandability is a good goal. What are some of the characteristics of the W3C suites that make them "acidic"?
On Mon, Jun 21, 2010 at 1:42 PM, Allen Wirfs-Brock <Allen.Wirfs-Brock at microsoft.com> wrote:
The intent is a suite that tests an implementation against the actual requirement of the Ecma-262 standard. It will be based upon and similar to what you can already see in the Sputnik and ES5conform suites. Those tests generally identify a specific requirement of a specific section of the spec. and then try to prove that the requirement is/isn't met by an implementation.
Give that the standard specification itself is not particularly understandable to the general public, it probably isn't reasonable to expect that all the tests of the specification would be understandable to the same public.
However, reasonable understandability is a good goal. What are some of the characteristics of the W3C suites that make them "acidic"?
I believe that the various "Acid" tests for rendering are what's being referred to here [1]. I think the relevant characteristc is that anyone can run them in their browser, and they display a visual rendering of how "good" a job the browser does, making them into easily-distinguishable markers of conformance to whatever is tested. It's not clear to me whether this would be good for this test suite.
(I'm going to try to shift this thread to the test262-discuss list, by bccing: es-discuss.)
I think we all agree that a web based test runner is part of what we want to have. It seems inevitable that some sort of pass % or similar score has to display even though it's not clear whether such a score is very meaningful for this style of test suite. You can get big fluctuation because a single bugs can cause many individual tests to fail while in other cases a really significant bugs might only show up in a small number.
If anybody has any ideas for a meaningful metric that doesn't have those sorts of sensitivities, let's hear it.
At the last TC39 meeting we agreed to launch a project to develop an official ECMAScript specification test suite that developers and users can use to verify that implementations accurately implement the standard. The hope is that we can integrate the various existing test suites along with new tests to create a single comprehensive test suite for the ECMAScript standard.
Because of the nature of Ecma's intellectual property policies, only Ecma members can directly contribute code to the project. However, the project code will be BSD licensed and development will be carried out in a publically accessible manner. The working name of the test suite is Test262 and I'll be serving as the initial project editor. The source code of the project will be hosted in the TS39 repository at hg.ecmascript.org (coming soon) and issues will be tracked at bugs.ecmascript.org/report/15 . In addition, I'm setting up a separate test262-discuss mailing list for topics relating to ECMAScript testing and the Test262 test suite. Test262-discuss is open to the public. You can subscribe at t mail.mozilla.org/listinfo/test262-discuss
I'm now in the early stages of organizing the project infrastructure and collecting initial code contributions from Ecma members. To kick things off, Microsoft is contributing the ES5Confrom test suite as well as a significant number of additional tests. Christian Plesner Hansen, leader of the Sputnik project , reports that Google will contribute their Sputnik code. Other TC39 members have all been supportive of this effort and I anticipate that we will receive additional contributions from them in the near future.
So, if you're interested in ECMAScript testing subscribe to test262-discuss and join in. After people have had a couple of days to subscribe, I'll be kicking things off with some of my ideas about how to structure and organize the Test262 suite. Please also share your thoughts.
Thanks, Allen Wirfs