I think it is a reasonable goal to have "complete" (possibly shallow) spec coverage in test262 prior to ES6 ratification. I am not sure if this goal is officially stated but it's something I'm working toward regardless. I will need significant help from other implementers and the community to reach this goal, though. I can't speak for other implementers, but any spec ambiguities or bugs we come across are brought up either on this mailing list or at a TC-39 meeting.

I understand the frustration with lack of Test262 progress; it would have been preferable to start this effort in earnest earlier. I'm at a point now where I can start doing more, and I hope you can help as well with making the test suite more useful for implementers and the community and getting as many high quality tests as we can in the suite. The thread linked by Rick is indeed just step 1. It makes sense to me to get some fundamentals in place before we start asking for contributions so we don't have to do more work than necessary. Hope this makes sense, but if not, feel free to chime in on test262-discuss!

From: es-discuss-bounces at mozilla.org [mailto:es-discuss-bounces at mozilla.org] On Behalf Of David Bruant
Sent: Wednesday, April 24, 2013 2:37 PM
To: Rick Waldron
Cc: EcmaScript
Subject: Re: ES6 __proto__ test suite

Le 24/04/2013 20:08, Rick Waldron a écrit :
On Wed, Apr 24, 2013 at 6:14 AM, David Bruant <bruant.d at gmail.com<mailto:bruant.d at gmail.com>> wrote:
Le 23/04/2013 23:47, Rick Waldron a écrit :
On Tue, Apr 23, 2013 at 1:34 PM, David Bruant <bruant.d at gmail.com<mailto:bruant.d at gmail.com>> wrote:
Hi,

Based on recent messages on es-discuss, I feel that both es-discuss and apparently even TC39 meetings with notes have left ambiguity in what people understood the TC39 agreement was. I believe this ambiguity is due to this pretty bad communication format called the English language (For anyone in doubt, French is as bad; I'm afraid it's a property inherent to natural languages :-) ).

This is wasting everyone time and energy. This generate frustration additional to the already existing frustration caused by standardizing __proto__ at all.
So I would like to encourage TC39 to discuss around and create consensus around a test suite.

Do you mean something other then the one that already exists?

http://test262.ecmascript.org/#<http://test262.ecmascript.org/>
If TC39 adds tests to this test suite *before* ES6 becomes an official standard, use this, yes. Please writes hundreds of tests for ES6.
For the anecdote, I've started a test suite for proxies [1] and the exercise led to feedback [2][3], some of which were spec bugs [4][5]. So I don't know, maybe there is some virtue to write tests before the spec is shipped. Not for the sake of writing tests or even the sake of getting a conformance test suite, but for the spec of spending time carefully reviewing the drafts and catching spec bugs early. For the sake of having a structured medium to discuss on and not just plain-text emails with occasional code snippets.

But I don't see TC39 adding tests. Worse, I see the same conversations happening over and over on __proto__. Even after the January TC39 meeting. Even with the notes, ambiguities and misunderstanding remain.

So, until tests are added to http://test262.ecmascript.org/, I propose using a test suite not as a conformance tool, but as a conversation medium. This is an attempt to move the conversation from words like "poisoned", "realm", "magic" (!) to a conversation where there are a bunch (20, 50, 100?) of test cases where TC39 says "for all of these cases, we agree the test must pass" and where people can have a very concrete medium to point out and say "I agree this test must pass, but this more subtle test case must pass too" (refining the previous test case)

The goal of the test suite I have started is not to run it. It's for human beings to discuss around it; to read it, explain why they disagree, correct an existing test or add one for each disagreement. A communication medium in essence.
At this point, I believe that a test suite would be an excellent complement to meeting notes to capture consensus.


Again, I suggest subscribing to https://mail.mozilla.org/listinfo/test262-discuss
I am already subscribed.


This conversation is happening right now, starting here: https://mail.mozilla.org/pipermail/test262-discuss/2013-April/000169.html
This thread is about how to organize ES6 tests (folders structure, Hg branches, etc.), not about using tests as a conversation medium as I have described.

Are you trying to say that TC39 is committed to write lots of tests before shipping the spec and that this thread is the first step?
And that whoever write these tests will be committed to spend a lot of time on es-discuss+bugs.ecmascript to report spec errors and discuss potential spec ambiguities?

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/es-discuss/attachments/20130425/1f57c5e4/attachment-0001.html>

I think it is a reasonable goal to have "complete" (possibly shallow) spec coverage in test262 prior to ES6 ratification. I am not sure if this goal is officially stated but it's something I'm working toward regardless. I will need significant help from other implementers and the community to reach this goal, though. I can't speak for other implementers, but any spec ambiguities or bugs we come across are brought up either on this mailing list or at a TC-39 meeting.

I understand the frustration with lack of Test262 progress; it would have been preferable to start this effort in earnest earlier. I'm at a point now where I can start doing more, and I hope you can help as well with making the test suite more useful for implementers and the community and getting as many high quality tests as we can in the suite. The thread linked by Rick is indeed just step 1. It makes sense to me to get some fundamentals in place before we start asking for contributions so we don't have to do more work than necessary. Hope this makes sense, but if not, feel free to chime in on test262-discuss!