testable specification
On Oct 26, 2011, at 12:08 PM, Michael Dyck wrote:
According to harmony:harmony goal #2 of Harmony is: Switch to a testable specification, ideally a definitional interpreter hosted mostly in ES5.
Is there much activity toward this goal? The current ES6 drafts are using mostly the same formalism as ES1 (although there was a marked de-spaghettification between ES3 and ES5).
My opinion is that the best we'll do on this goal, better than past ECMA-262 specs, is the test262.ecmascript.org project. I.e., we have a testsuite and it will be maintained and extended.
(I ask because, in my spare time, I'm developing a process that massages the ES spec into an executable/testable form. So I wonder if I'm duplicating existing work.)
Interesting -- you are writing an interpreter?
Michael,
It took me a while to find them the first time I looked, but they are downloadable here:
hg.ecmascript.org/tests/test262
Instructions here:
Brendan Eich wrote:
On Oct 26, 2011, at 12:08 PM, Michael Dyck wrote:
According to harmony:harmony goal #2 of Harmony is: Switch to a testable specification, ideally a definitional interpreter hosted mostly in ES5.
Is there much activity toward this goal? The current ES6 drafts are using mostly the same formalism as ES1 (although there was a marked de-spaghettification between ES3 and ES5).
My opinion is that the best we'll do on this goal, better than past ECMA-262 specs, is the test262.ecmascript.org project. I.e., we have a testsuite and it will be maintained and extended.
Certainly a laudable effort.
(I ask because, in my spare time, I'm developing a process that massages the ES spec into an executable/testable form. So I wonder if I'm duplicating existing work.)
Interesting -- you are writing an interpreter?
Yes, but not an interpreter for ES, rather an interpreter (more or less) for ES-spec-ese, the ad hoc notation that the ES spec's algorithms are written in. So then if you feed the ES spec to that interpreter, you do get an interpreter for ES. That's the idea, anyhow. It works, but so far only for really simple programs.
Rick Waldron wrote:
Michael,
It took me a while to find them the first time I looked, but they are downloadable here: hg.ecmascript.org/tests/test262
Instructions here: test262:command
Yup, I'm aware of the test262 project -- I even submitted a bug against it. And it's great that ES has a test suite. But from the context of the goal:
Switch to a testable specification, ideally a definitional interpreter hosted mostly in ES5.
I got the impression that "testable specification" meant that one would actually test the specification, not just test implementations of it.
Anyhow, that's the kind of activity I was asking about in my original post.
On Oct 26, 2011, at 5:02 PM, Michael Dyck wrote:
(I ask because, in my spare time, I'm developing a process that massages the ES spec into an executable/testable form. So I wonder if I'm duplicating existing work.) Interesting -- you are writing an interpreter?
Yes, but not an interpreter for ES, rather an interpreter (more or less) for ES-spec-ese, the ad hoc notation that the ES spec's algorithms are written in.
Right, our Visual Cobol, as it were ;-).
So then if you feed the ES spec to that interpreter, you do get an interpreter for ES. That's the idea, anyhow. It works, but so far only for really simple programs.
This could indeed satisfy the goal! Great to hear you are working on it. How is it going?
On Oct 26, 2011, at 5:02 PM, Michael Dyck wrote:
Brendan Eich wrote: ...
(I ask because, in my spare time, I'm developing a process that massages the ES spec into an executable/testable form. So I wonder if I'm duplicating existing work.) Interesting -- you are writing an interpreter?
Yes, but not an interpreter for ES, rather an interpreter (more or less) for ES-spec-ese, the ad hoc notation that the ES spec's algorithms are written in. So then if you feed the ES spec to that interpreter, you do get an interpreter for ES. That's the idea, anyhow. It works, but so far only for really simple programs.
I'm again considering creating a line-by-line translation of the ES6 spec algorithms into an executable evaluator of parse trees. this would be a non-normative translation of the spec. that could be used to test the algorithms.
I'd be interesting in hearing about the details of what you have been working on.
+1. Where the spec is already almost pseudo-code, its readability would improve if it was, in fact, pseudo-code. But would an extra interpreter be needed or couldn’t one just implement the ES-262 constructs (execution contexts etc.) in an existing language (Python, Rust, Scheme, Smalltalk, etc.)? Might make the algorithm testing harder, though.
Le 27/10/2011 12:08, Axel Rauschmayer a écrit :
+1. Where the spec is already almost pseudo-code, its readability would improve if it was, in fact, pseudo-code. But would an extra interpreter be needed or couldn’t one just implement the ES-262 constructs (execution contexts etc.) in an existing language (Python, Rust, Scheme, Smalltalk, etc.)?
Why choosing a completely different language? Why not ECMAScript 5.1? It will be one less language to learn as people who read the ES6 spec are very likely to be familiar with ES5.1. I personnally wouldn't feel comfortable reading a spec in any of the 4 languages you cited. Or maybe define the couple of things that can't be fully implemented in ES5.1 (proxies, private names) and use ES5.1 + these construct to define ES6.
+1. Where the spec is already almost pseudo-code, its readability would improve if it was, in fact, pseudo-code. But would an extra interpreter be needed or couldn’t one just implement the ES-262 constructs (execution contexts etc.) in an existing language (Python, Rust, Scheme, Smalltalk, etc.)? Why choosing a completely different language? Why not ECMAScript 5.1?
Right, I had forgotten about ECMAScript itself.
It will be one less language to learn as people who read the ES6 spec are very likely to be familiar with ES5.1. I personnally wouldn't feel comfortable reading a spec in any of the 4 languages you cited. Or maybe define the couple of things that can't be fully implemented in ES5.1 (proxies, private names) and use ES5.1 + these construct to define ES6.
I don’t think you need these constructs at the meta-level to implement them at the object level. However, I would in general be in favor of ES6, because it would help with implementing types (at the very least, subtyping would be easier via <| ).
On 27 October 2011 13:35, David Bruant <bruant.d at gmail.com> wrote:
+1. Where the spec is already almost pseudo-code, its readability would improve if it was, in fact, pseudo-code. But would an extra interpreter be needed or couldn’t one just implement the ES-262 constructs (execution contexts etc.) in an existing language (Python, Rust, Scheme, Smalltalk, etc.)?
Why choosing a completely different language? Why not ECMAScript 5.1? It will be one less language to learn as people who read the ES6 spec are very likely to be familiar with ES5.1. I personnally wouldn't feel comfortable reading a spec in any of the 4 languages you cited. Or maybe define the couple of things that can't be fully implemented in ES5.1 (proxies, private names) and use ES5.1 + these construct to define ES6.
To spec a beast like ES, you want something with a considerably simpler and cleaner semantics than ES. Otherwise, all you end up with is a circular definition.
Ideally, a good executable spec would become the normative spec at some point, so this is not just a philosophical point.
To spec a beast like ES, you want something with a considerably simpler and cleaner semantics than ES. Otherwise, all you end up with is a circular definition.
It mainly depends on how close you can get to true pseudo-code. With ES6’s cleaner semantics (e.g. block scoping), I think it could work well. Many of the details of how, say, the stack works can be pushed into a library and would not be shown in the spec.
You might not even need to have full meta-circularity (as in “implement all the features”), just a more programming-language-like notation would already be nice.
+1:
- existing test case IDs for all 10K+ tests on test262 would need to updated. This alone makes me very hesitant to suggest any modifications to the existing pseudo-code algorithms
- an implementer's dilemma here is that in many cases a developer implementing Harmony feature 'Xyz' is not going to be well-versed in all areas of ES5.1
That said, there's definitely value in spec'ing Harmony's new methods (e.g., 'String.prototype.startsWith') in ES5.1 from a test perspective. I.e., I can write the tests, mock the ES6 additions to an existing ES5.1 implementation, and most importantly validate correctness of the tests themselves long before a native implementation of the ES6 feature has become available. It's a great enabler for more TDD.
My best,
If you're curious about interpreters, I'd like to point out that we have one that we think is pretty decent:
It now has reasonable, though incomplete, coverage of Test262:
cs.brown.edu/~joe/public/results/summary.html
Further, it implements much of ES5 in a clearly defined specification language:
brownplt/LambdaS5/blob/master/envs/es5.env
It's built for understanding, not for performance, which makes it a good fit for playing around with new features (we have our eye on proxies), and ironing out test suites. We think it's a great implementation to tinker with--do have a look, and see what in the approach might help with the ES6 specification effort.
On Oct 27, 2011, at 4:40 AM, Axel Rauschmayer wrote:
+1. Where the spec is already almost pseudo-code, its readability would improve if it was, in fact, pseudo-code. But would an extra interpreter be needed or couldn’t one just implement the ES-262 constructs (execution contexts etc.) in an existing language (Python, Rust, Scheme, Smalltalk, etc.)? Why choosing a completely different language? Why not ECMAScript 5.1?
Right, I had forgotten about ECMAScript itself.
I would do it in ES5, or not at all. There is absolutely no need for any other language. Always eat your own dogfood!
It will be one less language to learn as people who read the ES6 spec are very likely to be familiar with ES5.1. I personnally wouldn't feel comfortable reading a spec in any of the 4 languages you cited. Or maybe define the couple of things that can't be fully implemented in ES5.1 (proxies, private names) and use ES5.1 + these construct to define ES6.
I don’t think you need these constructs at the meta-level to implement them at the object level. However, I would in general be in favor of ES6, because it would help with implementing types (at the very least, subtyping would be easier via <| ).
There would be no issues here. We aren't talking about a metacircular interpreter, but a direct line-by-line translation of the spec. algorithms. An ES6 objects would not be directly represented by a single ES5 object but instead by a ES5 based data structure that implemented the object semantics specified by ES6. ES5 is Turning-compete. Anything the spec. describes can be implemented in it.
+1. Where the spec is already almost pseudo-code, its readability would improve if it was, in fact, pseudo-code. But would an extra interpreter be needed or couldn’t one just implement the ES-262 constructs (execution contexts etc.) in an existing language (Python, Rust, Scheme, Smalltalk, etc.)? Why choosing a completely different language? Why not ECMAScript 5.1?
Right, I had forgotten about ECMAScript itself.
I would do it in ES5, or not at all. There is absolutely no need for any other language. Always eat your own dogfood!
Why not ES6? Because it’s not finished, yet and you want to start now?
On Oct 27, 2011, at 4:35 AM, David Bruant wrote:
Le 27/10/2011 12:08, Axel Rauschmayer a écrit :
+1. Where the spec is already almost pseudo-code, its readability would improve if it was, in fact, pseudo-code. But would an extra interpreter be needed or couldn’t one just implement the ES-262 constructs (execution contexts etc.) in an existing language (Python, Rust, Scheme, Smalltalk, etc.)? Why choosing a completely different language? Why not ECMAScript 5.1? It will be one less language to learn as people who read the ES6 spec are very likely to be familiar with ES5.1. I personnally wouldn't feel comfortable reading a spec in any of the 4 languages you cited. Or maybe define the couple of things that can't be fully implemented in ES5.1 (proxies, private names) and use ES5.1 + these construct to define ES6.
The purpose of this implementation would not be to provide more readable alternative to the ES6 pseudo-code. We already agreed to abandon that idea for a number of reasons.
The purpose would be to have a tested version of the algorithms so we can debug them prior to publishing the standard. The resulting interpreter would not be normative. Implementers should read the normative pseudo-code specification.
On Oct 27, 2011, at 12:07 PM, Axel Rauschmayer wrote:
+1. Where the spec is already almost pseudo-code, its readability would improve if it was, in fact, pseudo-code. But would an extra interpreter be needed or couldn’t one just implement the ES-262 constructs (execution contexts etc.) in an existing language (Python, Rust, Scheme, Smalltalk, etc.)? Why choosing a completely different language? Why not ECMAScript 5.1?
Right, I had forgotten about ECMAScript itself.
I would do it in ES5, or not at all. There is absolutely no need for any other language. Always eat your own dogfood!
Why not ES6? Because it’s not finished, yet and you want to start now?
Exactly.
On Oct 27, 2011, at 9:44 AM, Dave Fugate wrote:
+1:
- existing test case IDs for all 10K+ tests on test262 would need to updated. This alone makes me very hesitant to suggest any modifications to the existing pseudo-code algorithms
ES.next is a major revision of the ES specification. Much more so than ES5 was. Unfortunately many of the existing algorithms have to be modified to accommodate new semantics introduced in ES.next. In other cases, feature semantics may not have changed but the lower level semantics that the feature algorithms depend upon have had to change to accommodate new functionality unrelated to this specific feature and those changes force changes to the feature algorithms. Finally, the accumulated effect of all these changes leads to situations where both local and global refactoring of the specification are necessary in order to make in a readable/usable document.
You can already see some of this in the current draft. It will be even more apparent in the next draft that I should be able to post next week.
- an implementer's dilemma here is that in many cases a developer implementing Harmony feature 'Xyz' is not going to be well-versed in all areas of ES5.1
In order to properly interpreter the ES specification for a feature an implementer is going to need to be pretty comprehensively versed in the entire specification
That said, there's definitely value in spec'ing Harmony's new methods (e.g., 'String.prototype.startsWith') in ES5.1
no the normative specification must be the pseudo code in the actual specification. If the existence of a translation of it into executable code in some language by TC39 members is going to creating the impression that such code is a normative substitute for the actual specification then we shouldn't do it.d
from a test perspective. I.e., I can write the tests, mock the ES6 additions to an existing ES5.1 implementation, and most importantly validate correctness of the tests themselves long before a native implementation of the ES6 feature has become available. It's a great enabler for more TDD.
an executable translation of the specification is useful both for developing tests and using them to debug the specification.
I don't see where TDD enters into it. It is certainly not how we are developing the specification.
On Oct 27, 2011, at 3:08 AM, Axel Rauschmayer wrote:
+1. Where the spec is already almost pseudo-code, its readability would improve if it was, in fact, pseudo-code.
I don't understand? The current spec. is pseudo-code, not almost pseudo code (however, there are some sections that are not fully expressed using pseudo-code but instead depend upon English prose). It sounds like by "in fact, pseudo-code" you mean a semantically complete fully specified executable language. That doesn't sound like "pseudo code" to me, it sounds like "code". See en.wikipedia.org/wiki/Pseudocode
Allen Wirfs-Brock wrote:
I'm again considering creating a line-by-line translation of the ES6 spec algorithms into an executable evaluator of parse trees.
Cool. So my work might help you do the translation programmatically.
this would be a non-normative translation of the spec. that could be used to test the algorithms.
I'd be interesting in hearing about the details of what you have been working on.
Here's a quick overview:
(1) The first stage is to massage the ES spec into something easier for me to work with. I start with the MS Word file, import it into OpenOffice, and save as HTML. The resulting markup is fairly crufty and presentational, so I proceed to tweak the markup (via regex and dom-hacking) until it more-or-less reflects the semantic structure of the spec (using an ad hoc XML vocabulary).
For example, consider the second rule on page 22 of the current PDF (I tried to find something small but interesting), whose text is:
The MV of DecimalLiteral :: DecimalIntegerLiteral . DecimalDigits is
the MV of DecimalIntegerLiteral plus (the MV of DecimalDigits times
10–n), where n is the number of characters in DecimalDigits.
For this rule, the HTML from OpenOffice is:
<LI><P LANG="en-GB" STYLE="margin-top: 0.01in; margin-bottom: 0in; line-height: 100%">
<FONT FACE="Times New Roman, serif"><FONT SIZE=2><FONT FACE="Arial, sans-serif">The</FONT><FONT FACE="Arial, sans-serif">
MV of </FONT><I>DecimalLiteral </I><FONT FACE="Arial, sans-serif"><B>::</B></FONT><B>
</B><I>DecimalIntegerLiteral </I><FONT FACE="Courier New"><B>.</B></FONT><I>
DecimalDigits</I><I><B> </B></I><FONT FACE="Arial, sans-serif">is
the MV of </FONT><I>DecimalIntegerLiteral</I> <FONT FACE="Arial, sans-serif">plus
(the MV of </FONT><I>DecimalDigits</I><I><B> </B></I><FONT FACE="Arial, sans-serif">times
</FONT>10<SUP>–</SUP><SUP><I>n</I></SUP><FONT FACE="Arial, sans-serif">),
where </FONT><I>n</I><FONT FACE="Arial, sans-serif"> is the number
of characters in </FONT><I>DecimalDigit</I>s.</FONT></FONT></P>
Which I eventually turn into:
<li>
<para> The MV of
<prod1 n_colons="2"><lhs><nt>DecimalLiteral</nt></lhs>
<rhs>
<nt>DecimalIntegerLiteral</nt>
<t>.</t>
<nt>DecimalDigits</nt>
</rhs>
</prod1>
is the MV of <nt>DecimalIntegerLiteral</nt> plus
(the MV of <nt>DecimalDigits</nt> times
10<sup>-<var>n</var></sup>), where <var>n</var>
is the number of characters in <nt>DecimalDigits</nt>.
</para>
</li>
Here, -- A <prod1> is a production where only one RHS (of possibly many) appears. -- An <nt> is a non-terminal. -- A <t> is a terminal. -- A <var> is a metavariable [i.e. a variable in the pseudo-code].
Note that both nonterminals and metavariables are serif italic in the original, so when the massaging code sees a serif italic word, it has to make a decision between <nt> and <var>.
Note also that the final 's' in the final "DecimalDigits" is outside the <i> element in the HTML. (You can see it in the PDF too, if you enlarge
it sufficiently.) At various points in the process, I have lots of little tweaks to fix things like this, so that they get converted correctly. (Perhaps similar to the "preposterous concatenation of hacks" referred to at people.mozilla.org/~jorendorff/es5.html .) In some cases, I've pointed out the problem at bugs.ecmascript.org, but mostly I haven't.
(When the draft spec is made available in .docx format, I may recode this whole stage. There are some oddities in the HTML that take a lot of work to deal with, and I think they might be introduced by OpenOffice.)
(2) The second stage is (roughly speaking) to find <para>s that contain
pseudo-code, and convert them into a form that says the same thing, but reflects the syntactic structure in a way that's easily machine-parsable.
For the example rule, the pretty-printed result is: ( ( _the_MV_of ( (_nt DecimalLiteral) _:: [(_nt DecimalIntegerLiteral) (_t '.') (_nt DecimalDigits)] ) ) _IS ( ( (_the_MV_of (_nt DecimalIntegerLiteral)) _plus ( (_the_MV_of (_nt DecimalDigits)) _times (10 _to_the_power (_negative $n)) ) ) _where $n _is (_the_number_of_characters_in (_nt DecimalDigits)) ) )
(All the parentheses make it look like LISP, but that's mostly a coincidence.)
I invite you to read the original rule and the above form, and confirm that they say the same thing, using almost exactly the same words, but the latter makes explicit the syntactic grouping that (the spec assumes) a human reader can infer correctly in their head.
In addition to delimiting phrases with parentheses, the above form also identifies the 'key words' of each phrase, and precedes each with an underscore (gluing together multiple such if adjacent).
(3) The third stage is basically an interpreter for the language in which the above forms appear to be written.
(a) It reads in all the parenthesized forms, "parses" them (though the parsing is almost trivial), and builds an internal data structure. E.g., (10 _to_the_power (_negative $n)) is represented by a form whose 'key' is 'X_to_the_power_X', and whose two 'items' are the number 10 and the sub-form for (_negative $n).
(b) It reads in all the ES productions, and builds a parser for the ES language.
(c) It reads in some ES code, parses it as a Program according to the parser constructed in (b), and then tries to evaluate the Program using the structure it constructed in (a).
Some keys are primitive, e.g. X_to_the_power_X negative_X X_where_X_is_X and so are implemented directly in the interpreter. Other keys are defined by other rules, e.g. the_MV_of_X and so cause a search for an appropriate rule to execute.
I've got all of stage 1 done (though I have to adjust it slightly for each new draft). Stages 2 and 3 are in progress. I pick a simple ES program, then run the meta-interpreter until it hits a phrase that either wasn't (really) translated (in stage 2), or isn't implemented (in stage 3). I fix that, re-run, and things get a little farther. Sometimes it even completes!
Correct. My bad.
-- Dr. Axel Rauschmayer rauschma.de [Sent from a mobile device, please forgive brevity and typos]
According to harmony:harmony goal #2 of Harmony is: Switch to a testable specification, ideally a definitional interpreter hosted mostly in ES5.
Is there much activity toward this goal? The current ES6 drafts are using mostly the same formalism as ES1 (although there was a marked de-spaghettification between ES3 and ES5).
(I ask because, in my spare time, I'm developing a process that massages the ES spec into an executable/testable form. So I wonder if I'm duplicating existing work.)
According to http://wiki.ecmascript.org/doku.php?id=harmony:harmony goal #2 of Harmony is: Switch to a testable specification, ideally a definitional interpreter hosted mostly in ES5. Is there much activity toward this goal? The current ES6 drafts are using mostly the same formalism as ES1 (although there was a marked de-spaghettification between ES3 and ES5). (I ask because, in my spare time, I'm developing a process that massages the ES spec into an executable/testable form. So I wonder if I'm duplicating existing work.) -Michael