Discuss
  • more JSON spec questions

more JSON spec questions

# Hallvord R. M. Steen (16 years ago)

the feedback below refers to the "Initial ECMA formated ES5 draft" and
consists of a few things I've noticed while working on a JSON test suite.

First, I think considerable work has gone into the JSON part of the spec
since I first looked at it - it's beginning to look really good. My test
suite is a work in progress, and of course both the tests and my questions
might just show that there's something I haven't entirely understood
rather than problems with the specification. Feedback is welcome.

Now, the questions..

  1. Apparently the grammar still disallows literal tab inside JSONString. Implementations that pass this test allow it: testsuites.opera.com/JSON/correctness/035.html
  • from earlier discussions I had the impression that the spec would
    change..?

  1. The grammar and an existing test in the json.org test suite disallows
    final commas in array input (JSON.parse('[1,]') should apparently throw) -
    IE8, Safari 4 and Firefox 3.5 all seem to happily accept it though, as
    does the yet-to-go-public Opera implementation. I'd like some reassurance
    from other browser vendors that you consider this a bug and intend to fix
    it before I push for a fix here because this is the kind of thing that
    might cause compat problems :-o (Guess this is more a question for the
    implementers on the list than for the spec authors.)

  2. Should JSON.parse() without arguments throw? It's not obvious from the
    spec as far as I can tell, but it's what most browser implementations do.

  3. (Editorial) - spec says "The abstract operation Str(key, holder) has
    access to PropertyList and ReplacerFunction" - the algorithm does not actually use PropertyList (very trivial issue, of course)

  4. (Editorial) It's not apparent from the prose description of
    JSON.stringify() that if the replacer argument is an array, it will only
    be applied to objects. I expected to be able to do something like
    JSON.stringify( [0,1,2], [1] ) => '[1]'. Suggest a small clarification,

perhaps just saying "selecting object keys that will be stringified"
instead of "selecting the keys that will be stringified"

  1. (Editorial) Per the algorithm, passing in a "replacer" array to
    JSON.stringify() will cause it to also stringify dontEnum properties. An
    example would be JSON.stringify( Number, ['MAX_VALUE'] ) including the
    MAX_VALUE property where JSON.stringify(Number) would not. Perhaps clarify
    this in the prose too, not just in the algorithm? Calling it a "whitelist"
    seems it will only narrow down the number of elements that are included,
    not expand it..

  2. I also didn't understand why the "reviver" function was called once
    with "key" being the empty string and "value" being the entire output
    object until I read the algorithm carefully.

Quick word about the test suite: it's current home is
testsuites.opera.com/JSON where you can have a look at the README,
load runner.htm to run all tests (Warning: if you do that in IE8 something
makes it eat all your memory and make the PC entirely unusable!) or browse
through folders to individual tests. Hope it's useful to all you
implementers out there. Feedback welcome :)

# Oliver Hunt (16 years ago)
  1. The grammar and an existing test in the json.org test suite
    disallows final commas in array input (JSON.parse('[1,]') should
    apparently throw) - IE8, Safari 4 and Firefox 3.5 all seem to
    happily accept it though, as does the yet-to-go-public Opera
    implementation. I'd like some reassurance from other browser vendors
    that you consider this a bug and intend to fix it before I push for
    a fix here because this is the kind of thing that might cause compat
    problems :-o (Guess this is more a question for the implementers on
    the list than for the spec authors.)

This is definitely a bug in the WebKit implementation (we disallow the
trailing comma in object literals, i'm saddened that we allow the
trailing comma in [] :-/ )

  1. (Editorial) - spec says "The abstract operation Str(key, holder)
    has access to PropertyList and ReplacerFunction" - the algorithm does not actually
    use PropertyList (very trivial issue, of course)

PropertyList is used in the abstract function JO

Quick word about the test suite: it's current home is testsuites.opera.com/JSON where you can have a look at the README, load runner.htm to run all
tests (Warning: if you do that in IE8 something makes it eat all
your memory and make the PC entirely unusable!) or browse through
folders to individual tests. Hope it's useful to all you
implementers out there. Feedback welcome :)

There are also reasonably extensive tests in the WebKit repository --
unfortunately they are designed primarily as regression tests so their
output isn't astonishingly nice -- Browsers without a native JSON
implementation just fail everything somewhat catastrophically with no
obvious indication as to why. That said the tests do cover most edge
cases (although not the [] with terminating comma thing apparently :- ( ) such as order of execution, number of times properties are
accessed, etc.

trac.webkit.org/export/HEAD/trunk/LayoutTests/fast/js/JSON-stringify-replacer.html, trac.webkit.org/export/HEAD/trunk/LayoutTests/fast/js/JSON-parse.html, trac.webkit.org/export/HEAD/trunk/LayoutTests/fast/js/JSON-stringify.html (this one takes quite a while)

# Luke Smith (16 years ago)

I shared this on the JSON Yahoo! group list, but this seems relevant
here, now. I extracted tests from the YUI JSON test suite for Native
JSON support.

lsmith/JSON-test-suite/tree/master

Great to see the sharing of tests! Luke

# Oliver Hunt (16 years ago)

Quick word about the test suite: it's current home is testsuites.opera.com/JSON where you can have a look at the README, load runner.htm to run all
tests (Warning: if you do that in IE8 something makes it eat all
your memory and make the PC entirely unusable!) or browse through
folders to individual tests. Hope it's useful to all you
implementers out there. Feedback welcome :)

The spec only defines Date.prototype.toJSON now -- so the tests for
toJSON on String, Boolean and Number are wrong (technically they
should not be present, and stringification should work if they aren't).

Octal values are completely specified -- the lexer defined in ES5 and json.org disallows a leading 0 on any value not between -1 and 1.

You should also ensure comments are not allowed in the content.

# Hallvord R. M. Steen (16 years ago)

On Thu, 27 Aug 2009 00:52:20 +0200, Oliver Hunt <oliver at apple.com> wrote:

Quick word about the test suite: it's current home is
testsuites.opera.com/JSON where you can have a look at the
README, load runner.htm to run all tests (Warning: if you do that in
IE8 something makes it eat all your memory and make the PC entirely
unusable!) or browse through folders to individual tests. Hope it's
useful to all you implementers out there. Feedback welcome :)

The spec only defines Date.prototype.toJSON now -- so the tests for
toJSON on String, Boolean and Number are wrong (technically they should
not be present, and stringification should work if they aren't).

Aha, I was wondering why Safari failed all those tests - completely missed
that these were removed from the spec :-p Thanks!

Octal values are completely specified -- the lexer defined in ES5 and
json.org disallows a leading 0 on any value not between -1 and 1.

Right - I think the test itself is valid, so I just need to fix the
comment saying it's an unspecified assertation, right? (test is testsuites.opera.com/JSON/correctness/scripts/003.js )

# Oliver Hunt (16 years ago)

Octal values are completely specified -- the lexer defined in ES5
and json.org disallows a leading 0 on any value not between -1 and
1.

Right - I think the test itself is valid, so I just need to fix the
comment saying it's an unspecified assertation, right? (test is testsuites.opera.com/JSON/correctness/scripts/003.js )

Yup. You may also want tests for use of replacer arrays and functions,
etc

# Hallvord R. M. Steen (16 years ago)

On Thu, 27 Aug 2009 01:22:46 +0200, Oliver Hunt <oliver at apple.com> wrote:

Octal values are completely specified -- the lexer defined in ES5 and json.org disallows a leading 0 on any value not between -1 and 1.

Good to hear but - sorry! - I can't quite see where the spec supports this
conclusion. I tried looking at the lexical and syntactical grammars and
the JSON.parse() algorithm.

Just reported the bug on us parsing '[1,]' and reversed tests for toJSON()
on String, Boolean and Number (Safari surges to 90.5% pass - good for you
and well done ;) )

# Oliver Hunt (16 years ago)

On Aug 26, 2009, at 4:50 PM, Hallvord R. M. Steen wrote:

On Thu, 27 Aug 2009 01:22:46 +0200, Oliver Hunt <oliver at apple.com>
wrote:

Octal values are completely specified -- the lexer defined in ES5 and json.org disallows a leading 0 on any value not between -1 and 1.

Good to hear but - sorry! - I can't quite see where the spec
supports this conclusion. I tried looking at the lexical and
syntactical grammars and the JSON.parse() algorithm.

From sections 7.8.3 and 15.12.1.1 JSONNumber :: -<opt> DecimalIntegerLiteral JSONFraction <opt> ExponentPart <opt>

DecimalIntegerLiteral :: 0 NonZeroDigit DecimalDigits <opt>

eg. the only valid character left of the decimal point the begins with
0 is 0

# Mark S. Miller (16 years ago)

On Wed, Aug 26, 2009 at 5:02 PM, Oliver Hunt<oliver at apple.com> wrote:

From sections 7.8.3 and 15.12.1.1 JSONNumber ::  -<opt> DecimalIntegerLiteral JSONFraction <opt> ExponentPart <opt> DecimalIntegerLiteral :: 0 NonZeroDigit DecimalDigits <opt> eg. the only valid character left of the decimal point the begins with 0 is 0

As also shown by the railroad diagram: json.org/number.gif

# Oliver Hunt (16 years ago)

On Aug 26, 2009, at 4:22 PM, Oliver Hunt wrote:

Yup. You may also want tests for use of replacer arrays and
functions, etc

On this topic, i'm fixing an issue in our handling of reviver
functions in JSON.parse and noticed that json2.js and the JSON.parse
specification differ in the handling of properties being deleted from
the holder object, namely if a reviver function deletes a property
from holder before it is visited then the current spec behaviour would
imply that that property is still visited, only with the value
undefined.

While json2.js disagrees with this implementation the firefox
implementation does match spec here, so i'd just like to verify that
the currently specified behaviour is intended (this is sufficiently
edge-case like for me to doubt it has any significant compatibility
impact, i just want clarification)

# Allen Wirfs-Brock (16 years ago)

At this point it's hard to say whether or not the exact handling of this specific edge case was intentional or not. However, it isn't obvious that this behavior is any worse or better than the possible alternatives so I don't think we have a strong case for changing it. Particularly, since it has been specified this way for in drafts for quite a while.

We're about a year past json2.js serving as the reference implementation for the ES5 JSON spec. (although I believe Crock has done some json2.js updates to track the ES5 spec.) There are some things in the current algorithms that just can't be easily accomplished in pure ES3 code.

Allen

From: es-discuss-bounces at mozilla.org [mailto:es-discuss-bounces at mozilla.org] On Behalf Of Oliver Hunt Sent: Wednesday, August 26, 2009 8:23 PM To: Oliver Hunt Cc: es-discuss at mozilla.org Subject: Re: more JSON spec questions

On Aug 26, 2009, at 4:22 PM, Oliver Hunt wrote:

Yup. You may also want tests for use of replacer arrays and functions, etc

On this topic, i'm fixing an issue in our handling of reviver functions in JSON.parse and noticed that json2.js and the JSON.parse specification differ in the handling of properties being deleted from the holder object, namely if a reviver function deletes a property from holder before it is visited then the current spec behaviour would imply that that property is still visited, only with the value undefined.

While json2.js disagrees with this implementation the firefox implementation does match spec here, so i'd just like to verify that the currently specified behaviour is intended (this is sufficiently edge-case like for me to doubt it has any significant compatibility impact, i just want clarification)

# Robert Sayre (16 years ago)

On Wed, Aug 26, 2009 at 11:03 PM, Allen Wirfs-Brock<Allen.Wirfs-Brock at microsoft.com> wrote:

At this point it’s hard to say whether or not the exact handling of this specific edge case was intentional or not.

I think it is obvious. The list of keys is built before iteration takes place for a couple of reasons.

# Oliver Hunt (16 years ago)

Rob, the specific issue is whether you call the reviver for properties
that have been deleted during filtering -- eg. the behaviour of for(in)

# Allen Wirfs-Brock (16 years ago)

-----Original Message----- From: Robert Sayre [mailto:sayrer at gmail.com]

I think it is obvious. The list of keys is built before iteration takes place for a couple of reasons.

Yes, but there could be subsequent explicit tests to see whether the property still exists. However, I really don't think it would make any much differences as I don't think there are any reasonable use cases for a reviver that would be sensitive to this.

# Allen Wirfs-Brock (16 years ago)

-----Original Message----- From: es-discuss-bounces at mozilla.org [mailto:es-discuss- bounces at mozilla.org] On Behalf Of Hallvord R. M. Steen ...

  1. Apparently the grammar still disallows literal tab inside JSONString. Implementations that pass this test allow it: testsuites.opera.com/JSON/correctness/035.html
  • from earlier discussions I had the impression that the spec would change..?

The JSON grammar and the RFC has "always" disallowed all control characters (including tabs) from appearing unescaped inside JSONStrings. This has come up for discussion several times but I believe that we have always gone back to the RFC to guide us. Can you point to what made you think this was going to change.

  1. The grammar and an existing test in the json.org test suite disallows final commas in array input (JSON.parse('[1,]') should apparently throw)

IE8, Safari 4 and Firefox 3.5 all seem to happily accept it though, as does the yet-to-go-public Opera implementation. I'd like some reassurance from other browser vendors that you consider this a bug and intend to fix it before I push for a fix here because this is the kind of thing that might cause compat problems :-o (Guess this is more a question for the implementers on the list than for the spec authors.)

The spec. means what it says in this regard and additionally forbids conforming implementations from extending specified JSON grammar for use in JSON.parse (you can, of course, provide a different implementation specific parsing function that uses an extended grammar).

I think I already stated on another thread that Microsoft plans on ultimately bringing our JSON implementation into full conformance with the spec.

  1. Should JSON.parse() without arguments throw? It's not obvious from the spec as far as I can tell, but it's what most browser implementations do.

Parse Step 1: "Let JText be ToString(text)". If parse had no arguments, text will treated as the value undefined and do ToString(undefined) so JText is the string value 'undefined' step 2. "throw a SyntaxError exception if the JText did not conform to the JSON grammar for the goal symbol JSONText." The identifier undefined does not reduce to JSONText according to the grammar so a SyntaxError should be thrown.

  1. (Editorial) - spec says "The abstract operation Str(key, holder) has access to PropertyList and ReplacerFunction" - the algorithm does not actually use PropertyList (very trivial issue, of course)

Thanks

  1. (Editorial) It's not apparent from the prose description of JSON.stringify() that if the replacer argument is an array, it will only be applied to objects. I expected to be able to do something like JSON.stringify( [0,1,2], [1] ) => '[1]'. Suggest a small clarification, perhaps just saying "selecting object keys that will be stringified" instead of "selecting the keys that will be stringified"

I replaced "keys" with "object properties" (even though arrays are also objects). What's a "key" anyway.

In general, the algorithms are normative, not the prose descriptions.

  1. (Editorial) Per the algorithm, passing in a "replacer" array to JSON.stringify() will cause it to also stringify dontEnum properties. An example would be JSON.stringify( Number, ['MAX_VALUE'] ) including the MAX_VALUE property where JSON.stringify(Number) would not. Perhaps clarify this in the prose too, not just in the algorithm? Calling it a "whitelist" seems it will only narrow down the number of elements that are included, not expand it..

An interesting point that I don't recall having been mentioned before. These isn't any prose that says that only enumerable properties get stringified and steps 5 and 6 of JO are pretty clear about which properties are examined in both the list and no-list cases. Given that (and the late date) I'm disinclined to add a note. ...

# Hallvord R. M. Steen (16 years ago)

On Thu, 27 Aug 2009 05:23:04 +0200, Oliver Hunt <oliver at apple.com> wrote:

Yup. You may also want tests for use of replacer arrays and functions, etc

You meant octal/noctal syntax in replacer arrays and functions? Wasn't
quite sure what you were suggesting.

On this topic, i'm fixing an issue in our handling of reviver functions in JSON.parse and noticed that json2.js and the JSON.parse specification differ in the handling of properties being deleted from the holder object, namely if a reviver function deletes a property from holder before it is visited then the current spec behaviour would imply that that property is still visited, only with the value undefined.

Looks like that's the way the spec works, and I've added another edge case
test for it here: testsuites.opera.com/JSON/correctness/050.html