JSON support for BigInt in Chrome/V8

# Anders Rundgren (4 days ago)

var small = BigInt("5"); var big = BigInt("5555555555555555555555555500003"); JSON.stringify([big,small]); VM330:1 Uncaught TypeError: Do not know how to serialize a BigInt     at JSON.stringify (<anonymous>)     at <anonymous>:1:6

JSON Number serialization has apparently reached a new level (of confusion).

Personally I don't see the problem.  XML did just fine without hard-coded data types.

The JSON type system is basically a relic from JavaScript.  As such it has proved to be quite useful. However, when you are outside of that scope, the point with the JSON type system gets pretty much zero since you anyway need to map extended types.

Oracle's JSON-B solution which serializes small values as Number and large values as String rather than having a unified serialization based on the underlying data type seems like a pretty broken concept although indeed fully conforming to the JSON specification. "Like the Devil reads the Bible" as we say in Scandinavia :-)

Adding a couple of double quotes is a major problem?  If so, it seems like a way more useful project making quotes optional for keys (named in a specific way), like they already are in JavaScript.

Yeah, and of course adding support for comments.

Anders

# J Decker (4 days ago)

On Sat, Jul 14, 2018 at 1:36 AM Anders Rundgren < anders.rundgren.net at gmail.com> wrote:

var small = BigInt("5"); var big = BigInt("5555555555555555555555555500003"); JSON.stringify([big,small]); VM330:1 Uncaught TypeError: Do not know how to serialize a BigInt at JSON.stringify (<anonymous>) at <anonymous>:1:6

is BigInt the only way to create a BigInt ? Or did they also implement the 'n' suffix, which I noted here tc39/proposal-bigint#24 would easily distinguish bigint from other numbers; and be easy to add on the parsing side; and call BigInt(xxx) instead of Number(xxx).

var small = 5n; var big = 5555555555555555555555555500003n;

n suffix as from tc39/proposal-bigint

JSON Number serialization has apparently reached a new level (of confusion).

Personally I don't see the problem. XML did just fine without hard-coded data types.

The JSON type system is basically a relic from JavaScript. As such it has proved to be quite useful. However, when you are outside of that scope, the point with the JSON type system gets pretty much zero since you anyway need to map extended types.

Oracle's JSON-B solution which serializes small values as Number and large values as String rather than having a unified serialization based on the underlying data type seems like a pretty broken concept although indeed fully conforming to the JSON specification. "Like the Devil reads the Bible" as we say in Scandinavia :-)

Adding a couple of double quotes is a major problem? If so, it seems like a way more useful project making quotes optional for keys (named in a specific way), like they already are in JavaScript.

Yeah, and of course adding support for comments.

I'd rather not see numbers converted to strings; that would be required to allow application handling of values; at a layer higher than JSON core itself. It is nice that JSON keeps numbers as numbers and strings as strings without needing intimite knowledge about the actual 'types' they end up in.

Comparing numeric length would be a half/useless solution since bigints are required to interop with other bigints only; so small numbers couldn't be 'guessed' and the application would have to provide a reviver.

# Anders Rundgren (3 days ago)

On 2018-07-15 04:27, J Decker wrote:

On Sat, Jul 14, 2018 at 1:36 AM Anders Rundgren <anders.rundgren.net at gmail.com <mailto:anders.rundgren.net at gmail.com>> wrote:

var small = BigInt("5");
var big = BigInt("5555555555555555555555555500003");
JSON.stringify([big,small]);
VM330:1 Uncaught TypeError: Do not know how to serialize a BigInt
      at JSON.stringify (<anonymous>)
      at <anonymous>:1:6

is BigInt the only way to create a BigInt ?  Or did they also implement the 'n' suffix, which I noted  here tc39/proposal-bigint#24 would easily distinguish bigint from other numbers; and be easy to add on the parsing side; and call BigInt(xxx) instead of Number(xxx).

This problem is related to the BigInt object itself. If you create such using the 'n' notation you get the same result.

If you want to use BigInt with JSON you have to serialize it yourself:

var small = BigInt(5n); var big = BigInt(5555555555555555555555555500003n); JSON.stringify([big.toString(),small.toString()]);

which generates ["5555555555555555555555555500003","5"]

Anders

# J Decker (3 days ago)

On Sat, Jul 14, 2018 at 9:23 PM Anders Rundgren < anders.rundgren.net at gmail.com> wrote:

On 2018-07-15 04:27, J Decker wrote:

On Sat, Jul 14, 2018 at 1:36 AM Anders Rundgren < anders.rundgren.net at gmail.com <mailto:anders.rundgren.net at gmail.com>> wrote:

var small = BigInt("5");
var big = BigInt("5555555555555555555555555500003");
JSON.stringify([big,small]);
VM330:1 Uncaught TypeError: Do not know how to serialize a BigInt
      at JSON.stringify (<anonymous>)
      at <anonymous>:1:6

is BigInt the only way to create a BigInt ? Or did they also implement the 'n' suffix, which I noted here tc39/proposal-bigint#24 would easily distinguish bigint from other numbers; and be easy to add on the parsing side; and call BigInt(xxx) instead of Number(xxx).

This problem is related to the BigInt object itself. If you create such using the 'n' notation you get the same result.

If you want to use BigInt with JSON you have to serialize it yourself:

Yes; and I did forget to mentions erilaization side but the serlizer could do an additional type check and emit and appropriate thing. I thought the replacer could be used- but the output of replacer would have to type check to see if it's a bigint too.... v8/v8/blob/master/src/json-stringifier.cc#L305 case BIGINT_TYPE: hmm and digging some more there's lots of eexcpetions thrown...

does Number( "5n" ) ? result in a bigint? No....

Number( "5n" )
NaN
var a = 5n
a
5n

var small = BigInt(5n); var big = BigInt(5555555555555555555555555500003n); JSON.stringify([big.toString(),small.toString()]);

which generates ["5555555555555555555555555500003","5"]

Anders

# Anders Rundgren (3 days ago)

On 2018-07-15 08:17, J Decker wrote: <snip>

If you want to use BigInt with JSON you have to serialize it yourself:

Yes; and I did forget to mentions erilaization side but the serlizer could do an additional type  check and emit and appropriate thing.

It is the "appropriate thing" that is problem; the rest is trivial.

Anders

# Andrea Giammarchi (a day ago)

We miss a fundamental feature in JS, the ability to understand if a native constructor can be used with new or not.

BigInt("5555555555555555555555555500003"); 5555555555555555555555555500003n

new BigInt("5555555555555555555555555500003"); VM51:1 Uncaught TypeError: BigInt is not a constructor

Uint8Array([]) VM54:1 Uncaught TypeError: Constructor Uint8Array requires 'new'

new Uint8Array([]) Uint8Array []

Without that knowledge, any attempt to even think about a solution that would scale not only with BigInt but with everything else, is kinda futile.

Best .

# J Decker (a day ago)

On Tue, Jul 17, 2018 at 1:48 AM Andrea Giammarchi < andrea.giammarchi at gmail.com> wrote:

We miss a fundamental feature in JS, the ability to understand if a native constructor can be used with new or not.

BigInt("5555555555555555555555555500003"); 5555555555555555555555555500003n

new BigInt("5555555555555555555555555500003"); VM51:1 Uncaught TypeError: BigInt is not a constructor

typeof(5n)
"bigint"

Uint8Array([])

# Andrea Giammarchi (a day ago)

fair enough, so everything that is not object or null should never use new. Is this somehow an indirect rule based on current specs or actually part of the specification?

# Andrea Giammarchi (a day ago)

also, how would you "cast" a BigInt from string to BigInt ?

# Andrea Giammarchi (a day ago)

P.S. eval / Function is obviously not an answer

# Andrea Giammarchi (a day ago)

actually, never mind ... but I find it hilarious that BigInt('55555555555555555550000000000000000000001') works but BigInt('55555555555555555550000000000000000000001n') doesn't ^_^;;

Anyway, there is an easy pattern to go around this issue (and all others), will come back on this.

Best

# Anders Rundgren (a day ago)

On 2018-07-17 13:27, Andrea Giammarchi wrote:

actually, never mind ... but I find it hilarious that BigInt('55555555555555555550000000000000000000001') works but BigInt('55555555555555555550000000000000000000001n') doesn't ^_^;

This is how it works in Java as well. 55555555555555555550000000000000000000001n is a numeric literal.

Other issues like "new" are more difficult to understand the rationale behind.

# Cyril Auburtin (a day ago)

It would be great to have

JSON.stringify({x: 5n, y: BigInt('6')}) === '{"x":5n,"y":6n}'
JSON.parse('{"x": 3n}') // {x: 3n}

I don't know how feasible it would be, maybe have a new JSON5 object if JSON can't be changed for some reasons

# Andrea Giammarchi (a day ago)

My idea is that JSON.stringify(new Uint8Array([1, 2, 3])); is also an issue, not only BigInt.

if every value could be represented as {{type: 'BigInt', value: '55555555'}, {new: 'Uint8Array', value: [1, 2, 3]}, {type: 'string', value: 'dastring'}, {type: 'number', value: 123}, {new: 'String', value: 'as wrap'}} and resurrected later on it's basically problem solved except for Weak* and Symbol, unless it's a global one, and it'd be OK.

That transformation would be compatible also with all circular-json like libraries I know (or I've written).

# Andrea Giammarchi (a day ago)

actually, {type: 'String', value: ''} ... so the rule is: usable as class ? {new: 'ClassName', value: 'how to construct it'} : {type: 'Primitive', value: 'how to create it'} and the unserialized value uses new global[obj.new](obj.value) or global[obj.type](obj.value) to recreate the value.

# Anders Rundgren (21 hours ago)

On 2018-07-17 17:44, Cyril Auburtin wrote:

It would be great to have

JSON.stringify({x: 5n, y: BigInt('6')}) === '{"x":5n,"y":6n}'
JSON.parse('{"x": 3n}') // {x: 3n}

I don't know how feasable it would be, maybe have a new JSON5 object if JSON can't be changed for some reasons

If compatibility with other platforms which already have addressed this issue is an objective, there are essentially only three workable schemes:

  • Use JSON exactly as specified [*]; without restrictions on numbers. Example: Microsoft's Json.NET.

  • Use an adaptive I-JSON scheme. Example: Oracle's JSON-B.

  • Use use a normalized I-JSON scheme. Example: Close to a de-facto standard for systems targeting multiple platforms as well as for [probably all] IETF standards defining JSON structures.

Anders

*] Ignoring footnote-like interoperability considerations for JSON numbers.

# Jordan Harband (20 hours ago)

You already can't use new with Symbol, and you don't ever want to use new with a primitive's constructor anyways.

# Waldemar Horwat (16 hours ago)

On 07/17/2018 04:27 AM, Andrea Giammarchi wrote:

actually, never mind ... but I find it hilarious that BigInt('55555555555555555550000000000000000000001') works but BigInt('55555555555555555550000000000000000000001n') doesn't ^_^;;

That's no different from how other built-in types work. String('"foo"') doesn't give you the same string as the string literal "foo".

 Waldemar
# Rob Ede (16 hours ago)

The way I see it is that JSON is kind of a thing by itself even without JavaScript and we shouldn’t be beholden to the JS syntax of representing Bigints (123n vs 123) in JSON.

I think changing the behaviour of JSON.parse and introducing JSON5 namespace (or whatever) are both on the right track but meeting in the middle and extending the syntax of the existing JSON.parse/stringify with an options object that specified how to decode/encode bigints (string/number) would be the best approach.

For example, the Twitter API returns tweet ids as 64-bit ints. Granted they also return the is as a string for just this reason. But I don’t expect them to start including a third is field just to add an “n” onto it.

# Andrea Giammarchi (16 hours ago)

I guess a better example would've been Boolean('false') returns true, but yeah, I've moved slightly forward already with everything, if you read other messages.

.

# Michael J. Ryan (16 hours ago)

Out of bounds as you'd still have to parse it, but for encoding, could add BigInt.prototype.toJSON ...

# Jordan Harband (11 hours ago)

You can permanently fix any such type-checks by function isObject(x) { return x && (typeof x === 'object' || typeof x === 'function'); }, or function isObject(x) { return x && Object(x) === x; }, etc.

# Isiah Meadows (10 hours ago)

The way this conversation is going, we might as well just create a schema-based JSON serialization DSL for both parsing and stringifying. But I don't really see that as helpful in the language itself at least as a mandatory part of the spec (maybe an optional built-in module).

I've in the past few months seen similar things come up a few times already. I like the idea of a built-in schema-based JSON validator + parser (it'd be faster than what we currently have), but most things out there suck in some way, mostly just being boilerplatey, and there's a lot of design work to get out of the way first before you can come up with something that doesn't.

But as it stands, the only things I'd support for the JSON global itself are:

  1. Adding separate prototypes for JSON.stringify(source, options) and JSON.parse(source, options), so it's easier to extend and comprehend the arguments.
  2. Adding an option to parse anything consisting of purely digits (no exponent or decimal) as a BigInt, regardless of size.
  3. Adding an option to stringify BigInts into integer-only numbers and normal numbers into unconditional floats.

These could be either separate methods or part of a 4th options argument.


Isiah Meadows me at isiahmeadows.com, www.isiahmeadows.com

# Anders Rundgren (9 hours ago)

Serializing BigInts as quoted strings is compatible with Java JSON-B, Json.NET and can be poly-filled for older JS implementations.

In addition, this is also compatible with non-schema based parsing.

AFAIK, no IETF standard to date which depends on extended numbers (like JOSE) break the I-JSON recommendations.

Anders

# J Decker (5 hours ago)

On Tue, Jul 17, 2018 at 9:16 PM Isiah Meadows <isiahmeadows at gmail.com>

wrote:

The way this conversation is going, we might as well just create a schema-based JSON serialization DSL for both parsing and stringifying. But I don't really see that as helpful in the language itself at least as a mandatory part of the spec (maybe an optional built-in module).

I've in the past few months seen similar things come up a few times already. I like the idea of a built-in schema-based JSON validator + parser (it'd be faster than what we currently have), but most things out there suck in some way, mostly just being boilerplatey, and there's a lot of design work to get out of the way first before you can come up with something that doesn't.

But as it stands, the only things I'd support for the JSON global itself are:

  1. Adding separate prototypes for JSON.stringify(source, options) and JSON.parse(source, options), so it's easier to extend and comprehend the arguments.
  2. Adding an option to parse anything consisting of purely digits (no exponent or decimal) as a BigInt, regardless of size.

This won't work....

{ a : 123, b : 123n }
{ "a":123 ,
  "b":123 }

function test( json ) { var c = json.a * 5; var BIc = json.b * 5n; }

if long numbers only translate to bigint (and small as Number); the calc for 'BIc' fails. if everything is translated to bigint the calc for 'c' fails.

  1. Adding an option to stringify BigInts into integer-only numbers and

normal numbers into unconditional floats.

These could be either separate methods or part of a 4th options argument.

Well; Since JSON (JavaScript Object Notation) there's now available to JS a feature that an 'n' suffix can be applied to a number. Seems JSON should just inherit that.

Although I would like to see a better method than eval() to translate the input string from JSON for say { d : 123n }; right now within JS I'd have to use EVAL; in V8 I could just create a BigInt::new()....

(and RE: eval, not saying it's a GOOD solution; it's just apparently the only solution ATM; other than splicing the string.)