Why we need to clean up __proto__
The cleanest solution to the problem you outlined seems to be to never use objects as dictionaries(?) That is, one would separate the concerns program definition and application data. Allen’s object model reformation (allowing one to override the [] operator) would help, too. strawman:object_model_reformation
Doesn't Object.keys() solve this problem?
jjb
The cross-browser differences of {}.hasOwnProperty('proto') and {}.proto.hasOwnProperty('proto') are interesting and need to be ironed out.
As for the issue of using objects as dictionaries you can simply
prefix keys with some unique value, var uid = 'uid' + (+new Date);
,
and then access entries like dict[uid + key]
. This will allow value
lookups to work correctly cross-browser regardless of ES5+ or
proto support.
On Wed, Dec 28, 2011 at 11:39 AM, John-David Dalton < john.david.dalton at gmail.com> wrote:
As for the issue of using objects as dictionaries you can simply prefix keys with some unique value,
var uid = 'uid' + (+new Date);
, and then access entries likedict[uid + key]
. This will allow value lookups to work correctly cross-browser regardless of ES5+ or proto support.
At the expense of creating lots of garbage and slowing things down.
At the expense of creating lots of garbage and slowing things down.
A prefix is hardly lots of garbage
and the perf hit is not a real
concern (millions of ops/sec vs higher millions of ops/sec).
On Wed, Dec 28, 2011 at 12:43 PM, John-David Dalton < john.david.dalton at gmail.com> wrote:
At the expense of creating lots of garbage and slowing things down.
A prefix is hardly
lots of garbage
and the perf hit is not a real concern (millions of ops/sec vs higher millions of ops/sec).
In GWT, we did exactly this prefix operation, and we found GC became a problem on IE because of it, as each lookup or store in the map creates a new garbage string.
In GWT, we did exactly this prefix operation, and we found GC became a problem on IE because of it, as each lookup or store in the map creates a new garbage string.
GWT has larger problems if key prefixes are giving it perf grief (even in older IE).
I'd also like to add that proto allows valid JSON to change it's object type and allow functions within properties. There isn't a compelling exploit scenerio for this yet but who knows what is possible if setters come into the equation.
alert(({"proto":[]}).sort) alert(({"proto":function::['parent']}).location)
On Dec 28, 2011, at 17:13 , John J Barton wrote:
Doesn't Object.keys() solve this problem?
For getting elements, yes. For setting, the problem with the illegal own property name "proto" remains.
On Dec 28, 2011, at 11:58 AM, gaz Heyes wrote:
I'd also like to add that proto allows valid JSON to change it's object type and allow functions within properties. There isn't a compelling exploit scenerio for this yet but who knows what is possible if setters come into the equation.
I assume by "valid JSON" you really mean any object literal rather than the use of the built-in JSON.parse function.
The ES5.1 specification for JSON.parse, if strictly followed, should not set the the [[Prototype]] of the object created by JSON.parse('{"proto": []}') to anything other than Object.prototype. The JSON.parse spec. explicitly says that [[DefineOwnProperty]] is used to create for create properties. The spec. for JSON.parse also says "It is not permitted for a conforming implementation of JSON.parse to extend the JSON grammars". It is probably debatable whether or not giving "proto" special meaning would is iimplicitly a grammar change. However, the intent of that statement was that JSON.parse should must operate as specified and without extension.
Firefox (9 at least) appears to conform to the JSON.parse spec. in this regard.
TC39 has already agreed that as in ES.next is would provide a a normative specification for an "optional" proto feature. This is a good reminder that when we do that we need to carefully specify details such as requiring that JSON.parse does not recognize it.
alert(({"proto":[]}).sort) alert(({"proto":function::['parent']}).location)
The second example threw me for a while until I figured out that :: is an e4X construct. A conforming implementation of JSON.parse should reject such syntax.
The biggest problem with proto, as I see it, isn't that you can hit it accidentally, but that even if you know what you are doing, you can't avoid it (and have to do expensive workarounds).
It obviously depends on the implementation. In Mozilla Firefox/*Monkey, proto is an inherited accessor on Object.prototype (just as define[SG]etter). It's not an own-property and you can shadow it.
In Chrome/V8, it's an artificial own property on every object, and it can't be removed, shadowed or ignored. It tries to emulate Safari's original implementation, but probably fails some details (as your tests show, they aren't the same).
I like the Firefox way much better, both as a user and as an implementor. Since we probably can't get rid of proto totally, I would cheer if implementations would change to the Mozilla way.
/L 'entirely my personal opinion'.
[change of Subject: as this reply starts the promised thread]
Hi Lasse, good. That is the direction Dave Herman suggested and which I also find attractive. I have elaborated it at < strawman:magic_proto_property>,
which I've also placed on the Agenda for the January EcmaScript meeting.
The most surprising punchline is in requirement #4:
Object.create(null), creating an object that does not inherit from Object.prototype, also creates an object that does not inherit it’s proto, even if that property has not been deleted. With this change, objects-as-stringmaps for objects created by Object.create(null) would avoid the proto hazard, even in contexts where Object.prototype.proto has not been deleted. (FF already acts this way, so my previous message was wrong in claiming that Object.create(null) fails to avoid this platform on all non-IE browsers.)
Comments on this proposal appreciated.
On Wed, Dec 28, 2011 at 1:09 AM, Axel Rauschmayer <axel at rauschma.de> wrote:
The cleanest solution to the problem you outlined seems to be to never use objects as dictionaries(?) That is, one would separate the concerns program definition and application data. Allen’s object model reformation (allowing one to override the [] operator) would help, too. strawman:object_model_reformation
Hi Axel,
I agree that we should encourage programmers to stop using object directly as maps from strings to values because of these hazards. In fact, Caja and SES are moving all such uses over to using the StringMap abstraction at < code.google.com/p/es-lab/source/browse/trunk/src/ses/StringMap.js>.
My point was not that programmers should use objects-as-stringmaps, but rather that the hazards created by the diversity of unspecified magical proto behaviors is a hazard that has tripped up many, including expert JS programmers. If we pin down its normative-optional semantics, then when present, it will at least behave in a consistently bizarre manner. As FF, Dave, and Lasse suggest, the least magical way for this to act is as an accessor inherited only from Object.prototype.
Actually, there's no reason for the prefix or suffix to be random or unguessable or whatever. code.google.com/p/es-lab/source/browse/trunk/src/ses/StringMap.jssimply uses "$" as a suffix.
As for the performance issues of prefix or suffixing, I haven't measured. But performance measurements are only relevant when comparing different means of being correct. As (I think) Gerald Weinberg once said "If the program doesn't need to be correct, I can make it arbitrarily fast."
This exact issue is tested at < hg.ecmascript.org/tests/test262/file/c84161250e66/test/suite/ch15/15.12/15.12.2/S15.12.2_A1.js>.
Although Chrome and Safari fail this test for this reason, WebKit Nightly and Opera, both of which have magical proto, both still pass this test. Their magical proto does not corrupt their JSON implementation. Needless to say, IE's JSON has no such problem.
On Wed, Dec 28, 2011 at 1:15 PM, Axel Rauschmayer <axel at rauschma.de> wrote:
On Dec 28, 2011, at 17:13 , John J Barton wrote:
Doesn't Object.keys() solve this problem?
For getting elements, yes. For setting, the problem with the illegal own property name "proto" remains.
Doesn't this same problem face any property of an object-as-dictionary which is also a property needed for the dictionary to work properly? So fixing proto is not important, it's inadequate?
jj
On Dec 28, 2011, at 5:36 PM, Mark S. Miller wrote:
Comments on this proposal appreciated.
Mark,
Your proposal doesn't address the occurrence of proto as a property name in an object literal
Btw, test_JSON_PARSE_PROTO_CONFUSION(), currently at < code.google.com/p/es-lab/source/browse/trunk/src/ses/repairES5.js#1373>,
is how SES's initialization tests for this problem. We do indeed consider this bug a security hazard, so on those platforms that suffer from this bug, we repair it using repair_JSON_PARSE_PROTO_CONFUSION(), currently at < code.google.com/p/es-lab/source/browse/trunk/src/ses/repairES5.js#2195
Darn. I completely missed that. No easy fix comes to mind. Suggestions?
Actually, there's no reason for the prefix or suffix to be random or unguessable or
Right, I recently tweeted an example using a zero-width non-breaking space and at the time suggested it could be something as simple as using an underscore. twitter.com/#!/jdalton/status/140296487694381056, jdalton/fusejs/blob/2365ede5ac51c97f12b3e31fc6f16dff45552e92/src/lang/hash.js#L149
On Wed, Dec 28, 2011 at 5:59 PM, John J Barton <johnjbarton at johnjbarton.com>wrote:
On Wed, Dec 28, 2011 at 1:15 PM, Axel Rauschmayer <axel at rauschma.de>wrote:
On Dec 28, 2011, at 17:13 , John J Barton wrote:
Doesn't Object.keys() solve this problem?
For getting elements, yes. For setting, the problem with the illegal own property name "proto" remains.
Doesn't this same problem face any property of an object-as-dictionary which is also a property needed for the dictionary to work properly? So fixing proto is not important, it's inadequate?
Good uses of objects-as-stringmaps use the objects only as stringmaps. Aside from the use of "hasOwnProperty" as an instance method in Crock's original example, which he elsewhere shows how to correct, Crock's "registry" is used correctly. Together with the proposed proto as accessor behavior, Crock's code would actually be correct (though I'd still advise using an abstraction like StringMap that can encapsulate such hazards and present a simple interface).
On Wed, Dec 28, 2011 at 6:11 PM, Mark S. Miller <erights at google.com> wrote:
On Wed, Dec 28, 2011 at 5:59 PM, John J Barton < johnjbarton at johnjbarton.com> wrote:
On Wed, Dec 28, 2011 at 1:15 PM, Axel Rauschmayer <axel at rauschma.de>wrote:
On Dec 28, 2011, at 17:13 , John J Barton wrote:
Doesn't Object.keys() solve this problem?
For getting elements, yes. For setting, the problem with the illegal own property name "proto" remains.
Doesn't this same problem face any property of an object-as-dictionary which is also a property needed for the dictionary to work properly? So fixing proto is not important, it's inadequate?
Good uses of objects-as-stringmaps use the objects only as stringmaps. Aside from the use of "hasOwnProperty" as an instance method in Crock's original example, which he elsewhere shows how to correct, Crock's "registry" is used correctly. Together with the proposed proto as accessor behavior, Crock's code would actually be correct (though I'd still advise using an abstraction like StringMap that can encapsulate such hazards and present a simple interface).
Ok, but IMO, our system needs a built-in StringMap more than it needs to fix proto.
jjb
Le 29/12/2011 03:04, Mark S. Miller a écrit :
Darn. I completely missed that. No easy fix comes to mind. Suggestions?
I think 'proto' in object literals should work like any property. The ES.next proto operator [1] creates a standard alternative to using proto in object literal. As the proposal says:
Replace: var o = { proto : myProto, a:0, b: function () {} }
With: var o = myProto <| { a:0, b: function () {} }
proto would then act like a regular property (shadowing the one on the Object.prototype, very much like if it had been added with Object.defineProperty). The ability to later change the prototype is not lost as Lasse suggested.
I don't know to what extent my suggestion would break exisiting scripts. Neither do I know whether breaking existing proto-using scripts is a good or a bad thing. If in some browsers it breaks something, browsers, when changing proto semantics could issue a warning in their console ("you're using proto in object literals and we have changed its semantics...").
David
On Dec 29, 2011, at 4:58 AM, David Bruant wrote:
Le 29/12/2011 03:04, Mark S. Miller a écrit :
Darn. I completely missed that. No easy fix comes to mind. Suggestions? I think 'proto' in object literals should work like any property. The ES.next proto operator [1] creates a standard alternative to using proto in object literal. ... proto would then act like a regular property (shadowing the one on the Object.prototype, very much like if it had been added with Object.defineProperty). The ability to later change the prototype is not lost as Lasse suggested.
I don't know to what extent my suggestion would break exisiting scripts. Neither do I know whether breaking existing proto-using scripts is a good or a bad thing. If in some browsers it breaks something, browsers, when changing proto semantics could issue a warning in their console ("you're using proto in object literals and we have changed its semantics...").
The only reason to even consider a normative definition for proto is because it already exists on the web. If that definition doesn't support its common usage patterns than there isn't much point in doing it.
We really should be starting from an analysis of the actual intersection semantics of proto among the major web browsers. Mark's test in his original post is just a start at that analysis.
Personally, given that IE as yet to support it, I think there is a good argument that the intersection semantics is empty and at the very least proto in object literals it should be banned in ES.next Harmony mode.
On Thu, Dec 29, 2011 at 9:13 AM, Allen Wirfs-Brock <allen at wirfs-brock.com>wrote: [...]
Personally, given that IE as yet to support it, I think there is a good argument that the intersection semantics is empty and at the very least proto in object literals it should be banned in ES.next Harmony mode.
If we can agree that the intersection semantics is empty, then I would be overjoyed to dispense with this whole exercise. I hate proto with a passion I can hardly express.
However, at a recent EcmaScript meeting (I think it was the November one too), MS explained that the mobile space is currently so dominated by browsers honoring mutable proto that some popular mobile-only libraries now rely on it. From the history of web evolution, we can understand the pressures this creates: The authors of those libraries feel no pressure to change, since their use of proto works everywhere they care about.
By contrast, MS, currently attempting to become a mobile player, feels much pressure to change mobile IE to be able to run these libraries. If MS changes, it will simply change IE rather than fork mobile vs non-mobile. If IE adopts proto, then the intersection semantics really becomes standard, even if not de jure, so let's get out in front and codify what this standard should be. That codification must include current usage of "{proto:.." or, as Allen says, the whole exercise is pointless.
From a different history of evolution, we can also think of the emergence
of a separate mobile ecosystem as occurring on the other side of a chasm, leading to loss of genetic diversity (platforms not honoring proto) and partial speciation for the portion suddenly cut off from the main population. The chasm is not total, and the remixing of these partially speciated sub populations is complex.
Le 29/12/2011 18:13, Allen Wirfs-Brock a écrit :
On Dec 29, 2011, at 4:58 AM, David Bruant wrote:
Le 29/12/2011 03:04, Mark S. Miller a écrit :
Darn. I completely missed that. No easy fix comes to mind. Suggestions? I think 'proto' in object literals should work like any property. The ES.next proto operator [1] creates a standard alternative to using proto in object literal. ... proto would then act like a regular property (shadowing the one on the Object.prototype, very much like if it had been added with Object.defineProperty). The ability to later change the prototype is not lost as Lasse suggested.
I don't know to what extent my suggestion would break exisiting scripts. Neither do I know whether breaking existing proto-using scripts is a good or a bad thing. If in some browsers it breaks something, browsers, when changing proto semantics could issue a warning in their console ("you're using proto in object literals and we have changed its semantics..."). The only reason to even consider a normative definition for proto is because it already exists on the web. If that definition doesn't support its common usage patterns than there isn't much point in doing it.
We really should be starting from an analysis of the actual intersection semantics of proto among the major web browsers. Mark's test in his original post is just a start at that analysis.
Personally, given that IE as yet to support it, I think there is a good argument that the intersection semantics is empty and at the very least proto in object literals it should be banned in ES.next Harmony mode.
I'm not sure I understand. I suggest to ban 'proto' for object literals, you seem to disagree based on the fact that proto should be standardized based on the intersection of current implementation semantics, then you suggest to ban proto in object literals. Am I misinterpreting what you answered?
Also, you mention "in ES.next Harmony mode". What is the plan regarding proto? To standardize it also for ES5? What is the scope of the next version of the standard? Does it standardize ES5, ES5+strict mode and ES6 (or whatever number)? Will the standardization of proto also be part of an ES5.2?
If the idea is only to standardize proto and its interaction with ES6 and proto was not considered as a regular property, then, what would be the semantics of:
var o = myProto1 < { proto: myProto2, a: 1, };
that's a good point. ES-next will have three modes, IIRC currently called non-strict, strict, and extended. (ES5 and ES-next non-strict is effectively an "ES3 compatibility mode" and ES-next strict is effectively an "ES5-strict compatibility mode".) As with ES5's non-strict vs strict, these three modes are opt in per code, not per heap, and so can only condition code properties.
{proto:...} having a magic meaning is per code, and so can and should be conditioned on mode. I think it should only be allowed to be magic in non-strict mode. In strict code and extended code, it should either not be magic or it should be prohibited -- I'm not sure which. As with nested named functions, it would then help if current ES5 implementations did not threat this as magic in ES5 strict code, to prepare the ground for ES-next.
I will extend the proposal accordingly. Thanks.
On Dec 29, 2011, at 2:00 PM, David Bruant wrote:
Le 29/12/2011 18:13, Allen Wirfs-Brock a écrit :
On Dec 29, 2011, at 4:58 AM, David Bruant wrote:
Le 29/12/2011 03:04, Mark S. Miller a écrit :
Darn. I completely missed that. No easy fix comes to mind. Suggestions? I think 'proto' in object literals should work like any property. The ES.next proto operator [1] creates a standard alternative to using proto in object literal. ... proto would then act like a regular property (shadowing the one on the Object.prototype, very much like if it had been added with Object.defineProperty). The ability to later change the prototype is not lost as Lasse suggested.
I don't know to what extent my suggestion would break exisiting scripts. Neither do I know whether breaking existing proto-using scripts is a good or a bad thing. If in some browsers it breaks something, browsers, when changing proto semantics could issue a warning in their console ("you're using proto in object literals and we have changed its semantics..."). The only reason to even consider a normative definition for proto is because it already exists on the web. If that definition doesn't support its common usage patterns than there isn't much point in doing it.
We really should be starting from an analysis of the actual intersection semantics of proto among the major web browsers. Mark's test in his original post is just a start at that analysis.
Personally, given that IE as yet to support it, I think there is a good argument that the intersection semantics is empty and at the very least proto in object literals it should be banned in ES.next Harmony mode. I'm not sure I understand. I suggest to ban 'proto' for object literals, you seem to disagree based on the fact that proto should be standardized based on the intersection of current implementation semantics, then you suggest to ban proto in object literals. Am I misinterpreting what you answered?
The plan of record of TC39 is to specify proto in Annex B and to change the status of Annex B to Normative/Optional. This means that it doesn't have to be implemented, but if it is it must follow the specification.
The argument, in favor for specifying it is that it is a de facto feature of browsers particularly in the mobile space.
I'd prefer not having proto at all, but if we must have it then I would prefer to keep it out of "Harmony mode" code.
Also, you mention "in ES.next Harmony mode". What is the plan regarding proto? To standardize it also for ES5? What is the scope of the next version of the standard? Does it standardize ES5, ES5+strict mode and ES6 (or whatever number)? Will the standardization of proto also be part of an ES5.2?
This is about specifying it in "ES6". This has no normative impact on ES5.x but but future implementors of such might well follow the ES6 specification when implementing this non-standard (for ES5.x) feature.
I don't believe that we have previously discussed the possibility of variations of proto among the various "ES6" modes. That is what I am now suggesting, precisely to avoid issues like the one you mention below
If the idea is only to standardize proto and its interaction with ES6 and proto was not considered as a regular property, then, what would be the semantics of:
var o = myProto1 < { proto: myProto2, a: 1, };
If we disallow any special meaning of proto in object literals in ES6 Extended code ("Harmony mode") then the above would simply define proto as a regular data property of the new object. If we don't disallow the special meaning then we will need to define what exactly this means. I vote would be Syntax Error.
Also, note that any use of proto in a object literal that is expected to set [[Prototype]] requires special specification treatment (for any mode).
I don't think proto-supporting implementations will do anyone any favors by banning the special meaning of proto in ES5 strict mode. The likeliest outcomeis further non-adoption of ES5 strict mode by developers, who already avoid it because of deoptimization effects.
In particular, without an alternative literal form such as
null <| {...}
(the ... is meta), removing support for magic proto in object literals simply prevents "use strict"; from being added to code that makes "good" use of proto, e.g. predefine an object literal's [[Prototype]] to null.
If we want to defer strict mode adoption until <| is added, we'll have less time gaining feedback on the migration costs of ES5 strict mode. This could be worth that cost if "use strict"; is too hard to test both ways, compared to a true
use strict;
or
use version 6;
pragma that downrev browsers will fail to parse rather than silently ignore. But do we really want to add another reason for developers to avoid ES5 strict mode?
Separately from that, implementors won't want to evolve the meaning of ES5 strict much. Continuing to add restrictions to it, especially runtime semantic shifts instead of early errors, as in what's proposed below, smells very bad to me. Progressively more restrictive implementations risk developer blow-back for the hard-to-find bugs introduced by such shifts.
On 30 December 2011 00:00, Mark S. Miller <erights at google.com> wrote:
{proto:...} having a magic meaning is per code, and so can and should be conditioned on mode. I think it should only be allowed to be magic in non-strict mode.
What about {["proto"]: ...} then?
Or, "Even Crock's Code Doesn't Overcome All The Bad Parts."
In this message, I explain why I believe we need to clean up proto. I'll start a separate thread on how to clean it up -- which is just my documenting a suggestion from Dave Herman.
eleventyone.done.hu/OReilly.JavaScript.The.Good.Parts.May.2008.pdf>
PDF page 119
that.on = function (type, ....) { .... var registry = {}; .... if (registry.hasOwnProperty(type)) { registry[type].push(handler); } else { registry[type] = [handler]; } .... };
This "registry" is a classic use of an object as a string-to-value map. Note that the "type" parameter comes in from the client of the abstraction. The only validation performed on it is the "hasOwnProperty" check above. Elsewhere Crock points out that this hasOwnProperty check does not work if "hasOwnProperty" itself is ever used as a key, and so we should instead say:
This still doesn't yet work on most actual browsers for a reason explained below.
<aside on why ES5 does not really help>
Going beyond "Good Parts", EcmaScript 5 gives us two new opportunities to clean up this code further.
at initialization time, assuming that this module gets to initialize itself before such monkey patching may have occurred:
var bind = Function.prototype.bind; var uncurryThis = bind.bind(bind.call);
var hopFn = uncurryThis({}.hasOwnProperty);
So that we can say
We can replace the initialization of "registry" with
var registry = Object.create(null);
or, if we wish the same protection from post-initialization monkey patching
var create = Object.create; ... var registry = create(null);
It turns out that neither of these additional ES5-based paranoid steps helps with the worst problem remaining with Crock's corrected code. Likewise, nothing so far proposed for ES-next would help, except for Dave Herman's suggestion which I will write up shortly.
</aside on why ES5 does not really help>
As you've probably already guessed from the title of this email, the problem occurs on most platforms when the function is called with an argument of "proto". Since we do a "hasOwnProperty" check first, the way in which this goes wrong depends on the platform. Evaluating
{}.proto.hasOwnProperty('proto')]
on various browsers gives:
false,false // Chrome true,true // WebKit Nightly false,true // FF Nightly false,false // Opera 12 alpha throws TypeError // IE10 Preview 2
Since IE doesn't treat "proto" as special magic, no problem there. WebKit Nightly will take the branch that happens to fail safe here, but only for the reason that there's no "Object.prototype.push". All we can say is that we got lucky this time. When Chrome, FF, and Opera evaluate
they change registry to inherit from [handler] rather than Object.prototype. WebKit would do the same if it had executed this line.
Think this exercise is academic and doesn't arise in real systems? As observed at < www.google.com/support/forum/p/Google+Docs/thread?tid=0cd4a00bd4aef9e4>,
until recently, on all non-IE browsers, if you typed "proto" at the beginning of a new Google Doc, your Google Doc would hang. This was tracked down to such a buggy use of an object as a string map. (To avoid such problems, Caja is shifting to using < code.google.com/p/es-lab/source/browse/trunk/src/ses/StringMap.js>,
which does seem safe on all platforms.)