ArrayBuffer neutering
On May 20, 2014, at 7:37 AM, Anne van Kesteren wrote:
"ArrayBuffer instances whose [[ArrayBufferData]] is null are considered to be neutered" However, most algorithms only check for it being undefined (and sometimes missing).
The operations on ArrayBuffers all have lines that are the equivalent of:
1 Let block be arrayBuffer’s [[ArrayBufferData]] internal slot. 2 If block is undefined or null, then throw a TypeError exception.
undefined is used as an indication that the ArrayBuffer has not yet been initialized. null is an indication that it has been neutered.
See: people.mozilla.org/~jorendorff/es6-draft.html#sec-clonearraybuffer, people.mozilla.org/~jorendorff/es6-draft.html#sec-getvaluefrombuffer, people.mozilla.org/~jorendorff/es6-draft.html#sec-setvalueinbuffer, people.mozilla.org/~jorendorff/es6-draft.html#sec-get-arraybuffer.prototype.bytelength, people.mozilla.org/~jorendorff/es6-draft.html#sec-arraybuffer.prototype.slice
Anything that access the actual data of an ArrayBuffer ultimately goes through one of the above operations or methods.
Let me know if you see any path in the spec. that doesn't.
In addition, the behavior defined seems incompatible with what is in browsers:
The ES6 TypedArray/ArrayBuffer spec. was written closely following the Khronos spec. which is pretty vague about what happens when an ArrayBuffer is neutered.
software.hixie.ch/utilities/js/live-dom-viewer <script> var a = new ArrayBuffer(12) onmessage = function(e) { w(e.data); w(e.data.byteLength); w(a); w(a.byteLength) } postMessage(a, "software.hixie.ch", [a]) </script>
The above will log:
object "[object ArrayBuffer]" (0 props) 12 object "[object ArrayBuffer]" (0 props) 0
And not throw a TypeError as suggested.
Is this behavior actually specified anywhere. If not, do all significant browsers implemented this behavior. Does web content depend upon it. (What use could be made of knowing the (former) length of a neutered ArrayBuffer.
It's a trivial change to make to the ES spec. if it's needed. Mostly I'm trying to understand the requirements that would motivate the change.
On Tue, May 20, 2014 at 5:25 PM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
The ES6 TypedArray/ArrayBuffer spec. was written closely following the Khronos spec. which is pretty vague about what happens when an ArrayBuffer is neutered.
If you go through www.khronos.org/registry/typedarray/specs/latest and search for "neutered" it seems relatively clear. byteLength returns 0 and the other members follow from there.
Is this behavior actually specified anywhere. If not, do all significant browsers implemented this behavior. Does web content depend upon it. (What use could be made of knowing the (former) length of a neutered ArrayBuffer.
What I tested seems to be specified and implemented by Firefox, Safari, and Chrome.
On May 20, 2014, at 8:37 AM, Anne van Kesteren wrote:
On Tue, May 20, 2014 at 5:25 PM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
The ES6 TypedArray/ArrayBuffer spec. was written closely following the Khronos spec. which is pretty vague about what happens when an ArrayBuffer is neutered.
If you go through www.khronos.org/registry/typedarray/specs/latest and search for "neutered" it seems relatively clear. byteLength returns 0 and the other members follow from there.
Wow, I'm pretty sure all of those return 0 if neutered's weren't there when I used the Khronos spec. to generate the ES6 spec.
The change for byteLength of ArrayBuffer is trivial (and already done).
However, the changes for ArrayBufferView (note this doesn't actually exists in the ES spec.) and in particular the 'length' property of the various Typed Arrays has a lot of implications to consider.
In ES6, (almost) all of the Array.prototype methods are available as methods of typed arrays. Do you really want all of these, if applied to an Typed Array with a neutered ArrayBuffer to act as if they were a 0 length array. That would seem to just be a good way to obscure bugs as many of the array methods turn into no-ops when applied to 0-length arrays. With the current ES6 spec. they would throw as soon as they tried to access the underlying ArrayBuffer using the original length.
Also note, that ignoring this new requirement, a Typed Array's length, byteLength, and byteOffset are all constants and this fact is used in specifying the behavior of the methods that operate upon them. If they can change (even to 0) then this can occur on any operation that can trigger side-effects. (For example, consider calling the callback function on 'map' or similar methods). Do we really want to dynamically reconsider changes to ;length' as opposed to simply letting throws to occur on access to the neutered ArrayBuffer?
Finally, I note that the current Khronos spec. doesn't provide much guidance in this regard. The thing it has that is most similar to the other array methods is the 'subarray' method and it doesn't explicitly say anything about what happens when it is applied to a TypedArray with an underlying neutered ArrayBuffer.
Is this behavior actually specified anywhere. If not, do all significant browsers implemented this behavior. Does web content depend upon it. (What use could be made of knowing the (former) length of a neutered ArrayBuffer.
What I tested seems to be specified and implemented by Firefox, Safari, and Chrome.
Sorry, I missed read your test case. I thought you were saying that the original length was preserved.
On Tue, May 20, 2014 at 9:40 AM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
On May 20, 2014, at 8:37 AM, Anne van Kesteren wrote:
On Tue, May 20, 2014 at 5:25 PM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
The ES6 TypedArray/ArrayBuffer spec. was written closely following the Khronos spec. which is pretty vague about what happens when an ArrayBuffer is neutered.
If you go through www.khronos.org/registry/typedarray/specs/latest and search for "neutered" it seems relatively clear. byteLength returns 0 and the other members follow from there.
Wow, I'm pretty sure all of those return 0 if neutered's weren't there when I used the Khronos spec. to generate the ES6 spec.
The change for byteLength of ArrayBuffer is trivial (and already done).
However, the changes for ArrayBufferView (note this doesn't actually exists in the ES spec.) and in particular the 'length' property of the various Typed Arrays has a lot of implications to consider.
In ES6, (almost) all of the Array.prototype methods are available as methods of typed arrays. Do you really want all of these, if applied to an Typed Array with a neutered ArrayBuffer to act as if they were a 0 length array. That would seem to just be a good way to obscure bugs as many of the array methods turn into no-ops when applied to 0-length arrays. With the current ES6 spec. they would throw as soon as they tried to access the underlying ArrayBuffer using the original length.
Also note, that ignoring this new requirement, a Typed Array's length, byteLength, and byteOffset are all constants and this fact is used in specifying the behavior of the methods that operate upon them. If they can change (even to 0) then this can occur on any operation that can trigger side-effects. (For example, consider calling the callback function on 'map' or similar methods). Do we really want to dynamically reconsider changes to ;length' as opposed to simply letting throws to occur on access to the neutered ArrayBuffer?
First, some background. When typed arrays were designed, they were specified with Web IDL and its ECMAScript binding. There were attempts during typed arrays' development to throw exceptions on some operations -- like out-of-range indexing -- but one by one these were discovered to be incompatible with either Web IDL's or ECMAScript's semantics like property lookup.
When neutering was added, there were several discussions regarding how to detect that an object had been neutered. The current behavior of treating the ArrayBuffer and all of its views as though they became zero-length was arrived at after careful consideration of the alternatives. I also recall discussing this behavior in person with ECMAScript committee members.
Throwing an exception upon fetching these properties from a neutered ArrayBuffer or view might provide nice fail-fast behavior but it's also incompatible with the current specification. It's likely that web applications using web workers, typed arrays and Transferables will break if this change is made.
Finally, I note that the current Khronos spec. doesn't provide much guidance in this regard. The thing it has that is most similar to the other array methods is the 'subarray' method and it doesn't explicitly say anything about what happens when it is applied to a TypedArray with an underlying neutered ArrayBuffer.
It isn't clear to me that it needs to. Starting from a view which is pointing to a neutered ArrayBuffer, there is no way to create a subarray of any nonzero length.
below On May 20, 2014, at 4:28 PM, Kenneth Russell wrote:
On Tue, May 20, 2014 at 9:40 AM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
On May 20, 2014, at 8:37 AM, Anne van Kesteren wrote:
On Tue, May 20, 2014 at 5:25 PM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
The ES6 TypedArray/ArrayBuffer spec. was written closely following the Khronos spec. which is pretty vague about what happens when an ArrayBuffer is neutered.
If you go through www.khronos.org/registry/typedarray/specs/latest and search for "neutered" it seems relatively clear. byteLength returns 0 and the other members follow from there.
Wow, I'm pretty sure all of those return 0 if neutered's weren't there when I used the Khronos spec. to generate the ES6 spec.
The change for byteLength of ArrayBuffer is trivial (and already done).
However, the changes for ArrayBufferView (note this doesn't actually exists in the ES spec.) and in particular the 'length' property of the various Typed Arrays has a lot of implications to consider.
In ES6, (almost) all of the Array.prototype methods are available as methods of typed arrays. Do you really want all of these, if applied to an Typed Array with a neutered ArrayBuffer to act as if they were a 0 length array. That would seem to just be a good way to obscure bugs as many of the array methods turn into no-ops when applied to 0-length arrays. With the current ES6 spec. they would throw as soon as they tried to access the underlying ArrayBuffer using the original length.
Also note, that ignoring this new requirement, a Typed Array's length, byteLength, and byteOffset are all constants and this fact is used in specifying the behavior of the methods that operate upon them. If they can change (even to 0) then this can occur on any operation that can trigger side-effects. (For example, consider calling the callback function on 'map' or similar methods). Do we really want to dynamically reconsider changes to ;length' as opposed to simply letting throws to occur on access to the neutered ArrayBuffer?
First, some background. When typed arrays were designed, they were specified with Web IDL and its ECMAScript binding. There were attempts during typed arrays' development to throw exceptions on some operations -- like out-of-range indexing -- but one by one these were discovered to be incompatible with either Web IDL's or ECMAScript's semantics like property lookup.
When neutering was added, there were several discussions regarding how to detect that an object had been neutered. The current behavior of treating the ArrayBuffer and all of its views as though they became zero-length was arrived at after careful consideration of the alternatives. I also recall discussing this behavior in person with ECMAScript committee members.
Throwing an exception upon fetching these properties from a neutered ArrayBuffer or view might provide nice fail-fast behavior but it's also incompatible with the current specification. It's likely that web applications using web workers, typed arrays and Transferables will break if this change is made.
We can make those properties all return 0, without too much impact. But that doesn't necessarily mean all the new methods (over 20) that ES6 provides for Typed Arrays can't also fail fast when applied to an TypedArray with a neutered array buffer rather than trying to pretend that it just happens to be zero length. In particular, I don't want to have to scatter length change checks throughout thee algorithms in case one gets neutered as a side-effect of a callback or a proxy mediated access.
What I propose is that for these new methods we will do a neutered check on entry and immediately throw a TypeError if the this value is a neutered. If not the algorithms proceeds using the current length, etc. values. If the typed array gets neutered while in the middle of one of these algorithms, a TypeError will get thrown at the point where the algorithm next tries to read or write into the ArrayBuffer.
Oner issue, is that there is currently know way for ES code to test if an Buffer has been neutered. That means we while I can specify these new built-ins as during the neutered test there is no way to write a ES-hosted implementation of that functionality. Why didn't you provide a isNeutered predicate?
Finally, I note that the current Khronos spec. doesn't provide much guidance in this regard. The thing it has that is most similar to the other array methods is the 'subarray' method and it doesn't explicitly say anything about what happens when it is applied to a TypedArray with an underlying neutered ArrayBuffer.
It isn't clear to me that it needs to. Starting from a view which is pointing to a neutered ArrayBuffer, there is no way to create a subarray of any nonzero length.
No, but it seems highly unlikely that anybody doing myTypedArray.subarray(5,10) actually wants to get back a 0-length array is myTypedArray happens to be neutered.
On Wed, May 21, 2014 at 2:10 AM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
Why didn't you provide a isNeutered predicate?
Note that per www.whatwg.org/specs/web-apps/current-work/multipage/common-dom-interfaces.html#transferable there's more objects that can be neutered. Once dslomov-chromium/ecmascript-structured-clone is picked up TC39 should probably develop an overarching strategy for these kind of objects.
On Wed, May 21, 2014 at 2:10 AM, Allen Wirfs-Brock <allen at wirfs-brock.com>wrote:
below On May 20, 2014, at 4:28 PM, Kenneth Russell wrote:
On Tue, May 20, 2014 at 9:40 AM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
On May 20, 2014, at 8:37 AM, Anne van Kesteren wrote:
On Tue, May 20, 2014 at 5:25 PM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
The ES6 TypedArray/ArrayBuffer spec. was written closely following the Khronos spec. which is pretty vague about what happens when an ArrayBuffer
is neutered.
If you go through www.khronos.org/registry/typedarray/specs/latest and search for "neutered" it seems relatively clear. byteLength returns 0 and the other members follow from there.
Wow, I'm pretty sure all of those return 0 if neutered's weren't there when I used the Khronos spec. to generate the ES6 spec.
The change for byteLength of ArrayBuffer is trivial (and already done).
However, the changes for ArrayBufferView (note this doesn't actually exists in the ES spec.) and in particular the 'length' property of the various Typed Arrays has a lot of implications to consider.
In ES6, (almost) all of the Array.prototype methods are available as methods of typed arrays. Do you really want all of these, if applied to an Typed Array with a neutered ArrayBuffer to act as if they were a 0 length array. That would seem to just be a good way to obscure bugs as many of the array methods turn into no-ops when applied to 0-length arrays. With the current ES6 spec. they would throw as soon as they tried to access the underlying ArrayBuffer using the original length.
Also note, that ignoring this new requirement, a Typed Array's length, byteLength, and byteOffset are all constants and this fact is used in specifying the behavior of the methods that operate upon them. If they can change (even to 0) then this can occur on any operation that can trigger side-effects. (For example, consider calling the callback function on 'map' or similar methods). Do we really want to dynamically reconsider changes to ;length' as opposed to simply letting throws to occur on access to the neutered ArrayBuffer?
First, some background. When typed arrays were designed, they were specified with Web IDL and its ECMAScript binding. There were attempts during typed arrays' development to throw exceptions on some operations -- like out-of-range indexing -- but one by one these were discovered to be incompatible with either Web IDL's or ECMAScript's semantics like property lookup.
When neutering was added, there were several discussions regarding how to detect that an object had been neutered. The current behavior of treating the ArrayBuffer and all of its views as though they became zero-length was arrived at after careful consideration of the alternatives. I also recall discussing this behavior in person with ECMAScript committee members.
Throwing an exception upon fetching these properties from a neutered ArrayBuffer or view might provide nice fail-fast behavior but it's also incompatible with the current specification. It's likely that web applications using web workers, typed arrays and Transferables will break if this change is made.
We can make those properties all return 0, without too much impact. But that doesn't necessarily mean all the new methods (over 20) that ES6 provides for Typed Arrays can't also fail fast when applied to an TypedArray with a neutered array buffer rather than trying to pretend that it just happens to be zero length. In particular, I don't want to have to scatter length change checks throughout thee algorithms in case one gets neutered as a side-effect of a callback or a proxy mediated access.
What I propose is that for these new methods we will do a neutered check on entry and immediately throw a TypeError if the this value is a neutered. If not the algorithms proceeds using the current length, etc. values. If the typed array gets neutered while in the middle of one of these algorithms, a TypeError will get thrown at the point where the algorithm next tries to read or write into the ArrayBuffer.
Oner issue, is that there is currently know way for ES code to test if an Buffer has been neutered. That means we while I can specify these new built-ins as during the neutered test there is no way to write a ES-hosted implementation of that functionality. Why didn't you provide a isNeutered predicate?
Finally, I note that the current Khronos spec. doesn't provide much guidance in this regard. The thing it has that is most similar to the other array methods is the 'subarray' method and it doesn't explicitly say anything about what happens when it is applied to a TypedArray with an underlying neutered ArrayBuffer.
It isn't clear to me that it needs to. Starting from a view which is pointing to a neutered ArrayBuffer, there is no way to create a subarray of any nonzero length.
No, but it seems highly unlikely that anybody doing myTypedArray.subarray(5,10) actually wants to get back a 0-length array is myTypedArray happens to be neutered.
Maybe, but I believe we have been shipping 'subarray' with exactly this behavior for quite a while now, and speccing it otherwise will be a compat hazard.
On May 21, 2014, at 2:02 AM, Dmitry Lomov wrote:
Finally, I note that the current Khronos spec. doesn't provide much guidance in this regard. The thing it has that is most similar to the other array methods is the 'subarray' method and it doesn't explicitly say anything about what happens when it is applied to a TypedArray with an underlying neutered ArrayBuffer.
It isn't clear to me that it needs to. Starting from a view which is pointing to a neutered ArrayBuffer, there is no way to create a subarray of any nonzero length.
No, but it seems highly unlikely that anybody doing myTypedArray.subarray(5,10) actually wants to get back a 0-length array is myTypedArray happens to be neutered.
Maybe, but I believe we have been shipping 'subarray' with exactly this behavior for quite a while now, and speccing it otherwise will be a compat hazard.
I didn't say that the pre-existing subarray method should change its behavior. But I am proposing that the new methods being added by ES6 fail hard if they encounter a neutered object.
On Wed, May 21, 2014 at 5:19 PM, Allen Wirfs-Brock <allen at wirfs-brock.com>wrote:
On May 21, 2014, at 2:02 AM, Dmitry Lomov wrote:
Finally, I note that the current Khronos spec. doesn't provide much guidance in this regard. The thing it has that is most similar to the other array methods is the 'subarray' method and it doesn't explicitly say anything about what happens when it is applied to a TypedArray with an underlying neutered ArrayBuffer.
It isn't clear to me that it needs to. Starting from a view which is pointing to a neutered ArrayBuffer, there is no way to create a subarray of any nonzero length.
No, but it seems highly unlikely that anybody doing myTypedArray.subarray(5,10) actually wants to get back a 0-length array is myTypedArray happens to be neutered.
Maybe, but I believe we have been shipping 'subarray' with exactly this behavior for quite a while now, and speccing it otherwise will be a compat hazard.
I didn't say that the pre-existing subarray method should change its behavior. But I am proposing that the new methods being added by ES6 fail hard if they encounter a neutered object.
I think it would be weird if some of them fail hard and some would behave as if the length is zero. Consistency is always good. Why "fail hard" is more desirable?
Dmitry
On May 21, 2014, at 9:59 AM, Dmitry Lomov wrote:
On Wed, May 21, 2014 at 5:19 PM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
On May 21, 2014, at 2:02 AM, Dmitry Lomov wrote:
Finally, I note that the current Khronos spec. doesn't provide much guidance in this regard. The thing it has that is most similar to the other array methods is the 'subarray' method and it doesn't explicitly say anything about what happens when it is applied to a TypedArray with an underlying neutered ArrayBuffer.
It isn't clear to me that it needs to. Starting from a view which is pointing to a neutered ArrayBuffer, there is no way to create a subarray of any nonzero length.
No, but it seems highly unlikely that anybody doing myTypedArray.subarray(5,10) actually wants to get back a 0-length array is myTypedArray happens to be neutered.
Maybe, but I believe we have been shipping 'subarray' with exactly this behavior for quite a while now, and speccing it otherwise will be a compat hazard.
I didn't say that the pre-existing subarray method should change its behavior. But I am proposing that the new methods being added by ES6 fail hard if they encounter a neutered object.
I think it would be weird if some of them fail hard and some would behave as if the length is zero. Consistency is always good. Why "fail hard" is more desirable?
Because trying to invoke any of these methods (we're talking about thinks like map, filter, reduce, copyWithin, sort, etc.) on a neutered TypedArray is surely a logic bug (can you think of any case where it isn't). Throwing helps people find such bugs.
The only inconsistency is with the single legacy method 'subarray'. Personally, I'd be fine with trying to also change it to throw. Again, can you think of any case where doing someNeuteredArray.subarray(10) isn't a bug?
On Tue, May 20, 2014 at 5:10 PM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
below On May 20, 2014, at 4:28 PM, Kenneth Russell wrote:
On Tue, May 20, 2014 at 9:40 AM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
On May 20, 2014, at 8:37 AM, Anne van Kesteren wrote:
On Tue, May 20, 2014 at 5:25 PM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
The ES6 TypedArray/ArrayBuffer spec. was written closely following the Khronos spec. which is pretty vague about what happens when an ArrayBuffer is neutered.
If you go through www.khronos.org/registry/typedarray/specs/latest and search for "neutered" it seems relatively clear. byteLength returns 0 and the other members follow from there.
Wow, I'm pretty sure all of those return 0 if neutered's weren't there when I used the Khronos spec. to generate the ES6 spec.
The change for byteLength of ArrayBuffer is trivial (and already done).
However, the changes for ArrayBufferView (note this doesn't actually exists in the ES spec.) and in particular the 'length' property of the various Typed Arrays has a lot of implications to consider.
In ES6, (almost) all of the Array.prototype methods are available as methods of typed arrays. Do you really want all of these, if applied to an Typed Array with a neutered ArrayBuffer to act as if they were a 0 length array. That would seem to just be a good way to obscure bugs as many of the array methods turn into no-ops when applied to 0-length arrays. With the current ES6 spec. they would throw as soon as they tried to access the underlying ArrayBuffer using the original length.
Also note, that ignoring this new requirement, a Typed Array's length, byteLength, and byteOffset are all constants and this fact is used in specifying the behavior of the methods that operate upon them. If they can change (even to 0) then this can occur on any operation that can trigger side-effects. (For example, consider calling the callback function on 'map' or similar methods). Do we really want to dynamically reconsider changes to ;length' as opposed to simply letting throws to occur on access to the neutered ArrayBuffer?
First, some background. When typed arrays were designed, they were specified with Web IDL and its ECMAScript binding. There were attempts during typed arrays' development to throw exceptions on some operations -- like out-of-range indexing -- but one by one these were discovered to be incompatible with either Web IDL's or ECMAScript's semantics like property lookup.
When neutering was added, there were several discussions regarding how to detect that an object had been neutered. The current behavior of treating the ArrayBuffer and all of its views as though they became zero-length was arrived at after careful consideration of the alternatives. I also recall discussing this behavior in person with ECMAScript committee members.
Throwing an exception upon fetching these properties from a neutered ArrayBuffer or view might provide nice fail-fast behavior but it's also incompatible with the current specification. It's likely that web applications using web workers, typed arrays and Transferables will break if this change is made.
We can make those properties all return 0, without too much impact. But that doesn't necessarily mean all the new methods (over 20) that ES6 provides for Typed Arrays can't also fail fast when applied to an TypedArray with a neutered array buffer rather than trying to pretend that it just happens to be zero length. In particular, I don't want to have to scatter length change checks throughout thee algorithms in case one gets neutered as a side-effect of a callback or a proxy mediated access.
What I propose is that for these new methods we will do a neutered check on entry and immediately throw a TypeError if the this value is a neutered. If not the algorithms proceeds using the current length, etc. values. If the typed array gets neutered while in the middle of one of these algorithms, a TypeError will get thrown at the point where the algorithm next tries to read or write into the ArrayBuffer.
Perhaps that would work, though I agree with Dmitry's comment later in this thread that consistency is best.
Oner issue, is that there is currently know way for ES code to test if an Buffer has been neutered. That means we while I can specify these new built-ins as during the neutered test there is no way to write a ES-hosted implementation of that functionality. Why didn't you provide a isNeutered predicate?
Because when the concept of Transferable was formalized in the HTML5 spec, there was a goal to make the minimal possible changes. Transferable was basically a generalization of MessagePort, which was the only type that previously could be "transferred" to a web worker. Neutering is only a concept in spec text and not in the IDL. The Transferable typedef doesn't have any associated methods. The only way to neuter an object is to transfer it to a web worker. There were requests to provide a "close()" method and make Transferable a sub-interface of a new Closable interface. We resisted making those changes because they would have essentially introduced manual memory management to JavaScript. All of this can be revisited.
Finally, I note that the current Khronos spec. doesn't provide much guidance in this regard. The thing it has that is most similar to the other array methods is the 'subarray' method and it doesn't explicitly say anything about what happens when it is applied to a TypedArray with an underlying neutered ArrayBuffer.
It isn't clear to me that it needs to. Starting from a view which is pointing to a neutered ArrayBuffer, there is no way to create a subarray of any nonzero length.
No, but it seems highly unlikely that anybody doing myTypedArray.subarray(5,10) actually wants to get back a 0-length array is myTypedArray happens to be neutered.
I agree, but the compatibility impact has to be considered if subarray() is going to start throwing exceptions. It might well be minimal but it has to be measured.
What about typed arrays' indexed getters and setters? From reading these:
people.mozilla.org/~jorendorff/es6-draft.html#sec-getvaluefrombuffer, people.mozilla.org/~jorendorff/es6-draft.html#sec-setvalueinbuffer
it looks like in the ES6 spec they throw exceptions if called against neutered objects. What happens to the length of a neutered typed array view in the ES6 spec? Does it become 0 as in the Khronos typed array spec? What happens if an index which would have been out of range before the object was neutered is passed to the indexed getter or setter? What happens if the [] operator is used to get or set some random named property against a neutered typed array view?
On Wed, May 21, 2014 at 9:59 AM, Dmitry Lomov <dslomov at chromium.org> wrote:
I think it would be weird if some of them fail hard and some would behave as if the length is zero. Consistency is always good. Why "fail hard" is more desirable?
It is desirable because it allows for more efficient implementations in the common case. Quoting Allen Wirfs-Brock:
Also note, that ignoring this new requirement, a Typed Array's length,
byteLength, and byteOffset are all constants and this fact is used in specifying the behavior of the methods that operate upon them. If they can change (even to 0) then this can occur on any operation that can trigger side-effects. (For example, consider calling the callback function on 'map' or similar methods). Do we really want to dynamically reconsider changes to ;length' as opposed to simply letting throws to occur on access to the neutered ArrayBuffer?
[...]
In particular, I don't want to have to scatter length change checks throughout thee algorithms in case one gets neutered as a side-effect of a callback or a proxy mediated access.
As a concrete example, when iterating over a typed array inside the map()
implementation, it used to be possible to hoist the length of the array out
of the loop, since it is a constant, and the loop can be unrolled if the
length is known to be small. If the length of the array can change, the
length needs to be re-checked on every iteration and unrolling typically
won't happen. You can special-case this, since you know the only way the
length can change is if the array is neutered -- but all this special
casing adds up. It is cleaner (thus, in the absence of special-casing,
faster) to use the existing exception mechanism to fail fast where
necessary (for example, when you try to access the next element of an array
which has been neutered).
On May 21, 2014, at 10:50 AM, Kenneth Russell wrote:
On Tue, May 20, 2014 at 5:10 PM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
What I propose is that for these new methods we will do a neutered check on entry and immediately throw a TypeError if the this value is a neutered. If not the algorithms proceeds using the current length, etc. values. If the typed array gets neutered while in the middle of one of these algorithms, a TypeError will get thrown at the point where the algorithm next tries to read or write into the ArrayBuffer.
Perhaps that would work, though I agree with Dmitry's comment later in this thread that consistency is best.
I think it is better to have one legacy method that behaves differently from all the other (new) methods then to maintain consistency by introducing 20+ new bug farms.
Oner issue, is that there is currently know way for ES code to test if an Buffer has been neutered. That means we while I can specify these new built-ins as during the neutered test there is no way to write a ES-hosted implementation of that functionality. Why didn't you provide a isNeutered predicate?
Because when the concept of Transferable was formalized in the HTML5 spec, there was a goal to make the minimal possible changes. Transferable was basically a generalization of MessagePort, which was the only type that previously could be "transferred" to a web worker. Neutering is only a concept in spec text and not in the IDL. The Transferable typedef doesn't have any associated methods. The only way to neuter an object is to transfer it to a web worker. There were requests to provide a "close()" method and make Transferable a sub-interface of a new Closable interface. We resisted making those changes because they would have essentially introduced manual memory management to JavaScript. All of this can be revisited.
So you would have no objection to adding a isNeutered method to ArrayBuffer.prototype. As I mentioned, I think it is needed for self-hosting the currently specified typed array methods and any new methods that people might want to write the recognize the possibility that an arrays backing store can get neutered.
Finally, I note that the current Khronos spec. doesn't provide much guidance in this regard. The thing it has that is most similar to the other array methods is the 'subarray' method and it doesn't explicitly say anything about what happens when it is applied to a TypedArray with an underlying neutered ArrayBuffer.
It isn't clear to me that it needs to. Starting from a view which is pointing to a neutered ArrayBuffer, there is no way to create a subarray of any nonzero length.
No, but it seems highly unlikely that anybody doing myTypedArray.subarray(5,10) actually wants to get back a 0-length array is myTypedArray happens to be neutered.
I agree, but the compatibility impact has to be considered if subarray() is going to start throwing exceptions. It might well be minimal but it has to be measured.
This sounds like something that can be tried in a pre-beta browser build.
What about typed arrays' indexed getters and setters? From reading these:
people.mozilla.org/~jorendorff/es6-draft.html#sec-getvaluefrombuffer, people.mozilla.org/~jorendorff/es6-draft.html#sec-setvalueinbuffer
it looks like in the ES6 spec they throw exceptions if called against neutered objects.
Yes, as currently spec'ed [[Get]] and [[Set]] operations upon TypedArray instances throw if the backing ArrayBuffer has been neutered. But the reason is in people.mozilla.org/~jorendorff/es6-draft.html#sec-integerindexedelementget and people.mozilla.org/~jorendorff/es6-draft.html#sec-integerindexedelementset
(IntegerIndexed exotic objects are the ES underlying specification type that is used for all Typed Array instances).
Integer indexed property access (eg, ta[5] ) return undefined (on [[Get]]'s) that are outside of the 0..originalLength-1 range of the array. This test is specified in terms of an internal [[ArrayLength]] property that is immutably set to the original defined size of the array. So the spec. currently say an "in original array range" access on a TypedArray with a neutered ArrayBuffer will be "in range" but the call to GetValueFromBuffer will throw because the ArrayBuffer has been neutered.
What happens to the length of a neutered typed array view in the ES6 spec? Does it become 0 as in the Khronos typed array spec?
The length property of an Typed Array is an accessor property. I just updated my working draft of the specification of that accessor (people.mozilla.org/~jorendorff/es6-draft.html#sec-get-%typedarray%.prototype.length not yet updated) and also the accessors for byteLength and byteOffset so they return 0 when the backing ArrayBuffer is observed to be neutered.
However, that doesn't directly address the indexed access issues. Implementations presumably want indexed access to typed arrays to be has fast as possible, so they are unlikely to want to make a call to a 'length' accessor property on every indexed access. That's why we specify the [[ArrayLength]] as private state of the the typed array instance.
Instead I can either place a IsNeutered guard on the calls to GetValueInBuffer/SetValueInBuffer from the indexed accessors or change GetValueInBuffer/SetValueInBuffer to not throw on neutered buffered accesses. However, this also requires some care to make sure that DataView accesses continue to throw.
The bottom line of all of this is we want the fast path for a non-netured typed array access to be as fast as possible while still behaving safely (if not reasonably) in the presence of neuterable ArrayBuffers.
What happens if an index which would have been out of range before the object was neutered is passed to the indexed getter or setter?
Currently, in the ES spec. it is range checked against the original array length. All out of range accesses return undefined.
What happens if the [] operator is used to get or set some random named property against a neutered typed array view?
They just work (assuming that the property isn't an integer index). Neutering the backing ArrayBuffer doesn't do anything to the ordinary properties of a typed array instance.
I think it is better to have one legacy method that behaves differently from all the other (new) methods then to maintain consistency by introducing 20+ new bug farms.
Is it too late (or too unexpected) for aTypedArray.subarray(aNeuteredObject)to throw in new contexts (i.e., modules, generators, etc.)? Simply for the sake of reducing the footprint of the legacy method/behavior.
On May 21, 2014, at 5:51 PM, Jeremy Martin wrote:
I think it is better to have one legacy method that behaves differently from all the other (new) methods then to maintain consistency by introducing 20+ new bug farms.
Is it too late (or too unexpected) for aTypedArray.subarray(aNeuteredObject) to throw in new contexts (i.e., modules, generators, etc.)? Simply for the sake of reducing the footprint of the legacy method/behavior.
The behavior of build-in objects and methods cannot be context sensitive in this manner because object references flow across such contexts but methods are generally unware from what sort of context they are being called from. That's why strict mode restrictions are source code contextual and not object contextual.
I'm betting that the world doesn't end if subarray starts throw if applied to a typed array with a neutered buffer. I still try to see if one or more of the browser implementations are willing to try the experiment.
On Wed, May 21, 2014 at 9:04 PM, Allen Wirfs-Brock <allen at wirfs-brock.com>wrote:
Yes, as currently spec'ed [[Get]] and [[Set]] operations upon TypedArray instances throw if the backing ArrayBuffer has been neutered. But the reason is in
people.mozilla.org/~jorendorff/es6-draft.html#sec-integerindexedelementget and
people.mozilla.org/~jorendorff/es6-draft.html#sec-integerindexedelementset
(IntegerIndexed exotic objects are the ES underlying specification type that is used for all Typed Array instances).
Integer indexed property access (eg, ta[5] ) return undefined (on [[Get]]'s) that are outside of the 0..originalLength-1 range of the array. This test is specified in terms of an internal [[ArrayLength]] property that is immutably set to the original defined size of the array. So the spec. currently say an "in original array range" access on a TypedArray with a neutered ArrayBuffer will be "in range" but the call to GetValueFromBuffer will throw because the ArrayBuffer has been neutered.
What happens to the length of a neutered typed array view in the ES6 spec? Does it become 0 as in the Khronos typed array spec?
The length property of an Typed Array is an accessor property. I just updated my working draft of the specification of that accessor ( people.mozilla.org/~jorendorff/es6-draft.html#sec-get-%typedarray%.prototype.length not yet updated) and also the accessors for byteLength and byteOffset so they return 0 when the backing ArrayBuffer is observed to be neutered.
However, that doesn't directly address the indexed access issues. Implementations presumably want indexed access to typed arrays to be has fast as possible, so they are unlikely to want to make a call to a 'length' accessor property on every indexed access. That's why we specify the [[ArrayLength]] as private state of the the typed array instance.
Instead I can either place a IsNeutered guard on the calls to GetValueInBuffer/SetValueInBuffer from the indexed accessors or change GetValueInBuffer/SetValueInBuffer to not throw on neutered buffered accesses. However, this also requires some care to make sure that DataView accesses continue to throw.
The bottom line of all of this is we want the fast path for a non-netured typed array access to be as fast as possible while still behaving safely (if not reasonably) in the presence of neuterable ArrayBuffers.
What happens if an index which would have been out of range before the object was neutered is passed to the indexed getter or setter?
Currently, in the ES spec. it is range checked against the original array length. All out of range accesses return undefined.
What happens if the [] operator is used to get or set some random named property against a neutered typed array view?
They just work (assuming that the property isn't an integer index). Neutering the backing ArrayBuffer doesn't do anything to the ordinary properties of a typed array instance.
Wait, am I reading this correctly that after a backing array buffer is neutered, accesses to typed arrays within [0...original length) should throw and accesses outside that range should continue to return 'undefined'? This is definitely not what is shipping in Chrome, and looks like a compat hazard way bigger than 'subarray'.
Dmitry
On Wed, May 21, 2014 at 8:51 PM, C. Scott Ananian <ecmascript at cscott.net>wrote:
On Wed, May 21, 2014 at 9:59 AM, Dmitry Lomov <dslomov at chromium.org>wrote:
I think it would be weird if some of them fail hard and some would behave as if the length is zero. Consistency is always good. Why "fail hard" is more desirable?
It is desirable because it allows for more efficient implementations in the common case. Quoting Allen Wirfs-Brock:
Also note, that ignoring this new requirement, a Typed Array's length,
byteLength, and byteOffset are all constants and this fact is used in specifying the behavior of the methods that operate upon them. If they can change (even to 0) then this can occur on any operation that can trigger side-effects. (For example, consider calling the callback function on 'map' or similar methods). Do we really want to dynamically reconsider changes to ;length' as opposed to simply letting throws to occur on access to the neutered ArrayBuffer?
[...]
In particular, I don't want to have to scatter length change checks throughout thee algorithms in case one gets neutered as a side-effect of a callback or a proxy mediated access.
As a concrete example, when iterating over a typed array inside the
map()implementation, it used to be possible to hoist the length of the array out of the loop, since it is a constant, and the loop can be unrolled if the length is known to be small. If the length of the array can change, the length needs to be re-checked on every iteration and unrolling typically won't happen. You can special-case this, since you know the only way the length can change is if the array is neutered -- but all this special casing adds up. It is cleaner (thus, in the absence of special-casing, faster) to use the existing exception mechanism to fail fast where necessary (for example, when you try to access the next element of an array which has been neutered).
Good example, but it has nothing to do with 'subarray' (I didn't realize until I saw the other mail form Allan that we are introducing a much bigger change)
On May 22, 2014, at 4:01 AM, Dmitry Lomov wrote:
Wait, am I reading this correctly that after a backing array buffer is neutered, accesses to typed arrays within [0...original length) should throw and accesses outside that range should continue to return 'undefined'? This is definitely not what is shipping in Chrome, and looks like a compat hazard way bigger than 'subarray'.
Do you think any non-buggy code actually depends upon being able to index into a neutered typed array?
But, yes, that's the implication of the current ES6 spec. language. Essentially, TypedArray objects (or at least the actual indexed property access operations don't know anything about neutered ArrayBuffers so they do their normal bounds checks and and if an index is in bounds the actual access gets delegated to the ArrayBuffer which throws.
This can easily be changed (in the spec.) so, that TypedArrays have an explicit "is the buffer neutered" guard prior to bounds checking.
The more interesting question is the general concept of TypedArrays returning undefined on out-of-bounds accesses which (particularly after the neutering) seems to be a significant bug farm. Admittedly trying to changing it would be a breaking change, but would it break anything that wasn't already buggy.
I've been talking to Luke Wagner about this it at appears that one of the sources of out-of-bound accesses to Typed Arrays are Emscripten compilation bugs that should be fixed. (and this is a good example, of the danger of the current behavior. It's masking bugs that need to be discovered and fixed.).
Out-of-bounds access have also been observed in non-Emscripten code, for example 1, 2. Note that this is another example, of the current behavior masking logic bugs in the application code.
It's still early days for production use of TypedArrays and maybe its not too late to evangelize fixing any major apps that depend upon out-of-bounds accesses or accessed to neutered arrays. In TC39 we talk about having a "five fingers" breaking change budget for ES6. If we think a breaking change is important enough (and rare enough) we will try to evangelize away dependency upon it so we can make the change. But, we figure we can probably only get away with around 5 or less of such changes. Perhaps, throwing on out-of-bound typed array access should get allotted to a finger. I'm pretty sure that it would be the right thing to do from a long term perspective and that within a couple years we would have no regrets.
I actually think there are two risk levels of breaking changes we should consider (and experiment with)
-
Throw on all indexed accessed to typed arrays with neutered buffers. Note that this will only impact existing code that actually transfers typed arrays/ArrayBuffers and then (clearly for buggy reasons) try to access the transferred array. This actually seem like a pretty low risk change to me.
-
Throw on all out-of-bounds indexed accesses to typed arrays. This will impact existing code that contains out-of-bounds accesses (presumable either array creation size bugs or index computation bugs) including things like buggy Emscripten code. This is a higher risk change, but may be possible with evangelism.
It would be possible to do 1) and not do 2). But, both would be best.
We should at least consider these alternative rather accept without question another permanent JS WTF behavior.
Allen
On 5/22/14, 12:16 PM, Allen Wirfs-Brock wrote:
Out-of-bounds access have also been observed in non-Emscripten code, for example [1] [2].
Out-of-bounds reads are very common in code that works with canvas imagedata, because a lot of image-processing algorithms want to examine adjacent pixels, and when you get to the end you end up outside the array. Making out-of-bounds access throw on non-neutered Uint32ClampedArray is a non-starter, imo.
It's still early days for production use of TypedArrays
Uh... No, it's not. They've been shipping for years and people are using them all over, as far as I can tell.
and maybe its not too late to evangelize fixing any major apps that depend upon out-of-bounds accesses or accessed to neutered arrays.
Neutering is more recent, so I can't speak to compat issues for neutered arrays with certainty. I can say with certainty that out-of-bounds reads on Uint8ClampedArray can't change their behavior.
On May 22, 2014, at 9:28 AM, Boris Zbarsky wrote:
On 5/22/14, 12:16 PM, Allen Wirfs-Brock wrote:
Out-of-bounds access have also been observed in non-Emscripten code, for example [1] [2].
Out-of-bounds reads are very common in code that works with canvas imagedata, because a lot of image-processing algorithms want to examine adjacent pixels, and when you get to the end you end up outside the array. Making out-of-bounds access throw on non-neutered Uint32ClampedArray is a non-starter, imo.
I can believe it. Although I suspect a lot of those image-process algorithms were originally expressed in C code where hopefully they weren't dong out-of bounds accesses.
It's still early days for production use of TypedArrays
Uh... No, it's not. They've been shipping for years and people are using them all over, as far as I can tell.
and maybe its not too late to evangelize fixing any major apps that depend upon out-of-bounds accesses or accessed to neutered arrays.
Neutering is more recent, so I can't speak to compat issues for neutered arrays with certainty. I can say with certainty that out-of-bounds reads on Uint8ClampedArray can't change their behavior.
I'd be reasonably content if we could make accesses to neutered arrays throw and keep undefined as the result of out-of-bounds accesses for non-neutered typed arrays.
"ArrayBuffer instances whose [[ArrayBufferData]] is null are considered to be neutered" However, most algorithms only check for it being undefined (and sometimes missing).
In addition, the behavior defined seems incompatible with what is in browsers:
software.hixie.ch/utilities/js/live-dom-viewer
The above will log:
And not throw a TypeError as suggested.
"ArrayBuffer instances whose [[ArrayBufferData]] is null are considered to be neutered" However, most algorithms only check for it being undefined (and sometimes missing). In addition, the behavior defined seems incompatible with what is in browsers: http://software.hixie.ch/utilities/js/live-dom-viewer/ <script> var a = new ArrayBuffer(12) onmessage = function(e) { w(e.data); w(e.data.byteLength); w(a); w(a.byteLength) } postMessage(a, "http://software.hixie.ch", [a]) </script> The above will log: object "[object ArrayBuffer]" (0 props) 12 object "[object ArrayBuffer]" (0 props) 0 And not throw a TypeError as suggested. -- http://annevankesteren.nl/