Discuss
  • Lexically Scoped Object Extensions (was About private names)

Lexically Scoped Object Extensions (was About private names)

# Erik Arvidsson (15 years ago)

The thread about using private names is getting a bit unwieldy but I'd like to focus on the use case that I have been thinking of as "Lexically scoped monkey patching" or "Lexically scoped object extensions" instead of focusing on how to use "private names" to fit this scenario.

Extending built ins and modifying existing classes to work around bugs or to provide a better API is (or was) a common pattern. Today a lot of JS library shun this approach due to the risk of conflicts.

Let us assume that you could extend an object in your current lexical scope and that such extensions could be imported from a module into your current scope.

Given:

{ function largerThanN(obj, n) { return obj.filter(function(item) { return item > n; } }

var a = [0, 1, 2, 3, 4]; print(largerThanN(a, 2)); }

Now we would like to make largerThanN to work with Object objects. The naïve thing to do is to just to add a filter method to Object.prototype. However, this might cause conflicts with other code that uses objects. The idea here is that we can do this safely in our scope (ignore syntax except that it is important that it can be done statically).

{ extend Object.prototype with { filter: function(fun) { var retval = {}; for (var key in this) { if (fun(this[key]) retval[key] = this[key]; } return retval; } };

function largerThanN(obj, n) { return obj.filter(function(item) { return item > n; } }

var a = [0, 1, 2, 3, 4]; print(largerThanN(a, 2)); var o = {a: 0, b: 1, c: 2, d: 3, e: 4}; print(largerThanN(0, 2)); }

The above use case cannot be solved using private names because private names conflict with public names.

Can we agree that this is a use case that we care about and focus on this instead of whether private names can or cannot do this?

# Andrew Dupont (15 years ago)

Erik,

Yes, this is exactly what a framework like Prototype would need. Your code is highly reminiscent of Ruby's proposed "refinements" feature [1], one which is being debated for inclusion in Ruby 2.0. If this could be implemented in a way that avoids the dreaded namespace dragons, then I would lobby hard for a feature like this.

Cheers, Andrew

[1] timelessrepo.com/refinements

# Waldemar Horwat (15 years ago)

On 03/21/11 13:13, Erik Arvidsson wrote:

The thread about using private names is getting a bit unwieldy but I'd like to focus on the use case that I have been thinking of as "Lexically scoped monkey patching" or "Lexically scoped object extensions" instead of focusing on how to use "private names" to fit this scenario.

Extending built ins and modifying existing classes to work around bugs or to provide a better API is (or was) a common pattern. Today a lot of JS library shun this approach due to the risk of conflicts.

Let us assume that you could extend an object in your current lexical scope and that such extensions could be imported from a module into your current scope.

Given:

{ function largerThanN(obj, n) { return obj.filter(function(item) { return item> n; } }

var a = [0, 1, 2, 3, 4]; print(largerThanN(a, 2)); }

Now we would like to make largerThanN to work with Object objects. The naïve thing to do is to just to add a filter method to Object.prototype. However, this might cause conflicts with other code that uses objects. The idea here is that we can do this safely in our scope (ignore syntax except that it is important that it can be done statically).

{ extend Object.prototype with { filter: function(fun) { var retval = {}; for (var key in this) { if (fun(this[key]) retval[key] = this[key]; } return retval; } };

function largerThanN(obj, n) { return obj.filter(function(item) { return item> n; } }

var a = [0, 1, 2, 3, 4]; print(largerThanN(a, 2)); var o = {a: 0, b: 1, c: 2, d: 3, e: 4}; print(largerThanN(0, 2)); }

The above use case cannot be solved using private names because private names conflict with public names.

Can we agree that this is a use case that we care about and focus on this instead of whether private names can or cannot do this?

Yes, this is useful. If you follow this to its logical conclusion (in particular, supporting Allen's use cases), you'll get something analogous to ES4 namespaces.

 Waldemar
# Erik Arvidsson (15 years ago)

On Mon, Mar 21, 2011 at 16:45, Waldemar Horwat <waldemar at google.com> wrote:

On 03/21/11 13:13, Erik Arvidsson wrote:

The thread about using private names is getting a bit unwieldy but I'd like to focus on the  use case that I have been thinking of as "Lexically scoped monkey patching" or "Lexically scoped object extensions" instead of focusing on how to use "private names" to fit this scenario.

Extending built ins and modifying existing classes to work around bugs or to provide a better API is (or was) a common pattern. Today a lot of JS library shun this approach due to the risk of conflicts.

Let us assume that you could extend an object in your current lexical scope and that such extensions could be imported from a module into your current scope.

Given:

{   function largerThanN(obj, n) {     return obj.filter(function(item) {       return item>  n;     }   }

var a = [0, 1, 2, 3, 4];   print(largerThanN(a, 2)); }

Now we would like to make largerThanN to work with Object objects. The naïve thing to do is to just to add a filter method to Object.prototype. However, this might cause conflicts with other code that uses objects. The idea here is that we can do this safely in our scope (ignore syntax except that it is important that it can be done statically).

{   extend Object.prototype with {     filter: function(fun) {       var retval = {};       for (var key in this) {         if (fun(this[key])           retval[key] = this[key];       }       return retval;     }   };

function largerThanN(obj, n) {     return obj.filter(function(item) {       return item>  n;     }   }

var a = [0, 1, 2, 3, 4];   print(largerThanN(a, 2));   var o = {a: 0, b: 1, c: 2, d: 3, e: 4};   print(largerThanN(0, 2)); }

The above use case cannot be solved using private names because private names conflict with public names.

Can we agree that this is a use case that we care about and focus on this instead of whether private names can or cannot do this?

Yes, this is useful.  If you follow this to its logical conclusion (in particular, supporting Allen's use cases), you'll get something analogous to ES4 namespaces.

Yes, but we were hoping we could limit the scope of these significantly so we don't end up with something like ES4 namespaces.

  1. The scope of an extension is limited to a module and it is statically determined. This needs new syntax so it can be detected without executing any code.

  2. An extension is limited to an object and a property name (sure we would support importing * from a definition but that would still be something that can be statically determined before any code execution)

Given these 2 we believe that we can build the alternative lookup table for the object statically and associate that with the module scope, making property lookups not have the explosive performance overhead that ES4 namespaces had.

# Allen Wirfs-Brock (15 years ago)

On Mar 21, 2011, at 1:13 PM, Erik Arvidsson wrote:

The above use case cannot be solved using private names because private names conflict with public names.

Erik, I'm not sure that my understanding of the intended semantics of your extension statement is totally correct. But given what I I think you intended here is how I might imagine it desugaring using private names. Let me know what I misinterpretered:

{ private filter; Object.prototype.filter = function(fun) { var publicFilter= this["filter"]; if (publicFilter) return publicFilter.apply(this,arguments); var retval = {}; for (var key in this) { if (fun(this[key]) retval[key] = this[key]; } return retval; }

function largerThanN(obj, n) { return obj.filter(function(item) { return item > n; } }

var a = [0, 1, 2, 3, 4]; print(largerThanN(a, 2)); var o = {a: 0, b: 1, c: 2, d: 3, e: 4}; print(largerThanN(0, 2));

delete Object.prototype.filter }

# Waldemar Horwat (15 years ago)

On 03/21/11 17:42, Allen Wirfs-Brock wrote:

On Mar 21, 2011, at 1:13 PM, Erik Arvidsson wrote:

The above use case cannot be solved using private names because private names conflict with public names.

Erik, I'm not sure that my understanding of the intended semantics of your extension statement is totally correct. But given what I I think you intended here is how I might imagine it desugaring using private names. Let me know what I misinterpretered:

{ private filter; Object.prototype.filter = function(fun) { var publicFilter= this["filter"]; if (publicFilter) return publicFilter.apply(this,arguments); var retval = {}; for (var key in this) { if (fun(this[key]) retval[key] = this[key]; } return retval; }

function largerThanN(obj, n) { return obj.filter(function(item) { return item > n; } }

var a = [0, 1, 2, 3, 4]; print(largerThanN(a, 2)); var o = {a: 0, b: 1, c: 2, d: 3, e: 4}; print(largerThanN(0, 2));

delete Object.prototype.filter }

You'd still run into all of the issues caused by "private filter" behaving like a C++ #define. For example:

var foo = {filter: 34}; then pass foo to an outside client.

 Waldemar
# Allen Wirfs-Brock (15 years ago)

On Mar 21, 2011, at 5:55 PM, Waldemar Horwat wrote:

You'd still run into all of the issues caused by "private filter" behaving like a C++ #define. For example:

var foo = {filter: 34}; then pass foo to an outside client.

Trying to interpret your comment. Are you saying that the above appearing within the scope of the "private filter" would unintentionally use private foo instead of public foo?

That's true, but the whole point of the block (and the extension declaration) was to constrain the visibility of private filter, so it could be used as an extension property name in a limited scope. Defining a property using that name within that scope seem like a pretty clueless error. Sure, it will happen, but I don't see how this use of lexically scoping is any more or less error prone an any other use.

# Peter Hallam (15 years ago)

Hey Allen,

The use case runs something like this:

Suppose you use a collection library which has 'filter' implemented on all of their collections. So we have Set.prototype.filter, Map.prototype.filter, Tree.prototype.filter, etc. But the library doesn't add Array.filter because monkey patching is bad.

In your own code, you use a mix of the library collections and Arrays, and you want to use filter on all of your collections.

So you want to say something like:

{ var buttons = ...; // some Set var listItems = ...; // some Array buttons.filter(...); // filter works on Set ... listItems.filter(...); /// ... and filter works on Arrays in the same scope. }

# Brendan Eich (15 years ago)

On Mar 21, 2011, at 6:17 PM, Allen Wirfs-Brock wrote:

On Mar 21, 2011, at 5:55 PM, Waldemar Horwat wrote:

You'd still run into all of the issues caused by "private filter" behaving like a C++ #define.

#define exagerrates.

For example:

var foo = {filter: 34}; then pass foo to an outside client.

Trying to interpret your comment. Are you saying that the above appearing within the scope of the "private filter" would unintentionally use private foo instead of public foo?

That's true, but the whole point of the block (and the extension declaration) was to constrain the visibility of private filter, so it could be used as an extension property name in a limited scope. Defining a property using that name within that scope seem like a pretty clueless error. Sure, it will happen, but I don't see how this use of lexically scoping is any more or less error prone an any other use.

You could be right. But there is a difference, it is not as extreme as #define, but it's real. It is what Andrew described so clearly: that any foo after any dot in the scope of private foo is bound to the private name.

With let, const, or function in block, the lexical scoping affects the meaning of unqualified identifiers as primary expressions. It doesn't affect the meaning of identifiers (IdentifierName in the grammar) on the right of '.' in MemberExpressions.

Does this matter? In Java it matters much less because of types. In JS, it matters more but an IDE could still help (as in Java, but without types -- just lexical use-to-def connection-making by the IDE).

I'm not saying this characteristic of private names is "bad" or "good". It is distinct from lexical binding in Harmony without private names, though.

# Brendan Eich (15 years ago)

On Mar 21, 2011, at 1:13 PM, Erik Arvidsson wrote:

{ extend Object.prototype with { filter: function(fun) { var retval = {}; for (var key in this) { if (fun(this[key]) retval[key] = this[key]; } return retval; } };

function largerThanN(obj, n) { return obj.filter(function(item) { return item > n; } }

var a = [0, 1, 2, 3, 4]; print(largerThanN(a, 2)); var o = {a: 0, b: 1, c: 2, d: 3, e: 4}; print(largerThanN(0, 2)); }

I like it!

Just a reaction in brief. More this week at the meeting, I'm sure ;-).

The above use case cannot be solved using private names because private names conflict with public names.

I don't see this, though. Don't oversell!

Can we agree that this is a use case that we care about and focus on this instead of whether private names can or cannot do this?

Yes.

# Allen Wirfs-Brock (15 years ago)

As a followup, Refinements are derivative of ClassBoxes which for several year's was an area of focus within Oscar Nierstrasz's research group at the University of Bern. scg.unibe.ch/research/classboxes They started with Smalltalk and then went on to Java and C# implementations. They have a considerable publication record on the subject scg.unibe.ch/scgbib?query=classbox

The presentation deck that places Ruby Refnements into to context with classboxes is www.slideshare.net/ShugoMaeda/rc2010

# François REMY (15 years ago)

From: Brendan Eich Sent: 21 March 2011

On Mar 21, 2011, at 1:13 PM, Erik Arvidsson wrote:

The above use case cannot be solved using private names because private names conflict with public names.

I don't see this, though. Don't oversell!

I think it's true: you can't do this with private name.

private filter; Object.filter = function defaultFilter() { ... }; [0, 1, 2].filter(...) // will triggers Object.filter and not Array.filter, because filter is not "filter" anymore but a private name instead.

If you use a "normal name" with a reduced visibility instead, the classical prototype chain will continue to works as expected.

, François

# Brendan Eich (15 years ago)

On Mar 21, 2011, at 11:50 PM, François REMY wrote:

From: Brendan Eich Sent: 21 March 2011

On Mar 21, 2011, at 1:13 PM, Erik Arvidsson wrote:

The above use case cannot be solved using private names because private names conflict with public names.

I don't see this, though. Don't oversell!

I think it's true: you can't do this with private name.

private filter; Object.filter = function defaultFilter() { ... }; [0, 1, 2].filter(...) // will triggers Object.filter and not Array.filter, because filter is not "filter" anymore but a private name instead.

Sure, but the problem (as people keep pointing out) has two solutions: close the scope in which filfter binds to a private name, or use a renaming private declaration.

It's not perfect but there's no absolute inability to call public filter and private filter.

If you use a "normal name" with a reduced visibility instead, the classical prototype chain will continue to works as expected.

Is that what Erik et al. are proposing? From discussions today it seems either private names with strong encapsulation, or soft fields (weak maps), could be used to implement extend O with {...}. The crucial test would be when O is frozen. If the subjective extension still works in scope of the module declaring the extension, then soft fields or an equivalent must be what underlies the syntax.

# David Herman (15 years ago)

I wish you would make your proposal more precise; right now we have to infer it from your single example. In my conversations with several others on the committee, I'm already seeing lots of confusion about the semantics of what you are describing here. Can you write this up as a strawman in more detail?

Specifically:

  • you haven't made clear whether the semantics does a dynamic property add and property delete at the beginning and end of the scope;

  • you haven't made clear what the property key that 'filter' refers to in the example actually is; and

  • you haven't made clear whether there's any way that external code transitively called during the lifetime of the block could access the 'filter' property.

I also have to call a foul when you claim that this can do something that private names can't but then declare it out of bounds for anyone else to discuss the validity of that claim.

Sorry if this is a little grumpy. It's just that we're all very busy and it would help prevent runaway threads if we cut down on noise due to avoidable confusion.

# François REMY (15 years ago)

Sure, but the problem (as people keep pointing out) has two solutions: close the scope in which filfter binds to a private name, or use a renaming private declaration. It's not perfect but there's no absolute inability to call public filter and private filter.

Adding "filter" to the Object property was done to have a fallback when the object you work on don't have a "filter" method built-in. In case the object you're working on has a "filter" method, it should be used. In this case, if you use private name, you can't have this behavior right (ie: write "myObj.filter()" and use the filter method implemented in myObj, if any, or the temporary extension to Object.prototype). It's because your intent was not to create a private name. It was to define a temporary extension to an object.

If you want this behavior right using private names, you'll need "if(myObj["filter"]) { myObj"filter" } else { myObj.filter(); }" or "(myObj["filter"]||myObj.filter).call(myObj)" which is really ugly.

# Kam Kasravi (15 years ago)

On Mar 21, 2011, at 9:40 PM, Brendan Eich <brendan at mozilla.com> wrote:

On Mar 21, 2011, at 6:17 PM, Allen Wirfs-Brock wrote:

On Mar 21, 2011, at 5:55 PM, Waldemar Horwat wrote:

You'd still run into all of the issues caused by "private filter" behaving like a C++ #define.

#define exagerrates.

For example:

var foo = {filter: 34}; then pass foo to an outside client.

Trying to interpret your comment. Are you saying that the above appearing within the scope of the "private filter" would unintentionally use private foo instead of public foo?

That's true, but the whole point of the block (and the extension declaration) was to constrain the visibility of private filter, so it could be used as an extension property name in a limited scope. Defining a property using that name within that scope seem like a pretty clueless error. Sure, it will happen, but I don't see how this use of lexically scoping is any more or less error prone an any other use.

You could be right. But there is a difference, it is not as extreme as #define, but it's real. It is what Andrew described so clearly: that any foo after any dot in the scope of private foo is bound to the private name.

With let, const, or function in block, the lexical scoping affects the meaning of unqualified identifiers as primary expressions. It doesn't affect the meaning of identifiers (IdentifierName in the grammar) on the right of '.' in MemberExpressions.

Does this matter? In Java it matters much less because of types. In JS, it matters more but an IDE could still help (as in Java, but without types -- just lexical use-to-def connection-making by the IDE).

There would be several private names per type though each one preventing any occurrence of that name within any identifier in any member expression at that scope. This would complicate things like minifiers. Not to mention trying to mix say svg, canvas and Dom within some framework since they all use common member names such as element, node, children, attributes, etc.

# Brendan Eich (15 years ago)

On Mar 22, 2011, at 12:29 AM, François REMY wrote:

Sure, but the problem (as people keep pointing out) has two solutions: close the scope in which filfter binds to a private name, or use a renaming private declaration. It's not perfect but there's no absolute inability to call public filter and private filter.

Adding "filter" to the Object property was done to have a fallback when the object you work on don't have a "filter" method built-in. In case the object you're working on has a "filter" method, it should be used. In this case, if you use private name, you can't have this behavior right (ie: write "myObj.filter()" and use the filter method implemented in myObj, if any, or the temporary extension to Object.prototype). It's because your intent was not to create a private name. It was to define a temporary extension to an object.

That's not what Erik's example shows. He simply wants to call the pre-existing "filter" (public name) method on an array in the same scope that he calls the extension "filter" (public or private, does it matter?) name added to Object.prototype somehow and temporarily.

If you want this behavior right using private names, you'll need "if(myObj["filter"]) { myObj"filter" } else { myObj.filter(); }" or "(myObj["filter"]||myObj.filter).call(myObj)" which is really ugly.

As noted, you can also rename private names or close scopes.

But now I do not understand the proposal in detail. How does the extension method get invoked? Dave's right, we need more details.

# Brendan Eich (15 years ago)

On Mar 22, 2011, at 1:28 AM, Kam Kasravi wrote:

Does this matter? In Java it matters much less because of types. In JS, it matters more but an IDE could still help (as in Java, but without types -- just lexical use-to-def connection-making by the IDE). There would be several private names per type though each one preventing any occurrence of that name within any identifier in any member expression at that scope.

Unless spelled with brackets and quotes, yes.

This would complicate things like minifiers.

It would, but life's rough for minifiers already. I'm not dismissing this but it's not make or break.

Not to mention trying to mix say svg, canvas and Dom within some framework since they all use common member names such as element, node, children, attributes, etc.

You would not use those names in private declarations.

Remember the private declaration is optional. You can always index.

# Allen Wirfs-Brock (15 years ago)

I think a more precise statement is you can't do this with private names alone.

As the classbox work explains, you also have to have a modified property lookup algorithm that takes into account that a [[Get]] may be satisfied based upon either the global or local key value associated with the name . This was reflected in by counter example where I explicitly delegated to the public named property if it exists:

private filter; Object.prototype.filter = function(fun) { var publicFilter= this["filter"]; if (publicFilter) return publicFilter.apply(this,arguments); ...

In practice I don't think you would actually do it that way, but it does show that two distinct property key values are in play.

# Erik Arvidsson (15 years ago)

On Tue, Mar 22, 2011 at 00:21, David Herman <dherman at mozilla.com> wrote:

I wish you would make your proposal more precise; right now we have to infer it from your single example. In my conversations with several others on the committee, I'm already seeing lots of confusion about the semantics of what you are describing here. Can you write this up as a strawman in more detail?

I will write up a strawman with more details eventually. I just wanted to approach the problem from the use case angle instead of from the primitive building blocks.

Specifically:

  • you haven't made clear whether the semantics does a dynamic property add and property delete at the beginning and end of the scope;

  • you haven't made clear what the property key that 'filter' refers to in the example actually is; and

  • you haven't made clear whether there's any way that external code transitively called during the lifetime of the block could access the 'filter' property.

I also have to call a foul when you claim that this can do something that private names can't but then declare it out of bounds for anyone else to discuss the validity of that claim.

Sorry, I didn't mean it that way. If private names can do this. Great. My issue with the private names discussion was that it was trying to fit a very powerful primitive to a very specific use case and we had a hard time seeing a good fit. Instead I wanted to approach this from the other direction.

# Erik Arvidsson (15 years ago)

On Tue, Mar 22, 2011 at 08:27, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:

I think a more precise statement is you can't do this with private names alone.

As the classbox work explains, you also have to have a modified property lookup algorithm that takes into account that a [[Get]] may be satisfied based upon either the global or local key value associated with the name .  This was reflected in by counter example where I explicitly delegated to the public named property if it exists:

I was not aware of the classbox paper unitl yesterday. It looks very promising. Instead this was based on C# extension methods as well as on ES4 namespaces to some degree.

Just like for the classbox paper, property lookup is now also dependent on lexical scope. To ensure that the property lookup does not get slowed down too much it is important that this can be determined statically. Also, we believe that this should be limited to module scope to reduce the overhead even more.

The actual extension properties would probably be stored as a side table (using something like soft fields) and it is important that these extensions can work on frozen objects so that one could extend frozen primordials.