Discuss
  • Feature-Request: allow to iterate over WeakSet / WeakMap

Feature-Request: allow to iterate over WeakSet / WeakMap

# Michael Kriegel (8 years ago)

not sure, whether this is the right place for discussing a feature request - and how to find out, whether this was already proposed/discussed before and with which result... My google search did not bring up anything about it.

On my opinion it should be made possible to iterate over all elements in a WeakSet & WeakMap. I already found many cases, in which I'd like to have used the neat feature of "garbage collect the object, if it is not referred to by others anymore", but with the necessity to know, which objects are in the set (not only, if a given object is). An example:

Registering callbacks to objects:

class Foo { onBla(Handler) { this.BlaSuperWeakMap.push(Handler); }

someOtherMethod() { for (let X in this.BlaSuperWeakMap) { X(); } }

constructor() { this.BlaSuperWeakMap = new SuperWeakMap(); } }

const MyFoo = new Foo(); { let MyHandler1 = function(){ console.log('MyHandler1 called'); }; MyFoo.onBla(MyHandler1); console.log('Invokation1:'); MyFoo.someOtherMethod(); } let MyHandler2 = function(){ console.log('MyHandler2 called'); }; MyFoo.onBla(MyHandler2); MyFoo.onBla(function(){ console.log('MyHandler3 called'); }); console.log('Invokation2:'); MyFoo.someOtherMethod(); MyHandler2 = undefined; console.log('Invokation3:'); MyFoo.someOtherMethod();

Expected result: Invokation1: MyHandler1 called Invokation2: MyHandler2 called Invokation3:

Basically the example is: Invoke the callback of all objects which still "exist" (in the meaning of having "non-weak" references towards them).

I know this may have some caveats. It would be necessary to check the reference count of the object in the SuperWeakSet, because the object may have no "non-weak" references left but still was not yet garbage collected. I do not know, whether current ECMAScript-Implementations and their garbage collecting solutions could handle this easily - basically it is the same decision, which the garbage collector would make: If an object can be disposed, because it has no non-Weak references pointing to it, the object is by definition not in the WeakSet anymore. And implementations could in that case even give a hint to the garbage collector, which may improve performance of applications, which use the SuperWeakSet alot.

Greetings, Michael

# Isiah Meadows (8 years ago)

Could you explain a real world use case for this? I recall some that it would've simplified some, but the best implementation often didn't even require iteration, but just changing the data layout altogether.

# Bradley Meck (8 years ago)

This thread from 2011 might help understand why they should not be enumerable (it links to another thread as well inside). esdiscuss/2011-April/013604

# Michael Kriegel (8 years ago)

Sure.

I am using an MVC pattern, where model changes are notified via events. Multiple views, lets call two of them view1 and view2, are capable of displaying an object X, which is in the model. Whenever X is modified (in the sense of modifying a property of it) an event is fired. Views, which display an aspect of X register themselves to the modified-event of X to be refreshed. However, views may be closed, so a view which was closed shall be unsubscribed from that event automatically. Without an iterable WeakSet, I have to unsubscribe the view manually on disposal. This is more error prone, as different views subscribe to different model change events, so each of my views needs kind of a destructor, which unregisters it from all such event handlers, which it registered to - if you forget one, you have a memory leak.

Actually ECMAScript should introduce WeakReference, which is sth. like a variable, which may lose its object, when no "non-Weak" reference to it exists anymore. Losing the value means it returns undefined again. example:

let a = {}; let weak b = a; console.log(typeof(b)); // prints object a = {}; console.log(typeof(b)); // prints undefined

Having WeakReferences and iterable WeakSet/WeakMap would help a lot in preventing memory leaks.

Of course, as stated before: it would be preferred, if the WeakReference / WeakSet / WeakMap is predictable in the sense, that their state does not depend on "whether an object has actually been garbage collected", but on "whether an object still has 'non-weak' references pointing to it, aka 'is garbage-collectable". Of course this involves checking garbage-collectability on access to the WeakReference / WeakSet / WeakMap.

# Michael Kriegel (8 years ago)

I see the points in that thread, but I am not sure, whether I completely understand them. I can imagine, that there is an issue, because gc is somewhat unpredictable. But shouldn't it be predictable, whether there are still "non-weak" references to an object? So my intention slightly differs from what WeakSet/WeakMap (and WeakReference would) do: whether an object is in the SuperWeakSet (or a key in the SuperWeakMap, or a SuperWeakReference still points to the object) does NOT depend on, whether the object has actually been gc'ed, but whether the object does have "non-weak" references to it.

I am not sure, which kinds of garbage collectors the implementations of ECMAScript typically use. If they used a reference counter, which counts "non-weak" references only, a simple check, whether the reference count is 0 or not on access to a weak reference would be sufficient.

Of course, when I iterate through a WeakSet (respectively the keys of a WeakMap), in the beginning of the iteration, a keys-array needs to be retrieved, which is an array of all objects in the weak set, which have reference count > 0 at that point in time. As this array then holds hard

references to all key objects, the key array cannot be changed by gc during iteration.

# Boris Zbarsky (8 years ago)

On 7/13/16 10:18 AM, Michael Kriegel wrote:

But shouldn't it be predictable, whether there are still "non-weak" references to an object?

Maybe. The GC may include a machine stack scanner to root things referenced on the stack, which can give false positives depending on what integers happen to be hanging out on the stack.

does NOT depend on, whether the object has actually been gc'ed, but whether the object does have "non-weak" references to it.

Determining this basically requires running a full gc, no?

I am not sure, which kinds of garbage collectors the implementations of ECMAScript typically use. If they used a reference counter

They don't.

# Michael Kriegel (8 years ago)

On 13.07.2016 16:27, Boris Zbarsky wrote:

On 7/13/16 10:18 AM, Michael Kriegel wrote:

But shouldn't it be predictable, whether there are still "non-weak" references to an object?

Maybe. The GC may include a machine stack scanner to root things referenced on the stack, which can give false positives depending on what integers happen to be hanging out on the stack.

So what does gc do to determine, whether an object still has hard references pointing towards it?

does NOT depend on, whether the object has actually been gc'ed, but whether the object does have "non-weak" references to it.

Determining this basically requires running a full gc, no?

Depends on how the gc works. If "deleting the last reference to an object" leads to an immediate deletion of all references to other objects held by this object, then: no.

I am not sure, which kinds of garbage collectors the implementations of ECMAScript typically use. If they used a reference counter

They don't.

Any resources on what they do? E.g. chrome V8 or Firefox?

# Boris Zbarsky (8 years ago)

On 7/13/16 10:59 AM, Michael Kriegel wrote:

So what does gc do to determine, whether an object still has hard references pointing towards it?

The GC may be conservative, not be precise. That is, it may not collect some stuff that actually could be collected because the GC thinks there are references to it, while there actually aren't any. The stuff may then end up being collected on a later collection, again depending on the contents of things like the machine stack and machine registers.

Depends on how the gc works. If "deleting the last reference to an object" leads to an immediate deletion of all references to other objects held by this object, then: no.

I'm not aware of any JS GC that uses reference counting. It could be done, together with a cycle collector, but you'd really really need the cycle collector. And once you have a cycle collector, you need to run it to find out whether things really are unreachable; just looking at reference counts is no longer enough.

Any resources on what they do? E.g. chrome V8 or Firefox?

SpiderMonkey (the JS engine in Firefox) uses a tracing GC. In terms of the "implementation strategies" described at en.wikipedia.org/wiki/Tracing_garbage_collection it's a moving, mark-and-sweep, generational, incremental, precise collector.

# Allen Wirfs-Brock (8 years ago)

On Jul 13, 2016, at 7:18 AM, Michael Kriegel <michael.kriegel at actifsource.com <mailto:michael.kriegel at actifsource.com>> wrote:

I am not sure, which kinds of garbage collectors the implementations of ECMAScript typically use. If they used a reference counter, which counts "non-weak" references only, a simple check, whether the reference count is 0 or not on access to a weak reference would be sufficient.

Reference counting is not “garbage collection” because reference counting can not recover circular chains of object references.

Modern garbage collector are complex heuristic-based beasts that try to make good trade-offs between latency (how long does an object continue to consume storage after it actually becomes unreferenced) and performance (how much of the total processing cycles is consume trying to recover unreferenced objects). Each implementation’s GC uses different heuristics to optimize that trade-off. Thus when any given object would will be observably GC'ed is non-deterministic between implementations (and often even within a particular implementation).

If you want to drill deep into how garbage collectors work, there are lots of good resources at www.memorymanagement.org/index.html, www.memorymanagement.org/index.html

# Michael Kriegel (8 years ago)

Wouldn't it be preferable to discuss, whether the requested feature makes sense or not, before looking at whether it is possible to easily implement it in the current garbage collectors on the market?

On 13.07.2016 18:11, Allen Wirfs-Brock wrote:

On Jul 13, 2016, at 7:18 AM, Michael Kriegel <michael.kriegel at actifsource.com <mailto:michael.kriegel at actifsource.com>> wrote:

I am not sure, which kinds of garbage collectors the implementations of ECMAScript typically use. If they used a reference counter, which counts "non-weak" references only, a simple check, whether the reference count is 0 or not on access to a weak reference would be sufficient.

Reference counting is not “garbage collection” because reference counting can not recover circular chains of object references.

Of course not. However, reference counting can recover from circular chains. Example:

imagine, deleting a hard reference internally invokes deleteReference with following implementation:

deleteReferenceTo(Obj) { ReferenceCount[Obj]--; if (ReferenceCount[Obj] === 0) { for (Key in Object.getOwnProperties(Obj)) { delete Obj[Key]; } } }

delete Obj[Key] will call deleteReference. Once this delete call returns, there may be no further reference to the object Obj[Key] referred to. Then delete recursion occurs for that object. If there is a circle of objects without any further hard references to, the recursion will end, as the properties of the objects are deleted immediately. But as the object has no references to it anyway, deleting its properties is not an issue.

Modern garbage collector are complex heuristic-based beasts that try to make good trade-offs between latency (how long does an object continue to consume storage after it actually becomes unreferenced) and performance (how much of the total processing cycles is consume trying to recover unreferenced objects). Each implementation’s GC uses different heuristics to optimize that trade-off. Thus when any given object would will be observably GC'ed is non-deterministic between implementations (and often even within a particular implementation).

Sure and that is good as it is. Weak references as I'd like to see them, would depend on the information, whether there is still a hard reference to the object - not whether it has actually been gc'ed. However, I'll have to take a deeper look into thegc topic to find out, whether there is no way to find that out - still please note the first question in this message: does the requested feature make sense?

On my opinion: yes, because when the user can declare weak references explicitely, he can rely on that an object will be gc'ed even though he still has a weak reference to it. Of course he has to assure, that the resource still "exists":

let x = {}; let weak y = x

function someFunctionWhichUsesY() { if (y) y['a']++; // maybe: else throw ... }

function someOtherFunctionWhichDeletesTheOnlyHardReferenceToX() { x = {'a':0}; // user may not have noticed, that y is still pointing to the original object - but as y is weak, it will be undefined from now on -> no memory leak. }

If the user forgets to check for y's existence and someOtherFunctionWhichDeletesTheOnlyHardReferenceToX has been called before, an exception, but that is much better than having a memory leak (if you ask me). Again: On my opinion a weak reference should return undefined as soon as there is no hard reference to the object anymore, not as soon as the object becomes gc'ed.

# Allen Wirfs-Brock (8 years ago)

On Jul 13, 2016, at 9:45 AM, Michael Kriegel <michael.kriegel at actifsource.com <mailto:michael.kriegel at actifsource.com>> wrote:

Wouldn't it be preferable to discuss, whether the requested feature makes sense or not, before looking at whether it is possible to easily implement it in the current garbage collectors on the market?

But that discussion has already taken place within TC39 (and on es-discuss) during the development of WeakMap/WeakSet.

Also, this isn’t about markets, it’s about technical feasibility. Nobody knows how to implement a 0 latency, totally deterministic, low overhead GC. It’s likely that such a thing isn't even possible given current processor/memory/software architectures.

On 13.07.2016 18:11, Allen Wirfs-Brock wrote:

On Jul 13, 2016, at 7:18 AM, Michael Kriegel <michael.kriegel at actifsource.com <mailto:michael.kriegel at actifsource.com>> wrote:

I am not sure, which kinds of garbage collectors the implementations of ECMAScript typically use. If they used a reference counter, which counts "non-weak" references only, a simple check, whether the reference count is 0 or not on access to a weak reference would be sufficient.

Reference counting is not “garbage collection” because reference counting can not recover circular chains of object references.

Of course not. However, reference counting can recover from circular chains. Example:

That statement simply demonstrates that you aren’t well schooled in memory management technology. If you want to be taken seriously, please educate yourself.

imagine, deleting a hard reference internally invokes deleteReference with following implementation:

deleteReferenceTo(Obj) { ReferenceCount[Obj]--; if (ReferenceCount[Obj] === 0) { for (Key in Object.getOwnProperties(Obj)) { delete Obj[Key]; } } }

delete Obj[Key] will call deleteReference. Once this delete call returns, there may be no further reference to the object Obj[Key] referred to. Then delete recursion occurs for that object. If there is a circle of objects without any further hard references to, the recursion will end, as the properties of the objects are deleted immediately. But as the object has no references to it anyway, deleting its properties is not an issue.

Modern garbage collector are complex heuristic-based beasts that try to make good trade-offs between latency (how long does an object continue to consume storage after it actually becomes unreferenced) and performance (how much of the total processing cycles is consume trying to recover unreferenced objects). Each implementation’s GC uses different heuristics to optimize that trade-off. Thus when any given object would will be observably GC'ed is non-deterministic between implementations (and often even within a particular implementation).

consider:

var ref = {x: null}; //ref count of object is 1 ref.x=ref; //ref count of object is 2; ref = null; //after internally calling deleteReferenceTo, ref count of object is 1. // the object is now unreachable, but has a non-zero reference count. It has not been deallocated

# Michał Wadas (8 years ago)

What you think you need: iterable WeakMap What you really need: WeakReference, PhantomReference.

Personally I think we need built-in class "GarbageCollectorObserver" that emit events for each observed collected object (uniquely identified by strongly-held symbols), it will solve most of problems associated with binding models and views.

# Isiah Meadows (8 years ago)

You could also solve the iterability this way with weak references:

function *iter(wm) {
  for (const ref of wm._refs) {
    const key = ref.get()
    yield [key, wm.get(key)]
  }
}

function remove({wm, ref}) {
  wm._refs.remove(ref)
}

class IterableWeakMap extends WeakMap {
  constructor(list) {
    super()
    this._refs = new Set()
    for (const e of list) this.set(e)
  }

  set(key, value) {
    super.set(key, value)
    const arg = {wm: this, ref: undefined}
    arg.ref = makeWeakRef(e, remove, arg)
    this._refs.add(arg.ref)
    return this
  }

  forEach(callback, thisArg = undefined) {
    for (const [key, value] of iter(this)) {
      f.call(thisArg, value, key)
    }
  }

  [Symbol.iterator]() { return iter(this) }
  entries() { return iter(this) }

  keys() {
    for (const [key] of iter(this)) yield key
  }

  values() {
    for (const [, value] of iter(this)) yield value
  }

  clear() {
    for (const ref of this._refs) {
      this.delete(ref.get())
    }
    this._refs.clear()
  }
}