Discuss
  • First crack at a Streams proposal

First crack at a Streams proposal

# Tab Atkins Jr. (12 years ago)

Inspired by DOM adding Futures dom.spec.whatwg.org/#futures,

I've begun reworking several APIs in terms of them. While doing so, I've found several spots where Futures aren't appropriate, because I need something that can be "updated" multiple times, rather than just representing a single result like Futures do. I believe formalizing this notion will be as valuable as Futures, so I began by sketching out an API on my blog: www.xanthir.com/b4PV0.

I'd like to begin discussion of this API with es-discuss, because this feature is much larger than just DOM, and you people have a lot of relevant experience.

Here's a reproduction of the current state of my proposal (I'll update my blog as things progress if necessary):

callback StreamInit = void (StreamResolver resolver);
callback AnyCallback = any (optional any value);
typedef (Stream or Future or Iterable) StreamLike;

[Constructor(StreamInit init)]
interface Stream {
  Stream listen(optional AnyCallback? listenCB = null, optional
AnyCallback? completeCB = null, optional AnyCallback? rejectCB =
null);
  Future complete(optional AnyCallback? completeCB = null);
  Future catch(optional AnyCallback? rejectCB = null);
  Future next(optional any filter = null)

  static from(StreamLike value, optional transformerCB, optional thisArg);
  static of(any... values);
  static reject(optional any reason);

  Stream filter(filterCB or primitive);
  Stream map(mapCB);
  void   forEach(mapCB);
  Stream then(optional thenCB);
  Stream switch(optional switchCB);
  Stream throttle(milliseconds);

}

interface StreamResolver {
  void push(optional any value);
  void complete(optional any value);
  void continueWith(StreamLike continuation);
  void reject(optional any value);
};

The overall API was designed to be similar to DOM Futures, where the constructor returns the stream and passes the resolver object to a callback argument. In several aspects it was also designed to be similar to the Array API, because streams can usefully be treated as a push-based ordered collection.

The general form of a stream is zero or more updates, optionally followed by a single complete or reject event. Streams might not ever complete, and they may complete/reject without pushing any updates.

The resolver has a simple API. 'push' is used to put values into the stream. complete/reject fullfill the stream with a completion or rejection signal; these both kill the resolver so that none of the methods have any effect (maybe they throw?). 'continueWith' also kills the resolver, but doesn't fulfill the stream, instead delegating the stream's future progress to the continuation stream, so the stream pushes updates when the continuation does, and fulfills when the continuation does.

(I'm currently unsure whether completion should be just like an update, or not. That is, if I know that I'm about to push the very last value, should I do a push() then a complete(), or just a complete() with the value? This affects several APIs in the completion case.)

The Stream's own API is also simple, and very similar to that of Future. The basic API is the 'listen' function, which takes any of three callbacks, which are called when updates are pushed, the stream is completed, or the stream is rejected. It returns the same Stream, for chaining.

For convenience, Streams also have 'complete' and 'reject' functions which take just that particular callback. They return a Future which resolves when the stream is fulfilled - the 'complete' future accepts when the stream completes and rejects when the stream rejects, while the 'reject' future does the opposite.

The 'next' function returns a future for the next stream value. If the stream updates or completes, the future accepts with the same value; if it rejects, it rejects with the same value. (Alternately, should it reject if the stream completes?) Optionally, it can take a filter argument, which is either a JS primitive or a filter callback. If a primitive, it only completes for the next update that is === to the primitive; if a callback, it completes for the next update that the filter function returns true for.

Streams also have three convenience constructors: Stream.of() takes zero or more values, and produces a stream that pushes those values in order then completes (if passed a single value, this is the monadic lift operator); Stream.reject() takes a rejection reason and produces a rejected stream; Stream.from() takes a stream-like and turns it into a stream. Just like Array.from, it takes an optional converter callback.

(A stream is a stream-like, in which case Stream.from just creates a fresh stream that pushes the same values. A future is a stream-like, which either pushes a single value and immediately completes, or pushes no values and rejects, whenever the future accepts/rejects. An iterable is a stream-like, which immediately pushes all the values from the iterable into the stream.)

Finally, I have a start on a stream combinator library.

filter, map, and forEach do the same thing as they do on Array, just push-based rather than pull-based. ('filter' takes the same "primitive or callback" parameter that 'next' does.)

'then' is the monadic operator - it takes a stream of stream-likes, and produces a new stream which interleaves their updates. The optional callback is a transformer callback; "strm.then(cb)" is identical to "strm.map(cb).then()".

'switch' is another combiner - it also takes a stream of stream-likes and "interleaves" them, but stops listening to the updates from "earlier" streams as soon as "later" streams push their first update. A little hard to explain, but an example suffices - say you have a stream of text input values that's updated as the user types. Using a future-based XHR API, you send each value to your server for autocomplete suggestions. These futures may complete in any order, and once one completes, you no longer care about any pending futures from earlier values (and in fact, listening to them will screw up your UI, with suggestions showing up from old values). This is easily achieved with switch: "watchTextInput(input).switch(t=>getJSON(url,t)).forEach(updateUI)".

'throttle' is a filterer, which can either be idle or throttling. When idle, if it receives an update from its input stream, it enters throttling mode for the specified duration. While throttling, it remembers the value of the most recent update from its input stream, but doesn't update its output stream. At the end of the throttle period, it pushes the most recent update. This thins out a stream that may receive updates very quickly, so you don't end up doing expensive processing on lots of values that are already obsolete.

So! This was a long post, but hopefully it explains my proposal well. As far as I can tell, this Stream API is close to C#'s IObservable interface, which seemed powerful, easy-to-use, and very general when I was reading up on things. It also stays close to the existing Futures API, which it's intended to be a cousin of. It's also properly monadic, using the of/then pair used by Futures and proposed by Brian McKenna at brianmckenna.org/blog/category_theory_promisesaplus.

This API is not designed to directly handle IO streams, like Node's ReadableStream class nodejs.org/api/stream.html#stream_class_stream_readable.

However, such a thing should be implementable using streams, either as a subclass or something containing a stream.

Thoughts? Suggestions? Reasons I'm doing everything wrong?

# Nuno Job (12 years ago)

Node.js has multiple iterations on Streams, and those were made in the hope to make them better. It's hard and has been an ongoing effort to make it perfect.

That said it's quite surprising to see smart folks come to such different conclusions. Perhaps it's the lack of real life implementations and how these things affect real life programs and companies that use javascript. Almost all these concepts would have no chance to make it to node core. And some of these concepts are fairly advanced, when in node it took years to get get the basics right (the rest was left to npm and developers)

Nuno

# Andrea Giammarchi (12 years ago)

It's a nice effort but I agree with Nuno that fragmentation even on these things already in production (despite the versioning) out there and already working practically for the real-world isn't any good for the next future of the JS community.

Also, JS has been kept general purpose enough, I would not try to influence the language too much after DOM and W3C APIs and I thought this was also same idea of TC39.

Best

# Tab Atkins Jr. (12 years ago)

On Mon, Apr 15, 2013 at 1:51 PM, Andrea Giammarchi <andrea.giammarchi at gmail.com> wrote:

It's a nice effort but I agree with Nuno that fragmentation even on these things already in production (despite the versioning) out there and already working practically for the real-world isn't any good for the next future of the JS community.

Fragmentation of concept is exactly what standardization helps to solve - the role of standards is to, well, standardize, especially after the market has explored the problem space already. Futures were long-overdue when Anne finally specced them, having been around in lots of languages including JS in many forms, and I think Streams are in a similar boat.

Also, JS has been kept general purpose enough, I would not try to influence the language too much after DOM and W3C APIs and I thought this was also same idea of TC39.

I don't understand this sentence. It sounds like you're trying to say that Streams aren't general-purpose enough for JS to standardize, and should be left to DOM/W3C to do. Is this correct?

If so, I disagree, and think that several other people on es-discuss do as well. (For example, Alex, Yedua, and Dave have been responding to me about this proposal in Twitter.) In general, a Stream is just another container type, and is as general as an Array or Set. Plenty of specific instances or subclasses of Streams are appropriate for individual DOM/etc specs to define, but we need to agree on the general concept first, and es-discuss folks seem well-suited to help develop this.

~TJ

# Andrea Giammarchi (12 years ago)

I've implemented Alex Future proposal in JS before he did as prototype and for tests.

However, Futures aren't used that much in node.js and spec'd like that are nice.

However, I would not compare those two things.

If you propose Streams after Futures where Futures are less common in node but Streams are one of the major things I think you should consider more the current used node approach.

I don't see Future and Stream as different containers of the same thing with identical functionality, sorry.

This is all I am saying.

# Andrea Giammarchi (12 years ago)

of course, that is just my opinion. I am looking forward to read others ... meanwhile, in texas JS, a talk about node streams: 2013.texasjavascript.com/stream

# Nuno Job (12 years ago)

If fragmentation is bad why propose something that was never experimented and diverged completely from what we learned with node?

Would love to understand the technical merits on why this is better than node streams and why streams need to be in JS. I'm sure you have a reasoning, i for once would like to know what it is. It's likely that others do too.

Nuno

ps. would appreciate keeping the discussion civilized and fallacy free (eg. name dropping)

# Domenic Denicola (12 years ago)

I think a large part of the confusion here is that when Tab says "streams" he really means "observables" or "signals." As you point out, this would be completely inappropriate as a stream API, as it does indeed ignore many of the lessons Node learned. Surprisingly, it doesn't seem to draw on any of the existing observable or signal APIs that I've seen either, instead being some entirely new hybrid of promises and the nascent monads-in-JS movement.

# Andrea Giammarchi (12 years ago)

well, if the purpose is different and you need to describe streams as "observables" or "signals" then I would suggest to change naming convention so nobody will ever be confused again.

# Tab Atkins Jr. (12 years ago)

On Mon, Apr 15, 2013 at 2:36 PM, Domenic Denicola <domenic at domenicdenicola.com> wrote:

I think a large part of the confusion here is that when Tab says "streams" he really means "observables" or "signals." As you point out, this would be completely inappropriate as a stream API, as it does indeed ignore many of the lessons Node learned. Surprisingly, it doesn't seem to draw on any of the existing observable or signal APIs that I've seen either, instead being some entirely new hybrid of promises and the nascent monads-in-JS movement.

Not quite - I've seen things like what I describe called "Streams" before. The recent things I've been looking at while writing this up don't call it that, but that doesn't mean the term is unused. ^_^ But I'm not wedded to any particular term. I find "Observable" to be not a great term, but whatever.

I do somewhat resent the implication that I invented something wholly new without drawing on any existing APIs. I looked at the syntaxes of several similar APIs in other languages while designing this, and took lessons from them. Of course, I optimized it for JS. I did purposely cleave close to Futures, because Futures and Streams are very similar and will be used for very similar things in DOM, so it makes sense to make them have similar APIs, and I think the Futures API works well. It was also definitely influenced by some of the recent JS monads talk, but the two bits influenced by it are also very natural - Stream.of is identical to Array.of (as a simplified "just put this in the damn container" constructor), and Stream#then is one of the basic stream combinators in other languages (I just chose the name to match the JS-monads suggestion of using of/then as the monadic op names, which I think are quite reasonable).

# Domenic Denicola (12 years ago)

From: Tab Atkins Jr. [mailto:jackalmage at gmail.com]

I do somewhat resent the implication that I invented something wholly new without drawing on any existing APIs. I looked at the syntaxes of several similar APIs in other languages while designing this, and took lessons from them.

In that case I certainly apologize for my ignorance; I guess I'd never seen those existing APIs and thus, foolishly, assumed they were just your invention. Sorry about that.

It would probably be helpful to all involved to get pointers to these APIs, since for a task like this we'll need as much inspiration as possible. Currently Dart's streams and Node.js streams are what we are mostly looking at, so more sources are definitely welcome, especially if they have novel ideas like this but proven in the field of other languages.

# Tab Atkins Jr. (12 years ago)

On Mon, Apr 15, 2013 at 3:40 PM, Domenic Denicola <domenic at domenicdenicola.com> wrote:

From: Tab Atkins Jr. [mailto:jackalmage at gmail.com]

I do somewhat resent the implication that I invented something wholly new without drawing on any existing APIs. I looked at the syntaxes of several similar APIs in other languages while designing this, and took lessons from them.

In that case I certainly apologize for my ignorance; I guess I'd never seen those existing APIs and thus, foolishly, assumed they were just your invention. Sorry about that.

It would probably be helpful to all involved to get pointers to these APIs, since for a task like this we'll need as much inspiration as possible. Currently Dart's streams and Node.js streams are what we are mostly looking at, so more sources are definitely welcome, especially if they have novel ideas like this but proven in the field of other languages.

For example, examples and linked articles in queue.acm.org/detail.cfm?id=2169076, and several articles

discussing LINQ and IObservable in C#, and various flavors of Task in C# and other languages. (I happen to think C# did a pretty good job here. I'd like to look into the Haskell APIs for this kind of thing, but haven't done so so far.)

Dart's broadcast streams are extremely similar to my proposal, with just some naming differences. My crack at the API is designed for the DOM applications I have in mind, so I didn't consider single-listener cases, but that's probably easy enough to integrate (if you know you're a single-listener stream, you probably want to buffer updates until someone actually starts listening).

Node's streams are something quite different, though conceptually related - they're heavily specialized to binary IO. My proposal is designed more to handle reactive programming-style things, or first-class event streams from the DOM. (DOM Events are great for the tree-based abilities when you have a tree, but we currently use them for a lot of non-tree things, where a more specialized event stream API would be better and more powerful. IO streams might be integratable into this proposal, but I think we'll more likely end up with something that has a similar-shaped API but is a fundamentally separate thing.

# Jake Verbaten (12 years ago)

Would it help to split the sugar and combinator from the actual stream interface?

callback StreamInit = void (StreamResolver resolver);
callback AnyCallback = any (optional any value);

[Constructor(StreamInit init)]
interface Stream {
  Stream listen(optional AnyCallback? listenCB = null, optional
AnyCallback? completeCB = null, optional AnyCallback? rejectCB =
null);
}

interface StreamResolver {
  void push(optional any value);
  void complete(optional any value);
  void reject(optional any value);
};

One basic thing missing from this API is the ability to stop listening to a stream.

Other things missing are the ability to abort / cancel / close a Stream, the ability to pause or resume a stream.

It should also be made clear how and when a stream may emit values. Whether it is at any arbitrary time and it will send to whomever is listening at that moment. Whether it is only allowed to emit values after a listen call. Does every call to listen get the entire history independently of other calls? (Doing so would buffer all data and defeat the point of a stream). What happens when you push a value into the resolved and no-one is listening?

Another decision that needs to be made is whether it makes sense for a stream to emit multiple errors? For a Future it doesn't make sense because it can only be fulfilled to a single value. For a stream it may make sense for multiple errors to occur.

I personally like the listen(onChunk, onEnd) syntax as it matches a popular stream baseclass from the node community ( dominictarr/through#through ) in terms of simplicity.

# Tab Atkins Jr. (12 years ago)

On Mon, Apr 15, 2013 at 4:24 PM, Jake Verbaten <raynos2 at gmail.com> wrote:

Would it help to split the sugar and combinator from the actual stream interface?

callback StreamInit = void (StreamResolver resolver);
callback AnyCallback = any (optional any value);

[Constructor(StreamInit init)]
interface Stream {
  Stream listen(optional AnyCallback? listenCB = null, optional
AnyCallback? completeCB = null, optional AnyCallback? rejectCB =
null);
}

interface StreamResolver {
  void push(optional any value);
  void complete(optional any value);
  void reject(optional any value);
};

Thanks, I should have done that originally!

One basic thing missing from this API is the ability to stop listening to a stream.

Hm, yes, you need to be able to unlisten. I'd forgotten about that.

Other things missing are the ability to abort / cancel / close a Stream, the ability to pause or resume a stream.

I think all of these are only suited for single-consumer streams. My original proposal is explicitly for multi-consumer streams, as I designed it explicitly for DOM use-cases that I ran into.

If you have a single-consumer stream, though, I agree that you should be able to pause and cancel a stream. Same with Futures, for that matter.

It should also be made clear how and when a stream may emit values. Whether it is at any arbitrary time and it will send to whomever is listening at that moment. Whether it is only allowed to emit values after a listen call. Does every call to listen get the entire history independently of other calls? (Doing so would buffer all data and defeat the point of a stream). What happens when you push a value into the resolved and no-one is listening?

By virtue of not defining this, I implicitly answered your questions. ^_^ It's a purely async data source, and cares not for whether you're listening. (Again, the use-cases I was trying to solve are DOM-oriented.) If you're not listening when one comes through, too bad. Of course, recall that updates are processed async, so you can't miss any updates within any particular tick.

Another decision that needs to be made is whether it makes sense for a stream to emit multiple errors? For a Future it doesn't make sense because it can only be fulfilled to a single value. For a stream it may make sense for multiple errors to occur.

Good question. My design assumes that once something errors, it's because it's now invalid, and won't be producing any more. If you can emit errors and still keep updating, they probably shouldn't reject the stream, but just be a special update value.

I personally like the listen(onChunk, onEnd) syntax as it matches a popular stream baseclass from the node community ( dominictarr/through#through ) in terms of simplicity.

That's my syntax, except that I allow you to distinguish between the stream ending "normally" and with errors. If you ignore the reject callback, my listen() is exactly that syntax already.

# Sam Tobin-Hochstadt (12 years ago)

On Mon, Apr 15, 2013 at 7:35 PM, Tab Atkins Jr. <jackalmage at gmail.com> wrote:

It should also be made clear how and when a stream may emit values. Whether it is at any arbitrary time and it will send to whomever is listening at that moment. Whether it is only allowed to emit values after a listen call. Does every call to listen get the entire history independently of other calls? (Doing so would buffer all data and defeat the point of a stream). What happens when you push a value into the resolved and no-one is listening?

By virtue of not defining this, I implicitly answered your questions.

This really isn't right. Jake's questions were the first questions I had about your design. It defines a lot of the methods, but it leaves out the most important part of the design, which is the semantics. Can you please try to make this precise sooner rather than later in this discussion?

# Tab Atkins Jr. (12 years ago)

On Mon, Apr 15, 2013 at 4:43 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu> wrote:

On Mon, Apr 15, 2013 at 7:35 PM, Tab Atkins Jr. <jackalmage at gmail.com> wrote:

It should also be made clear how and when a stream may emit values. Whether it is at any arbitrary time and it will send to whomever is listening at that moment. Whether it is only allowed to emit values after a listen call. Does every call to listen get the entire history independently of other calls? (Doing so would buffer all data and defeat the point of a stream). What happens when you push a value into the resolved and no-one is listening?

By virtue of not defining this, I implicitly answered your questions.

This really isn't right. Jake's questions were the first questions I had about your design. It defines a lot of the methods, but it leaves out the most important part of the design, which is the semantics. Can you please try to make this precise sooner rather than later in this discussion?

Please note the fact that the quoted sentence was followed by a smilie, and an actual answer to his question. ^_^

I'm clarifying my blog post as we speak with some of the feedback from this thread.

# Jake Verbaten (12 years ago)

I think all of these are only suited for single-consumer streams. My original proposal is explicitly for multi-consumer streams, as I designed it explicitly for DOM use-cases that I ran into.

Most streams I've used have been single consumer streams. Whether I write FRP style programs with Signals or whether I write IO programs with streams the multi consumer use case comes up less.

A far more regular case I have is a single "sink" consuming from multiple inputs in parallel due to some kind of merge(manyStreams) => stream function.

The most common use-case for multiple consumers is a second consumer for printing / inspection purposes, but that is trivial to model as a map function that logs the value and returns it.

You may want to ask other's who have used Stream like abstractions how common the multiple consumer case is.

On Mon, Apr 15, 2013 at 4:35 PM, Tab Atkins Jr. <jackalmage at gmail.com>wrote:

On Mon, Apr 15, 2013 at 4:24 PM, Jake Verbaten <raynos2 at gmail.com> wrote:

Would it help to split the sugar and combinator from the actual stream interface?

callback StreamInit = void (StreamResolver resolver);
callback AnyCallback = any (optional any value);

[Constructor(StreamInit init)]
interface Stream {
  Stream listen(optional AnyCallback? listenCB = null, optional
AnyCallback? completeCB = null, optional AnyCallback? rejectCB =
null);
}

interface StreamResolver {
  void push(optional any value);
  void complete(optional any value);
  void reject(optional any value);
};

Thanks, I should have done that originally!

One basic thing missing from this API is the ability to stop listening to a stream.

Hm, yes, you need to be able to unlisten. I'd forgotten about that.

Other things missing are the ability to abort / cancel / close a Stream, the ability to pause or resume a stream.

I think all of these are only suited for single-consumer streams. My original proposal is explicitly for multi-consumer streams, as I designed it explicitly for DOM use-cases that I ran into.

If you have a single-consumer stream, though, I agree that you should be able to pause and cancel a stream. Same with Futures, for that matter.

It should also be made clear how and when a stream may emit values. Whether it is at any arbitrary time and it will send to whomever is listening at that moment. Whether it is only allowed to emit values after a listen call. Does every call to listen get the entire history independently of other calls? (Doing so would buffer all data and defeat the point of a stream). What happens when you push a value into the resolved and no-one is listening?

By virtue of not defining this, I implicitly answered your questions. ^_^ It's a purely async data source, and cares not for whether you're listening. (Again, the use-cases I was trying to solve are DOM-oriented.) If you're not listening when one comes through, too bad. Of course, recall that updates are processed async, so you can't miss any updates within any particular tick.

Another decision that needs to be made is whether it makes sense for a stream to emit multiple errors? For a Future it doesn't make sense because it can only be fulfilled to a single value. For a stream it may make sense for multiple errors to occur.

Good question. My design assumes that once something errors, it's because it's now invalid, and won't be producing any more. If you can emit errors and still keep updating, they probably shouldn't reject the stream, but just be a special update value.

I personally like the listen(onChunk, onEnd) syntax as it matches a popular stream baseclass from the node community ( dominictarr/through#through ) in terms of simplicity.

That's my syntax, except that I allow you to distinguish between the stream ending "normally" and with errors. If you ignore the reject callback, my listen() is exactly that syntax already.

~TJ

-------------- next part -------------- An HTML attachment was scrubbed... URL: esdiscuss/attachments/20130415/8a2d9427/attachment

# Kevin Gadd (12 years ago)

If this is really about multiple-consumer streams, the semantics of this proposed API are incredibly murky to me. What happens if each consumer calls next()? Do they all get the same value out of their Future when it's completed? Do they each randomly get one of the values pushed into the stream? Is the stream implicitly required to buffer data in order to be able to offer it at a slower rate to one consumer than it is offered to the other consumers? Does each consumer have a different 'view' of the state of the stream (i.e. its cancelled/ended state, when those concepts apply)?

# Tab Atkins Jr. (12 years ago)

On Mon, Apr 15, 2013 at 5:03 PM, Jake Verbaten <raynos2 at gmail.com> wrote:

I think all of these are only suited for single-consumer streams. My original proposal is explicitly for multi-consumer streams, as I designed it explicitly for DOM use-cases that I ran into.

Most streams I've used have been single consumer streams. Whether I write FRP style programs with Signals or whether I write IO programs with streams the multi consumer use case comes up less.

A far more regular case I have is a single "sink" consuming from multiple inputs in parallel due to some kind of merge(manyStreams) => stream function.

That's the Stream#then method. Just make a stream of stream-likes, and call .then() to flatten them into a single output stream.

The most common use-case for multiple consumers is a second consumer for printing / inspection purposes, but that is trivial to model as a map function that logs the value and returns it.

You may want to ask other's who have used Stream like abstractions how common the multiple consumer case is.

In the DOM, multi-consumer is the default case, for the same reason you can register multiple listeners for any event. Futures were designed to be multi-consumer for the same reason.

# Tab Atkins Jr. (12 years ago)

On Mon, Apr 15, 2013 at 5:06 PM, Kevin Gadd <kevin.gadd at gmail.com> wrote:

If this is really about multiple-consumer streams, the semantics of this proposed API are incredibly murky to me. What happens if each consumer calls next()? Do they all get the same value out of their Future when it's completed? Do they each randomly get one of the values pushed into the stream? Is the stream implicitly required to buffer data in order to be able to offer it at a slower rate to one consumer than it is offered to the other consumers? Does each consumer have a different 'view' of the state of the stream (i.e. its cancelled/ended state, when those concepts apply)?

Huh, I'm not sure what's unclear about it. (Though, obviously it must be.)

The future returned by Stream#next() resolves at the next update (or when the stream completes/rejects). If multiple consumers call next() repeatedly in the same tick (or in different ticks, but before an update gets pushed), all of the futures resolve at the same time, because they're all listening for the same "next update".

I'm not sure I understand how it could be required to buffer. Can you describe the kind of situation you think would cause that need?

Same with view - perhaps you're thinking that the state can be updated syncly, inbetween listen callbacks?

# Andrea Giammarchi (12 years ago)

On Mon, Apr 15, 2013 at 2:04 PM, Tab Atkins Jr. <jackalmage at gmail.com>wrote:

Also, JS has been kept general purpose enough, I would not try to influence the language too much after DOM and W3C APIs and I thought this was also same idea of TC39.

I don't understand this sentence. It sounds like you're trying to say that Streams aren't general-purpose enough for JS to standardize, and should be left to DOM/W3C to do. Is this correct?

I was rather thinking that if this proposal is for a general purpose, then the following should not be the case :-)

I designed it explicitly for DOM use-cases that I ran into

I see that this is going in the right direction though, more real-life cases for non DOM only things in the pot.

me like that

# Kevin Gadd (12 years ago)

OK, it sounds like this is sort of a 'lossy' stream concept, then, in that if you fail to call next() early enough, you can miss data from the stream? I can see how that would be acceptable for certain use cases. To me though, this limits use of the primitive.

Here's a hypothetical. I'm building an instant messaging webapp - think google talk - and I want a 'stream' that represents the status of my online buddies. So each time someone goes online/offline, that would come through the stream.

It sounds like with this design, were I to use next() to pull data out of the stream, any online status changes that have occurred before I called next() are lost forever, because at any given time all calls to next() return a future for the 'next update'. This means that if the portion of my service responsible from the contact list starts later than the actual network backend, it can miss status changes and end up with a partial picture of the state. Maybe that's ok.

Similarly, it sounds like given this model if I have 3 consumers, and they all call next() once to get values, they have to be absolutely certain to call next() again as soon as the Future from the previous next() gets data. If they do the 'right thing' that you normally do in Future-oriented scheduling, where a future being activated results in a task being scheduled to do work, it's possible to miss data, given a flow like this:

  1. The consumers call next() and get futures (we'll call these A)
  2. Data A is pushed into the stream.
  3. Futures A are all fulfilled with data A. The handlers responsible for responding to it push work items onto the message queue (or event pump, or setTimeout, or whatever).
  4. Data B is pushed into the stream. There are no next listeners at this point.
  5. The work items on the message queue run, respond to Data A, and then call next() to get futures (we'll call these futures B)
  6. Data C is pushed into the stream.
  7. Futures B are fulfilled with data C.

In this scenario, I don't think data B would ever be observed using next(), which feels like a real trap for the unwary. You would normally solve this with some sort of buffering, like you see in sockets and many threading setups, in order to handle cases where a consumer is not able to respond instantaneously to the flow of data.

Is this just an incorrect use of 'next' - are you supposed to instantaneously respond to your Future from next() completing by calling next() again? In many Future models this is inadvisable because you can end up with recursive completion of futures when the stream is full - each call to next() returns a future that is already complete, so as soon as you attach a callback to the future, that callback completes and you climb down the stack, recursively processing data until you run out of stack space. I can see how perhaps the implicit assumption here is that the stack overflow scenario is prevented by utilizing the event loop/message pump to fulfill futures, but honestly I think any design which depends on that is a rather questionable design.

Another question: Given this sort of 'synchronized' update model, what happens if two consumers both cause data to be pushed into the stream? There are two values to send out, but next() only has room for one value. Does the second value get thrown away? Does an exception get raised by the second push? I don't know how you can ensure that all the consumers will see the second value.

I think I will have to echo others' thoughts here that this really doesn't seem like a 'Stream' API. It does not match the semantics of any Stream primitive/concept I have ever encountered in an API.

# Jake Verbaten (12 years ago)

So one thing that is confusing is that next() has an API that looks like the stream is a "pull stream". A "pull stream" means that you call next() and it will give you the next value when it has it. This type of stream generally buffers values internally and only gives you them when you pull them out by calling next(). Node's streams2 API has a similar thing with a read() API.

The semantics of the stream are actually push, which means the stream will just push values out whenever it has them and it's the consumers responsibility to add listeners using listen(). The usage of next() is confusing as it's easy to mistake the stream for being pull based.

# Andrea Giammarchi (12 years ago)

how about .pull() then, to read, and .push() to write :D

# Tab Atkins Jr. (12 years ago)

On Mon, Apr 15, 2013 at 5:34 PM, Jake Verbaten <raynos2 at gmail.com> wrote:

So one thing that is confusing is that next() has an API that looks like the stream is a "pull stream". A "pull stream" means that you call next() and it will give you the next value when it has it. This type of stream generally buffers values internally and only gives you them when you pull them out by calling next(). Node's streams2 API has a similar thing with a read() API.

The semantics of the stream are actually push, which means the stream will just push values out whenever it has them and it's the consumers responsibility to add listeners using listen(). The usage of next() is confusing as it's easy to mistake the stream for being pull based.

Ah, I see the problem.

Yes, under my proposal's semantics, you shouldn't be using next as the standard way to consume data - that's what listen is for. next is for when you just need the next update only (or the next update that satisfies some filter) - it imposes no constraints on the stream, so it's possible to miss updates even if you call next again inside the future's accept callback.

For an example of where this is useful, check out my Font Futures proposal at www.xanthir.com/b4PV1. The FontList#ready

function is basically equivalent to getting an observer stream off of the loadStatus attribute, and then calling next("loaded") on it. All you care about is the very next time it's "loaded" - if it flips between "loading" and "loaded" more, you don't really care.

(Actually, to handle this properly you need a Stream subclass that remembers the stream's last update, and a value method that's identical to next except it first checks against the last update.)

My proposal isn't meant to be a lossless stream, though like I said, it's not hard to imagine a subclass that represents one, and a method that generates a lossless stream out of a lossy stream via buffering.

# Tab Atkins Jr. (12 years ago)

On Mon, Apr 15, 2013 at 5:37 PM, Andrea Giammarchi <andrea.giammarchi at gmail.com> wrote:

how about .pull() then, to read, and .push() to write :D

Node experience notwithstanding, long experience with Promises/etc showed that mixing the promise and the resolver in a single object is a bad idea. I expect the same to be true of streams, which is why I designed the constructor analogously to Future, as it produces a strict and hard-to-accidentally-violate separation.

# Tab Atkins Jr. (12 years ago)

On Mon, Apr 15, 2013 at 5:26 PM, Kevin Gadd <kevin.gadd at gmail.com> wrote:

OK, it sounds like this is sort of a 'lossy' stream concept, then, in that if you fail to call next() early enough, you can miss data from the stream? I can see how that would be acceptable for certain use cases. To me though, this limits use of the primitive.

Here's a hypothetical. I'm building an instant messaging webapp - think google talk - and I want a 'stream' that represents the status of my online buddies. So each time someone goes online/offline, that would come through the stream.

It sounds like with this design, were I to use next() to pull data out of the stream, any online status changes that have occurred before I called next() are lost forever, because at any given time all calls to next() return a future for the 'next update'. This means that if the portion of my service responsible from the contact list starts later than the actual network backend, it can miss status changes and end up with a partial picture of the state. Maybe that's ok.

Yes, my current model doesn't buffer changes before the first listener. This is the correct behavior for most DOM cases, but I see how for a lot of other cases it's less good. By default, you might want to have that buffer, so the first consumer gets to see all the history, at least. (If you want multiple consumers to all get the history, you should opt into that explicitly via some slightly less convenient API, as it means infinite buffering.)

Similarly, it sounds like given this model if I have 3 consumers, and they all call next() once to get values, they have to be absolutely certain to call next() again as soon as the Future from the previous next() gets data. If they do the 'right thing' that you normally do in Future-oriented scheduling, where a future being activated results in a task being scheduled to do work, it's possible to miss data, given a flow like this:

next is for when you just care about the very next time something happens. If you care about all the events that come out, use listen instead.

It looks like many existing APIs use next for more pull-based purposes, where the stream will buffer data until someone pulls it out with next. My current API is explicitly not pull-based, but it's easy to imagine a subclass that is.

Another question: Given this sort of 'synchronized' update model, what happens if two consumers both cause data to be pushed into the stream? There are two values to send out, but next() only has room for one value. Does the second value get thrown away? Does an exception get raised by the second push? I don't know how you can ensure that all the consumers will see the second value.

Same thing - consumers should be using listen if they want all the data. That's the "default" way to interact with a stream in my proposal.

I think I will have to echo others' thoughts here that this really doesn't seem like a 'Stream' API. It does not match the semantics of any Stream primitive/concept I have ever encountered in an API.

Yes, the name seems to be confusing people. On the other hand, I find a lot of the other existing names hopelessly opaque or awkward - Signal doesn't mean much to me, and Observable is too abstract. Stream is just the right kind of name to convey "a stream of updates", but unfortunately the name is co-opted by binary IO streams. :/

# Isaac Schlueter (12 years ago)

I'm not seeing what in this proposal can't be implemented in JavaScript as it is today. Is there an implementation of this somewhere? Are there any programs that use these streams?

# Tab Atkins Jr. (12 years ago)

On Sat, Apr 20, 2013 at 9:19 AM, Isaac Schlueter <i at izs.me> wrote:

I'm not seeing what in this proposal can't be implemented in JavaScript as it is today. Is there an implementation of this somewhere? Are there any programs that use these streams?

This is a fully-general counter-argument against literally everything that doesn't require new primitives, and so is useless as an actual argument. It would damn Promises/Futures, Sets, Maps, and a number of other new things. The valid version of this argument is about usefulness, and there being unable to implement in current JS is a supporting reason to add something, but not the only reason.

The concept of "event streams" shows up in lots of libraries, particularly those dealing with functional reactive programming. I'm going to start explicitly documenting these uses so we have more data for constructing and evaluating any final proposals.

Please note that the proposal topping this thread is already long-obsolete. I have a second thread explaining it more, and the most recent iteration of the proposal is being tracked on my blog: www.xanthir.com/b4PV0.

# Bjoern Hoehrmann (12 years ago)
  • Tab Atkins Jr. wrote:

On Sat, Apr 20, 2013 at 9:19 AM, Isaac Schlueter <i at izs.me> wrote:

I'm not seeing what in this proposal can't be implemented in JavaScript as it is today. Is there an implementation of this somewhere? Are there any programs that use these streams?

This is a fully-general counter-argument against literally everything that doesn't require new primitives, and so is useless as an actual argument.

It is unlikely that he meant to make a useless argument. I would take it as a request for references to running code so we can get a better idea of how this is implemented and used in practise to aid the discussion. I believe some references have already been given in the various threads, perhaps it would be useful to collect some of them on your website?

# Tab Atkins Jr. (12 years ago)

On Sat, Apr 20, 2013 at 1:23 PM, Bjoern Hoehrmann <derhoermi at gmx.net> wrote:

  • Tab Atkins Jr. wrote:

On Sat, Apr 20, 2013 at 9:19 AM, Isaac Schlueter <i at izs.me> wrote:

I'm not seeing what in this proposal can't be implemented in JavaScript as it is today. Is there an implementation of this somewhere? Are there any programs that use these streams?

This is a fully-general counter-argument against literally everything that doesn't require new primitives, and so is useless as an actual argument.

It is unlikely that he meant to make a useless argument. I would take it as a request for references to running code so we can get a better idea of how this is implemented and used in practise to aid the discussion. I believe some references have already been given in the various threads, perhaps it would be useful to collect some of them on your website?

Given that I said exactly that in the text following the bit you quoted, I obviously agree. ^_^

(I get annoyed at that particular argument because, a surprising amount of the time, people do make exactly that argument. They either think that everything should be done through libraries, or are trying to make an argument against the idea's usefulness but are accidentally proving too much.)

# Sean Silva (12 years ago)

On Sat, Apr 20, 2013 at 3:47 PM, Tab Atkins Jr. <jackalmage at gmail.com>wrote:

On Sat, Apr 20, 2013 at 9:19 AM, Isaac Schlueter <i at izs.me> wrote:

I'm not seeing what in this proposal can't be implemented in JavaScript as it is today. Is there an implementation of this somewhere? Are there any programs that use these streams?

This is a fully-general counter-argument against literally everything that doesn't require new primitives, and so is useless as an actual argument. It would damn Promises/Futures, Sets, Maps, and a number of other new things.

I'm pretty sure there is no way to implement Maps with arbitrary keys in current JS. As for Promises/Futures, I think that the argument is equally forceful against them as it is to the proposals in this thread.

The valid version of this argument is about usefulness, and there being unable to implement in current JS is a supporting reason to add something, but not the only reason.

My understanding from you wording is that you want this to be standardized into ECMAScript. That means that this change would be shipped in billions of devices and require implementation and testing effort across the globe and across every vendor. Are you sure it's utility would not be diminished if you just put a nice working implementation on github and register it with the various package managers that make installing and using such code dead simple? That would be a huge win since people would be able to start using it immediately.

Concretely, compared to putting it on github and registering with package managers, what percentage usefulness increase do you expect to see by having this standardized?

I think you're completely right, "The valid version of this argument is about usefulness", except it's about usefulness of standardizing it compared to other ways of making the code available, and not about the intrinsic usefulness of the API (which, I'll point out, does not appear to be convincingly established).

-- Sean Silva

# Tab Atkins Jr. (12 years ago)

On Sat, Apr 20, 2013 at 4:13 PM, Sean Silva <silvas at purdue.edu> wrote:

On Sat, Apr 20, 2013 at 3:47 PM, Tab Atkins Jr. <jackalmage at gmail.com> wrote:

On Sat, Apr 20, 2013 at 9:19 AM, Isaac Schlueter <i at izs.me> wrote:

I'm not seeing what in this proposal can't be implemented in JavaScript as it is today. Is there an implementation of this somewhere? Are there any programs that use these streams?

This is a fully-general counter-argument against literally everything that doesn't require new primitives, and so is useless as an actual argument. It would damn Promises/Futures, Sets, Maps, and a number of other new things.

I'm pretty sure there is no way to implement Maps with arbitrary keys in current JS.

It's possible, but lookup time is linear. You need some variety of language support to get constant-time lookups.

As for Promises/Futures, I think that the argument is equally forceful against them as it is to the proposals in this thread.

The valid version of this argument is about usefulness, and there being unable to implement in current JS is a supporting reason to add something, but not the only reason.

My understanding from you wording is that you want this to be standardized into ECMAScript. That means that this change would be shipped in billions of devices and require implementation and testing effort across the globe and across every vendor. Are you sure it's utility would not be diminished if you just put a nice working implementation on github and register it with the various package managers that make installing and using such code dead simple? That would be a huge win since people would be able to start using it immediately.

Concretely, compared to putting it on github and registering with package managers, what percentage usefulness increase do you expect to see by having this standardized?

I think you're completely right, "The valid version of this argument is about usefulness", except it's about usefulness of standardizing it compared to other ways of making the code available, and not about the intrinsic usefulness of the API (which, I'll point out, does not appear to be convincingly established).

Whether TC39 ends up taking over standardization of promises/futures or not, they're in the DOM and will be used by new standards.

I write standards for a living, so I'm well aware of the cost of adding features. I believe that addressing the use-cases of functional reactive programming is important enough to justify adding some new concepts to help them out, because it's an intrinsically useful and high-value abstraction for a lot of interactions with the DOM and with the user. Event streams are an early attempt by me to put something together for this, and I was using es-discuss as a sounding board and design help for it.

Now that I've got a handle on what I think a good design is, my plan is to retreat and spend some time studying and documenting existing stream patterns in libraries, to try and discover what I might be missing, or what areas I'm over/under-addressing. When I'm done, I'll report back with my findings.

# Sean Silva (12 years ago)

On Sat, Apr 20, 2013 at 8:10 PM, Tab Atkins Jr. <jackalmage at gmail.com>wrote:

On Sat, Apr 20, 2013 at 4:13 PM, Sean Silva <silvas at purdue.edu> wrote:

On Sat, Apr 20, 2013 at 3:47 PM, Tab Atkins Jr. <jackalmage at gmail.com> wrote:

On Sat, Apr 20, 2013 at 9:19 AM, Isaac Schlueter <i at izs.me> wrote:

I'm not seeing what in this proposal can't be implemented in JavaScript as it is today. Is there an implementation of this somewhere? Are there any programs that use these streams?

This is a fully-general counter-argument against literally everything that doesn't require new primitives, and so is useless as an actual argument. It would damn Promises/Futures, Sets, Maps, and a number of other new things.

I'm pretty sure there is no way to implement Maps with arbitrary keys in current JS.

It's possible, but lookup time is linear. You need some variety of language support to get constant-time lookups.

For ADT's, the complexity of an operation needs to be considered as part of the interface, since otherwise clients can't meaningfully use it. Every line of code a programmer writes requires a (possibly subconscious) evaluation of the complexity of the operation, and a factor of O(n) difference in complexity is enough to invalidate most of that reasoning and render the program broken. A linear lookup is equivalent to an incorrect implementation of the interface of a Map.

Now that I've got a handle on what I think a good design is, my plan is to retreat and spend some time studying and documenting existing stream patterns in libraries, to try and discover what I might be missing, or what areas I'm over/under-addressing. When I'm done, I'll report back with my findings.

Cool. Please make sure to involve the primary developers of the most-used FRP and event stream libraries in your initial prototyping and put your work up on github.

-- Sean Silva

# Isaac Schlueter (12 years ago)

On Sat, Apr 20, 2013 at 12:47 PM, Tab Atkins Jr. <jackalmage at gmail.com> wrote:

On Sat, Apr 20, 2013 at 9:19 AM, Isaac Schlueter <i at izs.me> wrote:

I'm not seeing what in this proposal can't be implemented in JavaScript as it is today. Is there an implementation of this somewhere? Are there any programs that use these streams?

This is a fully-general counter-argument against literally everything that doesn't require new primitives, and so is useless as an actual argument.

It's not fully-general counter-argument, because it's not an argument. It's a request for more information. Until I have that information, I don't have enough information to form a reasonable opinion.

I read your proposal. It appears to be something that is implementable with today's current JavaScript. Am I mistaken? If so, how so? (Ie, what new language semantics or primitives are required?)

If I'm not mistaken, then where's the reference implementation? I don't even see example code. I figured that I must be missing something.

You're wrong about what it would damn, in every case.

It would damn Promises/Futures

Promises had many, many iterations in working code long before a specification was ever proposed.

Sets, Maps

Both required new language semantics. But there WERE reference implementations of both, long before either of those reached a spec.

and a number of other new things.

Please be more specific. There are many numbers. Don't allude to the vastness of your point, just make it, or don't.

Search npm and github before making claims about what things would be damned by the request for a reference implementation in advance of a specification.

Please note that the proposal topping this thread is already long-obsolete.

Then please update the thread with the updated proposal, and preferably a link to the github account with the reference implementation, if such a thing exists.

If such a thing does not exist, please explain why it is not feasible to write an implementation of this with today's language semantics.

If I seem a bit impatient, it is because I am eager to have an opinion about this idea of yours, but I find myself incapable of responsibly having an opinion without seeing it in action.

As you say, the concept of event streams appears in many libraries. I work daily with Node.js, which is full of event emitters and binary data streams. You mentioned that it would be easy to implement binary streams on top of your streams idea. I'd like to see that. I know from firsthand experience that streams can be surprisingly hard to get right. In my opinion, if you can't easily implement TCP, TLS, and Zlib on a stream implementation, then you haven't gotten it right.

So, the fact that an implementation has not yet been shown seems quite strange to me, because that would be the first thing that I'd expect anyone would want to see, and it seems like it'd only take a few hundred lines of code to do. I say that because other "streaming" FRP pattern libraries have been implemented in just a few hundred lines. Look at the work of Dominic Tarr (dominictarr) or Irakli Gozalishvili (gozala) on npm and github.

Until there is an implementation, I don't really see what there is to talk about. As I said, this is not a counter-argument, because as far as I'm concerned, there's nothing to even argue about!

# Tab Atkins Jr. (12 years ago)

On Mon, Apr 22, 2013 at 4:13 PM, Isaac Schlueter <i at izs.me> wrote:

On Sat, Apr 20, 2013 at 12:47 PM, Tab Atkins Jr. <jackalmage at gmail.com> wrote:

On Sat, Apr 20, 2013 at 9:19 AM, Isaac Schlueter <i at izs.me> wrote:

I'm not seeing what in this proposal can't be implemented in JavaScript as it is today. Is there an implementation of this somewhere? Are there any programs that use these streams?

This is a fully-general counter-argument against literally everything that doesn't require new primitives, and so is useless as an actual argument.

It's not fully-general counter-argument, because it's not an argument. It's a request for more information. Until I have that information, I don't have enough information to form a reasonable opinion.

That request is nearly always followed with an (explicit or implied) "And if it can be, why should we do it? Leave it to libraries!" statement. If you did not intend this, then I apologize.

I read your proposal. It appears to be something that is implementable with today's current JavaScript. Am I mistaken? If so, how so? (Ie, what new language semantics or primitives are required?)

You need Object.observe to do reactive values as an event stream. You can do event streams (turning DOM Events into a stream) with today's JS.

If I'm not mistaken, then where's the reference implementation? I don't even see example code. I figured that I must be missing something.

Apologies for the mismatch in expectations. I recognize that reference implementations are more customary when proposing things in JS that don't depend on new primitives. Instead, I provided a spec that sketched out appropriate details (and can be filled in further as necessary). For the purpose of discussion of the idea, the two should be roughly equivalent.

Search npm and github before making claims about what things would be damned by the request for a reference implementation in advance of a specification.

My comment was not in response to a request for a reference implementation. (This side-trek is irrelevant, though - no need to respond to it.)

Please note that the proposal topping this thread is already long-obsolete.

Then please update the thread with the updated proposal,

To avoid any confusion in this thread, I started a second thread with my updated proposal. (Though the proposal in the OP of that thread is also obsolete, I updated the thread when I majorly changed the proposal. My blog post, which always has the current state of the proposal, is also in this thread and the other.

and preferably a link to the github account with the reference implementation, if such a thing exists.

If such a thing does not exist, please explain why it is not feasible to write an implementation of this with today's language semantics.

If I seem a bit impatient, it is because I am eager to have an opinion about this idea of yours, but I find myself incapable of responsibly having an opinion without seeing it in action.

As you say, the concept of event streams appears in many libraries. I work daily with Node.js, which is full of event emitters and binary data streams. You mentioned that it would be easy to implement binary streams on top of your streams idea. I'd like to see that. I know from firsthand experience that streams can be surprisingly hard to get right. In my opinion, if you can't easily implement TCP, TLS, and Zlib on a stream implementation, then you haven't gotten it right.

So, the fact that an implementation has not yet been shown seems quite strange to me, because that would be the first thing that I'd expect anyone would want to see, and it seems like it'd only take a few hundred lines of code to do. I say that because other "streaming" FRP pattern libraries have been implemented in just a few hundred lines. Look at the work of Dominic Tarr (dominictarr) or Irakli Gozalishvili (gozala) on npm and github.

Until there is an implementation, I don't really see what there is to talk about. As I said, this is not a counter-argument, because as far as I'm concerned, there's nothing to even argue about!

Again, I'm aware that JS proposals often come with a reference implementation, or at least a sketch of one. That's not strictly necessary to review and talk about a proposal, however. If you think that you are unable to even discuss the proposal without such a reference implementation, then I'm sorry that we will have to lose your voice until I have the time and engery to provide such.

# Isaac Schlueter (12 years ago)

That request is nearly always followed with an (explicit or implied) "And if it can be, why should we do it? Leave it to libraries!" statement. If you did not intend this, then I apologize.

Well, actually, I think that we should do it with libraries for a bit, because I have come to believe from repeated experience that this is the only reliable way to evaluate patterns, and that forcing acceptance of a specification in advance of trials leads to a lot of error. Again, qv promises, map/set, etc. That's how it's done because it works.

You need Object.observe to do reactive values as an event stream. You can do event streams (turning DOM Events into a stream) with today's JS.

Are you proposing reactive values as an event stream? It seemed like your proposal was mostly pretty simple interfaces.

Apologies for the mismatch in expectations. I recognize that reference implementations are more customary when proposing things in JS that don't depend on new primitives. Instead, I provided a spec that sketched out appropriate details (and can be filled in further as necessary). For the purpose of discussion of the idea, the two should be roughly equivalent.

I disagree. They're not equivalent at all. Not even roughly, not even close. One can be tested, reviewed, and subjected to experiments in the real world. The other is imaginary.

Again, I'm aware that JS proposals often come with a reference implementation, or at least a sketch of one. That's not strictly necessary to review and talk about a proposal, however.

To do so responsibly, I believe that it is.

If you think that you are unable to even discuss the proposal without such a reference implementation, then I'm sorry that we will have to lose your voice until I have the time and engery to provide such.

Actually, I strongly doubt anyone is able to adequately discuss the pros and cons of a proposal like this without an implementation.

I don't have anything else to contribute until there is a reference implementation. I'm not super motivated to write it myself, but I'd be happy to test it out, or consult about any sticky issues you run into, etc.