Explainer/Spec: Smart pipelines
ESDiscuss.org stripped much of the formatting from my original message. To give the links in plain text:
Readme explainer: js-choi/proposal-smart-pipelines
Formal spec: jschoi.org/18/es-smart-pipelines/spec
Issue tracker (please specify Proposal 4 in new issues): tc39/proposal-pipeline-operator/issues?utf8=✓&q=sort%3Aupdated-desc+
List of all current pipeline proposals: tc39/proposal-pipeline-operator/wiki
Warm , J. S. Choi
It looks like this is different to the existing pipeline proposal in
essentially only one way, in that it includes the # token (or lexical
topic as you call it). I like that the proposal addresses await and other
unary operators by default since it supports any expression on the
right-hand side given a lexical topic is involved.
One thing you may consider is automatic binding of unary operators similar
to the automatic binding of unary functions / methods / constructors. For
instance, this would enable x |> await |> fn or x |> typeof |> fn.
However, this may not be forwards-compatible. If someone had `x |> run |>
fnwhererunis a function name, and thenrun` was introduced as an
operator, you would have to define what takes precedence.
Overall, great job. I like it more that the existing proposal, and I have high hopes.
Can you explain how it solves the generator / async generator aspect of the proposal here: TheNavigateur/proposal-pipeline-operator-for-function-composition
I can't seem to find an example in the explainer.
Maybe you can formulate a way of doing it, then add it in the explainer. Thanks.
Thanks for the feedback, Peter and Naveen.
@Peter:
It looks like this is different to the existing pipeline proposal in essentially only one way, in that it includes the # token (or lexical topic as you call it). I like that the proposal addresses await and other unary operators by default since it supports any expression on the right-hand side given a lexical topic is involved.
To clarify: The way that bare style and topic style are distinguished are not by the absence/presence of a topic reference. Bare style has a simple and strict syntax: any identifiers, separated by .s, optionally preceded by new or await.
Any pipeline body that does not fulfill that simple and strict syntax is in topic style. However, any pipeline body that is in topic style also must contain a topic reference, or else it is an early syntax error. x |> f() is in topic style; it is also an early syntax error, because it is a topic-style pipeline that does not use the topic reference.
One thing you may consider is automatic binding of unary operators similar to the automatic binding of unary functions / methods / constructors.
I had considered that, but I decided against it. I want to keep the rules of bare style very simple and strict in order to minimize cognitive burden on the developer. Topic style is meant to also be terse: x |> await # |> fn, await x |> fn, x |> typeof # |> fn, or typeof x |> fn. There is also the forward-compatibility problem you mention.
Thanks again for the feedback, Peter.
@Naveen:
Can you explain how it solves the generator / async generator aspect of the proposal here: TheNavigateur/proposal-pipeline-operator-for-function-composition
For background, the TheNavigateur proposal can rephrase this:
const doubleThenSquareThenHalf = value =>
half(square(double(value)));
…as this:
const doubleThenSquareThenHalf =
double +> square +> half;
The smart-pipeline proposal plus Additional Feature PF would express this as:
const doubleThenSquareThenHalf =
+> double |> square |> half;
Likewise this:
const doubleThenSquareThenHalfAsync = async value =>
half(await squareAsync(double(value)));
…may be this in the TheNavigateur proposal:
const doubleThenSquareThenHalfAsync =
double +> squareAsync +> half;
…and this in the smart-pipeline proposal plus Feature PF:
const doubleThenSquareThenHalfAsync =
async +> double |> await squareAsync |> half;
Smart pipelines, including with Additional Feature PF, only address piping singular values through expressions, as well as method extraction, partial application, and composition on functions on singular values. They do not yet address piping plural values such as iterators and generators. This is because there are many ways to compose plural values—mapping, reducing, filtering, partitioning, and so forth. I do plan to try to address this in the futer with Clojure-style transducers. But currently, you would have to rely on higher-order helper functions from a library or written by yourself to combine plural values.
This:
const randomBetween0And100Generator = () => * {
for (randomBetween0And1Generator())
yield |> multiplyBy100;
};
…may be this in the TheNavigateur proposal:
const randomBetween0And100Generator =
randomBetween0And1Generator +> multiplyBy100;
…and this in the smart-pipeline proposal plus Feature PF and Feature PP:
const randomBetween0And100Generator = () => * {
for (randomBetween0And1Generator())
yield |> multiplyBy100;
};
…or this, using some map helper function,
const randomBetween0And100Generator = () =>
randomBetween0And1Generator()
|> map(+> multiplyBy100, #);
This is similar to how mapping arrays is not directly addressed by pipelines:
const arrayBetween0And100 =
arrayBetween0And1
.map(+> multiplyBy100, #);
There are many ways to compose plural values—mapping, reducing, filtering, partitioning, and so forth. It is currently out of scope for smart pipelines to address them; they are not even well standardized in JavaScript in general. I do plan to try to address uniformly composing plural-valued functions in the future with a Clojure-style transducer API – first as a library, then maybe as as a TC39 proposal if I can find an interested TC39 member. See Transducers.js for examples of this idea. But, no, currently, you would have to rely on higher-order helper functions from a library or written by yourself to combine plural-valued functions.
For now, I’m focusing on the low-hanging fruit that smart pipelines could provide, before the next TC39 meeting. This includes a Babel plugin that I am creating in cooperation with James DiGioia, who is also working on a proposal for Proposal 1: F-sharp Style Only (tc39/proposal-pipeline-operator/wiki#proposal-1-f-sharp-style-only). We will see what comes of all this in the future.
Warm , J. S. Choi
To clarify: The way that bare style and topic style are distinguished are
not by the absence/presence of a topic reference. Bare style has a simple
and strict syntax: any identifiers, separated by .s, optionally preceded
by new or await.
So x['hello'] would not be valid? Seems pretty inconsistent. I think that
checking for the lexical topic token would be simpler and easier for
developers.
Any pipeline body that does not fulfill that simple and strict syntax is
in topic style. However, any pipeline body that is in topic style also must
contain a topic reference, or else it is an early syntax error. x |> f()
is in topic style; it is also an early syntax error, because it is a topic-style pipeline that does not use the topic reference.
This seems to not support auto-currying functions. For instance, lodash/fp has auto-currying functions which take an iteratee as the first argument:
// `lodash/fp/filter` is iteratee-first data-last:
// (iteratee, collection)
var compact = fp.filter(Boolean);
compact(['a', null, 'c']);
// ➜ ['a', 'c']
So someone might want to do something like the following to utilize this behavior:
['a', null, 'c'] |> fs.filter(Boolean)
optionally preceded by
neworawait
This seems very arbitrary and not forwards-compatible.
By only allowing identifiers, you're requiring people to create unnecessary intermittent variables. I propose that you allow any expression on the right-hand side of the pipe operator, and decide whether it's in topic style or bare style based on the inclusion of the lexical topic token. Your argument that making it very restrictive reduced cognitive burden doesn't make sense to me at all, as remembering what is and isn't allowed is more difficult than just remembering "if it doesn't have the token, it will call the function specified by the expression with the argument of the result of the previous step in the pipeline".
On Sun, Mar 11, 2018 at 2:36 PM, Peter Jaszkowiak <p.jaszkow at gmail.com> wrote:
To clarify: The way that bare style and topic style are distinguished are not by the absence/presence of a topic reference. Bare style has a simple and strict syntax: any identifiers, separated by
.s, optionally preceded byneworawait.So
x['hello']would not be valid? Seems pretty inconsistent. I think that checking for the lexical topic token would be simpler and easier for developers.
I strongly disagree - I thing JS's approach is a large improvement over the previous pipeline proposal, for two reasons.
- As they already stated, it helps avoid accidental mis-types, like
x |> f()when you meantx |> f- you get an early syntax error
instead.
2. It purposely prevents point-free shenanigans, where you might
type x |> f() on purpose, intending f() to resolve to a function
that is then called with x. Instead you have to write x |> f()(#),
which has a much clearer intent, as it's written exactly how you'd
call it if you wrote it out without the pipeline. More importantly,
imo, it makes point-free stuff like flip or the like pointless (uh,
no pun intended) - instead of doing abstract manipulations of the
argument lists, you can just call the function normally, with # put
where you want.
Any pipeline body that does not fulfill that simple and strict syntax is in topic style. However, any pipeline body that is in topic style also must contain a topic reference, or else it is an early syntax error.
x |> f()is in topic style; it is also an early syntax error, because it is a topic-style pipeline that does not use the topic reference.This seems to not support auto-currying functions. For instance, lodash/fp has auto-currying functions which take an iteratee as the first argument:
// `lodash/fp/filter` is iteratee-first data-last: // (iteratee, collection) var compact = fp.filter(Boolean); compact(['a', null, 'c']); // ➜ ['a', 'c']So someone might want to do something like the following to utilize this behavior:
['a', null, 'c'] |> fs.filter(Boolean)
This works just fine, you just need to explicitly call it like:
['a', null, 'c'] |> fs.filter(Boolean)(#)
optionally preceded by
neworawaitThis seems very arbitrary and not forwards-compatible.
I agree with arbitrary, tho I think new/await ends up being reasonable. What part of it do you think isn't forwards-compatible? Afaict, we can definitely add new bare-style forms in the future, as attempts to use them today would be a syntax error.
By only allowing identifiers, you're requiring people to create unnecessary intermittent variables. I propose that you allow any expression on the right-hand side of the pipe operator, and decide whether it's in topic style or bare style based on the inclusion of the lexical topic token. Your argument that making it very restrictive reduced cognitive burden doesn't make sense to me at all, as remembering what is and isn't allowed is more difficult than just remembering "if it doesn't have the token, it will call the function specified by the expression with the argument of the result of the previous step in the pipeline".
There is never a need to create intervening variables, you just need to use topic style.
Thanks again for the reply, Peter. I’m a little confused by your latest questions, so I’ll try to clarify the questions at a time.
[me] To clarify: The way that bare style and topic style are distinguished are not by the absence/presence of a topic reference. Bare style has a simple and strict syntax: any identifiers, separated by
.s, optionally preceded byneworawait.[Peter] So `x['hello']`` would not be valid? Seems pretty inconsistent. I think that checking for the lexical topic token would be simpler and easier for developers.
value |> x['hello'](#) is a valid pipeline, in topic style. value |> x['hello'] is an invalid pipeline, in order to prevent infinite unrestricted lookahead, therefore simplifying interpretation for both human readers and computer readers.
By “simplicity”, I’m referring to several goals from the proposal explainer. Particularly relevant here is the goal of syntactic locality. Copying and pasting from there: “The syntax should minimize the parsing lookahead that the compiler must check. If the grammar makes garden-path syntax common, then this increases the dependency that pieces of code have on other code. This long lookahead in turn makes it more likely that the code will exhibit developer-unintended behavior.
“This is true particularly for distinguishing between different styles of pipeline body syntax. A pipeline’s meaning would often be ambiguous between these styles – at least without checking the pipeline’s body carefully to see in which style it is written. And the pipeline body may be a very long expression.
“By restricting the space of valid bare-style pipeline bodies (that is, without topic references), the rule minimizes garden-path syntax that would otherwise be possible – such as value |> compose(f, g, h, i, j, k, #). Syntax becomes more locally readable. It becomes easier to reason about code without thinking about code elsewhere.”
In addition, changing the grammar to depend on containment of a token would make it impossible to parse with a context-free grammar or by introducing a new grammar-production parameter, which multiplies the number of productions in JavaScript’s grammar. The pipeline syntax grammar alone would become considerably considerably more complicated – and this also applies to the grammars for every single other type of expression, all of which would have to be modified with an additional syntactic parameter.
At the very beginning, in fact, I actually considered making the grammar to depend on containment of a token – but then I realized many of the disadvantages above that it would bring.
The example value |> x['hello'] seems simple. But the danger of requiring unrestrictedly infinite lookahead is that people will also write things like value |> compose(x['hello'][symbol].propertyA.key(#).propertyB.propertyC, anotherFunction). It is easy for a human reader to miss the # in there, yet its presence or absence would completely change the pipeline’s semantics, from the value returned by compose(…) to a function call on that value. That’s the consequence of compromising syntactic locality. It means that a reader will have to check distant code in order to determine the style of a pipeline. Avoiding footguns is a paramount goal.
And value |> x['hello'](#) is still a valid pipeline.
[me] Any pipeline body that does not fulfill that simple and strict syntax is in topic style. However, any pipeline body that is in topic style also must contain a topic reference, or else it is an early syntax error. x |> f() is in topic style; it is also an early syntax error, because it is a topic-style pipeline that does not use the topic reference.
[Peter] This seems to not support auto-currying functions. For instance, lodash/fp has auto-currying functions which take an iteratee as the first argument:
My apologies – I’m a little confused by what your first sentence here means. By “this”, do you mean that smart pipelines do not support auto-currying functions, or do you mean that the bare style do not support them?
Auto-currying functions are indeed supported by smart pipelines, though not necessarily by bare style.
The purpose of bare style is merely to optimize a common use case – unary function calls. All other cases are intended to use topic style, which hopefully is still terse, yet explicit enough to avoid ambiguity.
In fact, several examples in the readme are drawn from Ramda’s cookbook, several of which, if I recall correctly, involve autocurrying functions (Ramda examples, part 1 and Ramda examples, part 2).
Your example using Lodash/FP,
['a', null, 'c'] |> fs.filter(Boolean),
is expressible using
['a', null, 'c'] |> fs.filter(Boolean)(#).
The latter is a valid topic-style pipeline.
In general, x |> f(a, b) is visually ambiguous between four reasonable interpretations:
x |> f(a, b, #),
x |> f(a, #, b),
x |> f(#, a, b), and
x |> f(a, b)(#).
The early error forces the writer to specify which of the four reasonable interpretations they mean, also reducing the parsing burden on the human reader.
[me] optionally preceded by
neworawait[Peter] This seems very arbitrary and not forwards-compatible.
My apologies again – I am also confused by what you mean here by forwards-compatible. What is there for bare style to be forward compatible with? New prefix operators? If a new operator is commonly used before unary function calls, then they could be added to bare style’s syntax later, but in the meantime topic style can handle them, as well as anything else, simply by appending (#).
Bare style is a special syntax, with special evaluation semantics, optimized for one common use case: unary functions. The new and await tokens are just flags for bare style because unary constructor calls and awaited unary async-function calls are also quite common, as my review of real-world codebases in Motivation shows. But in general, I expect topic style to be used more often, and topic style itself is very terse.
In fact, the new and await flags in bare style were originally not there. The only reason why I added them in the first place was because, during that review of real-world code, I found that unary constructor calls and awaited unary async-function calls are very common. I am still considering removing them from the Core Proposal, into their own separate Additional Features, so that their tradeoffs may be independently considered.
By only allowing identifiers, you're requiring people to create unnecessary intermittent variables. I propose that you allow any expression on the right-hand side of the pipe operator, and decide whether it's in topic style or bare style based on the inclusion of the lexical topic token. Your argument that making it very restrictive reduced cognitive burden doesn't make sense to me at all, as remembering what is and isn't allowed is more difficult than just remembering "if it doesn't have the token, it will call the function specified by the expression with the argument of the result of the previous step in the pipeline".
Hopefully, the quotation I pasted from the syntactic locality section from above is an adequate response to this paragraph. Bare style’s rules are simple: “”. The rules are simpler than humans and computers having to search a long pipeline expression, such as value |> compose(f, g, h, i, j, k, #, l, m, n, o) or value |> compose(x['hello'][symbol].propertyA.key(#).propertyB.propertyC, anotherFunction), for the absence or presence of a single topic reference #. In many reasonable cases, parsing would be considerably more complicated, for humans and computers alike.
Thanks again for the reply. Your x['hello']-method example and your Lodash/FP example are both already addressed by the current smart-pipelines proposal. I hope these explanations clarify your understanding of the current proposal.
I’ll have only intermittent free time in the next few days, but in the meantime James DiGioia and I will be working on the Babel plugin for both smart pipelines and F-sharp Style Only pipelines. I’ll try to reply to any reply here when I can.
Warm , J. S. Choi
On Sun, Mar 11, 2018 at 4:44 PM, Tab Atkins Jr. <jackalmage at gmail.com> wrote:
On Sun, Mar 11, 2018 at 2:36 PM, Peter Jaszkowiak <p.jaszkow at gmail.com> wrote:
optionally preceded by
neworawaitThis seems very arbitrary and not forwards-compatible.
I agree with arbitrary, tho I think new/await ends up being reasonable. What part of it do you think isn't forwards-compatible? Afaict, we can definitely add new bare-style forms in the future, as attempts to use them today would be a syntax error.
Actually, I'd like to elaborate here, as I think the inclusion of new/await is actually very important for understanding the motivations of bare style as JS has proposed, versus the "arbitrary topic-less expression that evaluates to a function" form that Dan's original has.
Ultimately, bare style is just a syntax nicety. Anything you can write in bare style (either one), you can write in topic style with a few more characters; it adds no additional power. As such, we should think about it in terms of what it allows vs what it disallows, and how useful each category would be.
In Dan's "arbitrary expression" form, you get:
-
easy pipelining of variables holding functions:
x |> foo |> bar -
easy pipelining of point-free expressions, or auto-curried functions:
x |> flip(foo)orxs |> filter(pred)
You do not get:
3. easy construction of new objects: x |> new Foo will call Foo()
and then try to call the result with x, when you probably wanted it to
call new Foo(x); you need to instead write x |> new Foo(#)
- easy awaiting of function results -
x |> await barwill await the
bar variable and then try to call the result with x, when you probably
wanted it to call await bar(x); you need to instead write `x |>
await bar(#)orx |> bar |> await #`
- easy method calls!!!:
x |> foo.barpullsbaroff offoo, then
calls it with x, without foo being bound to this! You probably
wanted it to call foo.bar(x), so you instead you need to write `x |>
foo.bar(#)`
JS's bare-form, on the other hand, gets you 1, 3, 4, and 5, and only
misses out on 2; you need to write xs |> filter(pred)(#) or `xs |>
filter(pred, #)` instead, depending on how the function is defined. I think that cases 3/4/5 are dramatically more important and common than case 2.
What's worse, in the "arbitrary expression" form, if you accidentally
leave off the #, you get a runtime error (hopefully) when your
expression is misinterpreted, and have to track down why something is
acting weird. (And each of the three possible mistakes acts weird in a
unique way - an object constructed without its argument, or a promise
showing up unexpectedly, or this being undefined unexpectedly.)
In JS's form, if you leave off the # you get a syntax error immediately, pointing you straight to where the problem is.
So overall I think JS's bare-form is very well motivated and shouldn't be changed. ^_^
If generator composition isn't directly supported somehow, then I'd have to say I personally find the function composition proposal more compelling on its own, even in the absence of a pipeline operator.
@Naveen:
If generator composition isn't directly supported somehow, then I'd have to say I personally find the function composition proposal more compelling on its own, even in the absence of a pipeline operator.
That’s all right, Naveen; thank you for reading the explainer and/or spec anyway. You may find the smart pipelines—or pipelines in general—still useful for all its other use cases, including method extraction, method application, function application, and partial application.
And, importantly, the smart pipeline proposal does not preclude the future addition of a composition operator that can compose generators. It is just out of the current scope. I myself am trying to think of ways that smart pipelines might be eventually extended to support generator composition.
As for TheNavigateur’s proposal in specifics, though—I will say that I think its approach to even generators has disadvantages, though I need to think more about someone might best address them.
Perhaps most seriously, generators are not intended by the designers to be distinguishable from functions that return iterators (I think I can find a citation from Domenic Denicola for that), just like how async functions are not intended to be distinguishable from promise-returning functions (I’m almost certain can find a citation from Denicola for that).
But in addition, the problem of plural/iterator/functor/whatever-type composition is not as easy as simply as treating generators and async generators with special behavior. There are so many types of plural or container objects—not only generators and async generators, but also maybes, streams, observables, and the iterators themselves. TheNavigateur’s approach does not address these, despite their being reasonable similar cases.
And there are also many ways to compose them. TheNavigateur’s approach only addresses composition with mapping. But I would want a pipeline approach that addresses plural things to also be able to compose functions with flattened mapping, filtering, taking/dropping, partitioning, reduction, and so many other types of transformations.
There is an alternative proposal for function composition, isiahmeadows/function-composition-proposal/blob/master/README.md, which tries another approach: having a well-known symbol Symbol.lift, which is used.
But I’m not satisfied with that proposal either. I want to see if there is a different approach to try, one that may integrate with smart pipelines. Something like generateRandomNumbers() |>> map f |>> filter g |>> take 5…but I don’t yet know precisely how. But smart pipelines might be extendable to handle them.
Either way, it can be dealt with later. There are many other benefits, as hopefully you can see from the real-world code examples. But we’ll see, once the Babel plugins are written and TC39 takes a look. Thanks for reading the explainer and/or spec.
Warm re-gards, J. S. Choi
I’d like to ask for feedback/criticism on a detailed explainer and specification for Smart Pipelines plus several possible extensions.
Readme explainer: js-choi/proposal-smart-pipelines Formal spec: jschoi.org/18/es-smart-pipelines/spec. There is a simple “actual” Core Proposal at Stage 0 championed by Daniel Ehrenberg, plus several optional Additional Features that extend the Core Proposal and address several other use cases. Daniel will present the Core Proposal at the next TC39 meeting, in several weeks at London.
The Core Proposal is a variant of the first pipeline-operator proposal also championed by Ehrenberg; this variant is listed as Proposal 4: Smart Mix in the pipe-proposal wiki. The variant resulted from previous discussions in the previous pipeline-operator proposal, discussions which culminated in an invitation by Ehrenberg to try writing a specification draft. A prototype Babel plugin is also being written.
I should stress that the Additional Features are separate, optional and mutually independent add-on proposals. The Additional Features show the potential of simply extending the Core Proposal to handle other use cases (such as composition, partial application, and method extraction). And I’ve attempted to keep the Core Proposal forward compatible with all of the additional features.
If you have any questions after reading the explainer and specification, please feel free to file an issue on the GitHub issue tracker. When you file an issue, please note in it that you are talking specifically about“Proposal 4: Smart Mix”. Or leave a comment here.
Warm , J. S. Choi