Discuss
  • Function composition vs pipeline

Function composition vs pipeline

# Naveen Chawla (6 years ago)

I was just thinking about the relative merits and coexistence (or not) of function composition operator and function pipeline operator features:

e.g. TheNavigateur/proposal-pipeline-operator-for-function-composition, tc39/proposal-pipeline-operator

They can of course co-exist, but there is overlap only in the respect that both allow function pipelines to be called from left to right (except the input parameter in the case of the composition feature, which requires existing bracket syntax to be used to call it). If one were to be chosen, would say that a function composition operator adds a whole new dimension of expressive power to the language, whereas a pipeline operator only offers a different way of calling functions.

I was wondering about all of your thoughts about whether you'd prefer only the pipeline operator, only the composition operator, or both, or neither to be added to the language (these are pretty much all the possibilities), and why.

# Jordan Harband (6 years ago)

How is either operator not "a different way of calling functions"?

# J. S. Choi (6 years ago)

See also recent discussions in these GitHub issues that touch on composition: tc39/proposal-pipeline-operator/issues?q=composition+sort%3Aupdated-desc.

Look particularly at the issues that were created after 2018-01-31 – when Gilbert/Mindeavor split the proposal into four competing proposals – since those issues reflect the current state of the art. To see those four competing proposals, look at the pipeline operator’s wiki, which has a list of the four proposals (tc39/proposal-pipeline-operator/wiki).

In particular, Naveem, you may be interested in tc39/proposal-pipeline-operator#93, which discusses the intersection of terse function application and terse function composition.

# Naveen Chawla (6 years ago)

The function composition operator composes function pipelines into functions for later use and/or further composition. Those functions still need to be called via the existing () syntax, so it doesn't offer a different way of calling functions as such.

The function pipeline operator calls the function pipeline immediately, so it is really only a different way of calling functions.

# Viktor Kronvall (6 years ago)

I don’t know the implications but I could easily imagine the pipeline proposal being extended to not taking any input on the left hand side and effectively represent composition in the opposite direction.

For example:

let h = |> f |> g

h(2) //g(f(2))

That said, the point holds for the proposal in its current state. Being able to compose functions leads to much more expressivity than if you have to call the pipeline (and collapse) where it is defined. 2018年2月24日(土) 14:32 Naveen Chawla <naveen.chwl at gmail.com>:

# Naveen Chawla (6 years ago)

That could be a problem for readability. I agree with the rest of what you said.

# Peter Jaszkowiak (6 years ago)

I'd like to point out the partial application operator: tc39/proposal-partial-application

Sounds like the combination of pipeline + partial application would result in what is essentially the same as function composition operator:

const h = ? |> f |> g;

Which results in h being the composition g • f.

On Feb 24, 2018 02:21, "Naveen Chawla" <naveen.chwl at gmail.com> wrote:

That could be a problem for readability. I agree with the rest of what you said.

# Naveen Chawla (6 years ago)

Although it doesn't allow composition with generator functions like the composition proposal does, otherwise it's a pretty good solution.

My only concern with pipeline is that since it offers a different way of calling functions than the () syntax, it can lead to mixed and hence slightly more confusing code when both () and |> are used. For example

multi arg and no-arg functions would still use (), and single arg functions may or may not use |> depending on whether or not they may

prospectively use a pipeline. The composition operator doesn't supersede the () syntax in any context, and so it could be argued it would lead to more consistent, more readable code.

# Isiah Meadows (6 years ago)

Just thought I'd point out that the proposal itself entertains the possibility of a corresponding composition proposal 1. Also, in my proposal, one of my "potential expansions" 2 would open a generic door for "lifting" over a type, addressing the concern of extensibility. (It's not ideal, and I just filed an issue in my repo for that, but that's orthogonal.)

Isiah Meadows me at isiahmeadows.com

Looking for web consulting? Or a new website? Send me an email and we can get started. www.isiahmeadows.com

# kai zhu (6 years ago)

my vote is for neither. exactly what industry painpoint or problem-space do either of these proposals solve?

rather, they compound an existing industry painpoint; where ocd-programmers have problems in deciding-and-choosing which es6 style/design-pattern to employ and stick with before coding even begins. many of us wish there were less choices, like python (and a more assertive tc39 that makes clear certain proposals are productivity-negative and not open for debate) so we could get on with the actual coding-part.

from a senior-engineer / technical-manager perspective, it also doesn't help in managing an entire web-project; comprised of dozens of sub-components that you didn't all write yourself; and having to context-switch for each sub-component's quirky es6/es7/es8/es9 style-guide/design-pattern.

# Peter Jaszkowiak (6 years ago)

Oh please,

This is an alternative syntax that's very useful for many people. If you want too simplify syntax yourself you can use a linter to disable alternatives.

# kai zhu (6 years ago)

@peter, put yourself in the shoes of a senior-programmer responsible for overseeing an entire web-project. the project is @ the integration-stage and you're busy debugging an async timeout/near-timeout bug preventing the frontend from talking to the backend (which btw, is one of the most common integration/qa javascript-bugs).

while trying to figure out what's causing the timeout-issue, you're debugging i/o code with operators that look like this:

const h = ? |> f |> g;

maybe it is useful for the small-picture sub-problem you were originally trying to solve. but now that you're a bigger-fish with bigger-picture integration i/o issues, doesn't this look alot like technical-debt that no one will have a clue how to debug once a month or two has passed?

# Peter Jaszkowiak (6 years ago)

Personally, I'd push my subordinates to learn this new syntax. But if you dislike it, you can blacklist it in your linter: it's one of the main features of a linter.

# Terence M. Bandoian (6 years ago)

In my opinion, one of the more significant advances in the C programming language was the increase in the maximum length of identifiers. To me, this translates to "less cryptic is better".

-Terence Bandoian

# Jordan Harband (6 years ago)

As someone who does wear the shoes of a senior programmer responsible (along with my team) for overseeing a very very large web project, the super trivial and easy answer to this is "use a linter" - eslint can be configured to restrict any syntax you like, and since surely your CI process is already gating any merges, so too can the linter be used to gate merges, which will prevent anyone from any using any syntax you deem unclean.

Tons of new syntax can be added to JavaScript forever and it need not have a single bit of impact on any of your project's code except a few lines in your eslint configuration.

# Mark Miller (6 years ago)

On Mon, Mar 12, 2018 at 11:33 PM, Jordan Harband <ljharb at gmail.com> wrote:

As someone who does wear the shoes of a senior programmer responsible (along with my team) for overseeing a very very large web project, the super trivial and easy answer to this is "use a linter" - eslint can be configured to restrict any syntax you like, and since surely your CI process is already gating any merges, so too can the linter be used to gate merges, which will prevent anyone from any using any syntax you deem unclean.

Tons of new syntax can be added to JavaScript forever and it need not have a single bit of impact on any of your project's code except a few lines in your eslint configuration.

Hi Jordan, while I agree with some of your overall point, I think this goes way too far. The larger the language, and the more diversity there is in which subset one shop chooses vs another, the more we loose the benefits of having many developers use a common language. No one shop writes all the JS they use. They use libraries written by others whose lint rules are different. They hire programmers from other shops. They read and post to stackOverflow, etc.

Much better is for the language to omit as much as possible, keeping it small. I am glad my "Tragedy of the Common Lisp" post is so widely cited and appreciated. Later in that thread, at esdiscuss.org/topic/the-tragedy-of-the-common-lisp-or-why-large-languages-explode-was-revive-let-blocks#content-22 I state a hierarchy of different parts of a language with different pressures towards minimality:

the force of my [minimality] point gets weaker as we move from core

language to standardizing libraries. The overall standard language can be seen as consisting of these major parts:

  • fundamental syntax -- the special forms that cannot faithfully be explained by local expansion to other syntax

  • semantic state -- the state than computation manipulates

  • kernel builtins -- built in library providing functionality that, if it were absent, could not be provided instead by user code.

  • intrinsics -- libraries that semantic state or kernel builtins depend on. For example, with Proxies, one might be able to do Array in user code. But other kernel builtins already have a dependency on Array specifically, giving it a privileged position over any replacement.

  • syntactic sugar -- the syntax that can be explained by local expansion to fundamental syntax.

  • global convenience libraries -- could be implemented by unprivileged user code, but given standard global naming paths in the primordial global namespace.

  • standard convenient library modules

I have listed these in order, according to my sense of the costs of growth and the urgency for minimalism. For all of these we still need to exercise discipline. But it is only for the last one that we should consider growth of absolute size to be unbounded; restricting ourselves only to the rate of growth as we wait for candidates to prove themselves first by the de facto process. Ideally, TC39 should stop being the bottleneck on the last bullet anyway, as external de facto and de jure processes should be perfectly capable of independently arguing about and evolving standard convenience modules.

Although syntactic sugar is low on the list, it is still costly and best avoided when there's no compelling need. "Just use a linter" is not a panacea.

# Jordan Harband (6 years ago)

All good points :-) I wasn't suggesting there's no cost to adding syntax to a language; I was suggesting that kai's blanket "please don't add new syntax because I don't want to have to deal with it in my project" isn't a useful objection on its own.

# Alexander Jones (6 years ago)

Straw man. The problem is variables named h, f and g, not the use of a composition operator.

# kai zhu (6 years ago)

On Mar 13, 2018, at 10:27 PM, Michael J. Ryan <tracker1 at gmail.com> wrote:

I'm jumping in late here, but being in that role, I can tell what it's doing mostly by looking at it... It looks like and is a pipeline. I actually find that less confusing than deeply nested calls...

a(b(c(d(e), f)))

which I've seen in real code... vs

e |> d |> (r => c(r,f)) |> b |> a

Which is a bit easier to reason with even if a little more verbose. Each step is in order instead of nesting which is reverse order.

@michael, i would argue a simple, magic-free, es5 recursive-callback of the following form is the easiest to read / debug / set-breakpoints-with:

var callbackState, recursiveCallback;
recursiveCallback = function (error, data) {
    // catch-all error-handler
    if (error) {
        ...
        return;
    }
    callbackState += 1;
    console.error('recursive-callback at case ' + callbackState);
    switch (callbackState) {
    case 1:
        dd(ee, recursiveCallback);
        break;
    // combine data with ff
    case 2:
        cc(data, ff, recursiveCallback);
        break;
    case 3:
        bb(data, recursiveCallback);
        break;
    case 4:
        aa(data, recursiveCallback);
        break;
    }
};
callbackState = 0;
recursiveCallback();

here's the full, standalone, working example for your use-case (that’s both browser and nodejs compatible) with attached screenshot of it running in browser (and another one showing how easy it is to step-through and debug with only a single breakpoint).

/*jslint
    bitwise: true,
    browser: true,
    maxerr: 8,
    maxlen: 256,
    node: true,
    nomen: true,
    regexp: true,
    stupid: true
*/
'use strict’;
var aa, bb, cc, dd, ee, ff, callbackState, recursiveCallback;
ff = 'goodbye world!';
ee = 'hello world!';
dd = function (data, recursiveCallback) {
    console.log('dd(ee) - simulating 300ms io-request with data=' + JSON.stringify(data) + ' ...');
    setTimeout(recursiveCallback, 300, null, data);
};
cc = function (data, data2, recursiveCallback) {
    console.log('cc(dd(ee), ff) - simulating 200ms io-request with data=' + JSON.stringify(data) + ' and data2=' + JSON.stringify(data2) +  ' ...');
    setTimeout(recursiveCallback, 200, null, data + ' ' + data2);
};
bb = function (data, recursiveCallback) {
    console.log('bb(cc(dd(ee), ff)) - simulating 100ms io-request with data=' + JSON.stringify(data) + ' ...');
    setTimeout(recursiveCallback, 100, null, data);
};
aa = function (data, recursiveCallback) {
    console.log('aa(bb(cc(dd(ee), ff))) - printing data ' + JSON.stringify(data));
    // simulate error
    recursiveCallback(new Error('this is a test error'));
};



recursiveCallback = function (error, data) {
    // catch-all error-handler
    if (error) {
        console.error('error occured in recursive-callback at case ' + callbackState);
        console.error(error);
        return;
    }
    callbackState += 1;
    console.error('recursive-callback at case ' + callbackState);
    switch (callbackState) {
    case 1:
        dd(ee, recursiveCallback);
        break;
    // combine data with ff
    case 2:
        cc(data, ff, recursiveCallback);
        break;
    case 3:
        bb(data, recursiveCallback);
        break;
    case 4:
        aa(data, recursiveCallback);
        break;
    }
};
callbackState = 0;
recursiveCallback();

/*
output:

recursive-callback at case 1
dd(ee) - simulating 300ms io-request with data="hello world!" ...
recursive-callback at case 2
cc(dd(ee), ff) - simulating 200ms io-request with data="hello world!" and data2="goodbye world!" ...
recursive-callback at case 3
bb(cc(dd(ee), ff)) - simulating 100ms io-request with data="hello world! goodbye world!" ...
recursive-callback at case 4
aa(bb(cc(dd(ee), ff))) - printing data "hello world! goodbye world!"
error occured in recursive-callback at case 4
Error: this is a test error
    at aa (/private/tmp/example.js:30:23)
    at Timeout.recursiveCallback [as _onTimeout] (/private/tmp/example.js:56:9)
    at ontimeout (timers.js:393:18)
    at tryOnTimeout (timers.js:250:5)
    at Timer.listOnTimeout (timers.js:214:5)
*/
# Michael J. Ryan (6 years ago)

And only 50x the amount of code too.

# kai zhu (6 years ago)

On Mar 15, 2018, at 10:49 PM, Michael J. Ryan <tracker1 at gmail.com> wrote:

And only 50x the amount of code too.

fair enough. but lets move from small-picture toy-cases to bigger-picture integration-level ones, where non-blocking code is common.

here's a working, simple but useful 40-sloc real-world electron-script [1] which employs a single recursive-callback (function onNext()) to step-by-step screen-capture websites/demos to png. how would you re-express the linear-steps in the example into something significantly more readable with function-composition or pipeline-operators?

/*
 * screen-capture.js
 *
 * this electron-script will screen-capture the website with the given url in the commandline
 *
 * exsmple usage:
 *    $ electron screen-capture.js https://www.pinterest.com/
 *
 * output:
 *    case 1: wait for electron to init
 *    case 2: open url https://www.pinterest.com/
 *    case 3: wait 5000ms for webpage to render
 *    case 4: screenshot webpage
 *    case 5: save file electron.screenshot.png
 *    case 6: exit electron
 */

/*jslint
    bitwise: true,
    browser: true,
    maxerr: 8,
    maxlen: 96,
    node: true,
    nomen: true,
    regexp: true,
    stupid: true
*/
(function () {
    'use strict';
    var options, modeNext, onNext;
    modeNext = 0;
    onNext = function (data) {
        modeNext += 1;
        switch (modeNext) {
        case 1:
            console.log('case ' + modeNext + ': wait for electron to init');
            // wait for electron to init
            require('electron').app.once('ready', onNext);
            break;
        case 2:
            console.log('case ' + modeNext + ': open url ' + process.argv[2]);
            // init options
            options = { frame: false, height: 768, width: 1024, x: 0, y: 0 };
            // init browserWindow;
            options.BrowserWindow = require('electron').BrowserWindow;
            options.browserWindow = new options.BrowserWindow(options);
            // goto next step when webpage is loaded
            options.browserWindow.webContents.once('did-stop-loading', onNext);
            // open url
            options.browserWindow.loadURL(process.argv[2]);
            break;
        case 3:
            console.log('case ' + modeNext + ': wait 5000ms for webpage to render’);
            // wait 5000ms for webpage to render
            setTimeout(onNext, 5000);
            break;
        case 4:
            console.log('case ' + modeNext + ': screenshot webpage');
            // screenshot webpage
            options.browserWindow.capturePage(options, onNext);
            break;
        case 5:
            console.log('case ' + modeNext + ': save file electron.screenshot.png');
            // save screenshot
            require('fs').writeFile('electron.screenshot.png', data.toPng(), onNext);
            break;
        case 6:
            console.log('case ' + modeNext + ': exit electron');
            // exit
            process.exit(0);
            break;
        }
    };
    onNext();
}());

[1] kaizhu256/node-electron-lite#quickstart-screenshot-example, kaizhu256/node-electron-lite#quickstart-screenshot-example

# Michael J. Ryan (6 years ago)

Does the existence of a pipeline operator stop such code from working?

Frankly, I'd probably structure your example more as an async function...

# kai zhu (6 years ago)

@michael, integration-level javascript rarely deals with blocking-code. its mostly tedious-work debugging and “fixing” endless piles of async-io timeout issues, without the help of a stack-trace :`( and the term “fixing” is oftentimes euphemism for “rewriting the carefully architected backend into something simpler-and-dumber, because it turns out to have too many timeout issues during integration”.

you can’t seriously call yourself a senior-developer if you insist on using irrelevant blocking-code design-patterns like function-composition or pipeline-operators at such a level, which mostly gets in the way of your debugging and “fixing" tasks. and using these features at the lower library-level only reinforce blocking-code-mindset bad-habits that will set you up for failure, when you're promoted to take on integration-level responsibilities.

btw, 2 of the async-steps in the example provided requires listening to triggered-events to proceed. subjectively, the stack-overflow workaround i found for async/await to handle events [1] looks a bit more hacky and less clean than my example:

async function fn() {
  await a();
  await b();
  const [{data}, cResult] = await Promise.all([
    new Promise(resolve => thing.once('myEvent', resolve)),
    c()
  ]);
  return data;
}

but if you think you can create cleaner-code than what i posted using async/await, then feel free to try.

-kai

[1] stackoverflow.com/questions/43084557/using-promises-to-await-triggered-events, stackoverflow.com/questions/43084557/using-promises-to-await-triggered-events

# Scott Sauyet (6 years ago)

I would probably structure this so differently that it's hard to even say where the changes should be. I don't know the Electron API, so the following is almost certainly wrong, but it should demonstrate the basic ideas:

const promisify = require('some-promisify-function');
const BrowserWindow = require('electron').BrowserWindow;
const options = { frame: false, height: 768, width: 1024, x: 0, y: 0 };

const delay = time => value => new Promise(resolve =>
  setTimeout(() => resolve(value), time)
)

const loadElectron = () => {
  console.log('wait for electron to init');
  return promisify(require('electron').app.once)('ready');
}

const open = () => {
  const url = process.argv[2];
  console.log(`open url ${url}`);
  window = new BrowserWindow(options);
  window.loadURL(url);
  return promisify(window.webContents.once)('did-stop-loading')
    .then(() => window);
}

const capture = window => {
  console.log('screenshot webpage');
  return promisify(window.capturePage)(options);
}

const save = screenshot => {
  console.log('save file electron.screenshot.png');
  return promisify(require('fs').writeFile)('electron.screenshot.png',
data.toPng())
}

const exit = () => {
  console.log('exit electron');
  process.exit(0);
}

loadElectron()
  .then(open)
  .then(delay(5000))
  .then(capture)
  .then(save)
  .then(exit)

And with one of the pipeline proposals, the final bit would look something like this:

await loadElectron()
  |> await open
  |> await delay(5000)
  |> await capture
  |> await save
  |> exit

I have no ideas if we could find one promisify function that could handle the various parts of the Electron API, but we could certainly manage this with individual wrappers if necessary.

I personally find this style much more readable.

I consider myself a senior-level developer, and I use Ramda's pipe and compose [1] all the time. While I don't have particularly strong feelings on the pipeline proposals, suggesting that they are only for toy problems seems absurd to me.

[1]: ramdajs.com/docs/?pipe, ramdajs.com/docs/?compose

# Terence M. Bandoian (6 years ago)

When "features" are added to the language, developers have to learn them. Either that or risk being relegated to second class status. That means more time learning about and testing and sorting out support for new features and less time actually developing an application. I like the idea of "keeping it small". To me, the ideal is a balance of simple and powerful.

-Terence Bandoian

# Jordan Harband (6 years ago)

Learning is a continuing requirement with or without new features in the language; any one feature not added to the language tends to mean you'll have to learn about more than one userland solution to that problem. Obviously there's a cost to adding anything to the language - but there's a cost to not adding things too - and in no case are you afforded the luxury of "no more learning".

# Terence M. Bandoian (6 years ago)

That's very true. However, every new feature is an added cost to the developer that can't be wished away with a linter.

-Terence Bandoian

# Michael J. Ryan (6 years ago)

Nobody is wishing away anything with a linter. The linter can only enforce a choice not to use a given language feature.

In any case, I feel this syntax is very valuable, fairly obvious in use, and similar to use in other languages.

Pipeline/composition are important features, touched on by several user libraries, none of which are as clean or obvious as the syntax additions proposed.

# Terence M. Bandoian (6 years ago)

The point is that there is an unavoidable cost to the developer when features are added to the language. My apologies if that wasn't clear.

-Terence Bandoian

# Bob Myers (6 years ago)

The point is that there is an unavoidable cost to the developer when features are added to the language. My apologies if that wasn't clear.

You don't really need to argue that there is an unavoidable cost to new features. Few would disagree. And this cost is already taken into account in discussions of new features.

Your argument seems to actually be that such cost should be weighted more heavily.

Opinions will differ on that point, of course, but I would say that people arguing for a heavy weighting of language complexity costs are underestimating the audience (as well as often overestimating those costs). Developers are people who have learned new languages, frameworks, and libraries their entire lives, and are probably learning new features in their existing stack as I write this. They ENJOY that. That's part of the job description. If you actually want to freeze time and stop the evolution of technology, of course, you'd have to not only stop new JS developments, but also new Web API and/or CSS features, since hey, those also can confuse people! If I am a manager, and my developers are incapable or unwilling to learn how to use these evolutionary improvements in technology, then I should replace them, or maybe I myself can find a new job at a company which is still writing some good old ASP+jQuery pages..

Bob

# Terence M. Bandoian (6 years ago)

The point is that there is an unavoidable cost to the developer

when features are added to the language.

My apologies if that wasn't clear.

You don't really need to argue that there is an unavoidable cost to new features. Few would

disagree. And this cost

is already taken into account in discussions of new features.

The point was made in reference to the assertion that new features may be linted away with the implication that doing so essentially eliminates the cost to the developer. I've seen that same assertion (in different forms) a number of times on this list but don't believe it to be true.
Apparently, neither do you.

Your argument seems to actually be that such cost should be weighted

more heavily.

Not really.

Opinions will differ on that point, of course, but I would say that

people arguing for a heavy weighting of language

complexity costs are underestimating the audience (as well as often

overestimating those costs). Developers are

people who have learned new languages, frameworks, and libraries their

entire lives, and are probably learning

new features in their existing stack as I write this. They ENJOY that.

That's part of the job description. If you actually

want to freeze time and stop the evolution of technology, of course,

you'd have to not only stop new JS developments,

but also new Web API and/or CSS features, since hey, those also can

confuse people! If I am a manager, and my

developers are incapable or unwilling to learn how to use these

evolutionary improvements in technology, then I

should replace them, or maybe I myself can find a new job at a company

which is still writing some good old

ASP+jQuery pages..

I didn't find this useful.

-Terence Bandoian