host object interface and host object plugins (Re: ECMA-262 purview? was re: FFT module)

# Claus Reinke (13 years ago)

That is all within the purview of a standard FFI to address. For instance, the Haskell FFI You can stop right there. We have one major Haskell implementation.

You mean like there is only one major browser implementation?-)

Just because one implementation currently has the most steam, that doesn't mean that there is only one implementation to consider.

"Standardizing" (I'll use your scare quotes cited below, and more fairly) an FFI for a single-implementation language "spec" is a different and far easier problem than trying to specify and agree on a single FFI for all JS engines in top N browsers today!

When the Haskell FFI was in development, there were three Haskell implementations participating, from a bytecode compiler and interpreter written in C -aiming for small and portable implementation-, over a compiler written in Haskell -aiming to generate memory-efficient code-, to a heavily optimizing compiler written in Haskell -aiming to generate time-efficient C.

These implementations' pre-standard approaches to FFI were so different that multiple tools had sprung up to preprocess code for the implementation-specific FFIs.

Today, I have no idea how many Haskell implementations there are, but at least three of them are still in development and support the FFI. Only one implementation is in both sets, currently the most popular (which implementation is most popular has changed several times over the years, as has the backend of the one implementation that was in both sets).

I'm not saying it cannot be done. I'm saying the odds of success are vanishingly small. The time to try this is when a market power or monopoly dominates the web, but even then, the security research required to restore safety may be too hard to yield practical (usable, sound) solutions, even when imposed by a monopoly. Remember Active X and its trust zones?

The odds depend on the goals. Your arguments imply that you think of the usual FFI, giving access to anything implemented in the underlying platform that has a C interface. That would explain your safety concerns.

As a minimalistic counterpoint, consider a FFI that only provides access to functionality from some standard device API (which I'll abbreviate as sdAPI here). That is, the existing sdAPI processes still decide what is available and safe, as they do now; the FFI describes how to use such safe functionality, and how to implement access to it.

Currently, sdAPI functions are agreed on and spec-ed, then implemented, hopefully by more than one implementation. Client code knows about functionality because coders read the spec and feature test whether using the spec-ed functionality works. In other words, the defacto FFI is dynamic. Invoking sdAPI functionality might trigger permission requests, unless sufficient permissions have been obtained beforehand. Without permissions, the functionality will fail. Permissions are also dynamic.

There are other batches of non-JS functionality that JS implementations currently provide JS code access to: DOM-related in browser JS code, OS-related in server-side JS code, chapter 15-related in some JS implementations. The functionality differs and is spec-ed by different bodies, but the FFI used to access that functionality should be similar, shouldn't it?

--- host objects as ES FFI The current basis for all this in the ES spec are host objects. ES5.1 talks about host objects themselves being beyond the scope of its specification but places constraints on such host objects. These constraints comprise the current (implicit) ES FFI. There is no comment on discoverability, on permissions, there is lots of "implementation-dependent". In particular, there is the implicit expectation that there is a single monolithic host environment - to change available host functionality, one changes the host. This is the foundation for current practices wrt importing host objects into JS code.

Now, one of the cornerstones of ES6 are modules, which -among other things- try to replace one level of runtime checks with loadtime checks: if a module loads successfully, its toplevel exports will be available and are known to support the client's imports.

One may wonder whether ES6, with modules, might not improve on the host object aspects of ES5. Perhaps in such a way that if a JS module loads successfully, it is known that the host objects it tries to use/import are indeed available (permission would still be dynamic).

To throw some wild ideas in the air: one could import host objects from a Host module; one could organize the Host namespace according to specification bodies (server-side, browser, DOM, device, ..); or one could go with the traditional FFI approach that host objects need to be declared in code, where the declaration compiles only if the host object is available; then, one could provide and import modules consisting only of such host object declarations, as needed; ..

Without fixing on concrete proposals, but code might look like this:

// host modules, each under control of a spec body
// if the JS implementation has access to code implementing
// such a host module, the import succeeds; at runtime, the
// host module might implement additional permission protocols

import document from DOM
import camera from Cordova
import read from CommonJS.Filesystem.fs

or like this:

// declare expected methods in host objects
function Array.prototype.forEach(cb,th) { "use host" }

Perhaps you prefer to call it host object interface (HOI) instead of foreign function interface (FFI). The point is that there is an interface that the ES spec has something to say on, that could be improved in line with the ES5 to ES6 transition, and that could be supported in JS implementations, without tc39 having to interfere with other existing standards bodies.

The host object interface is simply the way to give JS code access to the host objects standardized by other bodies.

The OS and compiler release schedules are decoupled from browser and other JS engine-bearing software cycles.

That's the whole idea of a standard FFI: it allows to specify a stable interface, so that clients (here JS code running on JS engines) and providers (here their host platform environments) can evolve independently, without breaking code.

You're missing my point: the OS and compiler vendors are not Ecma TC39, and they do not want to be constrained by JS's FFI this way.

What makes you think that the Haskell FFI group had any influence on OS or C compiler vendors?

However, we have a whole ecosystem of JS code that relies on webviews nested in native code, where the native code freely re-interprets webview actions to trigger arbitrary native actions and to feed back their results to the webview.

Are you referring to so-called "hybrid" apps on mobile devices? If so, the webview's embedding API is much safer than a raw JS FFI. There's no comparison.

If you mean hybrid apps contain native code that can call any loadable library, of course that's an unsafe set of APIs. More below, but observing that does not make the case for a standard JS FFI.

Yes, I was referring to hybrid apps. Using the Phonegap example, they prototype device API with the express intention of their prototype implementations becoming mere shims for standard APIs as those evolve.

The case for a standard JS FFI in this context is that they have APIS that have become standard, they have implementations of those APIs on devices, and they have JS implementations on the same devices, but they have no direct/standard way of extending the JS implementations with access to those API implementations.

If there was a standard JS FFI supported by JS implementations, including a way to extend the set of host objects without rewriting the JS implementation, then that could be used instead of the workarounds that current Phonegap implementations have to use. Most likely such an in-browser host object access would be more efficient and amenable to optimizations than the workarounds.

Even if I load JS code from the web that wants to import and use device access, and I happen to have code for accessing that device feature installed, the JS code will not get to use the feature unless I enable my browser's access to the device access code.

But if I decide to enable the feature, I do not have to wait for the browser to implement it, the browser can be extended by using "host object plugins".

tc39 does not have to join/repeat the security discussions going on in other committees - it specifies the host object interface. Browser implementers support the host object interface with a host object plugin API. As a first approximation, the plugin API would have a whitelist of standards that are known to have addressed safety concerns. Users install host object plugins from standards bodies they trust, and if their browsers share that trust, the plugins can be used with the security protocols defined by those standards. Without having to update the browser.

Let's get concrete in terms of JS engines and their FFIs.

If we take just the top open source engines' FFIs, we would have a smaller target to try to standardize, but again that leaves out too many players, and it defers security until "later", trying to enforce safety properties on top of an inherently unsafe API. And we still face huge diversity in, e.g., GC API details (exact vs. conservative vs. ref-counting or a mix).

I'm not saying it would be easy. Just that it would be worthwhile. And that it could be done in small steps, starting with the current host object specification in ES5.

With Emscripten and LLJS we're taking a different route: build on the browser security model by compiling low-level JS using typed arrays and more extensions in the future, but never throwing safety out and trying to restore it later.

Javascript special position as being transferred and compiled almost on every use mean that things like LLJS hold a lot of promise: just as minimizers optimize JS source for transfer, pre-compilation to LLJS, or to an explicitly typed future ES.harmony, can help final compilation at the usage sites. I am looking forward to those developments.

But that doesn't extend the set of accessible host objects.

When you write about "could be" or "ought", I suggest grounding what you write in what "is". There is zero chance of an FFI standard emerging in the foreseeable future in Ecma TC39.

The current ES FFI is the host object specification. It "is" and it constrains/hinders current practice and experimentation with emerging standards. If you do not want to improve that, that is your decision. I am merely asking about an opportunity I see for improving the foundations of current practice.

the multiplication principle means the likelihood of success is a tiny fraction near 0.

Also the path length in time/money to reach anything like the device and web APIs already in Gecko and going into WebKit is too long: indefinitely many years. Shorter paths win and they've already been taken. People route around insuperable odds, especially on the Web.

When do you expect to see those device APIs in IE? Shorter paths win in the short term, even if only few can walk them. But if they are successful, it is possible, as you like to say, to pave those cowpaths.

For Gecko and WebKit, it may just be a different way to implement the same specs, but if all browsers support a host object interface, further API development can be decoupled from browser versions.

A closing plea: please don't cite Haskell implementation experience freely as if it were by default relevant to JS standardization. It's not.

I cite Haskell experience because that is what I happen to be most familiar with. I only cite it when it is relevant. Please don't object to Haskell experience by default just because it is Haskell. A lot of good work has gone into that language.

Some posters here are familiar with Smalltalk, some with SML, some with Scheme, some with Haskell, some with other languages - I think it is great that ES design can profit from such a broad range of experience.

Claus

>> That is all within the purview of a standard FFI to address. For 
>> instance, the Haskell FFI
> You can stop right there. We have one major Haskell implementation.

You mean like there is only one major browser implementation?-)

Just because one implementation currently has the most steam,
that doesn't mean that there is only one implementation to consider.

> "Standardizing" (I'll use your scare quotes cited below, and more fairly) 
> an FFI for a single-implementation language "spec" is a different and far 
> easier problem than trying to specify and agree on a single FFI for all JS 
> engines in top N browsers today!

When the Haskell FFI was in development, there were three Haskell
implementations participating, from a bytecode compiler and interpreter
written in C -aiming for small and portable implementation-, over a
compiler written in Haskell -aiming to generate memory-efficient
code-, to a heavily optimizing compiler written in Haskell -aiming
to generate time-efficient C.

These implementations' pre-standard approaches to FFI were so
different that multiple tools had sprung up to preprocess code for
the implementation-specific FFIs.

Today, I have no idea how many Haskell implementations there are,
but at least three of them are still in development and support the FFI.
Only one implementation is in both sets, currently the most popular
(which implementation is most popular has changed several times
over the years, as has the backend of the one implementation that
was in both sets).

> I'm not saying it cannot be done. I'm saying the odds of success are 
> vanishingly small. The time to try this is when a market power or monopoly 
> dominates the web, but even then, the security research required to 
> restore safety may be too hard to yield practical (usable, sound) 
> solutions, even when imposed by a monopoly. Remember Active X and its 
> trust zones?

The odds depend on the goals. Your arguments imply that you think
of the usual FFI, giving access to anything implemented in the underlying
platform that has a C interface. That would explain your safety concerns.

As a minimalistic counterpoint, consider a FFI that only provides access
to functionality from some standard device API (which I'll abbreviate as
sdAPI here). That is, the existing sdAPI processes still decide what is
available and safe, as they do now; the FFI describes how to use such
safe functionality, and how to implement access to it.

Currently, sdAPI functions are agreed on and spec-ed, then implemented,
hopefully by more than one implementation. Client code knows about
functionality because coders read the spec and feature test whether using
the spec-ed functionality works. In other words, the defacto FFI is dynamic.
Invoking sdAPI functionality might trigger permission requests, unless
sufficient permissions have been obtained beforehand. Without permissions,
the functionality will fail. Permissions are also dynamic.

There are other batches of non-JS functionality that JS implementations
currently provide JS code access to: DOM-related in browser JS code,
OS-related in server-side JS code, chapter 15-related in some JS
implementations. The functionality differs and is spec-ed by different
bodies, but the FFI used to access that functionality should be similar,
shouldn't it?

--- host objects as ES FFI
The current basis for all this in the ES spec are host objects. ES5.1 talks
about host objects themselves being beyond the scope of its specification
but places constraints on such host objects. These constraints comprise
the current (implicit) ES FFI. There is no comment on discoverability, on
permissions, there is lots of "implementation-dependent". In particular,
there is the implicit expectation that there is a single monolithic host
environment - to change available host functionality, one changes the
host. This is the foundation for current practices wrt importing host
objects into JS code.
---

Now, one of the cornerstones of ES6 are modules, which -among other
things- try to replace one level of runtime checks with loadtime checks:
if a module loads successfully, its toplevel exports will be available and
are known to support the client's imports.

One may wonder whether ES6, with modules, might not improve on the
host object aspects of ES5. Perhaps in such a way that if a JS module loads
successfully, it is known that the host objects it tries to use/import are
indeed available (permission would still be dynamic).

To throw some wild ideas in the air: one could import host objects from
a Host module; one could organize the Host namespace according to
specification bodies (server-side, browser, DOM, device, ..); or one could
go with the traditional FFI approach that host objects need to be declared
in code, where the declaration compiles only if the host object is 
available;
then, one could provide and import modules consisting only of such host
object declarations, as needed; ..

Without fixing on concrete proposals, but code might look like this:

    // host modules, each under control of a spec body
    // if the JS implementation has access to code implementing
    // such a host module, the import succeeds; at runtime, the
    // host module might implement additional permission protocols

    import document from DOM
    import camera from Cordova
    import read from CommonJS.Filesystem.fs

or like this:

    // declare expected methods in host objects
    function Array.prototype.forEach(cb,th) { "use host" }

Perhaps you prefer to call it host object interface (HOI) instead of foreign
function interface (FFI). The point is that there is an interface that the 
ES
spec has something to say on, that could be improved in line with the
ES5 to ES6 transition, and that could be supported in JS implementations,
without tc39 having to interfere with other existing standards bodies.

The host object interface is simply the way to give JS code access to
the host objects standardized by other bodies.

>>> The OS and compiler release schedules are decoupled from browser and 
>>> other JS engine-bearing software cycles.
>>
>> That's the whole idea of a standard FFI: it allows to specify a stable
>> interface, so that clients (here JS code running on JS engines) and 
>> providers (here their host platform environments) can evolve 
>> independently, without breaking code.
>
> You're missing my point: the OS and compiler vendors are not Ecma TC39, 
> and they do not want to be constrained by JS's FFI this way.

What makes you think that the Haskell FFI group had any influence
on OS or C compiler vendors?

>> However, we have a whole ecosystem of JS code that relies on webviews 
>> nested in native code, where the native code freely re-interprets webview 
>> actions to trigger arbitrary native actions and to feed back their 
>> results to the webview.
>
> Are you referring to so-called "hybrid" apps on mobile devices? If so, the 
> webview's embedding API is much safer than a raw JS FFI. There's no 
> comparison.
>
> If you mean hybrid apps contain native code that can call any loadable 
> library, of course that's an unsafe set of APIs. More below, but observing 
> that does not make the case for a standard *JS* FFI.

Yes, I was referring to hybrid apps. Using the Phonegap example, they
prototype device API with the express intention of their prototype
implementations becoming mere shims for standard APIs as those
evolve.

The case for a standard JS FFI in this context is that they have APIS
that have become standard, they have implementations of those APIs
on devices, and they have JS implementations on the same devices,
but they have no direct/standard way of extending the JS
implementations with access to those API implementations.

If there was a standard JS FFI supported by JS implementations,
including a way to extend the set of host objects without rewriting
the JS implementation, then that could be used instead of the
workarounds that current Phonegap implementations have to use.
Most likely such an in-browser host object access would be more
efficient and amenable to optimizations than the workarounds.

Even if I load JS code from the web that wants to import and
use device access, and I happen to have code for accessing that
device feature installed, the JS code will not get to use the feature
unless I enable my browser's access to the device access code.

But if I decide to enable the feature, I do not have to wait for
the browser to implement it, the browser can be extended by
using "host object plugins".

tc39 does not have to join/repeat the security discussions
going on in other committees - it specifies the host object interface.
Browser implementers support the host object interface with
a host object plugin API. As a first approximation, the plugin
API would have a whitelist of standards that are known to have
addressed safety concerns. Users install host object plugins
from standards bodies they trust, and if their browsers share
that trust, the plugins can be used with the security protocols
defined by those standards. Without having to update the browser.

> Let's get concrete in terms of JS engines and their FFIs.
>
> If we take just the top open source engines' FFIs, we would have a smaller 
> target to try to standardize, but again that leaves out too many players, 
> and it defers security until "later", trying to enforce safety properties 
> on top of an inherently unsafe API. And we still face huge diversity in, 
> e.g., GC API details (exact vs. conservative vs. ref-counting or a mix).

I'm not saying it would be easy. Just that it would be worthwhile.
And that it could be done in small steps, starting with the current
host object specification in ES5.

> With Emscripten and LLJS we're taking a different route: build on the 
> browser security model by compiling low-level JS using typed arrays and 
> more extensions in the future, but never throwing safety out and trying to 
> restore it later.

Javascript special position as being transferred and compiled almost
on every use mean that things like LLJS hold a lot of promise: just as
minimizers optimize JS source for transfer, pre-compilation to LLJS,
or to an explicitly typed future ES.harmony, can help final compilation
at the usage sites. I am looking forward to those developments.

But that doesn't extend the set of accessible host objects.

> When you write about "could be" or "ought", I suggest grounding what you 
> write in what "is". There is zero chance of an FFI standard emerging in 
> the foreseeable future in Ecma TC39.

The current ES FFI is the host object specification. It "is" and it
constrains/hinders current practice and experimentation with emerging
standards. If you do not want to improve that, that is your decision. I am
merely asking about an opportunity I see for improving the foundations
of current practice.

> the multiplication principle means the likelihood of success is a tiny 
> fraction near 0.
>
> Also the path length in time/money to reach anything like the device and 
> web APIs already in Gecko and going into WebKit is too long: indefinitely 
> many years. Shorter paths win and they've already been taken. People route 
> around insuperable odds, especially on the Web.

When do you expect to see those device APIs in IE? Shorter paths
win in the short term, even if only few can walk them. But if they are
successful, it is possible, as you like to say, to pave those cowpaths.

For Gecko and WebKit, it may just be a different way to implement
the same specs, but if _all_ browsers support a host object interface,
further API development can be decoupled from browser versions.

> A closing plea: please don't cite Haskell implementation experience freely 
> as if it were by default relevant to JS standardization. It's not.

I cite Haskell experience because that is what I happen to be most
familiar with. I only cite it when it is relevant. Please don't object to
Haskell experience by default just because it is Haskell. A lot of good
work has gone into that language.

Some posters here are familiar with Smalltalk, some with SML,
some with Scheme, some with Haskell, some with other languages -
I think it is great that ES design can profit from such a broad range
of experience.

Claus

# Brendan Eich (13 years ago)

Claus Reinke wrote:

That is all within the purview of a standard FFI to address. For instance, the Haskell FFI You can stop right there. We have one major Haskell implementation.

You mean like there is only one major browser implementation?-)

No, I mean the reverse. How could this have been unclear?

Just because one implementation currently has the most steam, that doesn't mean that there is only one implementation to consider.

My point is that an FFI for the single definitive implementation of a programming language is much easier to "standardize" (it is often a de-facto standard just by virtue of its sole/lead definitive implementation status) than an FFI for all browser/OS combinations in play in the market.

Claus Reinke wrote:
>>> That is all within the purview of a standard FFI to address. For 
>>> instance, the Haskell FFI
>> You can stop right there. We have one major Haskell implementation.
>
> You mean like there is only one major browser implementation?-)

No, I mean the reverse. How could this have been unclear?

> Just because one implementation currently has the most steam,
> that doesn't mean that there is only one implementation to consider. 

My point is that an FFI for the single definitive implementation of a 
programming language is much easier to "standardize" (it is often a 
de-facto standard just by virtue of its sole/lead definitive 
implementation status) than an FFI for all browser/OS combinations in 
play in the market.

/be

# Brendan Eich (13 years ago)

Claus Reinke wrote:

When you write about "could be" or "ought", I suggest grounding what you write in what "is". There is zero chance of an FFI standard emerging in the foreseeable future in Ecma TC39.

The current ES FFI is the host object specification.

Oh please. "FFI" means more than any old API or interface between foreign or "host" code and native (in the ECMA-262 sense) code. An FFI means low-level access, the ability to call C or C++ code -- for instance an FFT implementation, the topic that spawned this thread.

In a vacuous sense, yes: the DOMs are generally implemented via C++, and the ECMA-262 spec imagines their objects as "host objects". But there are tons of loopholes and massive underspecification!

Indeed you cannot build a browser in pure JS right now, since JS cannot express all the magic that real-world DOMs require. You cannot talk about all the native data types (int64 and uint64 are particularly painful gaps).

Saying the "host object specification" (more ilke "lack of specification") is an "FFI" is almost a fraud. It debases what is usually meant by "FFI" to a much less meaningful and useful notion. In practice it does not help in your quixotic hope to standardize a real FFI among all browser vendors -- which is not gonna happen any time soon, and not via Ecma or the W3C.

Claus Reinke wrote:
>> When you write about "could be" or "ought", I suggest grounding what 
>> you write in what "is". There is zero chance of an FFI standard 
>> emerging in the foreseeable future in Ecma TC39.
>
> The current ES FFI is the host object specification.

Oh please. "FFI" means more than any old API or interface between 
foreign or "host" code and native (in the ECMA-262 sense) code. An FFI 
means low-level access, the ability to call C or C++ code -- for 
instance an FFT implementation, the topic that spawned this thread.

In a vacuous sense, yes: the DOMs are generally implemented via C++, and 
the ECMA-262 spec imagines their objects as "host objects". But there 
are tons of loopholes and massive underspecification!

Indeed you cannot build a browser in pure JS right now, since JS cannot 
express all the magic that real-world DOMs require. You cannot talk 
about all the native data types (int64 and uint64 are particularly 
painful gaps).

Saying the "host object specification" (more ilke "lack of 
specification") is an "FFI" is almost a fraud. It debases what is 
usually meant by "FFI" to a much less meaningful and useful notion. In 
practice it does not help in your quixotic hope to standardize a real 
FFI among all browser vendors -- which is not gonna happen any time 
soon, and not via Ecma or the W3C.

/be