ECMA-262 purview? was re: FFT module
On May 21, 2012, at 4:19 AM, Wes Garland wrote:
This thread brings up an interesting question: Once we have modules, should we have a place other than ECMA-262 to define standard host-environment libraries that ship with browsers?
Like the DOM?? ;-)
We are already working on one example, the Internationalization API. But note that is being developed so that it is not specifically browser dependent. So far, there have only been a few issues that have come up that potentially relate to browser integration and those have generally been bypassed as being out of scope.
Using that as an example, I can propose a couple criteria for which libraries might be within the prevue of TC39:
- it requires extensions to the base language syntax or semantics or core libraries
- it is of general utility in a variety of hosting environment, not just the browser
- it must be "baked in" to an implementation
- it raises significant interoperability issues that are solvable at the specification level
- TC39 has the appropriate domain expertise or experts are willing to join and participate in TC39
Also, we want to discourage libraries from depending upon "host object" semantic extensions permitted by ES <=5.1. If a library thinks it needs them it probably needs to have T39 involved.
Every library wouldn't match each criteria, but the more the better.
Once we have modules, should we have a place other than ECMA-262 to define standard host-environment libraries that ship with browsers?
Like the DOM?? ;-)
..criteria for which libraries might be within the prevue of TC39:
- it requires extensions to the base language syntax or semantics or core libraries
- it is of general utility in a variety of hosting environment, not just the browser
- it must be "baked in" to an implementation
- it raises significant interoperability issues that are solvable at the specification level
- TC39 has the appropriate domain expertise or experts are willing to join and participate in TC39
This reminds me of a different topic: foreign function interface.
Your first reaction might be: ES can't/shouldn't have any FFI!
In practice, FFI's exist in various forms already. Chapter 15 and DOM functions spring into existence seemingly out of nowhere (there is no ES code that says "there should be a 'createElement'", no standard prelude that says "'slice' exists and works on Array-like things"). Much of feature-testing tries to secure the gaps created by implicit FFI.
Projects needing device access from ES found work-arounds that support general, though not necessarily efficient, foreign function access. Things like Phonegap are entirely based on such work-arounds.
JS implementers experimenting with device access just provide foreign functions implicitly, implementation-specific.
If ES had a foreign function interface (FFI) to specify how foreign functionality could be imported into ES code, that might:
1 help to document current base functionality in code (there could be standard libraries listing "pre-defined" functions imported from the host environment)
2 provide a window for optimized, cross-browser implementation of device access, without the work-arounds currently employed in Phonegap/Cordova and related projects
Roughly, there would be ES source syntax for linking an ES identifier to foreign code, and ES engine support for calling that foreign code whenever the ES identifier is called in ES code.
Thoughts? Claus
This reminds me of a different topic: foreign function interface. [...] Thoughts?
- I'm not sure this belongs in ES
- If TC-39 starts looking at this, I would like to contribute somehow. I have significant experience in this area, and can share knowledge gleaned from an ES-centric implementation which is not related to Python's c-types.
Claus Reinke wrote:
1 help to document current base functionality in code (there could be standard libraries listing "pre-defined" functions imported from the host environment)
We don't have one notion of "foreign" in the real world. Nations and languages differ in vital ways. I'm reminded of Solzhenitsyn:
"In recent times it has been fashionable to talk of the levelling of nations, of the disappearance of different races in the melting-pot of contemporary civilization. I do not agree with this opinion, but its discussion remains another question. Here it is merely fitting to say that the disappearance of nations would have impoverished us no less than if all men had become alike, with one personality and one face. Nations are the wealth of mankind, its collective personalities; the very least of them wears its own special colours and bears within itself a special facet of divine intention."
Anyway, back to code ;-).
Different native-code compilers and OSes (not always tied, btw) will choose different FFIs, depending on local macro- and micro-economics. Even today we have a choice between calling conventions on Windows (not always obvious; Rust has changed at least once in its young life).
The OS and compiler release schedules are decoupled from browser and other JS engine-bearing software cycles.
FFIs are inherently unsafe if not crippled or lumbered with a complex research project such as NaCl or Xax.
So I doubt TC39 would ever agree on a single FFI.
2 provide a window for optimized, cross-browser implementation of device access, without the work-arounds currently employed in Phonegap/Cordova and related projects
You seem to have missed the work in the last 8 months in the W3C DAP and WebAPI groups:
brendaneich.com/2012/02/mobile-web-api-evolution
Gecko pioneered most of these for B2G but Samsung is getting WebKit patches in. FFIs are not the way to integrate device access, note well. We need the web JS security model, appropriately extended.
Different native-code compilers and OSes (not always tied, btw) will choose different FFIs, depending on local macro- and micro-economics. Even today we have a choice between calling conventions on Windows (not always obvious; Rust has changed at least once in its young life).
That is all within the purview of a standard FFI to address. For instance, the Haskell FFI has foreign import and export declarations that include calling conventions, and I've seen both ccall and stdcall in practice, depending on the native library and OS in question.
Haskell 2010> Chapter 8 - Foreign Function Interface
www.haskell.org/onlinereport/haskell2010/haskellch8.html#x15-1490008
Haskell code imports a Haskell library interface, it is up to the build phase of a library with foreign declarations to connect Haskell calls to native calls, sorting out the OS dependencies.
Sometime around 1997, I worked on connecting Haskell and Java code via their languages' native interfaces, before they arrived at today's standard FFIs. I found that the differing implementations (bytecode compiler/interpreter, native compiler) had come up with roughly similar solutions to similar problems.
The standardization processes merely removed accidental differences, and ironed out the kinks, based on practical experience. It also reduced the need for pre-processors that had been used to retarget a single Haskell code base to different implementation-specific FFI approaches.
Most high-level languages seem to have some form of FFI (SML, Java, ..).
The OS and compiler release schedules are decoupled from browser and other JS engine-bearing software cycles.
That's the whole idea of a standard FFI: it allows to specify a stable interface, so that clients (here JS code running on JS engines) and providers (here their host platform environments) can evolve independently, without breaking code.
FFIs are inherently unsafe if not crippled or lumbered with a complex research project such as NaCl or Xax.
Safety is indeed a major FFI issue, and the dynamic and pervasive nature of JS environments makes it critical.
However, we have a whole ecosystem of JS code that relies on webviews nested in native code, where the native code freely re-interprets webview actions to trigger arbitrary native actions and to feed back their results to the webview.
JS FFIs already exist in practice. One point of a standard ES FFI would be to improve and standardize the safety aspects of JS FFIs.
2 provide a window for optimized, cross-browser implementation of device access, without the work-arounds currently employed in Phonegap/Cordova and related projects
You seem to have missed the work in the last 8 months in the W3C DAP and WebAPI groups:
brendaneich.com/2012/02/mobile-web-api-evolution
Gecko pioneered most of these for B2G but Samsung is getting WebKit patches in. FFIs are not the way to integrate device access, note well. We need the web JS security model, appropriately extended.
I would indeed expect a JS FFI to include a security model as a central part of the spec.
Note that I'm not talking about, what, the third? or fourth? attempt to provide a mobile device api (Nokia phones provided device access before any of these "standard" device API efforts started), but about a standard language form in which such device apis could be specified.
Since this appears to be a point of confusion, let me restate:
-
tc39 would standardize an FFI as part of the ES language design, it would not specify any device APIs
-
phonegap, .., device and engine vendors would use the standard ES FFI to specify and implement their device APIs; there would still be different device APIs, and there would still be a need to standardize device APIs, outside tc39, but using the ES FFI
-
JS engine implementors might find it useful to use the ES FFI to connect to native host functionality required in the ES spec
Claus
Claus Reinke wrote:
Different native-code compilers and OSes (not always tied, btw) will choose different FFIs, depending on local macro- and micro-economics. Even today we have a choice between calling conventions on Windows (not always obvious; Rust has changed at least once in its young life).
That is all within the purview of a standard FFI to address. For instance, the Haskell FFI
You can stop right there. We have one major Haskell implementation. "Standardizing" (I'll use your scare quotes cited below, and more fairly) an FFI for a single-implementation language "spec" is a different and far easier problem than trying to specify and agree on a single FFI for all JS engines in top N browsers today!
Most high-level languages seem to have some form of FFI (SML, Java, ..).
SML is also effectively single-implementation as far as an FFI "standard" goes. For ES4 we used SML-NJ and dherman experimented with Milton but they don't implement the same "SML".
If by FFI you mean the JNI, Java is effectively single-implementation too, because it was single-implementation when JNI was standardized.
I recall the early days, competition between what became JNI and Netscape's JRI. By the time IBM and others tried advancing different Java VMs, JNI had won. No such happy history with a JS FFI (of course similar get-on-first game theory helped JS to become a standard).
I'm not saying it cannot be done. I'm saying the odds of success are vanishingly small. The time to try this is when a market power or monopoly dominates the web, but even then, the security research required to restore safety may be too hard to yield practical (usable, sound) solutions, even when imposed by a monopoly. Remember Active X and its trust zones?
The OS and compiler release schedules are decoupled from browser and other JS engine-bearing software cycles.
That's the whole idea of a standard FFI: it allows to specify a stable interface, so that clients (here JS code running on JS engines) and providers (here their host platform environments) can evolve independently, without breaking code.
You're missing my point: the OS and compiler vendors are not Ecma TC39, and they do not want to be constrained by JS's FFI this way.
FFIs are inherently unsafe if not crippled or lumbered with a complex research project such as NaCl or Xax.
Safety is indeed a major FFI issue,
Glad I brought it up! :-P
Seriously, another difference that you've neglected is the server-side or single-machine/admin nature of Haskell, SML, Java, etc. (ignoring research projects). A standard FFI in JS callable from the Web in the top N (4 or 5) browsers? That is a security nightmare on its face.
Security needs to be addressed up front, yet your first message did not even mention the word.
and the dynamic and pervasive nature of JS environments makes it critical. However, we have a whole ecosystem of JS code that relies on webviews nested in native code, where the native code freely re-interprets webview actions to trigger arbitrary native actions and to feed back their results to the webview.
Are you referring to so-called "hybrid" apps on mobile devices? If so, the webview's embedding API is much safer than a raw JS FFI. There's no comparison.
If you mean hybrid apps contain native code that can call any loadable library, of course that's an unsafe set of APIs. More below, but observing that does not make the case for a standard JS FFI.
JS FFIs already exist in practice. One point of a standard ES FFI would be to improve and standardize the safety aspects of JS FFIs.
There is no de-facto standard "webview FFI". Any "webview" embedding API is not a general FFI. You're abusing the "FFI" TLA and slapping it on a great many things to try to equate them all, when in practice those things differ in kind as well as degree.
Of course, hybrid and fully-native web apps are insecure. Some vendors claim to curate a single app-store to uphold quality standards including security. Others don't even review what's in their store. It's a mess. This is not a recommendation!
Let's get concrete in terms of JS engines and their FFIs.
If we take just the top open source engines' FFIs, we would have a smaller target to try to standardize, but again that leaves out too many players, and it defers security until "later", trying to enforce safety properties on top of an inherently unsafe API. And we still face huge diversity in, e.g., GC API details (exact vs. conservative vs. ref-counting or a mix).
With Emscripten and LLJS we're taking a different route: build on the browser security model by compiling low-level JS using typed arrays and more extensions in the future, but never throwing safety out and trying to restore it later.
2 provide a window for optimized, cross-browser implementation of device access, without the work-arounds currently employed in Phonegap/Cordova and related projects
You seem to have missed the work in the last 8 months in the W3C DAP and WebAPI groups:
brendaneich.com/2012/02/mobile-web-api-evolution
Gecko pioneered most of these for B2G but Samsung is getting WebKit patches in. FFIs are not the way to integrate device access, note well. We need the web JS security model, appropriately extended.
I would indeed expect a JS FFI to include a security model as a central part of the spec.
This is a non-starter. Of course, we're way past it in the W3C, whether you like it or not.
Note that I'm not talking about, what, the third? or fourth? attempt to provide a mobile device api (Nokia phones provided device access before any of these "standard" device API efforts started)
Nice dismissal of the recent standards work! Your scare quotes do not scare, though. The problem with past attempts were that they did not enlist browser vendors and get patches into the open source engines. That failed, indeed. (I won't digress on the broken economics and politics of DAP prior to last fall, or the WAC, but we've gotten past all of that.)
The specs listed in my linked blog post are moving forward. Some are in Last Call, about to become W3C RECs. It's happening, and an unsafe-up-front JS FFI, with some kind of super-science safety-property enforcing add-on is not happening.
This both a rude fact-pattern that monkey-wrenches anything like your proposal (equivalently, Google Native Client, note well), and something to study to understand the "why" as well as the "what", in order to follow the "laws of physics" constraining the possible outcomes.
, but about a standard language form in which such device apis could be specified.
When you write about "could be" or "ought", I suggest grounding what you write in what "is". There is zero chance of an FFI standard emerging in the foreseeable future in Ecma TC39.
Since this appears to be a point of confusion, let me restate:
- tc39 would standardize an FFI as part of the ES language design, it would not specify any device APIs
The confusion is all yours. No one will go the impossibly long way 'round of standardizing an unsafe FFI, then adding safety, then letting device APIs grow on it like mushrooms in the dark. That's not going to happen, and for good reasons adduced above.
- phonegap
... is becoming a shim library on top of the standard device and web APIs linked to from my blog. (From Brian Leroux of Phonegap/Adobe.)
, .., device and engine vendors would use the standard ES FFI to specify and implement their device APIs; there would still be different device APIs, and there would still be a need to standardize device APIs, outside tc39, but using the ES FFI
The shorter path is already being taken to route around the infeasibility of what you propose.
- JS engine implementors might find it useful to use the ES FFI to connect to native host functionality required in the ES spec
Lots of things "might" or "could" be useful. It's software, almost anything is possible with enough effort and (this is important) coordination or (among competing parties) cooperation. But let's be realistic.
When you propose six hard things,
- Unsafe FFI standard across JS engines, implying:
- OS vendor agreement.
- Compiler vendor agreement.
- "Security-last" restoration of important safety properties. This is a huge amount of work, technical and meta-tech/political.
- Device and other browser extension APIs built on top of the standard FFI.
- FFT libraries, etc. (this thread's subject).
the multiplication principle means the likelihood of success is a tiny fraction near 0.
Also the path length in time/money to reach anything like the device and web APIs already in Gecko and going into WebKit is too long: indefinitely many years. Shorter paths win and they've already been taken. People route around insuperable odds, especially on the Web.
A closing plea: please don't cite Haskell implementation experience freely as if it were by default relevant to JS standardization. It's not.
I don't see how a FFI could really help developers. What would be needed is a not empty intersection between developer.apple.com/library/safari/navigation, wiki.mozilla.org/WebAPI, WHAT WG, W3C, etc, unfortunately it is not the case at all
Or something like phonegap if it can become a standard (out of TC39)
The reality today is that it has became impossible to developp a web app supported by all platforms in a way that it looks like a native app (unless you get the fundings and appropriate team, or use something like phonegap, but what do you know about the future of phonegap ??). It requires too many efforts, my last projects are plenty of hack/hook (if ie, if webkit, if webkit & ios, if bada, if blackberry, if ff, if webkit & chrome, if, if, if...), and attempts to use build-in js or css features (animations, motion, ...) generally lead to very poor results (of course there are plenty of perfect examples showing that this works very well, but once the examples get modified to match reality, it never works correctly), on your way you can be blocked by some surprising implementation orientations (like bada deciding not to implement the synchronous xhr), then moving forward usually you have to decide to eliminate some platforms. At the end, finally, you can package your app for selected platforms and your are welcome to put it on hundred of stores, pay for it, wait for reviews, share revenues, maintain it, etc
B. Eich is mentionning the good unification efforts in terms of specifications but some parties are missing and some regressions did happen too.
Indeed, in all this process you might have decided to use some well known apis for your apps, and one day you might discover that the apis have been shut down or that you have to pay for it, or that you will have to pay for it, or that it will be shut down and replaced by something else less performant that you must pay too. This is not fair of course since the community did participate to build these apis and just get in return the fact that its apps will no longer work, this is exactly my case today for past projects and ongoing (now blocked) one.
Nothing to do with the initial subject ? Yes, it does, "the actual (web) world is (not) the best// of all// possible worlds" and I don't think we can afford continuing having exploding number of platforms, systems with different behaviors and no unified specs (not talking about tc39 here)
Maybe it's a dream or it's alreday too late
Le 22/05/2012 11:40, Claus Reinke a écrit :
On 22 May 2012 12:54, Aymeric Vitte <vitteaymeric at gmail.com> wrote:
I don't see how a FFI could really help developers.
FFIs are certainly helpful in many situations. I don't think this is one of them.
Let me relay a relevant xperience. We develop applications in "Server-Side JavaScript". We have an FFI that, with some magic compile-time shims and a few other tricks, lets us write close-to-the-bare-metal applications that are portable without platform detection across many operating systems -- in fact, I believe, to any conformant SUSv3 implementation. We write nearly-C-like code on Linux, Solaris, and Mac OS X, 32 and 64 bit, to give us nice JS libraries on top of the ugly, bare metal.
Our implementation is complete enough that we were able to write a complete WebSockets implementation in JS, that runs on many platforms, with no direct support in the host environment other than FFI, right down to the networking system calls, including the magic macros for select (FD_CLR et al).
Sounds great, right? Well, it is for us, but it would make a lousy direction for a standard: the resultant JS is completely non-portable to Windows. Or QNX. Or Gronch. Or a myriad of other operating systems.
And that's the real problem. We have overcome the typical "C porting" problems -- endianness, word size, whether fstat() is a function or a macro, etc..... but we're still nowhere close to being portable to where the web needs to run. If we want to run anywhere, we would need to FFI up one layer of abstraction, to something like APR or NSPR, and then guess what? We would be no better off in any way than what the standards guys have been up to -- and far, far, worse off in many.
That's why I believe TC-39 is not the right place for a JS FFI.
TC39 is obviously not the right place for FFI, but the subject can be discussed.
I and my former team did develop (modestly) quite a lot of stuff as well that could be called FFI, result : 0 in the middle/long term
Hopefully now the web gets some nice things like node.js at server side
But at device level, it should be up to vendors to propose a unified, performant and cross browser js interface, personnaly I will not go any more into the nightmare I described in my previous email, and I think less and less people will
It's a paradox, while the iphone brought light, shadow follows now with every vendor/platform thinking they can specify whatever they like, and others thinking they can shut down whatever they like (tip : the name starts by a G)
Le 22/05/2012 19:17, Wes Garland a écrit :
This thread brings up an interesting question: Once we have modules, should we have a place other than ECMA-262 to define standard host-environment libraries that ship with browsers?