Discuss
  • Function.prototype.toString to discover function dependencies

Function.prototype.toString to discover function dependencies

# James Burke (15 years ago)

First time posting to the es-discuss list:

Over on the CommonJS list, there is a thread about how to provide a module format that works in today's browsers. The traditional CommonJS module format cannot run in the browser via script tags without a transform, and some of us want a format that will work without transforms but can be easily hand-coded.

Tom Robinson made the following suggestion, is this an acceptable use of Function.prototype.toString:

require.def(function (require, exports, module) { var foo = require('foo'), bar = require('bar'); });

Where require.def would call .toString() on the function passed to it, regexp for the require() calls, then make sure to load those dependencies before executing the function that defines the module.

So the toString() does not have to be an exact copy of the function, just enough that allows the require() calls to be found.

I am leaving out a bunch of other stuff about the browser-friendly module format (for instance there is a transform for this format to allow grouping more than one module in a file for optimized browser delivery and that form does not use .toString() and this approach would not be used on minified code), but the main question for the es-discuss list is:

Is this an acceptable use of Function.prototype.toString? What problems would there be now or in the future?

What I have found so far:

The ECMAScript 3rd and 5th edition specs say the following:

15.3.4.2 Function.prototype.toString ( )

An implementation-dependent representation of the function is returned. This representation has the syntax of a FunctionDeclaration. Note in particular that the use and placement of white space, line terminators, and semicolons within the representation string is implementation-dependent.

The toString function is not generic; it throws a TypeError exception if its this value is not a Function object. Therefore, it cannot be transferred to other kinds of objects for use as a method.

In a 2008 es-discuss thread[1], Erik Arvidsson thought the "implementation-dependent" part allowed for low-memory devices to not keep a reversible implementation in memory.

A test was run in a few browsers to test the approach[2]: modern desktop browsers (including IE 6+) give a usable toString() value, and there are many mobile browsers that also work well: Android, iOS, Windows Mobile, webOS and latest BlackBerry. So far BlackBerry 4.6 and some version of Opera Mobile may not, and there are some like Symbian and Bada that have not been tested yet.

So one issue is not universal browser support, but enough of today's browsers both on desktop and mobile that it may make sense using it going forward. While the spec technically allows an implementation to produce something that might cause the above mentioned Function toString() approach to fail, it seems in practice it may be workable.

I am a bit wary of the approach, but initial testing seems to indicate it may work, so I am looking to have an expert review done before proceeding with the approach. If any of you have other other information or concerns it would be good to know.

James

[1 ]Thread starts with this post: esdiscuss/2008-September/007632

[2] Browser test page: requirejs.org/temp/fts.html

# Erik Arvidsson (15 years ago)

As far as I remember some Opera mobile versions don't return the code in Function.prototype.toString.

A lot of JS frameworks today use the Function.prototype.toString to find references to super/base but they do fallback on slower paths if the toString method does not provide the code of the function.

# Geoffrey Sneddon (15 years ago)

On 16/09/10 04:04, Erik Arvidsson wrote:

As far as I remember some Opera mobile versions don't return the code in Function.prototype.toString.

A lot of JS frameworks today use the Function.prototype.toString to find references to super/base but they do fallback on slower paths if the toString method does not provide the code of the function.

FWIW, there should be no difference between Desktop and Mobile with Carakan.

# Mark S. Miller (15 years ago)

On Wed, Sep 15, 2010 at 5:24 PM, James Burke <jrburke at gmail.com> wrote:

First time posting to the es-discuss list:

Over on the CommonJS list, there is a thread about how to provide a module format that works in today's browsers. The traditional CommonJS module format cannot run in the browser via script tags without a transform, and some of us want a format that will work without transforms but can be easily hand-coded.

Tom Robinson made the following suggestion, is this an acceptable use of Function.prototype.toString:

require.def(function (require, exports, module) { var foo = require('foo'), bar = require('bar'); });

Where require.def would call .toString() on the function passed to it, regexp for the require() calls, then make sure to load those dependencies before executing the function that defines the module.

So the toString() does not have to be an exact copy of the function, just enough that allows the require() calls to be found.

Hi James, at < strawman:function_to_string> I've

been trying to state a usefully stronger but not impractically stronger spec. I like your usage pattern of "require" above, especially that it can be reliably recognized without a full lexing or parsing by regexp tricks, without getting misled by comments or quotes. (I try a similar but different pattern at < code.google.com/p/es-lab/source/browse/trunk/src/ses/initSES.js#276>.

I like your's better.)

Given source code, of course, either recognition trick works. Given only meaning preserving source code, which is the most I was willing to propose on that strawman page, these recognition tricks fail < strawman:function_to_string#discussion>.

Would we be willing to specify enough fidelity with the original source code that your trick would work? I don't know. Perhaps AST preserving? Perhaps preserving of some abstraction over equivalent ASTs? I would like to clean up Function.prototype.toString for ES-Harmony. Opinions?

# Leo Meyerovich (15 years ago)

Perhaps worth considering for both this proposal and 'use strict' is the common (and surprising to me, actually!) use case of code partitioning tools. I suspect they might break some assumptions about how code is split into source files etc.

These tools, to minimize page/component load requirements (e.g., for large sites or bandwidth limited devices), essentially do fine grained partitions of source (generally at the function level), delaying their individual load for a more optimal order. The one I'm most familiar with, Doloto, replaces functions with stubs; the first invocation performs an on-demand load. I believe GWT and many other frameworks have something here as well.

So: would usage of 'require' and 'use strict' break these tools? Is there a feasible evolution path for updating these tools to not break use of these features? Perhaps these scenarios have already been considered -- if not, I suspect they might cause problems.

,

  • Leo

//also, for this particular proposal, I'd be a little worried about semantics -- e.g., should a 'require' call nestled within a conditional throw an exception?

# Mark S. Miller (15 years ago)

you might want to look at < groups.google.com/group/google-caja-discuss/browse_frm/thread/8df3d6419577c368?hl=en&tvc=1>.

As I've discussed with Ben Livshits, we gave up on the approach because it requires a synchronous xhr.

# James Burke (15 years ago)

On Sun, Sep 19, 2010 at 12:41 PM, Mark S. Miller <erights at google.com> wrote:

Given source code, of course, either recognition trick works. Given only meaning preserving source code, which is the most I was willing to propose on that strawman page, these recognition tricks fail strawman:function_to_string#discussion. Would we be willing to specify enough fidelity with the original source code that your trick would work? I don't know. Perhaps AST preserving? Perhaps preserving of some abstraction over equivalent ASTs? I would like to clean up Function.prototype.toString for ES-Harmony. Opinions?

If "meaning preserving source code" had qualifications like:

  1. If line returns are removed, comments must be removed.
  2. String literals need to stay string literals.
  3. The number of function arguments must stay the same, and function argument names and their use within the function must be preserved.

that might be enough to give implementors enough flexibility but still allow the require dependencies to be found? Although I am not a language expert, maybe there are still holes with those qualifications.

.#3 could be relaxed to "The number of function arguments need to stay the same. The name can change, but code inside the function needs to change to use the new name, and the use of those names needs to be preserved".

That would require a bit more work from my side. If the toString converted the function to something like function(r,e,m) {var f=r('foo'),b=r('bar');}, then it would require more work to pull out the r function arg. A bit less efficient, but doable. Probably not too bad given that this should just be used in a "source code loading" form, and not something that is used after optimization. For optimization it will be common to place many require.def calls into one file, so names for the modules and the dependencies can be pulled out from the source and injected into the optimized require.def call.

But if it is all the same to implementors, then I prefer the first, stronger wording of #3.

I went ahead and put support for this require.def toString approach into the RequireJS implementation to try it out. There are still some edges to clean up, but we will see how it goes. Right now I assume the above qualifications on the toString() form. This seems to work out, but more testing is needed. For the apparently small number of implementations that do not preserve some usable string for Function.prototype.toString, then the advice will be to use an optimized/built version of the code that has pulled the dependencies out already, or to use use the other syntax supported by RequireJS:

require.def(['foo', 'bar'], function(foo, bar){ //return a value to define the module. Alternatively, //as for 'exports' as a dependency in the dependency //array and specify it as a matching function arg //if dealing with a rare circular dependency failure case. return function(){}; });

James

# Brendan Eich (15 years ago)

Lars Hansen, back a few years and based on his Opera experience developing Futhark, said that EBay (at least -- possibly other sites) used Function.prototype.toString to decompile a function, edit the source with string and regexp hacks, and recompile with Function (or possibly eval), for self-modifying code in JS.

This means web compatibility rules out dropping high-fidelity toString on function objects just because the implementation is on a memory-poor device. If too memory-poor, no EBay for you! :

# Leo Meyerovich (15 years ago)

Do you mean code splitting is not a valid scenario and should be considered buggy? It seems to be a common technique in the web community. For example, Google supports an explicit form of it in its own framework: code.google.com/webtoolkit/doc/latest/DevGuideCodeSplitting.html . I'm actually not even the one originating the question -- someone else asked me it and I realized I didn't know the answer!

Is www.ecma-international.org/publications/files/ECMA-ST/ECMA-262.pdf the right place to look into this question or is there a more recent proposal on 'use strict'? I also see " doku.php?id=proposals:strict_and_standard_modes&s=use+strict " and John Resig's informal blog post about it.

,

# Chuck Jazdzewski (15 years ago)

#3 could be relaxed to "The number of function arguments need to stay the same. The name can change, but code inside the function needs to change to use the new name, and the use of those names needs to be preserved".

This weaker 3 runs into problems if the new name is a free variable in the body of the function (alpha-renaming name capture) as well as potentially breaking direct (closure capture) calls to eval(). Both would need to be excluded to allow argument renames, such as "...and the function does not contain a direct call to eval. The new name must not be the same as the name used in any other variable references in the function body." The last restriction is a bit stronger than necessary but weaker forms are more complicated without being more useful.

Chuck.

# James Burke (15 years ago)

On Wed, Sep 15, 2010 at 5:24 PM, James Burke <jrburke at gmail.com> wrote:

Is this an acceptable use of Function.prototype.toString? What problems would there be now or in the future?

For the record, the PlayStation 3's browser returns a Function.prototype.toString like:

function (require, exports, modue) { /-2134160600/ }

I went ahead and added support for this toString approach in the latest RequireJS release though, so it can be used in real projects, and to see if it helps smooth out the CommonJS/browser bumps.

For platforms that have give unusable Function.prototype.toString() values, there are two options to get the code to run:

  1. use the RequireJS optimization tool that will pull out the dependencies and normalize the modules to the base module format that has dependencies specified as an array outside the module definition function.

  2. code in the base module format.

James

# Juriy Zaytsev (15 years ago)

On Wed, Sep 29, 2010 at 3:13 PM, James Burke <jrburke at gmail.com> wrote:

On Wed, Sep 15, 2010 at 5:24 PM, James Burke <jrburke at gmail.com> wrote:

Is this an acceptable use of Function.prototype.toString? What problems would there be now or in the future?

For the record, the PlayStation 3's browser returns a Function.prototype.toString like:

function (require, exports, modue) { /-2134160600/ }

I made a list of few other representations some time ago — perfectionkills.com/those-tricky-functions — where you can see how they range from modified identifiers, to modified (or omitted) function bodies, to even crazier extensions like preserving parenthesis and comments when function is within a grouping operator (in IE)!

( /* foo bar / (((function f(){}) / baz qux / ))); // results in "( / foo bar / (((function f(){}) / baz qux */ )))"

(extension is gone in IE9, IIRC)

Non-standard nature of Function.prototype.toString return value, coupled with this variation in practice is the reason I avoid (and never recommend) function decompilation in context of general browser scripting.

P.S. It's worth mentioning that bound functions return host-objects-like "function(){ [native code] }" representation in at least 3 (IIRC) major implementations ATM — IE9, FF4 and one of nightly WebKit or Chrome.

(function(){}).bind({}) + ''; // returns something like "function () { /n [native code] /n }"

Perhaps something that could also use standardization.

[...]