Native Tensor support
Two questions:
- Would APIs that operate on existing data types (rather than tensors) work just as well?
- Could there instead be a kernel-like API that could work on things independently? Lower level APIs that enable equivalent high-level constructs is a much better place to start.
And two comments:
- You could easily emulate 2D arrays by simply using 1D arrays and storing data in row- or column-major order. This is pretty well-known at this point, and is how C/C++ allocate multi-dimensional arrays internally.
- Data parallelization requires special consideration, and I can assure you, machine learning isn't the only thing that could stand to benefit from this. It needs to be broad enpugh that other non-data applications can benefit from it. (Applying DOM changes from a static change list is an embarassingly parallel* problem. So anything that could speed this up by a substantial bit could be infinitely useful to anyone using Angular, React, Ember, or any other framework out there.)
* Yes, that's a technical term. Look it up on Wikipedia.
On 1/26/18 10:49 PM, Isiah Meadows wrote:
Applying DOM changes from a static change list is an embarassingly parallel* problem.
Only if you know there are no overlaps in terms of the nodes involved. And you ignore mutation events. And you ignore the fact that mutation records expose your order of operations. And you know that none of the insertions have immediate side-effects (e.g. none are <script> nodes).
And your style system doesn't try to do style recomputation in certain ways. And probably some other conditions that I'm forgetting.
I'm speaking from the framework's side. Yes, I did gloss over certain complexities and constraints, but yes, it's almost there in practice.
- If they don't expose the node through a ref or something, and no event handlers exist, they could just assume (unsafely) you don't reference it, and act accordingly. Note that I'm speaking from the framework level, which can take liberties browsers can't.
- I make the assumption you build change lists beforehand, and that
resolves all dependencies first. And in general, once you figure out what
changes need made, it doesn't actually matter as long as you unregister
blur/etc. handlers that fire on removal before applying the change list. - Script elements execute async, so you could ignore order here.
- For style elements and ref-accessible elements, you could still execute the changes in set slices, splitting on nodes/callbacks that require awareness of past effects. (This could just be a flag.) However, these are exceedingly rare.
every numerical webapp has unique requirements/slop for dealing with NaN's and i'm skeptical you can create a generalized numpy-like library in javascript than can handle them all in a user-friendly way. you will usually have a easier time debugging and fixing NaN-issues using specialized-code you write yourself (with insight about your app's use-case, numerical-range, and user-inputs), rather than from naively applying textbook-formulas to some blackbox tensor-library.
i wrote a numerical finance chrome-app 6 years ago (unfortunately it stopped working last may when yahoo shutdown their historical-stock-quoting api): chrome.google.com/webstore/detail/financejs/cmldinilpjimkpkdfmikeclmmbnjpdej, chrome.google.com/webstore/detail/financejs/cmldinilpjimkpkdfmikeclmmbnjpdej
source code here: kaizhu256/chrome-financejs/tree/alpha/public/financejs, kaizhu256/chrome-financejs/tree/alpha/public/financejs
it took historical stock-quotes from yahoo-finance, and fitted them against cosine-waves to see if there were any intrinsic periodicities in timeframes ranging from a few weeks to several decades. yes it used a multi-dimensional array library (i followed numpy’s striding approach, to avoid copy-on-transpose), and could take integers and floats of various widths, which were not needed at all for the product. the intention was to create a reusable library for other numerical projects, but in hindsight that was pretty naive of me. the library turned out too complicated to configure for reuse (surprise surprise for a javascript newcomer then transitioning from python/c). if i could do it again, i would just “hardcode” alot of stuff and use lists of contiguous 1d Float64Array’s, which would reduce the code-bloat by 4x, not to mention saving me a bit of nan-related headaches from naively writing and applying generalized matrix-operations in my original library (numerical performance turned out to be a complete non-issue in the product).
also in javascript, its generally easier to reuse code from previous zero-config projects with lots of hardcoding and minimal abstraction, than from one that's overly-configured. simply copy-paste the old code and grep-and-replace the hardcodes, rather than deal with any complicated config-tooling.
On 1/26/18 11:37 PM, Isiah Meadows wrote:
- Script elements execute async
Not in the inline script case, they don't.
That is true, but script elements generated via the DOM are not inline
script elements, and the DOM is the only thing that frameworks deal
with. They may feature an HTML-like syntax, like with Angular and
Ember, but they still build nodes via
document.createElement(tagName) and/or clone them via
elem.cloneNode(true), and they can't tell the browsers to not
schedule an async task for things the spec requires them to. (So in
other words, I didn't take into account HTML semantics outside the
DOM, because they don't apply to the use case I was stating.)
Isiah Meadows me at isiahmeadows.com
Looking for web consulting? Or a new website? Send me an email and we can get started. www.isiahmeadows.com
It seems you are implying that only parser-inserted scripts can run immediately. But scripts created using DOM methods can also do that, see html.spec.whatwg.org/multipage/scripting.html#script-processing-inline
var s = document.createElement("script");
s.text = "console.log(1);";
document.documentElement.appendChild(s);
console.log(2);
The output will be 1, 2. The script runs immediately, it's not async.
Okay. I stand corrected. It still can be handled similar to style elements and refs my fourth point, quoted from earlier.
- For style elements and ref-accessible elements, you could still execute the changes in set slices, splitting on nodes/callbacks that require awareness of past effects. (This could just be a flag.) However, these are exceedingly rare.
Isiah Meadows me at isiahmeadows.com
Looking for web consulting? Or a new website? Send me an email and we can get started. www.isiahmeadows.com
On 1/27/18 1:42 PM, Isiah Meadows wrote:
That is true, but script elements generated via the DOM are not inline script elements
Just to make sure we're on the same page:
var script = document.createElement("script"); script.textContent = 'console.log("script running")'; console.log("before"); document.body.appendChild(script); console.log("after");
That is an inline script, it's generated via the DOM, and it runs sync before the appendChild call returns, as you can see by the order of messages in your console.
and they can't tell the browsers to not schedule an async task for things the spec requires them to.
In my example there, the spec requires browsers to run the script sync. There's no "HTML semantics" involved, if you mean parser behavior by that; just DOM manipulation.
On 1/27/18 9:53 PM, Isiah Meadows wrote:
- For style elements and ref-accessible elements, you could still execute the changes in set slices, splitting on nodes/callbacks that require awareness of past effects. (This could just be a flag.) However, these are exceedingly rare.
This is not longer sounding embarrassingly parallel, fwiw, since you have to not do later things until earlier things are done under various conditions...
Q1: That's a good question. To me, tensors could have their own habitat within the JS scope, without handling existing data types, since they typically serve different purposes.
Q2: I don't quite get what you mean by kernel-like API. Could you go into more detail on that?
C1: Sure, that is what I said. It's easy to construct multi-dimensional arrays already, but neither is there a shortcut for such arrays, like in C++ with "new int[4][4][3]", nor is there a fast and parallel way of working with data in a certain dimension.
C2: Machine learning is just a momentary hype. Who knows what it will be in the next 5-10 years - nano*, quantum*, ... Whatever it is, the trend goes into the direction of calculations of high-dimensional data - and another trend is that all that will eventually land in the browser. So tensors as an abstract algebraic object right at the finger tips of a JS developer could facilitate all that.
I'm not sure if tensors are the best abstraction for DOM parallelism. Furthermore, I think static changes should not go beyond 2- or 3 dimensional spaces. And as Boris said, it only works if one can make sure there are no overlaps.
Robert
Am 27.01.18 um 04:49 schrieb Isiah Meadows:
Why should be it included in standard library?
Are there widely used libraries providing similar capabilities?
Why is it preferable to implementing tensor operations in Web Assembly?
On 27 Jan 2018 2:50 am, "Robert Eisele" <robert at xarg.org> wrote:
Hello,
Allocating multi-dimensional arrays in Javascript is only possible by building each dimension individually. In addition to being a very tedious job, a developer has no control over memory usage, which in general is likely to be very high.
Seeing an array algebraically as a vector, typed arrays have already created the ability to work more efficiently and memory-consciously with lists of numbers. A natural extension of this is not just a matrix, but a tensor.
I would like to suggest tensors as a native language construct in ES. This would have the advantage that developers could write highly parallelizable code independently of WebGL. As an API one could introduce the following classes in analogy to typed arrays:
- IntXTensor
- UintXTensor
- FloatXTensor
Where X is one of {8, 16, 32, 64}. To make these tensor objects really effective, it is necessary to introduce meaningful operations, maybe similar to the features of TensorFlow. I think by introducing tensors in the browser (but also node.js), a wide range of new applications open up. For example, working with deep learning right in the browser or calculating filters on images without having to write shaders for them.
The most important thing probably is having a way of storing high dimensional data in the browser without worrying about the memory footprint, even for complex applications.
What do you think about it?
Robert Eisele
The capabilities of the language itself are growing with each version. It's not that the first version of JavaScript would not be appropriate for today's web applications, but since we identify regular patterns in a language, we are able to add more layers of abstraction. But not only syntactic features can be abstracted to describe what the programmer wants with as little code as possible to give the engines more room for optimization, also the standard library should get extended. And I mean, it gets extended quite a lot - so tensors could be just one of these additions.
The link of J Decker is a good starting reference yiransheng/tensor-ops-js
It's not hard to use the features of JavaScript to implement a library like this. The hard part is to make an API like this as fast as possible to meet future demands for web applications. And this is the purpose of a standard library: 1) Providing regular functionalities without the need of external dependencies (e.g. libraries) and 2) Making them as fast and accurate as possible on a given machine.
And I think all these points make tensors a good fit for Web Assembly.
Robert
Am 28.01.18 um 15:08 schrieb Michał Wadas:
for stuff like machine-learning, using web-assembly / asm.js on an existing c/c++ library is probably more practical than trying to roll your own in javascript.
here are some pre-compiled (in asm.js) binaries using fann you can download and run directly in nodejs: kaizhu256/node-fann-lite/tree/examples/external/examples, kaizhu256/node-fann-lite/tree/examples/external/examples
kaizhu256/node-fann-lite/blob/examples/external/examples/Makefile#L20, kaizhu256/node-fann-lite/blob/examples/external/examples/Makefile#L20
performance-wise, the asm-variants running in nodejs train about 4x slower than their native c-compiled counterparts
I know that makes it no longer that way. But in the absence of such problematic parts, it is. In practice, trees usually only really have no more than a few refs, and style/script elements quite frequently don't even exist. So in the common case, it's almost.
Also, in reply to your other email, I did stand corrected on missing
the script with textContent set, but I can just add a case to that
particular point. (My experience and knowledge is far better with
JS than the DOM.)
Isiah Meadows me at isiahmeadows.com
Looking for web consulting? Or a new website? Send me an email and we can get started. www.isiahmeadows.com
Inline, so it's a little easier to read.
Isiah Meadows me at isiahmeadows.com
Looking for web consulting? Or a new website? Send me an email and we can get started. www.isiahmeadows.com
On Sun, Jan 28, 2018 at 8:27 AM, Robert Eisele <robert at xarg.org> wrote:
Q1: That's a good question. To me, tensors could have their own habitat within the JS scope, without handling existing data types, since they typically serve different purposes.
In practice, the data storage requirements are similar, and you could implement the tensor transforms in terms of static functions over typed arrays in the first place.
Q2: I don't quite get what you mean by kernel-like API. Could you go into more detail on that?
I was talking in reference to GPU compute kernels, which are
basically like only specifying the inner part of a for loop, but
also without the ability to break early. (This is an
over-simplification, but it should help you at least get an idea what
I mean.)
C1: Sure, that is what I said. It's easy to construct multi-dimensional arrays already, but neither is there a shortcut for such arrays, like in C++ with "new int[4][4][3]", nor is there a fast and parallel way of working with data in a certain dimension.
In JS, you could create a wrapper class to model typed array data in a multi-dimentional context, and in fact, that's how C/C++ (and most other languages with native support) do it under the hood.
However, you're right in that there's no realistic parallel method of working with data. There exists one runtime that has decent parallel threading support (based on Webkit), but that only covers low-parallelism needs that just need enough parallelism to not be I/O-bound or blocked on complex CPU computation elsewhere when rendering.
C2: Machine learning is just a momentary hype. Who knows what it will be in the next 5-10 years - nano*, quantum*, ... Whatever it is, the trend goes into the direction of calculations of high-dimensional data - and another trend is that all that will eventually land in the browser. So tensors as an abstract algebraic object right at the finger tips of a JS developer could facilitate all that.
Tensors aren't necessarily the ideal way to handle it, though. For example, a lot of high-parallelism needs, such as image processing and game rendering, need a more general framework, and they may find themselves needing to implement their own primitives. (For one, it's possible to do parallel sorting on the GPU with major speedups.)
I'm not sure if tensors are the best abstraction for DOM parallelism. Furthermore, I think static changes should not go beyond 2- or 3 dimensional spaces. And as Boris said, it only works if one can make sure there are no overlaps.
- They aren't. (I would really need a DOM API for async batch updates, but that's a whole different story.)
- You could in theory go into the hundreds of dimensions, but GPUs have hardware limitations above a certain limit, so the runtime would have to chunk accordingly.
- I'm aware. (More accurately, overlaps or dependencies, the latter is much easier to encounter.)
Hello,
Allocating multi-dimensional arrays in Javascript is only possible by building each dimension individually. In addition to being a very tedious job, a developer has no control over memory usage, which in general is likely to be very high.
Seeing an array algebraically as a vector, typed arrays have already created the ability to work more efficiently and memory-consciously with lists of numbers. A natural extension of this is not just a matrix, but a tensor.
I would like to suggest tensors as a native language construct in ES. This would have the advantage that developers could write highly parallelizable code independently of WebGL. As an API one could introduce the following classes in analogy to typed arrays:
Where X is one of {8, 16, 32, 64}. To make these tensor objects really effective, it is necessary to introduce meaningful operations, maybe similar to the features of TensorFlow. I think by introducing tensors in the browser (but also node.js), a wide range of new applications open up. For example, working with deep learning right in the browser or calculating filters on images without having to write shaders for them.
The most important thing probably is having a way of storing high dimensional data in the browser without worrying about the memory footprint, even for complex applications.
What do you think about it?
Robert Eisele
Hello, Allocating multi-dimensional arrays in Javascript is only possible by building each dimension individually. In addition to being a very tedious job, a developer has no control over memory usage, which in general is likely to be very high. Seeing an array algebraically as a vector, typed arrays have already created the ability to work more efficiently and memory-consciously with lists of numbers. A natural extension of this is not just a matrix, but a tensor. I would like to suggest tensors as a native language construct in ES. This would have the advantage that developers could write highly parallelizable code independently of WebGL. As an API one could introduce the following classes in analogy to typed arrays: - IntXTensor - UintXTensor - FloatXTensor Where X is one of {8, 16, 32, 64}. To make these tensor objects really effective, it is necessary to introduce meaningful operations, maybe similar to the features of TensorFlow. I think by introducing tensors in the browser (but also node.js), a wide range of new applications open up. For example, working with deep learning right in the browser or calculating filters on images without having to write shaders for them. The most important thing probably is having a way of storing high dimensional data in the browser without worrying about the memory footprint, even for complex applications. What do you think about it? Robert Eisele