persisting large wasm-sqlite3 datasets in browser (beyond kv-storage)

# kai zhu (5 years ago)

at work, we have browser-app that load-and-persist ~100MB (500k rows) csv-files into wasm-sqlite3 [1], (ingestion-time is ~15s for 100MB csv). we wish to go bigger, but chrome's indexeddb has a hard-limit of 125MB per key-value object (on windows).

i don't have anything actionable. just want people aware of datapoint, and think about javascript-language-design to improve UX-handling with sqlite3-persistence.

fyi, ignoring persistence, chrome can handle wasm-sqlite3 datasets as large as 300MB (1.5 million rows) in memory, before crashing (on 8gb windows10 machine).

[1] sql.js wasm-sqlite3 kripken/sql.js

at work, we have browser-app that load-and-persist ~100MB (500k rows)
csv-files into wasm-sqlite3 [1], (ingestion-time is ~15s for 100MB csv).
we wish to go bigger, but chrome's indexeddb has a hard-limit of 125MB per
key-value object (on windows).

i don't have anything actionable.  just want people aware of datapoint, and
think about javascript-language-design to improve UX-handling with
sqlite3-persistence.

fyi, ignoring persistence, chrome can handle wasm-sqlite3 datasets as large
as 300MB (1.5 million rows) in memory, before crashing (on 8gb windows10
machine).

[1] sql.js wasm-sqlite3
https://github.com/kripken/sql.js
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/es-discuss/attachments/20190904/d54ca8d9/attachment.html>

# Isiah Meadows (5 years ago)

Not sure this pertains to the ECMAScript spec in any way. You may have better luck with WICG, because they are who deal with those specs - those are specific to the web, while JS is used in places where those might not even make sense (like IoT sensors).

Isiah Meadows contact at isiahmeadows.com, www.isiahmeadows.com

Not sure this pertains to the ECMAScript spec in any way. You may have
better luck with WICG, because they are who deal with those specs -
those are specific to the web, while JS is used in places where those
might not even make sense (like IoT sensors).

-----

Isiah Meadows
contact at isiahmeadows.com
www.isiahmeadows.com

On Wed, Sep 4, 2019 at 12:40 PM kai zhu <kaizhu256 at gmail.com> wrote:
>
> at work, we have browser-app that load-and-persist ~100MB (500k rows) csv-files into wasm-sqlite3 [1], (ingestion-time is ~15s for 100MB csv).  we wish to go bigger, but chrome's indexeddb has a hard-limit of 125MB per key-value object (on windows).
>
> i don't have anything actionable.  just want people aware of datapoint, and think about javascript-language-design to improve UX-handling with sqlite3-persistence.
>
> fyi, ignoring persistence, chrome can handle wasm-sqlite3 datasets as large as 300MB (1.5 million rows) in memory, before crashing (on 8gb windows10 machine).
>
> [1] sql.js wasm-sqlite3
> https://github.com/kripken/sql.js
> _______________________________________________
> es-discuss mailing list
> es-discuss at mozilla.org
> https://mail.mozilla.org/listinfo/es-discuss