Using sql.js-httpvfs with browser <script type=module>

#sqlite #webassembly #javascript #webpack

https://www.npmjs.com/package/sql.js-httpvfs is an amazing package that lets us perform SQL queries against a remote database hosted anywhere range requests are supported. A special .wasm SQLite runs in the browser; a typical query might only need to fetch half a dozen 4kb pages from a 1GB database file.

It is normally used with webpack. What if we want to distribute it as a JavaScript module so we can just import it from our browser-native <script type=module> and develop a simple project in pure JavaScript?

I edited the example's webpack.config.js (https://github.com/phiresky/sql.js-httpvfs/tree/master/example) to output a module:

module.exports = {
  entry: "./src/index.ts",
  module: {
    rules: [
      {
        test: /\.tsx?$/,
        use: "ts-loader",
        exclude: /node_modules/,
      },
    ],
  },
  resolve: {
    extensions: [".tsx", ".ts", ".js"],
  },
  output: {
    filename: "sql-httpvfs.js",
    library: {
      type: "module" // output a JavaScript module
    },
    module: true, // truly
  },
  experiments: {
    outputModule: true  // yes, we really want one
  },
  optimization: {
    minimize: true
  },
};

index.ts is changed to export a useful function:

import { createDbWorker } from "sql.js-httpvfs";

const workerUrl = new URL(
  "sql.js-httpvfs/dist/sqlite.worker.js",
  import.meta.url
);
const wasmUrl = new URL("sql.js-httpvfs/dist/sql-wasm.wasm", import.meta.url);

async function load(url: string) {
  const worker = await createDbWorker(
    [
      {
        from: "inline",
        config: {
          serverMode: "full",
          url: url,
          requestChunkSize: 4096,
        },
      },
    ],
    workerUrl.toString(),
    wasmUrl.toString()
  );
  return worker;
}

export { load }; // only `load` is visible to the importer

Run webpack. In this example it will write 3 files to ./dist/. We can copy those files to wherever we want to use our new module.

Now we can import that module directly in index.html, and play around with loading database URLs in the browser console:

<script type="module">
    import { load } from "./dist/sql-httpvfs.js";
    window.loadDB = load;
</script>

Modules are automatically deferred, and won't run until the document has been parsed. Our module code can start manipulating the page right away without having to e.g. register a load or $(document).ready handler.