Shuttle

Posted on Mar 6 • Originally published at shuttle.rs

Writing a WASM module in Rust

#rust #javascript #tutorial #webdev

Hello world! In today’s post we’re going to talk about how you can write a WebAssembly module in Rust. WebAssembly is a portable compilation target for programming languages to be able to conveniently inter-op with JavaScript on the web. Rust being able to take advantage of this has made it extremely useful for many use cases, such as:

CPU intensive workloads (encryption)
GPU intensive workloads (image/video processing, image recognition)

This article will focus on writing a WASM module for image processing that can be used on the backend, as well as exploring common ways to deploy WASM and its targets.

Getting started

To get started, you’ll need Rust installed. If you don’t, you can install it here.

We’ll be focusing on trying out writing a WASM module in three different ways:

Using the wasm-bindgen CLI
Using wasm-pack
Using napi-rs

We will initially use wasm-bindgen-cli to create our application, then look at using wasm-pack. The focus of this article will be creating a simple module for image processing. Byte array manipulation and data processing is an area where Rust can significantly speed up your application.

Before we start, make sure you have the wasm32-unknown-unknown target installed. If you don’t, you can add it like so:

rustup target add wasm32-unknown-unknown

Note that for trying out our module, you’ll also additionally want npm (or any alternatives) installed.

Writing a WASM module

The basics

To set up our project, we’ll get started with using cargo init --lib wasm-example to create a new library project named wasm-example. We will then install our dependencies with a small shell snippet:

cargo add wasm-bindgen@0.2.91
cargo add js-sys@0.3.68
cargo add image@0.24.9

We will also want to add the dynamic library flag to our Cargo.toml file. Normally, it lets Cargo know that we want to make a dynamic system library - but when using it with the WebAssembly target, it simply means “make a *.wasm file without a start function”. To do this, we can add this small snippet below:

[lib]
crate-type = ["cdylib"]

JavaScript types in Rust

To be able to use JavaScript types in Rust, we need to use extern C in addition to using the wasm-bindgen macros. This allows us to import functions straight out of JavaScript and into Rust!

The Hello World application in WASM looks like this (from the book):

use wasm_bindgen::prelude::*;

#[wasm_bindgen]
extern "C" {
    fn alert(s: &str);
}

#[wasm_bindgen]
pub fn greet(name: &str) {
    alert(&format!("Hello, {}!", name));
}

Note that the alert function in the extern C is taken directly from JavaScript, and is what allows us to call it in our Rust function. If we were to compile this and execute it in a JavaScript file, it would be the same as if you had called alert() from regular JavaScript.

We can apply the same logic to be able to work with other types and functions - namely, buffers. Vec<u8> in JavaScript can either be represented in one of two ways:

The Uint8Array type (the direct JavaScript equivalent of Vec<u8>)
A Buffer type

Buffer is a subclass of Uint8Array. This is because when Node.js first released, there was no Uint8Array type - which is what led to the creation of the Buffer type. Later down the line when Uint8Arrays were introduced with ES6, both were eventually merged as it made sense to do so. Many JavaScript libraries still use Buffer. By using js-sys, we can get interoperability between JavaScript and Rust - which we can see below by defining the Buffer type and providing a method with the buffer() method:

use js_sys::ArrayBuffer;
// This defines the Node.js Buffer type
#[wasm_bindgen]
extern "C" {
    pub type Buffer;

    #[wasm_bindgen(method, getter)]
    fn buffer(this: &Buffer) -> ArrayBuffer;

    #[wasm_bindgen(method, getter, js_name = byteOffset)]
    fn byte_offset(this: &Buffer) -> u32;

    #[wasm_bindgen(method, getter)]
    fn length(this: &Buffer) -> u32;
}

Now when we write our WASM function, we can refer to the Buffer type directly!

Let’s write our Rust function for converting our image file format. We’ll make it require our Buffer and then have it return Vec<u8> - when we compile it through wasm-pack or another compiler, it will automatically get converted to a Uint8Array.

use js_sys::{ArrayBuffer, Uint8Array};
use wasm_bindgen::prelude::wasm_bindgen;
use image::ImageFormat;
use image::io::Reader;
use std::io::Cursor;

// .. extern C stuff goes here

#[wasm_bindgen]
pub fn convert_image(buffer: &Buffer) -> Vec<u8> {
    // This converts from a Node.js Buffer into a Vec<u8>
    let bytes: Vec<u8> = Uint8Array::new_with_byte_offset_and_length(
        &buffer.buffer(),
        buffer.byte_offset(),
        buffer.length()
    ).to_vec();

    let img2 = Reader::new(Cursor::new(bytes)).with_guessed_format().unwrap().decode().unwrap();

    let mut new_vec: Vec<u8> = Vec::new();
    img2.write_to(&mut Cursor::new(&mut new_vec), ImageFormat::Jpeg).unwrap();

    Ok(new_vec)
}

Building via wasm-bindgen-cli

Here, we need to compile from Rust to WASM by building our package for the wasm32-unknown-unknown target, which we can do like so:

cargo build --target=wasm32-unknown-unknown

Next, we need to use wasm-bindgen to generate the JS glue code to make it all work. We’ll use the nodejs target which will generate a CommonJS module and put it in the ./pkg folder which we can then implant anywhere we want.

wasm-bindgen --target nodejs --out-dir ./pkg \
./target/wasm32-unknown-unknown/release/wasm_example.wasm

Now we can either publish our WASM code as a package or implant it anywhere we want to use it!

I don’t want to use CommonJS!

If you don’t want to use CommonJS because you’re using ESM (EcmaScript modules, or ES6 modules), that’s cool! The CLI currently allows several targets:

bundler (produces code for usage with bundlers like Webpack)
web (directly loadable in a web browser)
nodejs (loadable via require as a CommonJS Node.js module)
deno (usable as a Deno module)
no-modules (like the web target but doesn’t use ES Modules).

There are specific docs for usage with ES which you can check out here. The easiest way to do it in terms of what compiler to use is typically with Webpack as it’s the most compatible. There is also an additional guide you can use for compiling to ES6 modules without a bundler here - though it involves initializing the WASM module manually before running which adds some overhead.

Test driving our new module

Now that we’ve written our code, let’s try it out! We will spin up a JavaScript backend server using Express.js. We will assume you’re running the following in the same folder as your Rust project (for convenience purposes). We’ll get started with the following shell snippet:

npm init -y
npm i express express-fileupload

Next, we’ll create a server.js file in our root directory and insert the following code:

const fileUpload = require('express-fileupload');
const express = require('express');
const { convert_image } = require('./pkg/wasmmeme.js');

const app = express();
const port = 3030;

app.get('/', (req, res) => {
  res.send(`
    <h2>With <code>"express"</code> npm package</h2>
    <form action="/api/upload" enctype="multipart/form-data" method="post">
      <div>Text field title: <input type="text" name="title" /></div>
      <div>File: <input type="file" name="file"/></div>
      <input type="submit" value="Upload" />
    </form>
  `);
});

app.post('/api/upload', (req, res, next) => {
        const image = convertImage(req.files.file.data)

  res.setHeader('Content-disposition', 'attachment; filename="meme.jpeg"');
  res.setHeader('Content-type', 'image/jpg');
    res.send(image);
  });

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

This snippet does the following:

We set up an Express server at port 3030
We have a route at / that will give us a HTML form when we visit it in the browser
We have an API route that will grab the data from our file upload, convert it to a new format, set the correct headers and return the new image.

If we use node server.js, head to [http://localhost:3030](http://localhost:3030) in our browser then fill the form out and attach an image, we should get an image download response back!

Note that depending on the settings you are using for your image file format conversion, your file size may increase post-conversion; this is because you may be using lossless conversion. If you want to use lossy conversion to decrease your file size, you want the new_with_quality method while instantiating the image encoder in your Rust code.

Building our app with alternative CLIs

While wasm-bindgen-cli is useful, it’s also the most low level CLI out of our options and you can run into spurious issues while using it such as wasm-bindgen version incompatibility issues. There are some additional quality of life changes that we could benefit from, such as automatic versioning and wasm-opt usage. Let’s take a quick look at some of these other options and see how they compare.

Wasm-pack

wasm-pack is a tool that aims to be a one-stop-shop for compiling Rust to WASM. It includes a CLI that you can use by installing it here. In comparison to using wasm-bindgen-cli, it has a number of quality of life upgrades:

Comes with wee_alloc, a WebAssembly allocator with a (pre-compression) 1kB code footprint.
Comes with a panic hook that allows you to debug Rust panic messages in the browser

To initialise our project, we can use wasm-pack new wasm-example which will do everything for us. Code-wise, our main function (and C/JS bindings) will remain the same as wasm-pack provides primarily tooling additions to make compilation easier and does not have any library code we can use.

napi-rs

napi-rs is a framework for building pre-compiled Node.js addons in Rust. If you find using wasm-bindgen too complicated to use and just want to write Node.js stuff, this is a good alternative. To use it, Node v0.10.0 or later is required. You can install it with the following shell snippet (requires npm or its alternatives):

npm install -g @napi-rs/cli

Once done, you can then use napi new wasm-example to build your new NAPI project!

napi-rs does come with some code changes, which you’ll be able to see below: we can finally get rid of the extern C block and instead use napi’s bindgen_prelude to include whatever we need.

use napi::bindgen_prelude::*;
use image::io::Reader;
use image::ImageFormat;
use image::ImageOutputFormat;
use std::io::Cursor

#[macro_use]
extern crate napi_derive;

#[napi]
pub fn convert_image(buffer: Buffer) -> Result<Buffer> {
    let bytes: Vec<u8> = buffer.into();

    let img2 = Reader::new(Cursor::new(bytes)).with_guessed_format().unwrap().decode().unwrap();

    let mut new_vec: Vec<u8> = Vec::new();
    img2.write_to(&mut Cursor::new(&mut new_vec), ImageFormat::Jpeg).unwrap();

    Ok(new_vec.into())
}

The advantages of this are clear:

We don’t need to manually import anything using extern C
We can easily use Node.js internals without trouble

Of course for all its advantages, napi-rs is only compatible with Node.js. If you want to write some WASM code for the browser, you would need to default to wasm-pack or wasm-bindgen. You additionally also need to use the Node ecosystem to keep your CLI updated, which from a Rust-first standpoint is a bit of a strange decision. Needless to say, napi-rs is a pretty easy way to start writing Node.js with Rust.

Finishing up

Thanks for reading! Rust has great inter-op with WASM, and there's no reason why we shouldn't be able to take advantage of this to help us when using other lanaugages.