Unlimited Actix power!
Welcome to the MoonZoon Dev News!
MoonZoon is a Rust full-stack framework. If you want to read about new MZ features, architecture and interesting problems & solutions - Dev News is the right place.
Chapters
News
Moon - Warp replaced with Actix. There are API changes to allow you to use Actix directly from your apps.
mzoon - Rewritten with Tokio, implemented
--open
parameter andwasm-pack
installer.The entire MoonZoon codebase should be clean enough now. Comments are still missing and there should be more tests but if you wanted to know how it really works, you don't have to be afraid to read the code in the MZ repo.
You can select the required mzoon version for heroku-buildpack-moonzoon by adding the file mzoon_commit to your repo with a MZ project.
You'll read about Moon and mzoon improvements mentioned above in the following chapters.
And I would like to thank:
- All Rust libraries maintainers. It's tough work but it allows us to write clean code and amazing products.
Actix
Why Actix?
- It's fast, async and popular.
- Supports HTTP/2 and probably also H3 in the future (related issue).
- Actix actor framework could be a good foundation for the first version of virtual actors.
- It uses Tokio under the hood. It's the most popular async runtime and we can use it also in mzoon.
- The API feels more intuitive than the Warp's one to me. And we were fighting with Warp during the Moon development.
- Tide supports only HTTP/1.x.
- Why not Rocket (detailed explanation to answer questions on the MZ chat):
- It's too much opinionated with too many batteries included to be used as just a library. Moon would fight with Rocket. Examples: We would need to disable Rocket's console logging, then explain to users they can't use
Rocket.toml
and hope that Rocket's live reloading won't break Moon's file watchers, etc. - It's still less popular / downloaded than Actix.
- We would need to find a compatible actor framework to write PoC of virtual actors.
- I've already rewritten Moon to Actix (before Rocket published the version
0.5.0-rc.1
). Actix works great so I don't see a real reason to sacrifice dozens hours of my free time and slow down Moon development by another rewriting. - Every framework has its own problems - there is a chance we would encounter a show-stopper during the Rocket integration.
- It doesn't really matter what framework we choose from the long-term view. The number of reasons why you want to communicate directly with the lower-level framework (Actix/Rocket) in Moon will decrease proportionally to the progress of Moon API development.
- Rocket and Actix (and other frameworks) have pretty similar performance and API in many cases so all Rust web developers should be able to learn the Actix API quickly and even migrate a project from other frameworks in a reasonable time.
- It's too much opinionated with too many batteries included to be used as just a library. Moon would fight with Rocket. Examples: We would need to disable Rocket's console logging, then explain to users they can't use
Moon API changes
The simplest Moon app:
use moon::*;
async fn frontend() -> Frontend {
Frontend::new().title("Actix example")
}
async fn up_msg_handler(_: UpMsgRequest) {}
#[moon::main]
async fn main() -> std::io::Result<()> {
start(frontend, up_msg_handler, |_|{}).await
}
main
is nowasync
so we no longer need theinit
function - you can write your async code directly to themain
's body.start!
macro has been rewritten to a simple functionstart
. The interesting is the third argument. See the next example:
use moon::*;
use moon::actix_web::{get, Responder};
async fn frontend() -> Frontend {
Frontend::new().title("Actix example")
}
async fn up_msg_handler(_: UpMsgRequest) {}
#[get("hello")]
async fn hello() -> impl Responder {
"Hello!"
}
#[moon::main]
async fn main() -> std::io::Result<()> {
start(frontend, up_msg_handler, |cfg|{
cfg.service(hello);
}).await
}
It's the code used in the GIF at the top.
cfg
in the example is actix_web::web::ServiceConfig. It allows you to create custom Actix endpoints and configure the server as you wish.Multiple crates and items are reexported from
moon
to mitigate dependency hell caused by incompatible versions and to simplify yourCargo.toml
. The current list looks like this:
pub use trait_set::trait_set;
pub use actix_files;
pub use actix_http;
pub use actix_web;
pub use actix_web::main;
pub use futures;
pub use mime;
pub use mime_guess;
pub use parking_lot;
pub use serde;
pub use tokio;
pub use tokio_stream;
pub use uuid;
MoonZoon.toml changes
port = 8080
# port = 8443
https = false
cache_busting = true
backend_log_level = "warn" # "error" / "warn" / "info" / "debug" / "trace"
[redirect]
port = 8081
enabled = false
[watch]
frontend = [
"frontend/Cargo.toml",
"frontend/src",
]
backend = [
"backend/Cargo.toml",
"backend/src",
]
-
There is a new property
backend_log_level
. It sets the env_logger log level.-
info
level is useful for debugging because it shows all requests (demonstrated in the GIF at the top). -
Note: There are also independent
404
and500
error handlers that calleprintl
with the error before they pass the response to the client. -
Note: fern looks like a good alternative if we find out
env_logger
isn't good enough. (Thanks azzamsa for the suggestion.)
-
[redirect_server]
has been renamed to[redirect]
because there is no longer a redirection server. The new RedirectMiddleware is activated when you enable the redirect.
Caching has been also improved:
-
cache_busting = true
:- mzoon generates files like
frontend_bg_[uuid].wasm
, whereuuid
is a frontend build id with the typeu128
. - Moon serves the files with the header
CacheControl
set toMaxAge(31536000)
(1 year).
- mzoon generates files like
-
cache_busting = false
- mzoon doesn't change the file names at all - e.g.
frontend_bg.wasm
. - Moon serves the files with the header
ETag
with a strong etag set to the frontend build id. (See MDN ETag docs for more info.)
- mzoon doesn't change the file names at all - e.g.
Server-Sent Events
Actix unfortunately doesn't have an official SSE API so I've decided to write a custom one. The current implementation is in the file crates/moon/src/sse.rs.
It sends a
ping
to all connections every 10 seconds to recognize the disconnected ones.-
Integration:
let sse = SSE::start();
App::new().app_data(sse.clone())
Moon's SSE connector:
async fn sse_responder(
sse: web::Data<Mutex<SSE>>,
shared_data: web::Data<SharedData>,
) -> impl Responder {
let (connection, event_stream) = sse.new_connection();
let backend_build_id = shared_data.backend_build_id.to_string();
if connection
.send("backend_build_id", &backend_build_id)
.is_err()
{
return HttpResponse::InternalServerError()
.reason("sending backend_build_id failed")
.finish();
}
HttpResponse::Ok()
.insert_header(ContentType(mime::TEXT_EVENT_STREAM))
.streaming(event_stream)
}
and the frontend reloader:
async fn reload_responder(sse: web::Data<Mutex<SSE>>) -> impl Responder {
let _ = sse.broadcast("reload", "");
HttpResponse::Ok()
}
Warning: Keep in mind that browsers can open only 6 SSE connections over HTTP/1.x to the same domain. It means when you open multiple browser tabs pointing to http://localhost
, you may observe infinite loadings or similar problems. The limit for HTTP/2 is 100 connections by default, but can be negotiated between the client and the server.
Moon endpoint changes
App::new()
// ...
.service(Files::new("_api/public", "public"))
.service(
web::scope("_api")
.route(
"up_msg_handler",
web::post().to(up_msg_handler_responder::<UPH, UPHO>),
)
.route("reload", web::post().to(reload_responder))
.route("pkg/{file:.*}", web::get().to(pkg_responder))
.route("sse", web::get().to(sse_responder))
.route("ping", web::to(|| async { "pong" })),
)
.route("*", web::get().to(frontend_responder::<FRB, FRBO>))
All backend endpoints are prefixed with _api
to prevent conflicts with frontend routes. There are other solutions like hash routing or moving the frontend endpoint to another domain or a prefix for frontend urls but these solutions often lead to many unpredictable problems. Let's keep it simple.
There is a new simple endpoint ping
. It's useful for testing if the server is alive. I can imagine we can also implement a heartbeat later (Moon would call a predefined endpoint in a configured interval).
mzoon
Async runtime
mzoon was rewritten with Tokio. The main goal was to remove spaghetti code and boilerplate caused by manual handling of threads, channels and signals. The secondary goal was error handling and improved performance.
There are also other async runtimes like async-std or smol but I've decided to choose the most battle-tested and popular one. Another reason for Tokio is Actix, because Actix is based on Tokio so there should be less context switching during the MoonZoon development.
Error handling
During the mzoon refactor, I've decided to integrate two nice libraries to eliminate boilerplate:
The first one is anyhow. It allows you to write ?
wherever you want to return an error early. No need to write error mappers or similar stuff.
anyhow
also provides the method context
(and its lazy version with_context
) and a macro anyhow!
for creating errors. An example:
// `anyhow::Result<T>` is an alias
// for a standard `Result<T, anyhow::Error>`
use anyhow::{anyhow, Context, Result};
pub async fn build_backend(release: bool, https: bool) -> Result<()> {
...
Command::new("cargo")
.args(&args)
.status()
.await
.context("Failed to get frontend build status")?
.success()
.err(anyhow!("Failed to build backend"))?;
...
}
- Notes:
The second error handling library is fehler. I've decided to integrate it into mzoon once I noticed that many functions were returning Ok(())
and their signature was ... -> Result<()>
. Ok(())
is a side-effect of anyhow
because you want to use ?
as much as possible to automatically convert concrete errors to anyhow::Error
. The second reason why there were many Ok(())
s is the fact that mzoon does many file operations.
I recommend to read these articles about fehler
- A brief apology of Ok-Wrapping and From failure to Fehler.
So when we combine both libraries, we can write a clean code without boilerplate:
use anyhow::Error;
use fehler::throws;
// ...
#[throws]
#[tokio::main]
async fn main() {
// ...
match opt {
Opt::New { project_name, here } => command::new(project_name, here).await?,
Opt::Start { release, open } => command::start(release, open).await?,
Opt::Build { release } => command::build(release).await?,
}
}
#[throws]
automatically converts the return type from-> ()
to-> Result<(), Error>
and you don't have to write uglyOk(())
or wrap the entirematch
intoOk()
.All errors before
?
are automatically converted toError
and nicely written to the terminal with their contexts thanks toanyhow
.
Let's look at another example from mzoon where we integrated the crate apply to help with chaining:
// ...
use anyhow::{Context, Error};
use apply::{Also, Apply};
use fehler::throws;
#[throws]
pub fn run_backend(release: bool) -> Child {
println!("Run backend");
MetadataCommand::new()
.no_deps()
.exec()?
.target_directory
.also(|directory| directory.push(if release { "release" } else { "debug" }))
.also(|directory| directory.push("backend"))
.apply(Command::new)
.spawn()
.context("Failed to run backend")?
}
Tip: Don't try to write "functional chains" at all costs. It's easy to get lost in long chains, they may be difficult to change and they may increase cognitive load because the reader has to keep intermediate steps/states in his working memory. The example above is very close to the case where clean code is uncomfortable to read.
Note: We have to find the
target
directory and call the Moon app binary (backend
) manually becausecargo run
always tries to build the project even if the project has been already built. It slows down the build pipeline and writes unnecessary messages to the terminal. Related issue.
File Watchers
While I was rewriting std
channels to the tokio
ones, I encountered the problem with the notify API. Also its event debouncing wasn't working properly in mzoon. Fortunately notify
maintainers are working on a new major version and they've already published 5.0.0-pre.x
versions. The API is more flexible but debouncing is still missing in the new notify
and in the crate futures-rs. So I had to write a custom debouncer.
The snippets below belong to the current ProjectWatcher
implementation in /crates/mzoon/src/watcher/project_watcher.rs.
use notify::{immediate_watcher, Event, RecommendedWatcher, RecursiveMode, Watcher};
use tokio::sync::mpsc::{self, UnboundedReceiver, UnboundedSender};
// ...
pub struct ProjectWatcher {
watcher: RecommendedWatcher,
debouncer: JoinHandle<()>,
}
impl ProjectWatcher {
#[throws]
pub fn start(paths: &[String], debounce_time: Duration) -> (Self, UnboundedReceiver<()>) {
let (sender, receiver) = mpsc::unbounded_channel();
let watcher = start_immediate_watcher(sender, paths)?;
let (debounced_sender, debounced_receiver) = mpsc::unbounded_channel();
let this = ProjectWatcher {
watcher,
debouncer: spawn(debounced_on_change(
debounced_sender,
receiver,
debounce_time,
)),
};
(this, debounced_receiver)
}
// ...
ProjectWatcher
is a general watcher based on thenotify
's watcher. It's used in mzoon'sBackendWatcher
andFrontendWatcher
.start_immediate_watcher
calls thenotify
'simmediate_watcher
function to register watched paths and the callback that is invoked whennotify
observes a file change. The callback sends()
(aka unit) through thesender
.The
sender
's second half -receiver
- is passed to thedebouncer
. It means the debouncer is able to listen for all registered file system events.The
debounced_sender
represents thedebouncer
's output - basically a stream of debounced units (we can replace units withEvent
s if needed in the future).
async fn debounced_on_change(
debounced_sender: UnboundedSender<()>,
mut receiver: UnboundedReceiver<()>,
debounce_time: Duration,
) {
let mut debounce_task = None::<JoinHandle<()>>;
let debounced_sender = Arc::new(debounced_sender);
while receiver.recv().await.is_some() {
if let Some(debounce_task) = debounce_task {
debounce_task.abort();
}
debounce_task = Some(spawn(debounce(
Arc::clone(&debounced_sender),
debounce_time,
)));
}
if let Some(debounce_task) = debounce_task {
debounce_task.abort();
}
}
async fn debounce(debounced_sender: Arc<UnboundedSender<()>>, debounce_time: Duration) {
sleep(debounce_time).await;
if let Err(error) = debounced_sender.send(()) {
return eprintln!("Failed to send with the debounced sender: {:#?}", error);
}
}
When the unit from the
notify
's callback is received, then a new task is spawned. The tasksleep
s for thedebounce_time
and then a unit is sent through thedebounced_sender
.When another unit is received, then the sleeping task is aborted and a new one is created. You can understand it as "debounce time reset".
Notice two same code blocks in the previous snippet:
if let Some(debounce_task) = debounce_task {
debounce_task.abort();
}
The first usage "resets debounce time", but the second one is basically an alternative to drop
. Unfortunately neither Rust nor tokio
is able to automatically clean all garbage so we have to do it manually - the task handle does nothing when dropped in most cases.
So... how we can stop the watcher?
The ProjectWatcher
doesn't have only one method (start
) - there is another one:
#[throws]
pub async fn stop(self) {
let watcher = self.watcher;
drop(watcher);
self.debouncer.await?;
}
- Drop
notify
'sRecommendedWatcher
. - Dropped
watcher
means that also oursender
has been dropped because it was closed by the closure used as a callback / event handler owned by the watcher. - When the
sender
is dropped, thenreceiver.recv().await.is_some()
returnsfalse
to break thewhile
loop in the debouncer. - The debounce task is aborted if there was one running.
Yeah, it's already quite complicated and error prone but we haven't finished yet.
FrontendWatcher
and BackendWatcher
have the similar relationship to ProjectWatcher
as ProjectWatcher
to notify
's Watcher
. Let's look at the FrontendWatcher
skeleton:
pub struct FrontendWatcher {
watcher: ProjectWatcher,
task: JoinHandle<Result<()>>,
}
impl FrontendWatcher {
#[throws]
pub async fn start(config: &Config, release: bool, debounce_time: Duration) -> Self {
let (watcher, debounced_receiver) =
ProjectWatcher::start(&config.watch.frontend, debounce_time)
.context("Failed to start the frontend project watcher")?;
// ...
Self {
watcher,
task: spawn(on_change(
debounced_receiver,
// ...
)),
}
}
#[throws]
pub async fn stop(self) {
self.watcher.stop().await?;
self.task.await??;
}
}
As you can see, there is another stop
method that calls the previous stop
method and the remaining code is very similar to the ProjectWatcher
implementation.
Let's look at the last snippet to know the whole watcher story (/crates/mzoon/src/command/start.rs):
#[throws]
pub async fn start(release: bool, open: bool) {
// ...
let frontend_watcher = build_and_watch_frontend(&config, release).await?;
let backend_watcher = build_run_and_watch_backend(&config, release, open).await?;
signal::ctrl_c().await?;
println!("Stopping watchers...");
let _ = join!(
frontend_watcher.stop(),
backend_watcher.stop(),
);
println!("Watchers stopped");
}
#[throws]
async fn build_and_watch_frontend(config: &Config, release: bool) -> FrontendWatcher {
if let Err(error) = build_frontend(release, config.cache_busting).await {
eprintln!("{}", error);
}
FrontendWatcher::start(&config, release, DEBOUNCE_TIME).await?
}
So I can imagine there are some opportunities for another refactor round:
- "Hide" loops and debouncer inside
Stream
s. - Use
notify
's debouncer once it's integrated into the library. - Use async drops once Rust supports them or an alternative.
- See the related article Asynchronous Destructors from the
fehler
's author.
- See the related article Asynchronous Destructors from the
- If you want to investigate the option "Wait until all task done" so we can just abort all tasks in a standard
drop
and then wait for async runtime to finish, there is the entrance to the rabbit hole.
Feel free to create a PR when you manage to simplify the code.
File Compressors
Frontend files are served compressed to get them quickly to users and to reduce network traffic and server load. Only app files (in the pkg
directory) are compressed at the moment but we'll probably compress the entire public
folder in the future.
mzoon compresses files when the app has been built in the release mode. The result is three files instead of one: file.xxx
(the original), file.xxx.gz
and file.xxx.br
. Then Moon serves them according to the ACCEPT_ENCODING
header sent by clients.
We would use only Brotli algorithm because it produces the smallest files but Firefox supports only Gzip over HTTP. All browsers support Brotli with HTTPS.
Note: If we decide to compress non-cacheable dynamic content - like messages between frontend and backend - then we will probably choose Gzip because it's faster than Brotli.
Let's look at the implementation. The first snippet is from /crates/mzoon/src/helper/file_compressor.rs:
use crate::helper::ReadToVec;
use async_trait::async_trait;
use brotli::{enc::backward_references::BrotliEncoderParams, CompressorReader as BrotliEncoder};
use flate2::{bufread::GzEncoder, Compression as GzCompression};
// ...
#[async_trait]
pub trait FileCompressor {
async fn compress_file(content: Arc<Vec<u8>>, path: &Path, extension: &str) -> Result<()> {
let path = compressed_file_path(path, extension);
let mut file_writer = fs::File::create(&path)
.await
.with_context(|| format!("Failed to create the file {:#?}", path))?;
let compressed_content = spawn_blocking(move || Self::compress(&content)).await??;
file_writer.write_all(&compressed_content).await?;
file_writer.flush().await?;
Ok(())
}
fn compress(bytes: &[u8]) -> Result<Vec<u8>>;
}
//...
// ------ Brotli ------
pub struct BrotliFileCompressor;
#[async_trait]
impl FileCompressor for BrotliFileCompressor {
fn compress(bytes: &[u8]) -> Result<Vec<u8>> {
BrotliEncoder::with_params(bytes, 0, &BrotliEncoderParams::default()).read_to_vec()
}
}
// ------ Gzip ------
pub struct GzipFileCompressor;
#[async_trait]
impl FileCompressor for GzipFileCompressor {
fn compress(bytes: &[u8]) -> Result<Vec<u8>> {
GzEncoder::new(bytes, GzCompression::best()).read_to_vec()
}
}
#[async_trait]
allows us to writeasync
methods in traits. (The crate async_trait, from the author ofanyhow
andthiserror
.)The combination of
async-trait
andfehler
is deadly for the Rust compiler. That's why you seeOk(())
+Result<()>
instead of#[throws]
. I'm not sure if it'sasync-trait
orfehler
problem, feel free to investigate it more and let me know.We need to call
spawn_blocking
instead ofspawn
to move compression to a new thread because both encoders / compressors are blocking. I was trying to use async-compression, but there was a bug probably somewhere close to theGzEncoder
- the MZ examplecounter
was producing a wasm file that had always only9KB
instead of16KB
. Also I had to useasync-compression
'sfutures
encoders with the compat layer to resolve the problem with incompatibletokio
versions. Feel free to investigate it more and let me know.Tip: Don't forget to call
.flush()
after.write_all()
. Sometimes it works without.flush()
, sometimes it doesn't, so it's difficult to debug.read_to_vec
is a custom helper - see /crates/mzoon/src/helper/read_to_vec.rs.Both encoders are set to compress in the best quality (i.e. to produce the smallest files at the cost of speed).
The second and the last snippet is from /crates/mzoon/src/build_frontend.rs:
use futures::TryStreamExt;
#[throws]
async fn compress_pkg(wasm_file_path: impl AsRef<Path>, js_file_path: impl AsRef<Path>) {
try_join!(
create_compressed_files(wasm_file_path),
create_compressed_files(js_file_path),
visit_files("frontend/pkg/snippets")
.try_for_each_concurrent(None, |file| create_compressed_files(file.path()))
)?
}
#[throws]
async fn create_compressed_files(file_path: impl AsRef<Path>) {
let file_path = file_path.as_ref();
let content = Arc::new(fs::File::open(&file_path).await?.read_to_vec().await?);
try_join!(
BrotliFileCompressor::compress_file(Arc::clone(&content), file_path, "br"),
GzipFileCompressor::compress_file(content, file_path, "gz"),
)
.with_context(|| format!("Failed to create compressed files for {:#?}", file_path))?
}
All files are compressed and generated in parallel thanks to
spawn_blocking
(explained before) and thanks totokio::fs
(we don't block the working thread by waiting for OS file operations).visit_files
is a stream of files (explained in the next section). It works nice with the function try_for_each_concurrent.
File Visitor
When you want to iterate over all files in the given directory and its nested folders, then it's relatively straightforward with the standard Rust library. Just go to the Rust docs for std::fs::read_dir
and copy the provided example. Also there is chance we'll see the function fs::read_dir_all in std
. Or you can use the crate walkdir from a very experienced maintainer of many Rust libraries.
However the Rust async world is still pretty new and messy. If I chose smol
instead of tokio
and was brave enough to use the library with only 602 downloads, then I would probably integrate the crate async_walkdir.
Another approach would be to use walkdir
to create a list of files and then process the list as needed in parallel. However it doesn't sound as a clean solution and in the case of a large directory tree, you want to return early when the processing fails or when your file search is complete.
I'm not a big fan or recursive functions because:
- They often lead to increased cognitive load.
- Stack overflow is difficult to catch and debug.
- Rust doesn't have a good support for TCO/TCE (tail call optimization / elimination), although there are some libraries like Tailcall and maybe promising news in rust-lang/rfcs.
- You often need to use
Box
in Rust recursive constructs (both functions and types need boxed items). The crate async-recursion basically just wraps theFuture
into aBox
. - Why does NASA not allow recursion?
Fortunately during intensive reading and searching for a better solution, I've found a nice answer on stackoverflow.com compatible with tokio
and futures
. I've refactored it a little bit and saved to /crates/mzoon/src/helper/visit_files.rs. The code:
pub fn visit_files(path: impl Into<PathBuf>) -> impl Stream<Item = Result<DirEntry>> + Send + 'static {
#[throws]
async fn one_level(path: PathBuf, to_visit: &mut Vec<PathBuf>) -> Vec<DirEntry> {
let mut dir = fs::read_dir(path).await?;
let mut files = Vec::new();
while let Some(child) = dir.next_entry().await? {
if child.metadata().await?.is_dir() {
to_visit.push(child.path());
} else {
files.push(child)
}
}
files
}
stream::unfold(vec![path.into()], |mut to_visit| {
async {
let path = to_visit.pop()?;
let file_stream = match one_level(path, &mut to_visit).await {
Ok(files) => stream::iter(files).map(Ok).left_stream(),
Err(error) => stream::once(async { Err(error) }).right_stream(),
};
Some((file_stream, to_visit))
}
})
.flatten()
}
(Let me know if you know a better solution or a suitable library.)
Wasm-pack installer
I hate complicated installations and configurations, especially if they aren't cross-platform. In an ideal world, we would just write cargo install mzoon
, hit enter and done. Unfortunately it isn't so simple even in the Rust + Cargo world.
The Rust compiler is pretty slow so if there is a chance to avoid compilation, we should use it. It applies especially for CI pipelines. So we have to download pre-compiled binaries. But to use binaries, we firstly have to answer these questions:
- What are available binary versions / supported platforms?
- What is our platform?
- Where we should download
wasm-pack
? - How should we download and unpack
wasm-pack
?
--
1) What are available binary versions / supported platforms?
wasm-pack
's repo has associated build pipelines for multiple platforms. So we can just look at the release assets.
The current list (version 0.9.1
):
-
wasm-pack-init.exe
(7.16 MB) -
wasm-pack-v0.9.1-x86_64-apple-darwin.tar.gz
(2.97 MB) -
wasm-pack-v0.9.1-x86_64-pc-windows-msvc.tar.gz
(2.69 MB) -
wasm-pack-v0.9.1-x86_64-unknown-linux-musl.tar.gz
(5.02 MB)
Note: wasm-pack-init.exe
is actually an uncompressed Windows binary with a different name.
--
2) What is our platform?
There are multiple ways to determine the platform. Two of them are used in mzoon:
There is a build script build.rs
in the mzoon crate with this code:
fn main() {
println!(
"cargo:rustc-env=TARGET={}",
std::env::var("TARGET").unwrap()
);
}
The only purpose is to "forward" the environment variable TARGET (available only during the build process) to the compilation. Then we can read it in the mzoon code (/crates/mzoon/src/wasm_pack.rs):
const TARGET: &str = env!("TARGET");
Unfortunately we can't use it directly because there are cases where it's too strict. For example, Heroku build pipeline is identified as x86_64-unknown-linux-gnu
but we have the binary only for x86_64-unknown-linux-musl
. However the available binary works even in that Heroku pipeline. So we need more relaxed platform matching in practice:
cfg_if! {
if #[cfg(target_os = "macos")] {
const NEAREST_TARGET: &str = "x86_64-apple-darwin";
} else if #[cfg(target_os = "windows")] {
const NEAREST_TARGET: &str = "x86_64-pc-windows-msvc";
} else if #[cfg(target_os = "linux")] {
const NEAREST_TARGET: &str = "x86_64-unknown-linux-musl";
} else {
compile_error!("wasm-pack pre-compiled binary hasn't been found for the target platform '{}'", TARGET);
}
}
-
Note: The macro
cfg_if
belongs to the crate cfg_if.
In the code above I assume mzoon will be compiled only on the most common platforms. When someone wants to compile mzoon on other platforms, we will need to add a fallback to cargo install
. There is also a small check to inform you that you may have incompatible platform:
if TARGET != NEAREST_TARGET {
println!(
"Pre-compiled wasm-pack binary '{}' will be used for the target platform '{}'",
NEAREST_TARGET, TARGET
);
}
The example output from the Heroku build log:
Building frontend...
Installing wasm-pack...
Pre-compiled wasm-pack binary 'x86_64-unknown-linux-musl' will be used for the target platform 'x86_64-unknown-linux-gnu'
wasm-pack installed
--
3) Where we should download
wasm-pack
?
wasm-pack
contains a self-installer, triggered when the executable name starts with "wasm-pack-init".
The self-installer copy itself next to the rustup
executable to make sure it's in PATH
.
- It isn't a bad idea but there will be problems when multiple MZ projects will need different
wasm-pack
versions.
wasm-pack
uses the crate binary-install to install its binary dependencies like wasm-bindgen
or wasm-opt
. Those binaries are saved into an OS-specific global cache folder, determined by the function dirs_next::cache_dir()
from the crate dirs_next.
- It's a better idea, but we still have to manage different
wasm-pack
versions and I don't like to use global caches too much because it's difficult to remove old files when they are no longer needed.
So the remaining option is to download wasm-pack
directly into the user project. We can store it in the target
directory. But I think the best option is the frontend
directory. Users can use wasm-pack
directly if they need it - e.g. to run tests until the test
command is implemented in mzoon.
--
4) How should we download and unpack
wasm-pack
?
/crates/mzoon/src/helper/download.rs
#[throws]
pub async fn download(url: impl AsRef<str>) -> Vec<u8> {
reqwest::get(url.as_ref())
.await?
.error_for_status()?
.bytes()
.await?
.to_vec()
}
-
reqwest is popular, universal, async and based on
tokio
.
/crates/mzoon/src/wasm_pack.rs
const DOWNLOAD_URL: &str = formatcp!(
"https://github.com/rustwasm/wasm-pack/releases/download/v{version}/wasm-pack-v{version}-{target}.tar.gz",
version = VERSION,
target = NEAREST_TARGET,
);
// ...
download(DOWNLOAD_URL)
.await
.context(formatcp!(
"Failed to download wasm-pack from the url '{}'",
DOWNLOAD_URL
))?
.apply(unpack_wasm_pack)
.context("Failed to unpack wasm-pack")?;
// ...
#[throws]
fn unpack_wasm_pack(tar_gz: Vec<u8>) {
let tar = GzDecoder::new(tar_gz.as_slice());
let mut archive = Archive::new(tar);
for entry in archive.entries()? {
let mut entry = entry?;
let path = entry.path()?;
let file_stem = path
.file_stem()
.ok_or(anyhow!("Entry without a file name"))?;
if file_stem != "wasm-pack" {
continue;
}
let mut destination = PathBuf::from("frontend");
destination.push(path.file_name().unwrap());
entry.unpack(destination)?;
return;
}
Err(anyhow!(
"Failed to find wasm-pack in the downloaded archive"
))?;
}
- The
formatcp!
macro is exported from the nice library const_format. - Decompressing and unpacking is sync (aka blocking), but we still need to wait for
wasm-pack
installation before we can do anything else (e.g. build frontend). So why complicate our lives with extra wrappers likespawn_blocking
? Also we can unblock it later when needed.
And that's all for today!
Thank You for reading and I hope you are looking forward to the next episode.
Martin
P.S.
We are waiting for you on Discord.
Top comments (0)