Implementing Webpack from Scratch, But in Rust - [4] Implement Plugin System

#webpack #rust

Referencing mini-webpack, I implemented a simple webpack from scratch using Rust. This allowed me to gain a deeper understanding of webpack and also improve my Rust skills. It's a win-win situation!

Code repository: https://github.com/ParadeTo/rs-webpack

This article corresponds to the Pull Request: https://github.com/ParadeTo/rs-webpack/pull/5

We know that Webpack uses Tapable to implement its plugin system. So, it seems reasonable to write a Rust version inspired by it. However, implementing it turned out to be not as simple as expected. Let's take SyncHook as an example. A very simple version of it can be implemented in JavaScript like this:

class SyncHook {
  constructor() {
    this.taps = []
  }
  tap(options, fn) {
    this.taps.push(fn)
  }
  call() {
    this.taps.forEach((fn) => fn(...arguments))
  }
}

const hook = new SyncHook(['param1', 'param2']) // Create a hook object
hook.tap('event1', (param) => console.log('event1:', param))
hook.tap('event2', (param) => console.log('event2:', param))
hook.tap('event3', (param, param2) => console.log('event3:', param, param2))
hook.call('hello', 'world')

Let's try to implement it in Rust. We might write code like this:

struct SyncHook<Arg: Copy, R> {
    taps: Vec<Box<dyn Fn(Arg) -> R>>
}

impl<Arg: Copy, R> SyncHook<Arg, R> {
    fn tap(&mut self, f: Box<dyn Fn(Arg) -> R>) {
        self.taps.push(f);
    }

    fn call(&self, a: Arg) {
        for tap in self.taps.iter() {
            tap(a);
        }
    }
}

fn main() {
    let mut sync_hook = &mut SyncHook{taps: vec![]};
    sync_hook.tap(Box::new(|arg| {
        println!("event {}", arg);
    }));
    sync_hook.call("hello")
}

Note that Arg must be constrained to be Copy, otherwise calling tap(a) will result in an error. The above code can run correctly, but it only supports the case where call takes a single argument. However, this can be solved using macros. We can pre-generate a set of structs that support different numbers of arguments:

struct SyncHook1<Arg1: Copy, R> {
    taps: Vec<Box<dyn Fn(Arg1) -> R>>
}

struct SyncHook2<Arg1: Copy, Arg2: Copy, R> {
    taps: Vec<Box<dyn Fn(Arg1, Arg2) -> R>>
}

However, there is still a problem: it does not support passing arguments of type &mut T:


struct Compiler {
    name: String
}

fn main() {
    let mut sync_hook = &mut SyncHook{taps: vec![]};
    sync_hook.tap(Box::new(|arg| {
        println!("event {}", arg);
    }));
    let compiler = &mut Compiler {name: String::from("test")};
    sync_hook.call(compiler)
}

The above code will throw an error: "the trait Copy is not implemented for &mut Compiler".

It seems that implementing a similar functionality is quite challenging for me as a beginner. So, let's take a look at Rspack and see how they approach it.

Upon investigation, I found that Rspack does not have a generic SyncHook. Instead, it defines each hook separately using macros. Let's try to incorporate it into our project.

First, let's copy the code from Rspack's source code, specifically the crates/rspack_macros and crates/rspack_hook directories, into the crates directory of our rs-webpack project. Make sure to rename the directories accordingly.

├── crates
│   ├── rswebpack_hook
│   └── rswebpack_macros

Then, let's add a new module called rswebpack_error as our unified error handling module:

// lib.rs
use anyhow::Result as AnyhowResult;

pub type Result<T> = AnyhowResult<T>;

Finally, we need to modify the dependencies in these libraries accordingly. For example, in rswebpack_hook, change Result from rspack_error::Result to rswebpack_error::Result.

Let's write a demo to test it:

use rswebpack_macros::{define_hook, plugin, plugin_hook};

struct People {
    name: String
}

define_hook!(Test: SyncSeries(people: &mut People));

#[plugin]
struct TestHookTap1;

#[plugin_hook(Test for TestHookTap1)]
fn test1(&self, people: &mut People) -> Result<()> {
    people.name += " tap1";
    Ok(())
}

#[plugin]
struct TestHookTap2;

#[plugin_hook(Test for TestHookTap2)]
fn test2(&self, people: &mut People) -> Result<()> {
    people.name += " tap2";
    Ok(())
}

fn main() {
    let mut test_hook = TestHook::default();
    test_hook.tap(test1::new(&TestHookTap1::new_inner()));
    test_hook.tap(test2::new(&TestHookTap2::new_inner()));
    let people = &mut People { name: "ayou".into() };
    test_hook.call(people);
    println!("{}", people.name); // ayou tap1 tap2
}

Let me explain the code above. First, we define a hook using define_hook!(Test: SyncSeries(people: &mut People));. It expands to something like this:

pub trait Test {
    fn run(&self, people: &mut People) -> rswebpack_hook::__macro_helper::Result<()>;
    fn stage(&self) -> i32 { 0 }
}
pub struct TestHook {
    taps: Vec<Box<dyn Test + Send + Sync>>,
    interceptors: Vec<Box<dyn rswebpack_hook::Interceptor<Self> + Send + Sync>>,
}
impl rswebpack_hook::Hook for TestHook {
    type Tap = Box<dyn Test + Send + Sync>;
    fn used_stages(&self) -> rswebpack_hook::__macro_helper::FxHashSet<i32> { rswebpack_hook::__macro_helper::FxHashSet::from_iter(self.taps.iter().map(|h| h.stage())) }
    fn intercept(&mut self, interceptor: impl rswebpack_hook::Interceptor<Self> + Send + Sync + 'static) { self.interceptors.push(Box::new(interceptor)); }
}
impl std::fmt::Debug for TestHook { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { write!(f, "TestHook") } }
impl Default for TestHook { fn default() -> Self { Self { taps: Default::default(), interceptors: Default::default() } } }
impl TestHook {
    pub fn call(&self, people: &mut People) -> rswebpack_hook::__macro_helper::Result<()> {
        let mut additional_taps = std::vec::Vec::new();
        for interceptor in self.interceptors.iter() { additional_taps.extend(interceptor.call_blocking(self)?); }
        let mut all_taps = std::vec::Vec::new();
        all_taps.extend(&self.taps);
        all_taps.extend(&additional_taps);
        all_taps.sort_by_key(|hook| hook.stage());
        for tap in all_taps { tap.run(people)?; }
        Ok(())
    }
    pub fn tap(&mut self, tap: impl Test + Send + Sync + 'static) { self.taps.push(Box::new(tap)); }
}

You can see that TestHook is similar to our previous implementation, and it also generates the type Test for the tap function of TestHook.

Next, let's use a macro to implement a Tap for Test:

#[plugin]
struct TestHookTap1;

#[plugin_hook(Test for TestHookTap1)]
fn test1(&self, people: &mut People) -> Result<()> {
    people.name += " tap1";
    Ok(())
}

It expands to something like this:

struct TestHookTap1 {
    inner: ::std::sync::Arc<TestHookTap1Inner>,
}
impl TestHookTap1 {
    fn new_inner() -> Self { Self { inner: ::std::sync::Arc::new(TestHookTap1Inner) } }
    fn from_inner(inner: &::std::sync::Arc<TestHookTap1Inner>) -> Self { Self { inner: ::std::sync::Arc::clone(inner) } }
    fn inner(&self) -> &::std::sync::Arc<TestHookTap1Inner> { &self.inner }
}
impl ::std::ops::Deref for TestHookTap1 {
    type Target = TestHookTap1Inner;
    fn deref(&self) -> &Self::Target { &self.inner }
}
#[doc(hidden)]
pub struct TestHookTap1Inner;

#[allow(non_camel_case_types)]
struct test1 {
    inner: ::std::sync::Arc<TestHookTap1Inner>,
}
impl test1 { fn new(plugin: &TestHookTap1) -> Self { test1 { inner: ::std::sync::Arc::clone(plugin.inner()) } } }
impl TestHookTap1 {
    #[allow(clippy::ptr_arg)]
    fn test1(&self, people: &mut People) -> Result<()> {
        people.name += " tap1";
        Ok(())
    }
}
impl ::std::ops::Deref for test1 {
    type Target = TestHookTap1Inner;
    fn deref(&self) -> &Self::Target { &self.inner }
}
impl Test for test1 {
    #[tracing::instrument(name = "TestHookTap1::test1", skip_all)]
    fn run(&self, people: &mut People) -> Result<()> {
        TestHookTap1::test1(&TestHookTap1::from_inner(&self.inner), people )
    }
}

It may seem a bit convoluted at first, but with a few more readings, it becomes understandable.

In this way, we have implemented Tapable-like functionality in Rust. However, we have only demonstrated the usage of the simplest SyncHook for now. We will introduce other hooks in the future.

Next, let's build a plugin system based on this. As we know, implementing a plugin for webpack is typically done like this:

const pluginName = 'ConsoleLogOnBuildWebpackPlugin'

class ConsoleLogOnBuildWebpackPlugin {
  apply(compiler) {
    compiler.hooks.run.tap(pluginName, (compilation) => {
      console.log('The webpack build process is starting!')
    })
  }
}

module.exports = ConsoleLogOnBuildWebpackPlugin

We follow the same pattern and start by defining a Plugin trait to specify the characteristics a plugin should have:

#[async_trait::async_trait]
pub trait Plugin: std::fmt::Debug {
    fn name(&self) -> &'static str {
        "unknown"
    }

    fn apply(&self, _ctx: PluginContext<&mut ApplyContext>) -> Result<()> {
        Ok(())
    }
}

#[derive(Debug, Default)]
pub struct PluginContext<T = ()> {
    pub context: T,
}

#[derive(Debug)]
pub struct ApplyContext<'c> {
    pub compiler_hooks: &'c mut CompilerHooks,
}

define_hook!(BeforeRun: SyncSeries(compiler: &mut Compiler));

#[derive(Default, Debug)]
pub struct CompilerHooks {
    pub before_run: BeforeRunHook,
}

As you can see, we pass the before_run hook as context to the plugins.

Next, we define a PluginDriver to drive the plugins:

pub struct PluginDriver {
    pub plugins: Vec<Box<dyn Plugin>>,
    pub compiler_hooks: CompilerHooks,
}

impl PluginDriver {
    pub fn new(plugins: Vec<Box<dyn Plugin>>) -> Arc<Self> {
        let mut compiler_hooks = CompilerHooks::default();
        let mut apply_context = ApplyContext {
            compiler_hooks: &mut compiler_hooks,
        };

        for plugin in &plugins {
            plugin
                .apply(PluginContext::with_context(&mut apply_context))
                .expect("failed to apply plugin context");
        }

        Arc::new(Self {
            plugins,
            compiler_hooks,
        })
    }
}

Here, we initialize the parameters passed to each plugin, iterate over the plugins, and call their apply method.

Finally, the PluginDriver is used in the Compiler:

impl Compiler {
    pub fn new(mut config: Config, plugins: Vec<BoxPlugin>) -> Compiler {
        let plugin_driver = PluginDriver::new(plugins);

        Compiler {
            root: config.root.clone(),
            entry_id: "".to_string(),
            config,
            modules: HashMap::new(),
            assets: HashMap::new(),
            plugin_driver,
        }
    }
    ...

We temporarily modify the run method of Compiler:

pub fn run(&mut self) {
    self.plugin_driver.clone().compiler_hooks.before_run.call(self);
    // let resolved_entry = Path::new(&self.root).join(&self.config.entry);
    // self.build_module(resolved_entry, true);
    // self.emit_file();
}

Now, let's write a demo to test it:

use rswebpack_core::compiler::Compiler;
use rswebpack_core::config::{Config, Output};
use rswebpack_core::hooks::BeforeRun;
use rswebpack_core::plugin::{ApplyContext, Plugin, PluginContext};
use rswebpack_macros::{plugin, plugin_hook};
use rswebpack_error::Result;

#[plugin]
struct BeforeRunHookTap;

#[plugin_hook(BeforeRun for BeforeRunHookTap)]
fn before_run(&self, compiler: &mut Compiler) -> Result<()> {
    println!("Root is {}", compiler.root);
    Ok(())
}

#[derive(Debug)]
struct TestPlugin;

impl Plugin for TestPlugin {
    fn apply(&self, _ctx: PluginContext<&mut ApplyContext>) -> Result<()> {
        _ctx.context
            .compiler_hooks
            .before_run
            .tap(before_run::new(&BeforeRunHookTap::new_inner()));
        Ok(())
    }
}

fn main() {
    let config = Config::new(
        "test".to_string(),
        "test".to_string(),
        Output {
            path: "out".to_string(),
            filename: "bundle".to_string(),
        },
    );
    let compiler = &mut Compiler::new(config, vec![Box::new(TestPlugin {})]);
    compiler.run(); // Root is test
}

There you have it! However, in actual development, custom plugins are usually developed in JavaScript. How can we integrate these JavaScript plugins? We'll reveal the answer in the next part.

Please kindly give me a star!