Hyperimport - Import c, rust, zig etc. files in TypeScript

"What? Did I read the title correct?"

Yes! Let's jump into an example right in the beginning, 'cause why not?

You'll be able to do this,

index.ts

import { add } from "./add.rs";
console.log(add(5, 5)); // 10

add.rs

#[no_mangle]
pub extern "C" fn add(a: isize, b: isize) -> isize {
    a + b
}

...and even more, like importing native C functions from libc in typescript. Check out the guide.

"Wait! What?? How is that even possible!?"

Long back, when I was working on the project webview-bun, which essentially is an FFI wrapper of the webview library APIs for Bun. It randomly struck my mind, why can't I import the webview C source file directly into typescript, if through FFI API we can import functions from shared libraries which are essentially compiled from a source file like c, rust, zig, etc., how about I create a way to bridge importing functions in typescript with the source file and automate and the steps in between, in a way that it looks to the end user that they are directly importing from the source file while all the hard parts are managed automatically internally.

I wrote a simple function called calc in zig to add two numbers. In typescript, I wrote an import function that would take the path to that zig file, spawn a child process to invoke the zig compiler to compile that file into a shared library file then open that shared library using FFI API and return the symbols which essentially contains the calc function. So when I used the function to import the zig file, those internal steps happened behind the scenes and the calc function worked. Then when I changed the operation inside the zig function from addition to subtraction and executed the typescript file, those steps happened again essentially recompiling the file and the new output reflected the changes. This is how the typescript file looked.

const { calc } = $import("./calc.zig");
console.log(calc(1, 2));

It appeared as if it was black magic in that function, importing the function directly from Zig but under the hood, the file is compiled into a shared library and it is importing from the library and not the source file itself. But, the syntax looked super clean and easy to comprehend. I recorded my screen showing this experimental prototype and posted it on Bun's discord server.

Soon, Jarred came across the message and replied that I could make use of the buntime (we'll call runtime as buntime, 'cause why not) Plugin API and implement my logic as a plugin which would allow me to use ES6 imports instead of that weird looking import function.

Honestly, before this, I never actually used the Plugin API, so I started diving into it. With some fair complications and a few rewrites, I was finally able to port that logic over to use the Plugin API. Now I could easily import the zig file using ES6 import syntax. Even though Typescript was still shouting at me because it doesn't know what calc.zig is or what the calc function is, it still blew my mind away because it looked terrifying.

import { calc } from "./calc.zig";
console.log(calc(1, 2));

So, I decided to make it even more terrifying. I added types.

Using typescript's wildcard module declaration feature, I created a types.d.ts where I declared that zig file path as a module, inside which I added type definitions for the calc function. Now typescript is happy and when I hovered over the calc function, the types are working great as expected. The whole combination looked perfect, but it was still a static prototype and not anything people would be able to use in their projects. I recorded my screen showcasing this scary syntax with black magic happening in the background and even showed when I changed the operation from addition to subtraction, the changes were still reflected. I recorded for both a Zig and a Rust file and posted them on the server again. Soon, Jarred reposted both of the videos on Twitter here showcasing the power of bun. Everyone in the comments of the tweet went crazy to see something like this even possible.

Primeagen commented, "This is super cool. Do you have any articles or anything I can read on this?", so I decided to write this article for him :)

People kept blowing up my discord asking me when I'll release it. I apologize for making them wait but its now out, finally!

The most difficult part was to make things dynamic because, in that prototype, I had to manually declare the FFI symbol definitions like argument and return types for the calc function and also manually wrote the type definitions too. I wanted things to be automated as much as possible and the user to have full control of every aspect when they would use it in their project. I wanted flexibility which put me into thinking what's the best approach to making this prototype into a real thing people would use in their projects. I experimented with various approaches but all failed in some way which decreased my motivation to work further on this project. I abandoned it for a long time and I also had university stuff going on in between.

My goal was, that I wanted the users...

• ...to add support for any other kind of other language which isn't available by default, easily.
• ...to be able to swap implementations with their custom logic.
• ...to import any other kind of plugin and not just be restricted to loading other language files.

Three months later, last week I started from scratch. It was my fourth or fifth attempt at rewriting from scratch. But every time, I started all over again I had experience of the old failed ideas. This final time, I decided to approach this by taking advantage of inheritance, basically classes. Where I split the whole logic into their functions allowing the user to extend and override them essentially swapping the implementations with their custom ones. Took me a week to get everything working as I wanted it to. My most important goals are achieved.

• The Loader class can be extended by the user and functions can be overridden with custom implementations to customize the behavior.
• Any kind of plugin can be imported through hyperimport.

Not to mention but, this idea was also featured on the official Bun 1.0 launch video. Watch here.

"Can I use it? Is it on github??"

Absolutely! Check out the Hyperimport repository and browse the wiki for a comprehensive documentation with guides.

Check out Importing a rust file, for a step by step guide on setting up a basic hyperimport project.

Feel free to join the discord server, if you have any questions.

If you have come this far, thank you so much for reading the article. The story behind a crazy experimental idea turned into reality. I am really excited and will be looking forward to what mind blowing ideas people can make use of it and push it's limits.