Please note

Since this blog was published a lot has happened. The Hooks Amendment is now running on a public testnet and has a public "Builder" where you (devs) can code Hooks and deploy them to the testnet, straight from your browser.

This is a progress update blog for the up-coming XRPL Hooks amendment. See previous blogs:

• Hooked #1: Smart Contracts on the XRP ledger
• Hooked #2: Hooks & Security (Smart Contracts on the XRP ledger)
• Hooked #3: Tech Preview Release (Play with Smart Contracts on the XRP ledger!)

Hooks runtime engine

During initial prototyping we were laser focused on getting any Web Assembly Runtime engine connected into xrpld (rippled) as quickly as possible, to preview how the Hooks Amendment might look (see blog 3). To develop the amendment we use an iterative design methodology. This helps us discover the hurdles we don't yet know about early in the project at the expense of producing ultimately unused code.

Wasmer Issues

Thus far we have used a Rust project called Wasmer as the Hooks runtime engine. This is the library that would take the Hook bytecode, which has been submitted by a user using a HookSet transaction on their XRPL account, and evaluate it, producing the Hook's output.

There emerged a few problems with this engine:

1. We do not have experienced Rust developers on staff, so updates to Wasmer (custom fork) would be generally infeasible.
2. Wasmer needs to either be built externally to the xrpld build engine and drawn in as a library (which makes Hooks less efficient than they could be) or it needs to be compiled inside the xrpld build system which would result in additional build dependencies, namely a Rust compiler toolchain.
3. Wasmer is an ongoing project with certain important missing C interfaces that the core developers are in no hurry to add. For example there is currently no way to terminate early from inside a host function without forking Wasmer.

This is not to say Wasmer or Rust is bad, it just wasn't a great fit for our use-case.

Hello SSVM

The writing was on the wall: we needed a C++ web assembly runtime engine. This would allow us to have a unified build system without additional toolchains, and allow for optimal integration (Hooks would run at the best possible speed) and allow us to modify the engine as needed.

Enter SSVM a high performance engine written in modern C++ with profiling and gas metering already built in.

Development proceeding with the SSVM engine can be found on the hooks-ssvm branch of the public Hooks repository.

Speed Is Everything

To profile the new engine we ran the Carbon hook (which takes 1% of your outgoing transaction and creates a new transaction to a carbon offset account).

At time of writing the following is the average runtime profile on commodity hardware (n=100 runs):

Note: . = decimal point, us = microseconds

Total execution time: 487.37 us
Wasm instructions execution time: 177.53 us
Host functions execution time: 309.84 us
Executed wasm instructions count: 4267
Instructions per second: 24483623.94

This is already 2-10x faster than the same code running on Wasmer. However the majority of execution time is now spent running Host functions. This means SSVM is already very efficient spending just 177 microseconds to execute the hook code itself. The Host functions are our code (the Hook API) which can always be optimised further.

Build Times

xrpld build times are notoriously slow. Upto 20 minutes on commodity hardware, depending on resources available to the compiler. This is partially because of the complexity of the codebase and partially because of the complexity of the dependencies.

For work-flow purposes, particularly debugging, long build times are undesirable. Therefore in an ongoing effort to remain optimally productive we are modifying the build processes we use.

Much of what the compiler spends its time doing during a build is re-compiling code it already compiled last build. This could be fixed by an improved build system (that doesn't incorrectly rebuild sources it's already built). However the build system is large and complex and doing this is its own project. Think of the build system as low interest technical debt to which we want an inexpensive bandaid to avoid diverting development resources away from the main goal.

xrpld's unit tests are only needed in selective builds but are currently built every build. Stripping these out reduced our compile times to around 5 minutes.

Adding a caching compiler was the next approach. Installing ccache on our development machine has made a huge difference to build times which are now typically under 60 seconds. A caching compiler keeps a hash of source code and compiler flags and the output that produced last time. If the same inputs are provided it skips the compile altogether and provides the same output as last time.

These efforts are ongoing.

What Next?

The development road ahead for Hooks is still long. Writing major changes to public blockchains requires care and attention. While it appears we have made huge strides thus far with the release of the Tech Preview last year, the reality of iterative design is that 80% of the work is after the first fully formed prototype. The next major milestone is the Hooks public testnet to be released this quarter (Q1 2021). Stay tuned!

Please check the other blogs, the source & README's and ... just a little more patience as we're on track for a Q1 public testnet release :)

Discussion

Discussions: here »

Photo by Veri Ivanova on Unsplash