Cover Photo by Markus Spiske on Unsplash
Disclaimer: I work at Keyrock but views expressed here are my own. However, if you like the views, do go checkout the company as it is full of awesome people and I have learned much whilst working there, including Rust.
If you haven't read Parts 1 & 2, I highly recommend you start with those:
- Syntax
- Separation of Data and Code
- Type safety
- Declarative Programming and Functional Composition features
- Evolved package management
- Concurrency ready std lib and great async support
- Clear, helpful compiler errors
- Zero cost abstractions (Optimisation)
- Energy efficiency
- WASM
- Tauri - High performance, secure, cross-platform desktop computing
Part 3
- Macros!
- Corporate Engagement
- Cryptocurrency and Blockchains
- Everybody else's tooling
- Speed
- No garbage collection
- Memory safety features
Macros!
Some people may be scarred and jaded from out of control macro usage in certain languages (especially low-level languages). In the world of Ruby people would decry "Rails" developers and later in the Elixir universe, people were encouraged to start without Phoenix, that giant macro-driven-code-generating beauty, for fear that they would never really explore the language properly and become "Phoenix" developers; I understand why - but in these examples - like with Elixir metaprogramming (using macros to generate and change code) - Macros were generally used for good, and supercharged the language the right way. Usually.
Macros in Rust are generally very sensible, and easy to identify - either with their exclamation mark identifiers for declarative style macros (like println!() in the std lib), or used with derive or as custom function attributes, which usually generate code from your code.
Generally the macro system is a platform for extended functionality and simplifying the interface to libraries. This hints at the potential growth still to come.
So how is this a major factor? I believe Macros are very similar to a custom type system. If the type is heavily documented and openly shared, it becomes part of the language and general knowledge of the system, known by all that claim to be its adherents. Where macros are not maintained, not tested well, not documented well, and are part of a closed system, they can become a knowledge-barrier to those attempting to change/fix/test that system's code. - hence the resistance many people have.
The Rust community has, through common practice, set some pretty heavy expectations for people producing macros that become part of the general knowledge of the language: As an example, the Serde library, known to almost anyone that will use Rust for web service development, is a heavily documented set of macros that allow for very granular transformation when serialising and derserialising data. There is even a Serde "book". Approached in this disciplined way, Rust Macros are going to provide an explosion of metaprogramming in the years to come, that will amplify the existing Rust growth.
More info:
https://happens.lol/posts/a-love-letter-to-rust-macros/
https://jstrong.dev/posts/2020/productive-rust-implementing-traits-with-macros/
Corporate Engagement
We're way past the initial gradual incline slope of Rust growth. At this point, we're approaching hypergrowth. Try and find a company interested in performance systems programming that's not looking at Rust or starting to teach it themselves. With Microsoft and AWS sponsoring major rust packages, it feels very much like all the ducks are already neatly lined up and ready for take-off.
Let's take a tiny peek at how this is happening and what this corporate engagement looks like:
“We’re not just hiring a couple of folks for Tokio and the Rust compiler,” Shane Miller says. “Rust is a critical component of our long-term strategy, and we’re investing to deliver Rust engineering at Amazon scale. That includes developer tools, infrastructure components, interoperability, and verification.”
— AWS Open Source Blog, November 2020
News headline:
"AWS's Shane Miller to head the newly created Rust Foundation"
The Rust Foundation Community Grants Program is made possible by the generous support of corporate partners, with Amazon Web Services (AWS), Google, and Huawei amongst the first contributors...
“A large part of the current success of the Rust Project is due to the vibrant community and volunteers who contribute not only to engineering tasks, but to all aspects of making the Project successful,” said Lars Bergstrom, Google Director of Engineering. “We believe that one way that the Rust Foundation can best support the Rust Project is through providing Community Grants to those critical volunteers.”
News headline:
Google's Android Team is backing an effort to introduce Rust as a second programming language in the Linux kernel.
News headline:
In Rust we trust: Shoring up Apache, ISRG ditches C, turns to wunderkind lang for new TLS crypto module
Subheading: Elder server httpd to be revitalized with Google-funded memory-safe add-on
News headline:
Fuchsia OS uses Rust to write code extensively
"According to the overall code analysis of Fuchsia, Rust has the most lines of code."
— Infotech News, meterpreter, May 2021
News headline:
"Announcing three new Rust Foundation Board Members"
Article detail:
The Rust Foundation is delighted to announce that three new individuals have joined the Board of Directors.
The members:
– Ryan Levick, Developer Advocate, Microsoft
– Andrew Wafaa, Senior Director of Software Communities, Arm
– Eric Garcia, Software Engineering Manager, Meta
— Rust Foundation news blog, March 2022
This is really a tiny proportion of the Rust news out there. Additionally all of the tech giants are launching their own Rust onboarding and training material. As an idea, introducing Rust at a giant enterprise might sound experimental but when everyone's doing it, well, then it's at worst, a fashionable trend; at best, corporates jumping onboard to position themselves around a technology that constitutes part of the core competitive advantage of tomorrows solutions.
More info:
https://docs.microsoft.com/en-us/learn/paths/rust-first-steps/
https://developer.ibm.com/articles/os-using-rust/
Cryptocurrency and Blockchains
Assuming you were not trapped in a CIA black-site for the last 5 years, you've probably seen another unstoppable tidal wave, besides Rust; call it what you want: Web 3.0, The blockchain revolution, The rise of DEFI, NewFI, GameFI, The New Fiat... Cryptocurrency has arrived and it's here to stay (note: I might be slightly biased - see my history in Part 1). And what else has come along with this new industry? A demand for performance: Performance transaction/block processing, performance historical data processing, and performance contract execution - all on various distributed platforms, each claiming superiority over the others, all burning millions in energy (see 'Energy efficiency' in Part 2), and each not wanting live code to fail (see 'Safety features', below) - all so some other excited people can buy $2m (two million dollar) cartoon apes, or a piece of virtual land etc etc. If you were starting one of these projects today, what technology might you decide to bet on?
Here's hint - checkout Substrate, a blockchain framework, written in Rust.
Some Blockchain projects using Rust:
Article title:
"Solana Uses Rust to Pull in Developers and Avoid Copypasta"
News Headline:
"Concordium Launches DevX Initiative to Support Rust Devs"
And when the many-named, many-faced hydra of social data wanted to make a cryptocurrency (Libra/Diem), they also went full Rust, back in 2019.
Everybody else's tooling
There is a naive argument that can be made at some point: "So why doesn't everyone just use Rust?" - well, there are lots of reasons: Legacy code (Cobol, C++ that no-one wants to touch but they want to extend - see all Banks), availability of developers, capability of engineering managers (😇), context and requirements (do we need quick code, or need to code quick?), suitability of application to the context (Some front-end frameworks are very mature - why fix what is not borked?), availability of specialist libraries (e.g. Python/Julia), architecture priority (e.g. we want 100% functional, with Actors), addiction to pain (Haskell, Java, C++), etc, etc.
However, The tooling around other languages does not need to be limited by the limits of those languages. An outcome of energy efficiency requirements, speed, and concurrency features, for example, might lead to someone writing the compiler/interpreter for another language in Rust 🙂 . More interestingly, once it is possible to transpile a language into Rust, it can be recompiled into WASM and run in the browser!
This means the Rust will likely form the core of other languages' tooling as we enter the next decade.
Examples:
A Node.js interpreter written in Rust: https://deno.land/
A web tool pipeline written in Rust: https://swc.rs/
The fastest dataframe library with Python links, written in Rust: https://pola-rs.github.io/polars-book/user-guide/index.html
A Clojure interpreter written in Rust: https://github.com/clojure-rs/ClojureRS
A Python interpreter written in Rust: https://rustpython.github.io/
The Erlang BEAM (VM) written in Rust: https://getlumen.org/
Ok, now we get to the "big three". Often when I am out and about and professionally participating in the local requirement to support the testing of grain, potato, and fruit fermentation operations, I'm often asked "Why Rust?". As the previous 15 factors in these articles would involve a rather extended evening of technical discussions, which can be challenging considering my previously mentioned professional activities, and not to mention immensely boring for non-developers, I usually default to the following three factors.
Speed
It may be obvious that low level compiled languages are faster; but it may not be obvious that they are faster - I mean hundreds and sometimes thousands of times faster than interpreted languages (I didn't realise that this was not common knowledge until speaking to a JS dev recently who said "Thousands? Are you sure?"). Rust speed is generally on par with C++ and in some cases faster - usually when C++ is not "cheating" with what Rust would call unsafe code. There are rules in the world of Rust this is going to add a few nanoseconds of overhead. For most application contexts, the difference will be a moot point.
More interesting is the meeting point of high-level abstractions (see Part 1) and speed. This allows for very high speed code that is easy to maintain, fix, test, and learn (knowledge transition amongst a team).
More info:
https://www.techempower.com/benchmarks/
https://www.logdna.com/blog/coding-for-performance-why-we-chose-rust
https://programming-language-benchmarks.vercel.app/c-vs-rust
https://programming-language-benchmarks.vercel.app/cpp-vs-rust
No garbage collection
Interpreted or compiled, high level languages are usually very memory safe, due to the common language feature of a garbage collector. Memory is tracked and when variables are no longer in scope, the memory is tidied up and we never see a null pointer or experience segmentation fault/memory access violation. In fact most people with a high level language background that I speak with about this subject say the same thing every time: I've never really had to think about memory, memory safety or how I use memory. You can thank your friendly neighbourhood garbage collector.
So what's the issue? Well, the JavaScript garbage collector (GC) can take around 150ms in some cases to finish its "sweep" of memory. That is a long time in a realtime system. Go-lang claims to have the fastest garbage collection and they have managed to eliminate large memory scanning pauses; they claim to have been able to get a GC pause down to 100 microseconds. This is impressive and likely makes GC pauses insignificant for most contexts.
However, what if we are creating an autopilot for an airplane, or a self-driving car? Do we ever want GC pauses? Unless we like to make extremely dangerous (and borderline psychotic) gambles, probably not [There are exceptions, in the case of a well designed system where GC is distributed and controlled, like Pony - but they start with the words "If you know what you are doing..."]. Similarly most High Frequency Trading systems in the traditional financial trading world are written in C++ for the same reason. Even a 100 microsecond pause could lead to a significant loss from that period of "blindness".
This sounds great and you may be thinking "Sure, but I'm making a web-server so I don't think I need to worry about this GC stuff." You are probably right - except maybe from one other perspective; I'm going to hit someone's buzzword bingo here - machine empathy. The lack of a garbage collector makes the code more efficient, as seen by Rust's energy profile, because memory is not being intermittently scanned/deallocated/reallocated etc. This can be understood intuitively: a GC pause is a performance hit that takes place because (usually) all threads must be suspended (stopping of the world) to ensure no memory access takes place during the sweep [Except in the Erlang/Elixir and Pony universes of course]. Multi-core systems can even experience a detrimental impact on throughput with an increase in number of threads (as time spent collecting garbage increases).
Some popular options to avoid GC are then C/C++/Rust/Objective-C/Swift/some other ancient languages like Cobol/Pascal/Fortran. Objective-C & Swift are generally thought of as an Apple device language with associated/relevant tools (although Server-side Swift is apparently a thing now) but not as cross platform general use languages (yet) - and Swift reference counting is thought of as a type of GC (the compiler inserts release/retain code at compile time) so we can remove those. So then, eliminating the ancient languages leaves C/C++... and Rust.
Taking these other factors into consideration, which would you choose?
More info:
https://groups.google.com/g/golang-dev/c/Ab1sFeoZg_8/m/_DaL0E8fAwAJ?pli=1
https://www.oracle.com/java/technologies/javase/gc-tuning-6.html
https://wiki.c2.com/?LanguagesWithoutGarbageCollection
https://docs.swift.org/swift-book/LanguageGuide/AutomaticReferenceCounting.html
Safety features
Rust provides an incredible amount of code safety for a low-level language (more than just the glossed-over object-lifetime tracking that is often mentioned as the main feature). The first line of defence comes from the strong static typing that is use to check all type usage at compile time. Similarly, the Trait system creates a structured way to check for characteristics of generic types in "dynamic" code, meaning a function can accept a generic as long as the generic type expresses (or implements in the Rust terminology) specific Traits. A simple example is that the println!() function (which is actually a std-lib macro) can accept any type but to print it to stdout, the type must implement either the Display trait or the Debug trait (which has different usage but with this same macro). This means the code will never crash because an undisplayable type was sent to this macro - because that code would not compile in the first place!
Memory safety comes from a feature called Ownership. The theory behind ownership is two-fold; no two pieces of code should be able to write to the same piece of memory at once time, and no code should be able to access a memory location that is no longer in scope in its parent/calling function - or rather is "alive" long enough for the usage (the concept of reference Lifetimes). This is primarily expressed by "passing" ownership of variables from one part of the code to another, and by tracking the lifetime of a reference at compile time. References can be borrowed as well, so they can be read but they cannot be written to unless borrowed as mutable, which is an alternative to ownership. As you can imagine, this brand new paradigm gets complicated really fast when it is first encountered and is often the source of frustration for those new to rust.
I think there is something interesting in encountering a new paradigm like this. We assume we know how code works. We come across a new language and look for "hooks" that are familiar paradigms and philosophies, or architectures, that we can use as the foundation of our learning. Along comes a new, unique, and unfamiliar paradigm. Now, unless we take step back and accept we do not know how this feature works and have almost no existing basis for understanding it, we will start to stumble around and make mistakes, or worse, avoid that area of the language entirely because it is difficult to grok.
So is it worth learning all this stuff? Is Rust actually safer? A couple of examples and a proof:
i.
Nick Mathewson, co-founder of the Tor project started learning Rust by rebuilding Tor with Rust. He soon found that:
"Roughly half of Tor’s security issues since 2016 would have been impossible in Rust, and many of the other issues would have been much less likely, based on our informal audit."
There now exists a Rust based Tor project (ARTI) that's in development as we speak.
ii.
Microsoft has produce some research that has become almost iconic in the rust community. They examined bugs and vulnerabilities in their software going back to 2006 and stated that:
"roughly 70% of the security issues that the MSRC assigns a CVE to are memory safety issues. This means that if that software had been written in Rust, 70% of these security issues would most likely have been eliminated."
Note: CVE = "Common Vulnerabilities and Exposures"
The proof:
Beyond simple reasoning around the description of these features, how do we know they contribute towards "safety" - specifically memory safety in the language. Well, you could write lots of tests and set up many different scenarios to demonstrate this but you can further (if you really want to): In 2018, Ralf Jung, Jacques-Henri Jourdan, Robbert Krebbers, and Derek Dreyer went a step further and documented a formal safety proof for the language, which included verification conditions for libraries which, even if they contain unsafe code features, can be used to demonstrate their safety and inclusion as an extension to the language.
This is important because without these verification conditions, libraries that contain C++ could be seen as inherently unsafe. If there there is an "external" set of conditions that the library needs to meet, it's inclusion into the ecosystem becomes a less contentious process - and could provide for much more crossover with other low level systems programming languages that have decades of history solving problems that may not have been solved by Rust as yet.
More info:
https://doc.rust-lang.org/nomicon/ownership.html
https://devopedia.org/rust-language
https://msrc-blog.microsoft.com/2019/07/22/why-rust-for-safe-systems-programming/
https://msrc-blog.microsoft.com/2019/07/18/we-need-a-safer-systems-programming-language/
https://cacm.acm.org/magazines/2021/4/251364-safe-systems-programming-in-rust/fulltext
https://people.mpi-sws.org/~dreyer/papers/rustbelt/paper.pdf
https://plv.mpi-sws.org/rustbelt/
https://etaps.org/user-profile/archive/47-etaps-2018/etaps-2018/373-speaker-6
https://pvs-studio.com/en/blog/posts/0733/
In summary
Is Rust my favourite language? No. There are plenty of areas for improvement: Excessively implicit patterns (From/Into, implicit return values that are easy to miss, implicit traits like Display), some more complex internal architecture that requires "putting in the work" (e.g. Futures & "Pinning" memory when using async self-referencing futures), considerable boiler plate around custom Errors (some libraries help, but also have their own odd quirks) - I can continue - but none of them are dealbreakers. These things are being improved at an acceptable rate by a dedicated core team and a giant, motivated global community of devs (rustaceans 🦀).
The Rust revolution is not only about Rust being a very enjoyable language learn and code with - actually it's sometimes the opposite when getting started - It's about inevitability. Rust is a solution that meets several (around 18) needs, bridges programming paradigms, and provides enough added value benefits and tooling that it's difficult not to consider it for everything from WASM, Data Science, performance distributed computing, and embedded systems.
So where are going? I had a fellow JavaScript dev challenge my passion and conviction for Rust about three years ago, when I was fanboying the technology with a less explicit understanding of it. He said "So you think everyone will be using Rust in the next 5 years?" I said "More or less, yes, I think that's possible." He said "Wanna make a bet?" Today I really wish I took that bet. The last language that had so much breadth, and that was also similarly championed by its community... was JavaScript. And that is where I think we're going with Rust: Ubiquity; Rust will be soon growing everywhere.
Top comments (1)
Thanks for the extremely interesting series Kushal 👍