I recently started contributing to the Rust standard library to start fleshing out networking support for the WebAssembly system interface (WASI). The issue is that building rustc and its component parts like LLVM takes two and a half hours on my 2015 MacBook Pro.
If I was to be able to be productive, I needed a different solution!
Moar Cores!
The easiest answer when your machine isn't running something fast enough is to try throw more cores at it and see if it helps. Luckily a large part of building rustc is building llvm which massively speeds up when it has access to more cores.
So my co-worker, Duncan and I spun up a F64s_v2 virtual machine on Azure which has 64 total cores. The Fsv2 series of VMs on Azure are compute optimized making them the right VMs for the job.
With this machine, the build times went down dramatically to a total of around 17 minutes!
Wondering what would happen with less cores, we ran the same setup on a F32s_v2 with 32 cores and a F16s_v2 with 16 cores. These builds ran in 21 and 30 minutes respectively - not too bad!
Estimated Cost
Azure VMs are usually a bit cheaper in the U.S. so we hosted them there. I'm located in Europe, but in my experience latency isn't really an issue as the dev tools I use (ssh and VS Code remote extension) handle the latency fine.
The cost for the 64 core machine at the time of this writing $2.73 per hour, while the 32 core machine is $1.38 per hour and the 16 core machine is $0.71 per hour.
Given that the smaller machines are much cheaper, I opted for doing an initial build on a 32 core machine and then for incremental builds transferring the disk to a B8MS virtual machine. B8MS is an 8 core machine that is optimized for "burst" workloads (i.e. high CPU usage that happens every once in a while - like compiling). It only costs $0.36 per hour. Of course, you should still remember to shut down the machine when you're not using it.
You'll also have to pay for storage. Using a standard SSD with 64GB of RAM, only costs $5 per month.
If we put all this together, I estimate it costs around $2.00 for typical dev session of around 4 or 5 hours. Not too bad if it means I spend almost all that time coding instead of waiting for the thing to compile.
Dev Experience
Obviously getting a machine setup takes a bit of time, but with simple shell scripts using the Azure CLI, I now can easily reproduce the setup of both machines and the swapping the disk between them.
As for, writing actual code, I've been really enjoying VS Code's Remote Development Extensions. Most of the time I don't notice, I'm not working on my machine. I've had similar experiences as a vim user in the past, but this is next level, even making file transfer to the remote machine, super simple.
Hope this was an interesting look into remote dev on a big project like Rust. Let me know what you think by commenting below or reaching out on Twitter
Top comments (2)
So if 64 cores gets it down to 17 minutes, what would happen if you add even more cores? Or is 64 the max? It would be I interesting to see how far this can be taken. Is there a point at which adding more cores doesn't make it faster?
If there is a maximum number of cores per virtual machine, could multiple machines work together to compile it faster?
That would all depend on how parallelizable the task is (for example if its too small to subdivide its not worth it) and how well the algorithms work.
But in general huge tasks that split up easily have no limit. But at some point you're going to hit a RAM limits or other bottlenecks. Especially if the number of cores starts exceeding what can go into one processor. Soon the network becomes the bottleneck.