DEV Community

Cover image for Workers pool
Paweł Piwosz for AWS Community Builders

Posted on

Workers pool

In this episode we will create workers pool, to run workloads on "our" servers. This process requires a few steps, let's see how quickly and easily we can add new worker to the app.

During this episode I will heavily use Spacelift documentation.

Create private key

First step is to create the key which wil be used to authenticate worker with application.

I'll use WSL2 for it.

openssl req -new -newkey rsa:4096 -nodes -keyout spacelift.key -out spacelift.csr
Enter fullscreen mode Exit fullscreen mode

These files we need to setup our pool.

Create workers pool

Navigate to Worker pools in the Spacelift app. You should see something similar like on the screen below.

Create workers pool

Click New worker pool.

Create workers pool

Add the csr file generated in previous step. Also, I changed space to root. Yes, you can have pools set by space.

Please remember, that one worker is able to run one execution. If this is good or bad, I leave it to you, however it doesn't sound very appealing if I will use virtual machines, am I right? But! We can use this VM as containers platform and it starts to look a little bit better.

Ok, we can create our pool.

My pool is created and the file with token is downloaded automatically. Well, I am not fully sure here. I almost missed this download, my browser didn't show anything. I knew there will be a token, so I was careful, but I think it will be nice if Spacelift improve the UX here.

Create worker

I decided to go easy way and I run the EC2 on AWS using Spacelift image.

Select Spacelift AMI on AWS

I've changed the default volume only. The type I selected is gp3 and I increased the size to 20G.

The provided AMI is not completed. I mean, it contains everything what we need to download and configure the launcher. Let's do it then!

First, we need to encode the key we generated earlier

cat spacelift.key | base64 -w 0
Enter fullscreen mode Exit fullscreen mode

Now we can login to the newly created machine and do

sudo -i
install spacelift-launcher /usr/bin/spacelift-launcher
chmod +x /usr/bin/spacelift-launcher
export SPACELIFT_TOKEN="<token_from_file>"
export SPACELIFT_POOL_PRIVATE_KEY="key_from_encode_operation>"
Enter fullscreen mode Exit fullscreen mode

What I do not like - to make the activity quickly I had to use root account.

What we've just done is very unstable and should be treated as "let me check it" way only. When we do it properly, all these should be automated and run without any access from human side. In fact... Spacelift provides a repository with IaC to provision the infrastructure!

Let's see our workers pool

Copnfigured workers pool

Yep, we are done!

The 'hmmm' moment

I need to dig deeper into the workers pool. I saw couple of things which made me think.

  • What will happen if I stop my worker node?
  • Logs for spacelift-launcher are set to debug by default (at least, this is what I see after quick check).
  • In logs I saw information about S3. And this is something what I have to explore. What S3? Why? Who owns it? I have some ideas, what is behind, but I have to confirm it.


Spacelift allows us to create private pools of workers which can be used for different workloads. Creation is simple and management is quite effective.

Top comments (0)