The first article in this series went over an example of how to set up the Salt configuration management tool to automate a computer infrastructure.
Salt is very powerful and in this next section of the series, we'll take a look at two important components, grains and the pillar. At a first glance grains and the pillar seem to serve the same purpose but there is an important distinction between them.
What are grains and the pillar?
Grains and the pillar are used by Salt to store data about the minion it is controlling. They are both in the end a key value store. In fact they are what is called in Python a dictionary (if you're into Perl or Ruby, they're known as a hash).
For example, the following is an example of some grains on the alpine minion we created in the first article in the series:
ipv4:
- 127.0.0.1
- 172.25.0.3
ipv6:
kernel:
Linux
kernelrelease:
4.15.0-65-generic
If you want to check out all of the grains, you can execute the following command in the salt master docker container (to get a shell, type this in the same directory as you keep your docker compose file: docker-compose exec salt bash
):
salt 'locomotive-*' grains.items
The pillar data looks the same except the keys and values are different. However if you run the command to list the pillar items on your salt master, you won't get a similar result now:
salt 'locomotive-*' pillar.items
You'll get nothing as a result:
locomotive-03f0f2574592:
----------
That's because while many grains are automatically generated when the salt minion starts up, the pillar data isn't. In fact the salt minion doesn't ever generate the pillar data. That's the big distinction between grains and the pillar.
Grains are generated and/or stored on the minions themselves. Pillar data are generated and/or stored on the salt master.
This distinction allows the pillar data to contain sensitive data you don't want lying around on the minion. It could be minion configurations or variables you want to manage in a central place, well or just any other data.
Grains on the other hand are useful for information you won't necessarily know when you configure the minion like IP addresses, operating system, CPU information etc. It can also be arbitrary data, like a role you want to configure and store on the minion itself.
For example you could on the machine create a grain called role
with value webserver
. Then the salt master could pick up that grain and from there decide how to configure it, without changing anything in the salt master configuration files. The pillar data for that machine may then include information such as software licenses you need for a particular application, users allowed to log into web servers etc.
One good thing to keep in mind is that grains are fairly static, they change infrequently because they're just information about the system (either automatically generated or custom data). The pillar on the other hand is better suited for data that will change more frequently.
Configuring grains
Most of the grains you normally need are automatically generated. However, if you ever find yourself in a position where you want to configure them you can store them in the minion configuration file on the machine. Location depends on operating system, for example on Unix systems you find it at /etc/salt/minion
except in FreeBSD where it's stored at /usr/local/etc/salt/minion
. On Windows machines the configuration is found at C:\salt\conf\minion
.
The configuration file is a YAML file and you can add grains
as a root element in the configuration file with the grains you want to store. For example you could add this to the minion config:
grains:
roles:
- webserver
This would make the grain roles
available with a list as its value, containing the webserver role (it doesn't have to be a list of strings but I try to choose a data structure applicable to the context and a server could have multiple roles so that's why I'd use a list for the roles).
Another place where you can store grains is in /etc/salt/grains
. I like explicit so I like this location more than the minion config. It's a YAML file but if you want to use /etc/salt/grains
you don't need the root grains
element. So the equivalent /etc/salt/grains
file to the one above is:
roles:
- webserver
Let's adapt the example from the first post in this series and configure grains for the two salt minions.
Let's create two grains files, one for our alpine container and another for the Ubuntu container, defining roles for the machines. Create a file called alpine.grains
in the same location as your docker-compose.yml
file (we're going to mount it later on). Here are the contents of the alpine.grains
file:
roles:
- locomotive
- fortuneteller
Create a similar file for the Ubuntu container called ubuntu.grains
with these contents:
roles:
- cow
- fortuneteller
Now let's update our docker-compose.yml
We just have to mount the two files we created at /etc/salt/grains
on both machines so the docker-compose.yml
file should look like this:
version: "3"
services:
salt:
image: salt-master
volumes:
- ./master.config:/etc/salt/master
- ./states:/srv/salt
alpine-minion:
image: alpine-minion
- ./alpine.grains:/etc/salt/grains
ubuntu-minion:
image: ubuntu-minion
volumes:
- ./ubuntu.grains:/etc/salt/grains
Let' get all of them up and running using these new grains and therefore their new roles. Run this command from the same directory:
docker-compose up -d
Still running first post containers?
If you shut down and removed the containers you created in the first post, you'll just have to accept keys like you did in the first post.
If you are still running the same containers, you should see the minions being updated (recreated), but not the salt master (it's up to date).
If you try to check things out with salt master now, you'll notice that it doesn't manage your salt minions any longer. The reason is that because we recreated the docker containers they now have a new minion id (it's a new machine basically). So if you're not following this guide adamantly and want to play around, you'll have to accept the keys as shown in the first blog post in the series.
Now we're ready to use those newly defined roles somehow.
Usefulness of grains
Targeting minions
Grains are very useful for targeting minions. Remember in the first article how we targeted the machines using the minion id? We can also target them using grains.
Let's update the configuration file for the salt master to target based on grains instead of minion ids. First let's create a special state file for the fortune
program. Remove the line in both cow.sls
and locomotive.sls
where we tell salt to install the fortune
package. Then create a similar state file called fortune.sls
:
install fortuneteller tools:
pkg.installed:
- pkgs:
- fortune
Alright. Now we can change the top.sls
file to target based on grains instead of minion ids. The pattern for it is to say key:value
. Then you can, before you list what state files to include, begin by telling salt what you want to match with your pattern (in our case the grain).
So to apply the apache.sls
file to machines with the role webserver
you would write something like this:
'roles:webserver':
- match: grain
- apache
Notice the '
around roles:webserver
. This is so that the YAML file gets processed correctly. There is a shorthand version to achieve the same. You could instead write this:
'G@roles:webserver':
- apache
Let's modify the top file so we target each role specifically and apply the appropriate state files. The top file could look like this then (I choose the more explicit targeting pattern):
base:
'roles:cow':
- match: grain
- cow
'roles:fortuneteller':
- match: grain
- fortune
'roles:locomotive':
- match: grain
- locomotive
Logic in state files
Grains can also be useful in the state files themselves. For example, we can check the operating system and do different things based on the operating system. You'll see more on this when we go through templating in the Salt files but to give you a taste, let's solve the problem we had in the first article of the series, where the Ubuntu machine installed the packages in /usr/games/
.
To solve that problem we'll put an if statement that checks the os
grain to see if it is an Ubuntu operating system. If it is, then we make a symbolic link to /usr/bin
. We'll need to change all state files because Ubuntu does this with all packages.
Before we change the files, let's first see how we look up the grain value in the state file. The grain we're after to check the operating system is the os
grain. It becomes available as grains['os']
(if you know Python, you'll understand that this is just a dictionary lookup).
So the trick is to add a Python if
statement inside {% %}
then include the state configurations you want before you end with a {% endif %}
. I'll cover this better in a later post but to start cow.sls
should have this content now:
install cow tools:
pkg.installed:
- pkgs:
- cowsay
{% if grains['os'] == 'Ubuntu' %}
make cowsay available on Ubuntu:
file.symlink:
- name: /usr/bin/cowsay
- target: /usr/games/cowsay
{% endif %}
To dissect this a little bit, first you see the state we defined in the first post of the series and modified a few minutes ago (to remove fortune
). Then we create our if statement which checks if the os
grain is equal to Ubuntu
. If it is, then we create a state called make cowsay available on Ubuntu. This states uses the file state module to create a symlink to /usr/games/cowsay
(the target) at /usr/bin/cowsay
(defined in name of the symlink). Then we close the if statement.
fortune.sls
is going to be similar
install fortuneteller tools:
pkg.installed:
- pkgs:
- fortune
{% if grains['os'] == 'Ubuntu' %}
make fortune available on Ubuntu:
file.symlink:
- name: /usr/bin/fortune
- target: /usr/games/fortune
{% endif %}
So will locomotive.sls
install locomotive tools:
pkg.installed:
- pkgs:
- sl
{% if grains['os'] == 'Ubuntu' %}
make sl available on Ubuntu:
file.symlink:
- name: /usr/bin/sl
- target: /usr/games/sl
{% endif %}
Trying out the grain based configuration
Let's test our new grain based configuration. First let's stop and remove all of our docker containers because we want a fresh slate (this way we'll also get around the new minion ids discussed earlier and easily remove the older containers.
Type in these docker-compose commands to get stop containers, remove them, start up new ones and finally enter bash shell on the salt master (answering yes when prompted):
docker-compose stop
docker-compose rm
docker-compose up -d
docker-compose exec salt bash
Once you've entered the bash shell on the new salt master, accept the minion keys with this command (pressing Enter for yes when prompted):
salt-key -A
Now we're ready to test our grain based targeting in the top.sls
. Let's apply the states to all machines (targeting them with the glob *
):
salt '*' state.apply
This will install all the programs, based on the roles defined in the grains and create the symlinks on the Ubuntu machines thanks to the if statement in the states. If you look at the output report, you'll see 2 states succeeded in the Alpine container while four changed in the Ubuntu container.
Targeting in the command line
Another place you can use to target based on grains is on the command line. Instead of using the glob or minion id when you run the salt
command on the salt master, you can target based on grain by using the -G
option.
As an example, let's run the fortune command on all fortuneteller minions (both Ubuntu and Alpine containers). To run a command via the salt
command on the salt master you type cmd.run
instead of the state.apply
we've used earlier and then you tell salt which command to run (in our case fortune
). Go ahead and type this in your salt master's bash shell:
salt -G roles:fortuneteller cmd.run fortune
As you can see, we give salt the -G
option followed by the grain key and value we want to target (much like we targeted them in the top.sls
file. Then we tell salt to run the fortune
command with the cmd.run fortune
. The output should be some nice words of wisdom:
cow-e54f99c12817:
Don't feed the bats tonight.
locomotive-68af4abb8a98:
Put no trust in cryptic comments.
We can also only ask Alpine minions to deliver fortunes:
salt -G os:Alpine cmd.run fortune
This returns only a single wisdom because the grain only matched a single minion:
locomotive-68af4abb8a98:
The meek shall inherit the earth -- they are too weak to refuse.
OK. That's enough about grains for now, let's quickly turn to the pillar.
Configuring the pillar
As I said previously. The pillar offers the same data structure as grains but instead of being generated and hosted on the salt minions, the pillar data are generated on the salt master and handed to the minion.
That means that one big difference between the pillar and grains is that the pillar needs targeting, so that the pillar data can be handed to the right minion. Besides that, they're just the same.
Let's move the exact programs we need to install for each role into the pillar. That makes it more flexible because we don't have to hard code the programs to install on each minion based on the role. It's going to simplify maintenance and our state code.
Configuring the pillar is a mix between states and grains. We have a top.sls
file for the pillar where we do the targeting. There we include files, which should return a data structure similar to grains.
Pillar configuration files
We begin by creating a folder called pillar
. This folder we're going to mount on our salt master. This is the folder that will hold our pillar data. Let's first create files for each role. The root element in each file will be packages
(could be anything we want but we want it to be explicit right?). Followed by the name of the program as a key with a value which is just a description of the package we install. Something like this:
packages:
package-name: package description
We won't really use the package description, we're only interested in the package name. However, salt will automatically merge dictionaries in a smart way (which is useful and cool) but unfortunately not append to lists. That's fine though because this gives us a nice place to put a description or comment with an Easter egg.
So first create a file called cow.sls
with this content:
packages:
cowsay: A program with a cow
The second file called fortuneteller.sls
with this content:
packages:
fortune: A program for a fortune teller
Lastly create the third file called locomotive.sls
with this content:
packages:
sl: Tech Model Railroad Club
Finally we need our top.sls
file. It's going to be exactly the same as our state file (you can copy it if you want). First we define the environment we want it to be used in, then we target the roles and include the sls files we just created. So go ahead and create the top.sls
with this content:
base:
'roles:cow':
- match: grain
- cow
'roles:fortuneteller':
- match: grain
- fortune
'roles:locomotive':
- match: grain
- locomotive
As you can see we can match grains when targeting our pillar. That's pretty cool. We use the roles
grain to match the roles.
Be careful, because matching grains in the pillar comes with a caveat. When you use grains to match because they can be configured on the minion itself which means a minion can send a grain to get data it shouldn't get. In our case we're just declaring packages to install, so that's not so harmful.
Configuring the master
Next we'll reconfigure our salt master to read from our newly created pillar directory. For that we have to mount the pillar directory with docker compose and configure the master to read the pillar files from that directory. We begin by changing the master configuration.
We only need to add a new configuration variable called pillar_roots
which is similar to the preexisting file_roots
. It tells in what folder we keep the pillar files for a particular environment (in our case base
).
We will mount the directory with the pillar data at /srv/pillar
. Update the master.config
file and add the pillar_root
and point the base environment to /srv/pillar
, so your master.config
file should look like this:
file_roots:
base:
- /srv/salt/
pillar_roots:
base:
- /srv/pillar
Then we need to change our docker-compose.yml
file to mount our new pillar
directory at /srv/pillar
.
version: "3"
services:
salt:
image: salt-master
volumes:
- ./master.config:/etc/salt/master
- ./states:/srv/salt
- ./pillar:/srv/pillar
alpine-minion:
image: alpine-minion
volumes:
- ./alpine.grains:/etc/salt/grains
ubuntu-minion:
image: ubuntu-minion
volumes:
- ./ubuntu.grains:/etc/salt/grains
That's it. That's all you need.
Do we have pillar data?
Let's have a look at the pillar data we've created. Again, we want to start with a clean slate so we stop our docker containers and remove them (press y when prompted if we're sure we want to remove them). Then we spin them back up again and attach to the bash shell of salt master:
docker-compose stop
docker-compose rm
docker-compose up -d
docker-compose exec salt bash
Once we're attached to the salt master, we accept the salt keys for the new minions (press enter to accept them when prompted):
salt-key -A
To have a look at the pillar data we just created type the following command
salt '*' pillar.items
You should now see the packages listed based on the roles of the machines. Something like this:
cow-e9e01577bbd8:
----------
packages:
----------
cowsay:
A program with a cow
fortune:
A program for a fortune teller
locomotive-023378ead2d9:
----------
packages:
----------
fortune:
A program for a fortune teller
sl:
Tech Model Railroad Club
We still aren't using this pillar data. To do that properly, we'll have to dive into the next part of this series: templating (I've teased it a little bit in this article though).
Little by little we'll get a better understanding of many of Salt's features.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Cover image by Yair Aronshtam: Salt pillar at the Dead Sea, Israel
Top comments (0)