As of January 1st, 2020, Python 2 has officially reached the end of life and will no longer receive any updates (except the last one in April). Various Linux distributions will support it for a while; however, many organizations already switched to Python 3 or plan to do so in the near future.
It is time to switch your Salt install to Python 3:
- Salt Nitrogen (2017.7.0) is the first release that supported Py3
- Salt Oxygen (2018.3.0) gained significant Py3 improvements
- Salt Fluorine (2019.2.0) can be considered more-or-less mature
- Salt 3001 Sodium (the one after Salt 3000 Neon) is going to deprecate Python 2.7
This guide describes how to switch the repo from Py2 to Py3, identify and uninstall the old packages, then install the Py3 version. It contains detailed checklists that will help you to plan and perform the migration. It can be also applied to regular upgrades. However, it is not a 100% automatic foolproof recipe you can apply blindly.
- 10 key principles
- Plan the upgrade process
- Survey your master and minion fleet
- Upgrade the Salt Master
- Upgrade Salt on Windows
- Upgrade Salt on Ubuntu and Debian
- Upgrade Salt on CentOS and Amazon Linux
- Upgrade Salt on other operating systems
- Monitor the upgrade process
- Upgrade automation
- Future developments
10 key principles
Over the years of using Salt, I developed a set of ten opinionated guiding principles that will help you maintain and upgrade it. You do not have to follow these guidelines, and I often break them too. However, they are implied throughout the article.
1. Have a QA environment
QA environment can be manual or automated. The goal is to be able to test the required combinations of Operating Systems, Salt versions (master & minion), and possible highstates. Always do staged/gradual rollouts. You’ll save a lot of time if your QA environment supports snapshots.
2. Run the same Salt version everywhere
Master and minion versions should be the same (the only exceptions are [1] and [5]). This is the most traveled path. You won’t need to adapt your states to different feature sets. Also, the upgrade process is much easier.
3. Avoid rare operating modes
Less traveled paths and settings that significantly change how Salt operates (transport, multiprocessing, zmq filtering, caching, undocumented things), tend to result in nasty surprises. If you want to change a setting, give it a try in your QA environment for a while.
4. Control the bootstrap/upgrade process
- Do not use the “latest” repo prefix (except in a QA environment), and do not rush to upgrade to a latest major version
- Use the “archive” repo prefix, or pin Salt packages down to a minor version (you never know when another 2019.2.1 will happen)
- Always install specific version when you bootstrap new minions (e.g., through the
script_args
Salt Cloud option)
Yes, a minor upgrade will require more work. But it is work you can control and decide when to do. You can’t control work that appears out of the blue when minions automatically upgrade and something breaks.
5. Upgrade the master first
When you upgrade Salt, upgrade the master first. Older master can (and eventually will) break things.
6. Back up the keys
Set up periodic backups of the /etc/salt/pki
master directory. Other than that, your Salt master should be disposable, and states/pillars should be kept in a version control system (or pulled from external data sources).
7. Have a backup access path
Have a plan B to access your servers when Salt goes down. Use salt-ssh
, fabric
, ansible
, ssh
, winrm
, or any other tool. It is also helpful to keep a list of minion IPs (you can periodically extract them from the master grain cache).
8. Use Salt as an O&M control plane
Avoid making your production workload depend on Salt. If you uninstall Salt, your production environment should remain operational (at least for a while).
If you have the resources, you may also want to consider the last two principles:
9. Mirror the Salt repo
I never had to do this, but I’ve seen rare connectivity issues with the official repo. Also, for some folks, the minion bootstrap process was broken when SaltStack removed (and then quickly returned back) the buggy 2019.2.1 release. This is less of an issue if you follow the principle [4].
10. Build your own packages
Bugs will happen, and you need to fix them quickly without waiting for the next release. Also, you may want to integrate new features before they are shipped. Custom packages are a way to achieve that. They give you full control over the Salt code you run.
I ran a patched salt-master from a virtualenv for some time in 2012 or 2013, but I no longer do that. Instead, I prefer the more lightweight ways to patch Salt.
Plan the upgrade process
1. Survey your master and minion fleet
A list of useful commands is provided below. There are many dimensions you should inspect:
- Operating systems, their architectures, and versions
- Available repositories
- Python versions
- Master and minion versions
- Salt dependencies
The main goal is to find all the existing combinations, so you can exhaustively test the upgrade process. Also check out the Py3 support list.
2. Prepare the QA environment
Ideally, you want to be able:
- Test the same OS/Salt versions
- Test the existing highstates
- Test the actual workload that runs on top of your infra
You’ll need to test the upgrade process multiple times, so a fast feedback cycle is necessary (e.g., it would be nice to quickly snapshot/restore your QA environment).
3. Update the states
Make sure your states are compatible with the target Salt version. Read the official release notes and search for any open issues.
4. Decide on the upgrade process
For example:
- Perform the backups (master/minion keys, alternate connection paths, etc.)
- Upgrade OS packages, run autoclean/autoremove. Optionally reboot if you have new kernels.
- Disable any periodic Salt jobs and other stuff that can interfere with the upgrade process. Try minion blackouts
- Upgrade master to the target version, while staying on Py2
- Ensure that the existing process to bootstrap new minions uses the target Salt version (Py3)
- Upgrade minions to the target version, while staying on Py2. If that is not possible, upgrade to the latest available for each distro.
- Upgrade Salt master to Py3
- Upgrade Salt minions to Py3 for each OS separately
Consider the minion upgrade order. Your environment may require some specific upgrade sequence.
The goal is to remove the old Salt version and its dependencies (to avoid having packages from different repos), then bootstrap the Py3 one from scratch.
If your minions are disposable, you can avoid the in-place upgrade process and instead rebuild them from scratch. Another option is blue/green deployment.
5. Test the upgrade process multiple times
- Manually upgrade your masters
- Make sure their minions are reachable after the upgrade
- Manually upgrade a single minion for each OS
- Make sure the upgraded minions are reachable, the highstates can be applied, and the business apps are functional
- Automate the above steps
- Test the automation multiple times
- Test the backup options (master/minion keys, alternate connection paths, etc.)
- Test the bootstrap process for new minions
6. Do the upgrade
- Decide how many upgrade stages you want
- Decide on the appropriate schedule
- Inform the team(s)
- Perform and verify the backups (master/minion keys, alternate connection paths, etc.)
- Switch the bootstrap for new minions to the target version
- Upgrade the master
- Upgrade the minions
- Monitor the result
Survey your master and minion fleet
Listing versions
% sudo salt '*' test.version
% sudo salt '*' pkg.version salt-minion
% sudo salt '*' test.versions_report
(you may find a lot of variation in the installed dependencies!)
% sudo salt '*' grains.item pythonversion
% sudo salt-run manage.versions
Master:
2019.2.3
Minion requires update:
----------
minion1:
2018.3.5
minion2:
2018.3.5
minion3:
2018.3.5
win10:
2018.3.5
sudo salt-run survey.hash '*' test.version
|_
----------
pool:
- minion1
- minion2
- minion3
- win10
result:
2018.3.5
Listing repos and keys
% sudo salt '*' pkg.list_repos
% sudo salt '*' pkg.get_repo_keys
Combine with jq
to get more compact output:
% sudo salt -G "os:Ubuntu" pkg.list_repos --static --out json | jq '.[] | with_entries(select(.key|contains("saltstack"))) | .[][].line'
"deb https://repo.saltstack.com/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic main"
"deb https://repo.saltstack.com/apt/ubuntu/18.04/amd64/archive/2019.2.3 bionic main"
% sudo salt -G "os:Debian" pkg.list_repos --static --out json | jq '.[] | with_entries(select(.key|contains("saltstack"))) | .[][].line'
"deb https://repo.saltstack.com/apt/debian/9/amd64/archive/2018.3.5 stretch main"
"deb https://repo.saltstack.com/apt/debian/9/amd64/archive/2019.2.3 stretch main"
% sudo salt -G "os:CentOS" pkg.list_repos --static --out json | jq '.[] | with_entries(select(.key|contains("saltstack"))) | .[].baseurl'
"https://repo.saltstack.com/yum/redhat/7/$basearch/archive/2018.3.5/"
"https://repo.saltstack.com/yum/redhat/7/$basearch/archive/2019.2.3/"
Targeting py2 minions
% sudo salt -C "G@os:Ubuntu and G@pythonversion:0:2" grains.item pythonversion saltversion
% sudo salt -C "G@os:Debian and G@pythonversion:0:2" grains.item pythonversion saltversion
% sudo salt -C "G@os:CentOS and G@pythonversion:0:2" grains.item pythonversion saltversion
% sudo salt -C "G@os:Windows and G@pythonversion:0:2" grains.item pythonversion saltversion
Other useful grains for targeting
cpuarch
lsb_distrib_codename
lsb_distrib_id
lsb_distrib_release
os
osarch
oscodename
osfinger
osmajorrelease
osrelease
osversion
pythonexecutable
pythonversion
saltpath
saltversion
Surveys
Any module calls (test.version
, cmd.run
, grains.item
, etc.) can be combined with the survey
runner:
sudo salt-run survey.hash '*' grains.item os osfinger pythonversion saltversion
|_
----------
pool:
- minion3k
result:
{u'osfinger': u'Ubuntu-18.04', u'saltversion': u'2018.3.5', u'os': u'Ubuntu', u'pythonversion': [2, 7, 17, u'final', 0]}
|_
----------
pool:
- minion2
result:
{u'osfinger': u'CentOS Linux-7', u'saltversion': u'2018.3.5', u'os': u'CentOS', u'pythonversion': [2, 7, 5, u'final', 0]}
|_
----------
pool:
- minion1
result:
{u'osfinger': u'Debian-9', u'saltversion': u'2018.3.5', u'os': u'Debian', u'pythonversion': [2, 7, 13, u'final', 0]}
|_
----------
pool:
- win10
result:
{u'osfinger': u'Windows-10', u'saltversion': u'2018.3.5', u'os': u'Windows', u'pythonversion': [2, 7, 14, u'final', 0]}
Listing package origins
The goal is to find packages that were installed from a specific origin, so that you can remove them during the upgrade process.
Ubuntu/Debian
I found the following snippet on Ask Ubuntu. If you know a more straightforward way to do the same without installing aptitude or other packages, I’m all ears.
% apt-cache policy $(dpkg -l | awk 'NR >= 6 { print $2 }') |
awk '/^[^]/ { split($1, a, ":"); pkg = a[1] }
nextline == 1 { nextline = 0; printf("%-40s %-50s %s\n", pkg, $2, $3) }
/\*\*\*/ { nextline = 1 }' | grep saltstack
The output can vary depending on the OS:
salt-common https://repo.saltstack.com/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic/main
salt-minion https://repo.saltstack.com/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic/main
python-jinja2 https://repo.saltstack.com/apt/debian/9/amd64/archive/2018.3.5 stretch/main
salt-common https://repo.saltstack.com/apt/debian/9/amd64/archive/2018.3.5 stretch/main
salt-minion https://repo.saltstack.com/apt/debian/9/amd64/archive/2018.3.5 stretch/main
Additionally, you can inspect how a Salt minion was installed in the first place. You need to figure out which packages are installed explicitly (the rest of the dependencies can be auto removed). Check out the /var/log/apt/history.log
:
Debian:
Start-Date: 2020-02-18 05:15:02
Commandline: apt-get install -y -o DPkg::Options::=--force-confold procps pciutils python-yaml
Requested-By: vagrant (1000)
Install: python-yaml:amd64 (3.12-1), libyaml-0-2:amd64 (0.1.7-2, automatic)
End-Date: 2020-02-18 05:15:02
Start-Date: 2020-02-18 05:15:05
Commandline: apt-get install -y -o DPkg::Options::=--force-confold wget gnupg2 apt-transport-https ca-certificates
Requested-By: vagrant (1000)
Install: gnupg2:amd64 (2.1.18-8~deb9u4), apt-transport-https:amd64 (1.4.9)
End-Date: 2020-02-18 05:15:06
Start-Date: 2020-02-18 05:15:35
Commandline: apt-get install -y -o DPkg::Options::=--force-confold salt-minion
Requested-By: vagrant (1000)
Ubuntu:
Start-Date: 2020-02-18 05:15:08
Commandline: apt-get install -y -o DPkg::Options::=--force-confold wget gnupg apt-transport-https ca-certificates
Requested-By: vagrant (1000)
Install: apt-transport-https:amd64 (1.6.12)
End-Date: 2020-02-18 05:15:08
Start-Date: 2020-02-18 05:15:49
Commandline: apt-get install -y -o DPkg::Options::=--force-confold python2.7 python-apt python-requests python-yaml procps pciutils
Requested-By: vagrant (1000)
Install: python-six:amd64 (1.11.0-2, automatic), python-openssl:amd64 (17.5.0-1ubuntu1, automatic), python-yaml:amd64 (3.12-1build2), python-requests:amd64 (2.18.4-2ubuntu0.1), python-certifi:amd64 (2018.1.18-2, automatic), python-chardet:amd64 (3.0.4-1, automatic), python-enum34:amd64 (1.1.6-2, automatic), python-cryptography:amd64 (2.1.4-1ubuntu1.3, automatic), python-cffi-backend:amd64 (1.11.5-1, automatic), python-ipaddress:amd64 (1.0.17-1, automatic), python-pkg-resources:amd64 (39.0.1-2, automatic), python-apt:amd64 (1.6.5ubuntu0.2), python-urllib3:amd64 (1.22-1ubuntu0.18.04.1, automatic), python-idna:amd64 (2.6-1, automatic), python-asn1crypto:amd64 (0.24.0-1, automatic)
Upgrade: python2.7-minimal:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), libpython2.7:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), python2.7:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), libpython2.7-minimal:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), libpython2.7-stdlib:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04)
End-Date: 2020-02-18 05:15:56
Start-Date: 2020-02-18 05:16:11
Commandline: apt-get install -y -o DPkg::Options::=--force-confold salt-minion
The bootstrap-salt.sh
script also logs everything to /tmp/bootstrap-salt.log
:
pciutils is already the newest version (1:3.5.2-1).
procps is already the newest version (2:3.3.12-3+deb9u1).
...
Selecting previously unselected package python-yaml.
Preparing to unpack .../python-yaml_3.12-1_amd64.deb ...
Unpacking python-yaml (3.12-1) ...
Setting up libyaml-0-2:amd64 (0.1.7-2) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Setting up python-yaml (3.12-1) ...
Reading package lists...
Building dependency tree...
Reading state information...
ca-certificates is already the newest version (20161130+nmu1+deb9u1).
ca-certificates set to manually installed.
wget is already the newest version (1.18-5+deb9u3).
The following NEW packages will be installed:
apt-transport-https gnupg2
...
The following additional packages will be installed:
dctrl-tools debconf-utils dirmngr javascript-common libjs-jquery
libjs-sphinxdoc libjs-underscore libpgm-5.2-0 libsodium18 libzmq5 python-apt
python-backports-abc python-cffi-backend python-chardet
python-concurrent.futures python-croniter python-cryptography
python-dateutil python-enum34 python-idna python-ipaddress python-jinja2
python-markupsafe python-msgpack python-openssl python-pkg-resources
python-psutil python-pyasn1 python-requests python-setuptools
python-singledispatch python-six python-systemd python-tornado python-tz
python-urllib3 python-zmq salt-common
Suggested packages:
debtags dbus-user-session pinentry-gnome3 tor apache2 | lighttpd | httpd
python-apt-dbg python-apt-doc python-cryptography-doc
python-cryptography-vectors python-enum34-doc python-jinja2-doc
python-openssl-doc python-openssl-dbg python-psutil-doc doc-base
python-socks python-setuptools-doc python-mysqldb python-pycurl
python-tornado-doc python-twisted python-ntlm python-augeas
Recommended packages:
sfdisk e2fprogs
The following NEW packages will be installed:
dctrl-tools debconf-utils dirmngr javascript-common libjs-jquery
libjs-sphinxdoc libjs-underscore libpgm-5.2-0 libsodium18 libzmq5 python-apt
python-backports-abc python-cffi-backend python-chardet
python-concurrent.futures python-croniter python-cryptography
python-dateutil python-enum34 python-idna python-ipaddress python-jinja2
python-markupsafe python-msgpack python-openssl python-pkg-resources
python-psutil python-pyasn1 python-requests python-setuptools
python-singledispatch python-six python-systemd python-tornado python-tz
python-urllib3 python-zmq salt-common salt-minion
So, for Debian/Ubuntu you may want to remove the salt-minion
, python-requests
, python-yaml
, and python-apt
explicitly, then auto remove the dependencies.
CentOS/Amazon
% yum list installed | grep saltstack
PyYAML.x86_64 3.11-1.el7 @saltstack
python-zmq.x86_64 15.3.0-3.el7 @saltstack
salt.noarch 2018.3.5-1.el7 @saltstack
salt-minion.noarch 2018.3.5-1.el7 @saltstack
zeromq.x86_64 4.1.4-7.el7 @saltstack
The /var/log/yum.log
is not very informative:
Feb 18 08:15:37 Updated: rpm-libs-4.11.3-40.el7.x86_64
Feb 18 08:15:37 Updated: rpm-4.11.3-40.el7.x86_64
Feb 18 08:15:38 Updated: rpm-build-libs-4.11.3-40.el7.x86_64
Feb 18 08:15:38 Updated: rpm-python-4.11.3-40.el7.x86_64
Feb 18 08:15:38 Installed: python-chardet-2.2.1-3.el7.noarch
Feb 18 08:15:38 Installed: python-kitchen-1.1.1-5.el7.noarch
Feb 18 08:15:38 Installed: libyaml-0.1.4-11.el7_0.x86_64
Feb 18 08:15:38 Updated: libxml2-2.9.1-6.el7_2.3.x86_64
Feb 18 08:15:38 Installed: libxml2-python-2.9.1-6.el7_2.3.x86_64
Feb 18 08:15:38 Updated: python-urlgrabber-3.10-9.el7.noarch
Feb 18 08:15:39 Updated: yum-3.4.3-163.el7.centos.noarch
Feb 18 08:15:39 Installed: yum-utils-1.1.31-52.el7.noarch
Feb 18 08:15:39 Installed: PyYAML-3.11-1.el7.x86_64
Feb 18 08:15:39 Updated: chkconfig-1.7.4-1.el7.x86_64
Feb 18 08:16:15 Installed: lz4-1.7.5-3.el7.x86_64
Feb 18 08:16:15 Updated: systemd-libs-219-67.el7_7.3.x86_64
Feb 18 08:16:15 Installed: python-ipaddress-1.0.16-2.el7.noarch
Feb 18 08:16:15 Installed: python-markupsafe-0.11-10.el7.x86_64
Feb 18 08:16:15 Installed: python-six-1.9.0-2.el7.noarch
Feb 18 08:16:15 Installed: libtommath-0.42.0-6.el7.x86_64
Feb 18 08:16:15 Installed: libtomcrypt-1.17-26.el7.x86_64
Feb 18 08:16:16 Installed: python2-crypto-2.6.1-16.el7.x86_64
Feb 18 08:16:16 Installed: python-backports-1.0-8.el7.x86_64
Feb 18 08:16:16 Installed: python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch
Feb 18 08:16:16 Installed: python-tornado-4.2.1-5.el7.x86_64
Feb 18 08:16:16 Installed: python-urllib3-1.10.2-7.el7.noarch
Feb 18 08:16:16 Installed: python-requests-2.6.0-8.el7_7.noarch
Feb 18 08:16:17 Installed: python-babel-0.9.6-8.el7.noarch
Feb 18 08:16:18 Installed: python-jinja2-2.7.2-4.el7.noarch
Feb 18 08:16:18 Installed: libsodium-1.0.18-1.el7.x86_64
Feb 18 08:16:18 Installed: python2-msgpack-0.5.6-5.el7.x86_64
Feb 18 08:16:18 Updated: cryptsetup-libs-2.0.3-5.el7.x86_64
Feb 18 08:16:20 Updated: systemd-219-67.el7_7.3.x86_64
Feb 18 08:16:20 Installed: systemd-python-219-67.el7_7.3.x86_64
Feb 18 08:16:20 Installed: openpgm-5.2.122-2.el7.x86_64
Feb 18 08:16:20 Installed: zeromq-4.1.4-7.el7.x86_64
Feb 18 08:16:21 Installed: python-zmq-15.3.0-3.el7.x86_64
Feb 18 08:16:21 Installed: python2-futures-3.1.1-5.el7.noarch
Feb 18 08:16:21 Updated: pciutils-libs-3.5.1-3.el7.x86_64
Feb 18 08:16:21 Installed: pciutils-3.5.1-3.el7.x86_64
Feb 18 08:16:21 Installed: python2-psutil-2.2.1-5.el7.x86_64
Feb 18 08:16:26 Installed: salt-2018.3.5-1.el7.noarch
Feb 18 08:16:26 Installed: salt-minion-2018.3.5-1.el7.noarch
Feb 18 08:16:26 Updated: systemd-sysv-219-67.el7_7.3.x86_64
Feb 18 08:16:26 Updated: libgudev1-219-67.el7_7.3.x86_64
However, the /tmp/bootstrap-salt.log
shows precisely which packages were requested and which are merely dependencies:
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
PyYAML x86_64 3.11-1.el7 saltstack 160 k
yum-utils noarch 1.1.31-52.el7 base 121 k
Updating:
chkconfig x86_64 1.7.4-1.el7 base 181 k
Installing for dependencies:
libxml2-python x86_64 2.9.1-6.el7_2.3 base 247 k
libyaml x86_64 0.1.4-11.el7_0 base 55 k
python-chardet noarch 2.2.1-3.el7 base 227 k
python-kitchen noarch 1.1.1-5.el7 base 267 k
Updating for dependencies:
libxml2 x86_64 2.9.1-6.el7_2.3 base 668 k
python-urlgrabber noarch 3.10-9.el7 base 108 k
rpm x86_64 4.11.3-40.el7 base 1.2 M
rpm-build-libs x86_64 4.11.3-40.el7 base 107 k
rpm-libs x86_64 4.11.3-40.el7 base 278 k
rpm-python x86_64 4.11.3-40.el7 base 83 k
yum noarch 3.4.3-163.el7.centos base 1.2 M
Transaction Summary
================================================================================
Install 2 Packages (+4 Dependent packages)
Upgrade 1 Package (+7 Dependent packages)
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
salt-minion noarch 2018.3.5-1.el7 saltstack 37 k
Below is a similar log for Amazon Linux:
Installing:
m2crypto x86_64 0.31.0-4.el7 saltstack-repo 287 k
python-crypto x86_64 2.6.1-2.el7 saltstack-repo 470 k
python-msgpack x86_64 0.4.6-1.el7 saltstack-repo 73 k
python-zmq x86_64 15.3.0-3.el7 saltstack-repo 520 k
Updating for dependencies:
PyYAML x86_64 3.11-1.el7 saltstack-repo 160 k
procps-ng x86_64 3.3.10-26.amzn2 amzn2-core 292 k
python-requests noarch 2.6.0-7.amzn2 amzn2-core 95 k
Installing for dependencies:
libsodium x86_64 1.0.16-1.el7 saltstack-repo 140 k
openpgm x86_64 5.2.122-2.el7 saltstack-repo 172 k
python2-typing noarch 3.5.2.2-4.el7 saltstack-repo 39 k
zeromq x86_64 4.1.4-7.el7 saltstack-repo 555 k
So, for CentOS/Amazon you may want to remove the salt-minion
, PyYAML
, m2crypto
, python-zmq
, python-crypto
, and python-msgpack
explicitly, then auto remove the dependencies.
Listing dependencies using pip
If you installed some Salt dependencies via pip
, it is worth inspecting them too:
% sudo salt -C "G@kernel:Linux" cmd.run 'pip freeze --local'
% sudo salt -t 30 -C "G@os:Windows" cmd.run 'pip.exe freeze --local' cwd='C:\salt\bin\scripts\'
On CentOS, Amazon, and Windows, the command shows a full list of python packages, so you’ll need to compare it with a default one to find what is different.
Upgrade the Salt Master
From the FAQ:
Q: Can I run different versions of Salt on my Master and Minion?
A: This depends on the versions. In general, it is recommended that Master and Minion versions match.When upgrading Salt, the master(s) should always be upgraded first. Backwards compatibility for minions running newer versions of Salt than their masters is not guaranteed.
There are two common ways to upgrade a Salt master:
Option 1
The in-place upgrade process:
- Backup the
/etc/salt/pki
directory (just in case) - Remove the old Salt repo
- Uninstall the old salt-master package and all of its dependencies
- Add the new repo (py3)
- Install the new salt-master
Option 2
If you can’t disconnect all minions at once during the upgrade process, it is possible to bring up an additional master server and migrate minions one-by-one:
- Set up new salt-master (py3)
- Copy the
/etc/salt/pki
directory from old master to the new one, to ensure that master and minion keys are the same - Migrate the minions:
# migrate_minion.sls
migrate_to_new_master:
file.replace:
- name: "{{ salt['config.get']('conf_file') }}"
# Alternative variant if you keep master address in a separate config file
# - name: "{{ salt['file.join'](salt['config.get']('config_dir'), 'minion.d', 'master.conf') }}"
- pattern: '^master:\s+{{ salt["config.get"]("master") | regex_escape }}$'
- repl: 'master: new-master.example.com'
# Restart salt minion after the the config option is set
# A naive module.run with service.restart stopped working in 2019.2.1
# (https://github.com/saltstack/salt/issues/55201), so we have to use salt-call
# https://docs.saltstack.com/en/latest/faq.html#restart-using-states
restart_salt_minion:
cmd.run:
- name: 'salt-call service.restart salt-minion'
- bg: true
- onchanges:
- file: migrate_to_new_master
To be safe, you can migrate minions in batches:
% sudo salt -G 'master:old-master.example.com' --batch-size 10% state.apply migrate_minion
Upgrade Salt on Windows
With winrepo
Things are very easy in Windows land if you use winrepo-ng
:
1. Update your winrepo definitions
If you do not want to bring the whole winrepo-ng, just put the following two files into salt://win/repo-ng
:
2. Rebuild the package database
% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.refresh_db
3. List the currently installed version
% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.version salt-minion
3. Make sure the target version is available
% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.list_available salt-minion-py3
4. Run the upgrade command
% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.install salt-minion-py3 version=2019.2.3
You might want to add the --batch-size X
argument for a staged rollout.
Without winrepo
Do not want to use winrepo at all? Below is an alternative way to upgrade Windows minions.
1. Fetch the installer
% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2 and G@cpuarch:AMD64" \
cp.get_url http://repo.saltstack.com/windows/Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe
win10:
c:\salt\var\cache\salt\minion\extrn_files\base\repo.saltstack.com\windows\Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe
You can also include a checksum to prevent redownloads (works in Salt 2018.3.0 or greater):
% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2 and G@cpuarch:AMD64" \
cp.get_url http://repo.saltstack.com/windows/Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe \
source_hash=$(curl -s http://repo.saltstack.com/windows/Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe.sha256 |\
iconv -f utf-16 -t utf-8 | awk '{print $1;}' | tr A-Z a-z)
win10:
c:\salt\var\cache\salt\minion\extrn_files\base\repo.saltstack.com\windows\Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe
The default download location is fine, because the Salt installer doesn’t remove the cache dir since 2017.7.2.
2. Run the upgrade command
% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2 and G@cpuarch:AMD64" cmd.run \
'START /B c:\salt\var\cache\salt\minion\extrn_files\base\repo.saltstack.com\windows\Salt-Minion-2019.2.3-Py3-AMD64-Setup.exe /S'
Upgrade Salt on Ubuntu and Debian
There are a few key pieces:
1. Remove the old repo and key
remove_salt_repo:
pkgrepo.absent:
- name: deb https://repo.saltstack.com/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic main
# - name: deb https://repo.saltstack.com/apt/debian/9/amd64/archive/2018.3.5 stretch main
- keyid: 0E08A149DE57BFBE
It is also possible to delete all existing Salt repos and keys on both Ubuntu and Debian:
{% for repo_key, repo_data in salt['pkg.list_repos']().items() -%}
{% for repo in repo_data -%}
{% if not repo.get('disabled') and 'saltstack' in repo.get('line', '') -%}
Remove repo {{ repo['line'] }}:
pkgrepo.absent:
- name: {{ repo['line'] }}
{% endif %}
{% endfor %}
{%- endfor %}
{% for key_id, key_data in salt['pkg.get_repo_keys']().items() -%}
{% if 'saltstack' in key_data.get('uid', '') -%}
Remove key {{ key_data['keyid'] }}:
module.run:
{%- if 'module.run' in salt['config.get']('use_superseded', []) %}
- pkg.del_repo_key:
- keyid: {{ key_data['keyid'] }}
{%- else %}
- name: pkg.del_repo_key
- kwargs:
keyid: {{ key_data['keyid'] }}
{% endif %}
{% endif %}
{%- endfor %}
2. Stop the salt-minion service
systemctl stop salt-minion.service
3. Remove the packages
The snippet below should be adapted to your specific environment (i.e., remove all the dependencies, but do not touch any packages that are critical for production workload):
apt-get -y remove --auto-remove salt-minion
apt-get -y remove --auto-remove python-requests python-yaml python-apt
4. Install the minion
While it is possible to just apt-get install
the necessary packages:
apt-get -y -o DPkg::Options::=--force-confold install python3-requests python3-yaml python3-apt
apt-get -y -o DPkg::Options::=--force-confold install salt-minion
I feel that using salt-bootstrap.sh
is simpler and supports more platforms:
rm -rf /var/cache/salt/minion
curl -Ls https://bootstrap.saltstack.com | sh -s -- -x python3 stable 2019.2.3
Removing the cache dir is optional, however it was recommended some time ago on Windows (no longer so since 2017.7.2). Also, this comment hints on possible incompatibilities between cache serialization formats. And finally, it could be important if you use version-specific custom modules.
5. The resulting state
# upgrade/ubuntu_debian.sls
{% for repo_key, repo_data in salt['pkg.list_repos']().items() -%}
{% for repo in repo_data -%}
{% if not repo.get('disabled') and 'saltstack' in repo.get('line', '') -%}
Remove repo {{ repo['line'] }}:
pkgrepo.absent:
- name: {{ repo['line'] }}
# - keyid: 0E08A149DE57BFBE
{% endif %}
{% endfor %}
{%- endfor %}
{% for key_id, key_data in salt['pkg.get_repo_keys']().items() -%}
{% if 'saltstack' in key_data.get('uid', '') -%}
Remove key {{ key_data['keyid'] }}:
module.run:
{%- if 'module.run' in salt['config.get']('use_superseded', []) %}
- pkg.del_repo_key:
- keyid: {{ key_data['keyid'] }}
{%- else %}
- name: pkg.del_repo_key
- kwargs:
keyid: {{ key_data['keyid'] }}
{% endif %}
{% endif %}
{%- endfor %}
install_at:
pkg.installed:
- name: at
upgrade_salt_minion:
cmd.run:
- name: |
echo "systemctl stop salt-minion.service
apt-get -y remove --auto-remove salt-minion
apt-get -y remove --auto-remove python-requests python-yaml python-apt
rm -rf /var/cache/salt/minion
curl -Ls https://bootstrap.saltstack.com | sh -s -- -x python3 stable 2019.2.3" | at now
The at
trick was once recommended in the official docs. See the following issues if you like some archaeology: #5721, #7997, #22993, #32593, #39952, #43340, #46709.
You can also try the systemd-run --scope
method.
6. Run the upgrade command
% sudo salt -C "( G@os:Ubuntu or G@os:Debian ) and G@pythonversion:0:2" state.apply upgrade.ubuntu_debian
You might want to add the --batch-size X
argument for a staged rollout.
Upgrade Salt on CentOS and Amazon Linux
There are a few key pieces:
1. Remove the old repo
remove_salt_repo:
pkgrepo.absent:
- name: saltstack
It is also possible to delete all existing Salt repos on any rpm-based distro:
{% for key, repo in salt['pkg.list_repos']().items() -%}
{% if repo.get('enabled', '1') == '1' and 'saltstack' in key -%}
Remove repo {{ key }}:
pkgrepo.absent:
- name: {{ key }}
{% endif -%}
{% endfor %}
2. Stop the salt-minion service
systemctl stop salt-minion.service
3. Remove the packages
The snippet below should be adapted to your specific environment (i.e., remove all the dependencies, but do not touch any packages that are critical for production workload):
yum -y remove salt-minion
yum -y remove PyYAML m2crypto python-zmq zeromq python-crypto python-msgpack
yum -y autoremove
4. Install the minion without starting it
rm -rf /var/cache/salt/minion
curl -Ls https://bootstrap.saltstack.com | sh -s -- -X -x python3 stable 2019.2.3
Removing the cache dir is optional, however it was recommended some time ago on Windows (no longer so since 2017.7.2). Also, this comment hints on possible incompatibilities between cache serialization formats. And finally, it could be important if you use version-specific custom modules.
5. Reuse the old config
The snippet below should be adapted to your specific environment:
[-f /etc/salt/minion.rpmsave] && \
( cp -f /etc/salt/minion /etc/salt/minion.bak; \
mv -f /etc/salt/minion.rpmsave /etc/salt/minion )
6. Start the salt-minion service
systemctl is-enabled salt-minion.service || \
(systemctl preset salt-minion.service && \
systemctl enable salt-minion.service)
systemctl start salt-minion.service
7. The resulting state
# upgrade/centos_amazon.sls
{% for key, repo in salt['pkg.list_repos']().items() -%}
{% if repo.get('enabled', '1') == '1' and 'saltstack' in key -%}
Remove repo {{ key }}:
pkgrepo.absent:
- name: {{ key }}
{% endif -%}
{% endfor %}
install_at:
pkg.installed:
- name: at
atd_service:
service.running:
- name: atd
# - enable: True
upgrade_salt_minion:
cmd.run:
- name: |
echo "systemctl stop salt-minion.service
yum -y remove salt-minion
yum -y remove PyYAML m2crypto python-zmq zeromq python-crypto python-msgpack
yum -y autoremove
rm -rf /var/cache/salt/minion
curl -Ls https://bootstrap.saltstack.com | sh -s -- -X -x python3 stable 2019.2.3
[-f /etc/salt/minion.rpmsave] && \
( cp -f /etc/salt/minion /etc/salt/minion.bak; \
mv -f /etc/salt/minion.rpmsave /etc/salt/minion )
systemctl is-enabled salt-minion.service || \
(systemctl preset salt-minion.service && \
systemctl enable salt-minion.service)
systemctl start salt-minion.service" | at now
The at
trick was once recommended in the official docs. See the following issues if you like some archaeology: #5721, #7997, #22993, #32593, #39952, #43340, #46709.
You can also try the systemd-run --scope
method.
8. Run the upgrade command
% sudo salt -C "( G@os:CentOS or G@os:Amazon ) and G@pythonversion:0:2" state.apply upgrade.centos_amazon
You might want to add the --batch-size X
argument for a staged rollout.
Upgrade Salt on other operating systems
If you have done this on any other operating system (Red Hat, Fedora, SUSE, Gentoo, Arch, Raspbian, macOS, Solaris, FreeBSD, OpenBSD, etc.), feel free to share your experience and possible gotchas. I’ll be happy to update the article with any specific recipes!
Monitor the upgrade process
The upgrade process can take some time. There are three ways to monitor minions as they reconnect:
Watch the master log
% sudo tail -f /var/log/salt/master
Watch the master event bus
% sudo salt-run state.event tagmatch='salt/minion/*/start' pretty=true
List the Py2/Py3 minion count
% sudo salt "*" grains.get pythonversion --out json --static | jq '.[] | "\(.[0]).\(.[1]).\(.[2])"' | sort | uniq -c
1 "2.7.13"
1 "2.7.17"
1 "2.7.5"
1 "3.5.3"
1 "3.6.8"
1 "3.6.9"
% sudo salt "*" grains.get pythonversion --out json --static | jq '.[] | "\(.[0])"' | sort | uniq -c
3 "2"
3 "3"
Upgrade automation
If you need more control over the upgrade process (handle retries and upgrade failures, define specific orchestration sequence, etc.) you can use Salt orchestration or preferably write a custom runner. Another variant is to use salt-api
and pepper
.
An out-of-band upgrade using fabric
or salt-ssh
is also an option.
Future developments
In 2019 SaltStack introduced salt-bin and pop-build projects to make self-contained Salt binaries. Hopefully, they will simplify the dependency handling and make the upgrade process easier.
Need to run different Salt versions simultaneously?
The next post will explain how to write backward-compatible Salt states (e.g., to deal with module.run syntax differences, or for public formulas). Subscribe to read it as soon as the draft is done.
Top comments (2)
this seems like a crazy complicated upgrade path
I install salt master and agents using pip,
heres my bootstrap script,
then if i need to upgrade, just do pip install -U salt-minion on each host
the entire thing is in a virtualenv which can be wiped out and recreated easily
Un grand merci pour cette ressource fort utile, du moins pour moi et mon projet actuel :)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.