My home lab - aka the menagerie of systems in my attic - has a bunch
of different systems in it. No two are identical. A couple of them
have radios attached, a couple of them have spinning rust or SSD
drives attached, one has some environmental sensors, and the one
that's in the box is destined to have another radio. Two systems
attached to the home lab are in data centers elsewhere.
They are all different, but I want to manage them from
a single command line at times. I also want to be able to
set up a new system from scratch to my specifications with
a minimum of manual keying, knowing of course full well that
they are all essentially different by virtue of hardware but
the same by virtue of common management.
The story tonight is of setting up Saltstack as my management
control plane for this set of systems. This is not my first
go-around with Salt for arm64; when I
tried to get it to go in 2016
there were a few hurdles. The experience is better now,
though the salt-bootstrap
process does still throw warnings as arm64 support is "experimental".
It does install cleanly using the bootstrap script on the
five slightly different arm64 systems I have, corresponding
to Ubuntu 18.04, Ubuntu 20.04, and Raspberry Pi OS "Buster".
I am not yet ready to try Salt on the Pi Zero I have that's my
MMDVM DMR radio gateway. It will probably work, but it's more
important to me that the system move radio waves around right now
than necessarily have the cleanest management interface.
There are several sets of common tasks that I do want to reduce the
headache of, illustrated below.
Addressing the general problem of staying up to date with package updates,
the cycle of "apt update; apt upgrade" should be minimally a hassle.
I'm running Tailscale on all of these
systems, and Tailscale is rapidly releasing new versions. I want to
be able to determine what I'm running right now, and to prompt an
update as needed.
Each of these systems has a Go compiler on it, and the Go
support for arm64 has gotten a lot better over time even if the
install packaging has not markedly improved. I'd love to have a way to
push a new compiler to all systems as needed, or at least to figure
out how far away I am from latest.
In order, then:
Package upgrades
Using the aptpkg module documentation,
this set of commands does a wholesale refresh on all systems.
salt '*' pkg.refresh_db
salt '*' pkg.list_upgrades
salt '*' pkg.upgrade
Note that if the upgrade takes a long time you might get a "did not
finish" response with instructions on how to fetch the results.
Tailscale audit
salt '*' cmd.run 'tailscale version'
Hopefully this returns the same version across all
of your supported systems; if not, best to check why.
Go audit
salt '*' cmd.run 'go version'
It may seem quaint to admit this, but right now I have
five different versions of Go installed across six machines.
Clearly this needs some more substantial automation and
thought to fix.
