It turns out to be efficient and easy to use Prometheus to scrape a bunch of metrics + a custom sensor reading and feed it into Grafana. Key to this is Tailscale to keep everything on a private net. But first, I had to go through some detours and to consider some other perfectly reasonable software that was not as well setup for this particular task at this particular level of scale and polish. Here's a start-to-finish account of the effort. Note that everything happens to run on Arm processors, mostly because the menagerie of little systems I have in the attic are Arm machines of one kind or another.
Temperature sensors: It turns out that even the simplest little single-board computer has at least one and often many more temperature sensors on board. These are used to throttle back the CPU to a lower speed if it gets too hot, or provide input controls to fans or other active cooling. You will find these sensor readings in the /sys/class/thermal filesystem exposed by the kernel. There's kernel docs on the General Thermal Sysfs Driver, but mostly what you need to know is that you don't need to have any program more substantial than cat(1) to read those temps.
Temperature sensors (2). There are also standalone sensors you can get that talk over any of a myriad of protocols and functions. I have one module that talks over the I2C bus using the qwiic connector as an interface, and there are Bluetooth and ISM band radio protocols used to transmit temperature in any of dozens of ways. For the purpose of this exercise, you'll want to find a remote sensor reading that you can extract by typing a command at the command line prompt. The specific code I am using is derived from Sparkfun's Qwiic_BME280_Py because I have that sensor. If you have a radio-based weather input you can take inspiration from rtl_433 by Benjamin Larsson.
Metrics gathering. Temperature sensors are just one of many sensor inputs that a modern computer might have for you. It's reasonable to want to also keep tabs on load average, disk space free, network bandwidth used, and any of 100s of other potentially useful values. Rather than collect each of them one at a time, run something like Prometheus's node_exporter to gather metrics in bulk. This is where I diverged from last night's foray into MQTT and NATS, which are both more about prompt delivery of individual messages than they are about wholesale bulk delivery of metrics data.
Metrics gathering (2). There will be some metrics that Prometheus's node_exporter won't have native drivers for, so you'll need to write your own. Thankfully the Prometheus text file format is incredibly simple and you can easily schedule a shell script to run out of cron to fill a directory with metrics that will be gathered at the same time the system metrics are harvested. I found the tutorial on the Robust Perception weblog, Using the Textfile Collector from a Shell Script, to be just what I needed to make a teeny tiny script based on the BME280 code above to dump out one line of text.
Metrics aggregation. Prometheus has a very straightforward and easy to install configuration for collecting the metrics produced by node_exporter and assembling them into a time series database for querying and simple graphical reporting and charting. With Prometheus, you can collect lots of data from lots of remote systems, polling as frequently as necessary for you to get prompt notice of changes in your measurements.
Graphing and reporting and alerting. Grafana takes the data out of Prometheus and provides neat dashboards and flexible alerting. I got as far as installing it to prove that I could put some charts up on the screen, but I haven't touched most of its functions.
Private network. Everything is harder when you're trying to provide access to systems that are remote behind firewalls, network address translations, and other obstacles. It's also harder when the endpoints you are interested in are on the wide-open, hostile public Internet. An essential part of my setup is Tailscale which lets you build a network overlay that makes a few machines on your private network reachable to each other even through firewals but without exposing them to the world. When there are long-running tasks on this network, bind their listening addresses to their Tailscale address, which will allow you private access.
Not everything went perfectly on the way to getting things going. I ran into a couple of minor bugs, which should be addressed in due course. Sharing those here to help the next person who looks for them.
Grafana 7.0.0 has prebuilt binares for arm64 for Ubuntu 20.04 which will fail to work on Ubuntu 18.04 with an error message version GLIBC_2.28 not found (required by /usr/sbin/grafana-server).See issue 24844. Work around by installing an earlier version of Grafana on Ubuntu 18.04, or the latest version on Ubuntu 20.04 or by compiling from source. There is a fix in the works.
Prometheus 1.0.0 node_exporter will emit error parsing mdstatus: error parsing mdstat /proc/mdstat on Raspbian, which doesn't stop it from working but does keep the error log filling up. See issue 1719. You can manually disable the collector with --no-collector.mdadm to avoid the log noise.
The easiest way to get started on a project like this is not the way I started it and not the way I'd do it again. From the user interface down, the sensible order is to get Prometheus up and running first, set up node_exporter to collect some metrics, and then set up Grafana once you have some data to make pretty. From the sensor up, this effort is best begun by producing the simplest shell script that will emit a single number that's your sensor reading of interest. Once you have those two ends of the puzzle done, the missing middle is the "textfile collector".
I assure you in your search for how to get from sensor to screen that a "textfile collector" would not be any search term you would be likely to use. Once you get this far, though, it might be obvious that's a key component to the effort. This repo of node exporter textfile collector scripts will be helpful in unlocking some of the power of the platform.
