Logbook

running Kubernetes on underpowered hardware

I have a single-node Kubernetes system running on a 4GB arm64 Jetson Nano, using kind (Kubernetes in Docker). This is the smallest configuration I can reasonably expect to have a functioning Kubernetes control plane on. The device regularly throws errors of the type

2021-05-23 19:00:21.333267 W | wal: sync duration of 2.25553095s, expected less than 1s

in its etcd log files.

This particular device is using an SD card for its backing store for Docker. etcd has a very heavy write load, and when its writes are too slow it will spit out this error. The solution (once I am ready to fund it) will involve restarting the cluster on an SSD storage device, which is generally regarded as fast enough to etcd to stay happy.

Is there a sweet spot, cheaper than SSD but more performant than an SD card? Marcel Wiget has a 3 node CM4 cluster using the onboard eMMC memory of the Pi CM4 modules for backing store.

When etcd is unhappy, your Kubernetes cluster is unhappy. That process is responsible for managing consensus among nodes, and when it can't keep up, all manner of things are said to fail. If your cluster has mysterious timeouts, check etcd first.

IBM Cloud has a tutorial on using fio to benchmark your storage to help predict whether it will be fast enough to run etcd. You are looking to benchmark writes, specifically the 99% percentile time for the fdatasync(2) system operation to keep it under 10ms.

Are there ways to reduce the write load of etcd, and what are the tradeoffs? If you're willing to rebuild the cluster from scratch every time there's a crash or a reboot, you can look at the unsafe-no-fsync flag, or put etcd backing store on tmpfs. Both are suitable only for very ephemeral clusters, like the sorts of setups you would do for CI/CD testing.

With only 4GB of memory, the Nano is no powerhouse - in use it eats up about 1.5GB of runtime memory for applications that keep Kubernetes going, leaving only about 2.5GB free for actual application memory. The biggest process, etcd, makes very heavy use of memory-mapped files. The working plan is to set up a few low-impact OpenFaas functions on the Nano once it's stabilized.

kind has arm64 support only fairly recently. It's best thought of as a quick and convenient way to set up a temporary cluster, especially given its warnings that there are no provisions for cluster upgrades and no specific security hardening.

Acknowledgements

Thanks to the following folks for assistance in this bring up process

• Alex Ellis (tutorials, OpenFaaS, "arkade" package manager)
• Matteo Olivi (IBM tutorial on fio)
• Ben Elder (kind and arm64 support)
• Thomas Strömberg (etcd on ramfs, "half kidding")
• the Portainer team (easy access to log files)
• Jason DeTiberus (warnings about cascading failures)
• Marcel Wiget (CM4 cluster with eMMC)

None of this should be thought of as recommendations for a production cluster, but everyone should have access to a teaching cluster that's woefully underpowered so that you can see what fails when you run out of some scarce resource.

Using BindToDevice in systemd socket files to serve network-specific content

It's summer, and that means the attic gets hot. I'm looking to get some kind of easy way to check the attic temperature from afar, so that I can keep tabs on it, and so that I can adjust the fans and the windows appropriately. This is really basic telemetry, a task that has been done many times before, using nearly every network protocol ever invented.

I decided for 2021 that I wanted to do this monitoring with some old-fashioned protocols, so the query tool of choice is "finger". I found a very nice modern implementation called finger2020 from Michael Lazar (mozz), which is designed for single-user systems. This runs under systemd, which makes it easy to install.

A simple crontab pulls temperature data from wherever it can find it, which on a Linux system could be lots of things. From past experience I have found that simply tracking the /sys/class/thermal values from the kernel give you CPU temperature for whatever computer you're running on, and that if your computer is hot, the rest of your space is probably hot too. As a bonus it's easy to grab with no additional external sensors needed.

I'm running Tailscale, which means there's a trustable inside network on tailscale0, and an untrusted outside network on ethernet or wifi or whatever interfaces are exposed. This led to the question: can you deliver different content to the world depending on which interface the traffic comes in on, without putting any of that logic into the program serving up the data?

After some spelunking through the systemd documentation and not a lot of searchable examples from the net, I came up with either a brilliant hack or an awful kludge to make this work. systemd has a BindToDevice directive in its socket files that lets you associate a socket with an interface (not just its address, which could change, but the interface itself). So I took the unit files from finger2020, replicated them three ways, and for each one added an appropriate BindToDevice=tailscale0 or whatever interface I had and then created a service file that pointed to that socket.

The slightly weird part is that if you are on a machine querying itself, even if you specify an IP address of an external interface, you'll get packets on the loopback (lo) interface. So one socket handles loopback, one handles eth0 (for queries on a public network), and one handles tailscale0 (for queries on the private VPN).

A more elegant approach if you want to change your server is to look up the address from Tailscale, using the function tailscale.WhoIs. An example of this logic is in the hello service run by Tailscale which switches on the results of the WhoIs call and lets you further serve individual content because you know not just the interface the user is coming in on but the user name! So very handy if you have more than one user on your network.

The BindToDevice approach should work for any VPN you are using, and it should also let you do deliver services that are reachable only from private addresses and that don't ever show up on the public Internet, even if the code you are running is ignorant of that distinction.

Brilliant hack or awful kludge? Hard to say yet.