Logbook

privacy-preserving telemetry isn't real

I don't actually believe that privacy-preserving telemetry is even a thing that can be real. Telemetry is designed to leak information about what's happening on a system, and by doing that you inevitably leak bits of privacy-adjacent material, and if you accumulate enough bits you get reidentification of previously "anonymous" information.

This is in reference to our favorite elephant in the room's favorite hat, see https://discussion.fedoraproject.org/t/f40-change-request-privacy-preserving-telemetry-for-fedora-workstation-system-wide/85320/1 for the relevant discussion bits.

My favorite old-school reading on the topic of reidentification is the 33 Bits blog https://33bits.wordpress.com from Arvind Narayanan.

Now this is the right way to do opt-in telemetry:

https://dmesgd.nycbug.org/index.cgi?do=index

Launched in 2004, dmesgd aims to provide a user-submitted repository of searchable *BSD dmesgs. The dmesg(8) command displays the system message buffer's content, and during boot a copy is saved to /var/run/dmesg.boot. This buffer contains the operating system release, name and version, a list of devices identified, plus a whole host of other useful information. We hope others find this resource useful and further contribute to its growth.

Not only is this telemetry opt-in, but it's also public, and the contributors can edit their contributions as appropriate to elide inconvenient information or add more annotations to the system details. With a nearly 20 year track record, this is an important collection.

the elephant and the hobbyist

I very much hesitate to say anything about the elephant in the room regarding open source. The one thing I do think I can say without question is that anyone who characterizes reproducibly rebuilding an operating system from the provided source code as a trivial task of "hobbyists" grossly underestimates the amount of time, attention, effort and agility needed by any software distribution's infrastructure team to successfully produce a pile of bits that actually boots and runs on modern hardware.

I am referring of course to https://www.redhat.com/en/blog/red-hats-commitment-open-source-response-gitcentosorg-changes which reads in part (in bold type no less)

Simply rebuilding code, without adding value or changing it in any way, represents a real threat to open source companies everywhere. This is a real threat to open source, and one that has the potential to revert open source back into a hobbyist- and hackers-only activity.

Reproducible builds are very hard. There are so many things that can go wrong in building software before it works, and so many precise elements (compiler versions, libraries, headers, build-time instructions and variables) that all have to line up exactly right for a system to be able to be reconstructed "without changing it in any way". I look at NixOS as one example of a system that does rebuild cleanly, but boy howdy it's hard work to get all those pieces parts working exactly right together.

The other use of the term "elephant" has to do with software projects that have one large institutional maintainer. "Elephant risk" is characterized not by any of the obvious metrics of project health - contributor count, bug handling, vulnerability mitigation, good documentation etc - but by the hard to measure risk that the "elephant" will change its mind due to changing corporate priorities. I have seen perfectly reasonable software systems become suddenly untouchable as a consequence of private equity buyouts, mergers and acquisitions, or even personal problems in a founder or project leader's life. Unfortunate decisions by leadership can prompt people who were otherwise comfortable in their previous path to look around and look hard at options.

As to hobbyists, I have seen home labs that have put enterprise operations to shame.

removing nulls in a JSON structure with jq

jq is "like sed for JSON data", but instead of using exclusively regular expressions as its reason for being, it has a compact and powerful programming and text rewriting language.

The simplest jq program,

jq .

simply copies input to output, formatting things nicely along the way.

The question came up, how do you deal with APIs that print out lots and lots of data that are mostly nulls? The example is from Azure, but it could be anything. There is probably a very small, simple, (largely inscrutable) "jq" pipeline that would drop all of the null/false items and just show you the ones with data. And sure enough there is.

Reference these two posts for details:

Remove all null values - Stack Overflow

How can I exclude all keys with a specific value inside a JSON with jq? - Stack Exchange

The winner comes from Michael Homer, who writes that as of jq 1.6:

If your values are true nulls, rather than the string "null", you can use the nulls builtin function in place of the whole select: del(..|nulls).

jq works on JSON data. If your world is full of YAML data, then yq might be your thing; it has a comparable feature set and problem space, and it also knows how to cope with CSV, TSV, and TOML data.

"Write one sentence." - implementing "sisyphus" across multiple platforms

My magnum opus of published open source software is sisyphus, which is a simple little shell script that produces a random pair of suggestions of things to do next. The observation is that the world is full of repeated tasks that never make it onto todo lists, and that being mindful of same helps get you away from the keyboard and into real life.

My modest goal for sisyphus is to implement it into every environment that I touch. I think of it as randomness as a microservice, or perhaps just an exercise in connecting a tiny shell script to some hulking bit of complex infrastructure.

Some list of places this might show up if I get the right combination of time and energy:

• an OpenFaas function on top of faasd on a Pi 4
• a Jamstack function in Netlify
• a dynamic page in Gopher or Gemini
• the page https://sisyphus.annarbortelegraph.com somehow
• at the end of a curl command
• implemented in Go, Python 3, or Rust
• as a Telegram bot or a Slack command
• in a protected environment requiring two factor auth to access
• via a function call that generates a postcard

Fortunately the algorithm at the heart of sisyphus is very simple, as implemented by the GNU "shuf" command. And also fortunately the data set it shuffles from is small enough that there is no risk of any kind of performance problem, no matter how hard I try to make things fast or slow it shouldn't matter much.

More at the sisyphus repo on Github.

sustainability and open source

How do you incentivize people to act or work towards sustainability of an OSS project? -

Adhita, query posted to Twitter, August 30 2020.

Twitter is too small to answer such a good question.

There's a lot of variety within open source, and a lot of levels of project maturity. Sustainability for a personal project with a few casual followers is really different from the same question about the Linux kernel, because the kernel has already been through more than 25 years of sustained effort, and your initial commits have not.

25 years! And it's still not done. So perhaps that's one sustainability key objective - never admit to being finished with anything, so that there's always a free surface to experiment on or to take new contributions.

Sustained effort on a project is the combination of lots of factors - financial, for one, but also personal and structural. It's easy to be motivated by work if your next paycheck depends on it, and it's easy to stop being motivated by work on open source if your circumstances change and it's no longer sensible for you to know or care about how that particular bit of software works in order to make your living.

It's very hard to get sustained effort on anything that doesn't provide someone with sustenance. You have to believe that core maintainers of projects have figured out how to get an income out of their efforts, or to believe that they have figured out how to get an income out of something else that pays for this particular itch to be scratched. If you can't believe either scenario then it's likely that the project is not sustainable, not at least to the extent that it depends on the original effort to continue.

"Form follows funding" is pithy; but also "Follow your passion". It's hard to divide your passion among several competing projects, especially projects in a state where there's lots of interest but few contributions. The most sustainable way to keep a project moving forward may be to abandon it once it becomes a burden! The inrush, if any, of people and resources applied to keeping it going ahead may not accrue to you directly as ex-maintainer, but the project itself may work out better with someone new in the lead.

There are lots of open source stories, of projects that have had continuous sustained effort by one person, and projects that have changed hands a few times. Sustaining your personal effort might not be healthy, even if it's good for the project and the net. Open source gets abandoned, reinvented, forked, appropriated, and buried as dependencies in other projects. If you can see your way forward to a personal or career goal that is improved by open source, you can sustain that effort throughout a career.