Logbook

typing a reply into a little box, thinking better of it, deleting the post

Rather than post a reply to a thread I will start one here. It's always best to be at the top of the page. Never read the comments.

Not everything that someone else types into a little rectangle deserves a reply, even if you have one handy. Networks that are optimized for engagement always bias towards inflammatory posts that trigger replies, you're just feeding the beast if you take the bait there.

But, some things should be said, even if not directly in reply to the thing that prompted them. Let's go through it in order.

On the current state of AI: A surprising number of mature products are returning AI-generated results, with something between terrible and hilarious results. Amazon's recommendation agent badly summarizes user-generated content, Google's search engine promotes goofy text from normally reliable sources as if it were correct, and more. It's a full-on Gish Gallop where machines can spew out nonsense faster than we can correct it.

On nostalgia for the "small web", the "old web": A few of us remember when the net was different. It was a lot slower, a lot more expensive to use and to operate, and the public web was tiny by modern standards. There was a culture which was kept in check by acceptable use policies that threatened you with banishment if you went beyond them. That old web barely exists now in little pockets where it hasn't been overrun. If you weren't there for that, I'm sorry. But don't criticize people who go out of their way to recreate that experience, even at great personal expense, because it was really different and not like the net we have now.

On internal communications within organizations: an incredible amount of useful information gets produced in the course of people interacting online and responding to questions from their peers. Even the best of intentions does not turn that ongoing conversation into reusable artifacts without a bunch of work. The problem is compounded by ever-changing organizational structures that do not map neatly to well-known names of things or existing external knowledge. I do know that when a well-understood task is carefully documented for external use that it can be incredibly helpful for transfer of knowledge. Unfortunately the questions come so thick and fast that there's also tendency to assert that the customer could and should do something complicated that we barely understand ourselves how to do in a repeatable fashion.

On complexity: I have seen one too many architectural diagrams where a complex system is built out of complex parts to solve what should be a simple problem. Better yet all of the complex parts come from the same vendor, so that in order to understand what the system is doing you need to commit to deep knowledge of that single vendor's product space. Contributing to these sorts of projects is very difficult as a lot of the necessary knowledge is in-house in the vendor and often the scale of the solution is well beyond the single computer or home lab to prototype a design. Big-budget complexity doesn't look more likely to succeed on its technical merits, but it might have a better business reason for existing because there's money to be made when your customers have to be all-in on your products.

(OK enough ranting for now. Some fragments of this might get cut and pasted into smaller rectangles.)

Obsidian as a daily note taker, a year out

I started using Obsidian about a year ago, and switched away from Github Issues about six months ago. So far so good. As is my habit, here's some notes on the process that will probably be more useful to me than they will be to you, but that's the nature of blogging these days to write notes to yourself mostly.

Obsidian has completely captured the "daily diary" part of my daily routine, as well as the "write down what you're going to do today before you do it", and the "take notes in wiki format". All of that is about 80% of what I need, and so it gets repeated daily use on my laptop.

Surprisingly, even though I can and do sync my laptop with my Android phone, I don't really use Obsidian much on my phone! To the extent that I do, it's for lookup rather than typing. I guess part of that is that the computer is so much faster as an input device that for any writing of any length I'd rather be at a full sized good keyboard.

When I gave up on Github Issues for daily tracking, there were 10000 issues that I had gone through, and I could tell you with some certainly how long various items had been on my todo list. And you know what, that's not a knowledge about myself that I miss having. I write things down and then I do them or don't do them. The process leaves enough of a trail that I can reconstruct my thoughts. I'm not really reporting out to anyone else the progress towards any one specific goal, so there's less unfinished business to worry about.

Is there unfinished business? Of course. It's just that's accepted as a normal course of events, and you get to things because you remember them and not because there's some Master List Of All Undone Tasks that you consult before taking on the next iteration of your day.

I love Obsidian as a daily note pad, and I really like it that I can bop around in about 1600 separate pages that are a fair capture of a year's worth of doing things and feel at least to a certain degree that it's a connected whole.

At 1600 pages a year, it's about six years when I hit the 10000 page mark. That would be a nice body of work, and by then some other system for writing will probably be available on some shiny new computer and I'll want to start over again from scratch. But until then - this is pretty nice, and I intend to keep at it. (Check back in 2027 and see if I still am thinking the same way.)

moving from Github Issues to Obsidian for daily issue tracking

I started using Github Issues as a personal issue tracker in 2012. The first issue was "Replace mobile phone with new device - iPhone, Android, or burner". It took me months to decide what to do. Since then, I've opened over 10,000 other issues with small or large details of things I'd like to do.

I've decided to give up on Github Issues as a personal info tracker, and replace it with Obsidian. While the thought is fresh in mind, here's a few reasons why.

Speed. Github Issues is available only through a web browser, while Obsidian is an app that stores the entire database in memory. As a consequence there are round-trips to the net for every transaction and every browsing effort on Issues, while Obsidian lets me bop around from node to node with no visible lag.

Portability. Github Issues isn't easy to download as a database. The whole thing can be pulled down to Markdown by a third-party tool (I'll want to do that). Obsidian is easy to sync to Git with a plugin, so I have the whole thing locally backed up, and I can browse and edit when I'm offline.

Native apps. The several third-party native apps I've tried for Issues have been really nice and have had a too-short shelf life. Bee.app was discontinued by its maker, and Gitscout seems to have died before a proper 1.0 release. Having to do everything through a browser feels clunky and slow and distant.

Community. Github Issues is designed for teams working on software. Obsidian is full of individuals with quirky knowledge management habits. I'm much more likely to run into someone with a cool idea about how to traverse their state of consciousness in Obsidian-land, and much more likely to find a product manager burning down a backlog in Issues-land.

Fresh start. Like I said, I had over 10,000 issues completed, which is my signal that I've learned enough about the environment to make some decisions about how well it works for me and whether the limits of it are ever going to change. My Obsidian page-count is nearly 600 right now, which means it has plenty of room to grow and morph as needed.

Wiki nature. I fell hard for Wiki ideals and wiki naming a couple of times (Socialtext, Arborwiki, Localwiki). I like typing a word in double brackets and having it make a hyperlink ready to click on. With Github Issues it is possible to cross-link issues by referring to items by number, but then you have to remember a lot of numbers, and who can do that?

The "law of two feet" from Open Space says that if you're not happy where you are, it's your responsibility to remove yourself from that location and put yourself somewhere that you are happy. I'm ready to explore a new challenge, cross everything off my todo list all at once, and recreate it as needed in a new place. We'll see how long this enthusiasm lasts.

bit rot in distributed systems

Work at $DAYJOB has been very busy, so much so that the time for hacking on the menagerie of systems on my desk has been limited. Naturally there has been some bit rot here - systems that used to work, that no longer do.

Fortunately, that's exactly the reason that you keep a blog - to write down how to do the things that you once knew how to do, so that you can repeat them again. Even more so, that's the reason that you drive towards automated installs and setup and provisioning, so that you can put another system on the net in minimum time.

My menagerie is deliberately all of different kinds, so it poses an extra set of challenges. Still, with some elbow grease and careful rereading of past posts, I should be able to refresh the setup over the holiday break.

friction and systems management

Imagine bricks made of molasses mixed with pumice, applied to systems in order to protect them but rough to deal with.

From John Gall's "Systemantics"

When Charles Babbage, early in the Nineteenth Century, attempted to build the world's first large calculating machine he made the parts of wood and promptly discovered the importance of Internal Friction. Briefly, his machine wouldn't go -- and he couldn't Push It hard enough to Make It Go without breaking it....This was not a Planned Discovery, and it was not one that Babbage enjoyed making.

And also Piet Hein from Grooks 1

Problems worthy of attack prove their worth by fighting back.

When building systems, inevitably there will be some friction between you and the desired end state. Anything you can do to recognize that there is friction and address it is worthwhile. You get stuck; that stuck feeling is friction.

Observe that friction happens at all layers of the stack, and that having friction at one level can obscure the difficulties you are seeing at adjacent levels. Take for instance the venerable corporate firewall. If you can't reach a remote site that you need to get your work done, it's impossible to know whether this is as a consequence of corporate policy that is hard to change, firewall vendor software that is buggy, or even a simple typo in some config file somewhere that doesn't intend to make policy. It's unclear whether something simple needs to be sanded down to size, or whether an entire layer is made of sticky grit.

Friction in an ecosystem shows up in the form of seemingly simple tasks that end up taking way longer than they should take - and as a consequence, filtering out anyone with a dopamine timer that's too short. Note that people who are used to quick response times and rapid turnaround rarely have much patience for tasks that take forever.

Friction wears you down; it makes the easy tasks laborious; it entraps you in its gritty debris; it turns work into heat. Recognize it where it happens, and do your best to smooth the path when it throws sand (or wood) in the gears.

---

Postscript: also noted, Strategy by Design on Friction as part of a discussion of von Clausewitz's "On War".

He used this term to describe the “myriad of small, but collectively numerous things that happen that cannot be foreseen or planned for, and which cause leaders to spend time on unforeseen decision making.”

Logbook: robustness, gray failure, and emergent misbehavior in complex systems

Reading: Robustness in Complex Systems, Steven D. Gribble, 2001. Noted by Cindy Sridharan. " We show that although the system behaved correctly when operating within its design assumptions, small perturbations sometimes led to the violation of these assumptions, which in turn lead to systemwide failure." I am always up for reading good stories about failure in complex systems, and this paper has several of these, including several good root causes of failures under load.

More reading: Emergent (Mis)Behavior vs Complex Software Systems, Jeffrey Mogul, HPL-2006-2. Mogul looks to produce a taxonomy of misbehavior, to look for common causes and common symptoms of things that go wrong. One key set of observations is the role of delay, especially when coupled with message loss, in creating conditions where centralized control systems break down, and where distributed control systems get confused.

Still more reading: Gray failure: the Achilles’ heel of cloud-scale systems Huang et al., HotOS’17. Adrian Colyer's writeup is a very good start to understand the "gray failure", the misbehavior of the system that looks like it's up to one observer but registers as down to another. This is "differential observability", or "I don't know why you think it's down, it works for me" wreaks havoc on control systems that assume they have one source of truth. In a cloud environment, many systems are complex, and the down detector may erroneously point the blame at one component when the root cause is somewhere else.

As a side note, the literature on failure in complex systems gets more complex over time. We now know, for example, that memory leaks are bad, and that uncontrolled memory leakage leads to all kinds of failures. What no one even thought about years ago was the role that hypervisors and their resource allocation (or resource starvation) strategies under load would play in how big systems exhibited unpredictable behavior. Don't put your key components on nodes that throttle when they get too busy! 

Way too much reading: Kubernetes Failure Stories (k8s.af). If you want to get better at failure, then you should fail more often. Kubernetes gives you that opportunity! Thanks to Henning Jacobs for that list.