The issue of the {day, week, month, year} is log4j, a logging library
for Java that turned out (unbeknownst to the world) to have a lot more
subtle power than was obvious on the surface. When looked at closely,
this subtle power turned into a vector for remote code execution (through
the Java JNDI interface) and exfiltration of secrets (through variable
expansion). The world is scrambling to figure out what their exposure
is to this newly found threat, and in many cases also trying to figure out
how they can safely update deeply embedded buggy dependencies.
If log4j were the product of a well-funded, scrappy, pre-unicorn startup,
the question of what to do would be different. As is, the committer
base for the product is very small, it's old in Internet terms, and
the original feature that triggered this issue was added years ago.
The pernicious thing about this particular exploit is that to be sure
you're not vulnerable you have to audit not only all your own code but
also all the code of all the systems you interact with. This includes
embedded systems, control planes for routers and switches, software
that you consume as a service rather than running on-premises, even
device firmware. Java is everywhere, and log4j is very popular, so
the attack surface is huge.
I am not sure what we as an industry can do about this kind of issue.
Software lifecycles are very tricky to manage. If code is feature
complete and not being actively added onto, people walk away from it
to do more interesting things, and you get the risk of abandonware.
If on the other hand the code tries to keep up with the pace of change
of the modern Internet, it can very easily accumulate a lot of cruft
and complexity. Scrapping a fully working system to replace it with
something brand new is very risky.
The question of funding comes up time after time. The very small team
that continues to work away at log4j after lo these many years is
not paid directly by the foundation (Apache) that hosts their effort.
For most big corporations, it's very hard to spend small amounts of money.
Larger amounts of corporate money might be available for feature development
(and added complexity), but see above re cruft.
I have seen foundations try to tackle this conundrum, e.g by paying
outside parties to do a security audit of code that's under their
umbrella. This is an expensive but controlled way of finding bugs
and design flaws in a system, and if it's done right it can contribute
to a sense of confidence about one specific component. But no one
is talking now about auditing "all code in Maven" or "all code in npm",
that's near enough to an infinite task to be considered impossible.
Even automated audits and fuzz testing only go so far to find known classes
of bugs.
I leave this without any simple solutions. The "software bill of materials"
(SBOM) approaches being actively promoted right now will help to a certain
extent, but even if they can accurately enumerate your dependencies they
won't necessarily tell you how to fix them if something goes haywire.
GitOps and other automated deployment tools will assist you in recreating
your infrastructure if need be, but won't always have a simple answer about
how to safely shift a single embedded component.
No one really understands their full stack (from pretty UI all the way down to BMC firmware).
Complex systems are in a state of perpetual partial failure. It might
have been a mistake to teach sand how to think.
