The upstream parable
Stop me if you’ve heard this one before.
Three friends are relaxing beside a river. Suddenly, they hear the sound of someone crying out for help.
Looking out into the river, they see a child flailing in the middle of a strong current. The first friend moves quickly, jumping in and swimming as fast as they can to get the child and pull them to safety.
Before the first friend is able to get back to shore, there are more cries for help. Three additional children are in the river, in need of urgent rescue. The second friend, knowing he must rescue all three at once, grabs a nearby raft and paddles out into the river.
Meanwhile more children keep popping up, all needing rescue. This second friend, overwhelmed, desperately looks around for the third friend, who is nowhere to be seen.
Finally, he spots her, in the river, swimming upstream.
“Where are you going, these children are going to drown!” calls out the second friend.
The third friend keeps swimming and calls back:
“I’m going upstream to find out who is throwing all of these children in the river!”
What is upstream thinking?
This parable is at the heart of what is known as upstream thinking. In his recent bestseller Upstream: The Quest to Solve Problems Before They Happen, author Dan Heath introduces the concept this way:
“So often in life, we get stuck in a cycle of response. We put out fires. We deal with emergencies. We stay downstream, handling one problem after another, but we never make our way upstream to fix the systems that caused the problems. Cops chase robbers, and doctors treat patients with chronic diseases, and call-center reps address customer complaints. But crime and chronic disease and customer complaints are preventable! So why do our efforts skew so heavily toward reaction rather than prevention?”
So if downstream thinking focuses on solving problems after they occur (fishing children out of the water, in our parable), upstream thinking focuses on efforts to prevent problems before they occur (“I’m going to find out who is throwing all of these children in the river!”).
Those who practice upstream thinking, also known as upstreamists, are systems-level thinkers, seeking to understand the root causes of downstream emergencies.
Upstream thinking and the White House cybersecurity executive order
Which brings me to the recent White House cybersecurity executive order, which we’ve done a full debrief on here.
TL;DR: the cybersecurity executive order is an attempt by the United States government to use its purchasing power to create positive changes to the way cybersecurity is addressed around the world.
Recent high profile breaches like the Colonial Pipeline ransomware attack or the SolarWinds software supply chain attack have shown that our cybersecurity defenses are woefully inadequate. This executive order forces a higher standard of cybersecurity for any organization selling software to the federal government, which in turn makes it the de facto global standard for all software in the future.
As I read the executive order for the first time, I couldn’t help but think it provides an incredible opportunity to apply upstream thinking to software.
In open source, the focus has been downstream for too long
Sticking with our parable for a second, when we look closely at the problems that open source software has faced when it comes to cybersecurity, most work over the past few years has gone into developing downstream solutions that help those already in crisis.
In fact, an entire industry has cropped up around software composition analysis (SCA) and container scanning tools that will tell you what is wrong with your software from a security or licensing perspective. This kind of open source urgent care has become a big business, with companies raising hundreds of millions of dollars and grabbing the attention of frustrated CIOs who don’t want to go down in flames as “the next Equifax.”
But are we really content to continue being overwhelmed by a never-ending parade of open source health crises?
Or should we instead turn our attention upstream?
Spurring an upstream movement in open source
On June 7, Tidelift will be hosting a completely free, virtual event to celebrate open source and the people who create it, and we’ve named it Upstream (sign up here!).
From the Wikipedia definition of upstream, as it relates to software development:
“In software development, upstream refers to a direction toward the original authors or maintainers of software that is distributed as source code, and is a qualification of either a version (released by the original authors, based on their upstream source code), a bug or a patch.”
Our original intent with this name was to honor the work of the people who create and maintain the open source code that all of our applications rely on: the upstream maintainers.
But we also loved the dual meaning, that this event could serve as an opportunity to spur a movement of upstream thinking in our open source corner of the universe.
While Dan Heath’s book shares examples of upstreamists operating in many different fields, healthcare is probably the field where upstream thinking has seen the most traction.
For some broader perspective and inspiration, we’ve invited one of the leading upstreamists in healthcare, Dr. Rishi Manchanda, to give a keynote at Upstream. If you haven’t seen his TED Talk on upstream thinking, you can view it here.
But why wait? We can start applying upstream thinking to open source software security and maintenance issues right now!
Applying upstream thinking to open source
Let’s do a little exercise in upstream thinking.
Problem: managing open source effectively and keeping it secure is an enormous challenge
For most organizations, it has become a difficult—if not practically impossible—task to effectively manage the security and maintenance of all of the open source code they use.
In fact, a recent Tidelift survey found that only 16% of organizations with over 10,000 employees are extremely confident that their open source is up to date, secure, and well maintained. Meanwhile a whopping 39% were not very or not all confident in their practices for managing open source.
A downstream solution: scanning
Many of these organizations have attempted to solve their problem with software composition analysis tools that point out security, maintenance, and licensing issues with their open source dependencies. But too often these tools are introduced late in the development process, and report false positives or issues with no easy resolution. They are in essence crying out, “hey, there are children in the river!!”
Yeah, we know.
This leaves developers stuck with bad choices.
- Replace problematic packages (with the associated delay, and only possible if good alternatives exist)
- Rewrite the code yourself (if you have time)
- Work with with an open source maintainer to get the issue fixed upstream (if you can get their attention)
- Ignore the scanner result (if you want to roll the dice)
Heading upstream to discover root cause
It begs the question: why are there so many issues for scanning tools to find? And could we reduce the open source equivalent of emergency room visits by examining the environmental factors that are causing them?
There are a lot of root causes we could consider for issues in open source, from what motivates scanners to report false positives to how package managers deliver updates. But today I want to focus on one root cause that is staring us right in the face. It was perhaps most poignantly captured by this recent xkcd cartoon.
In open source, the majority of people writing the code that modern digital infrastructure depends on are volunteers or are grossly under-compensated for their work. In other words, random people in Nebraska, thanklessly maintaining projects since 2003.
Sneak preview of our upcoming maintainer survey results: almost half of maintainers are paid nothing—not one cent—for their work, while more than half are paid less than $1,000 US per year. Only 6% are paid what might be considered a living wage (more than $50,000 US per year).
Which means that for most maintainers, open source is more of a hobby than a profession. It just happens to be a hobby that all organizations are counting on them to keep doing, to an enterprise standard, without compensation.
It doesn’t take a neurosurgeon to uncover the root cause here.
An upstream solution: partner with maintainers and users to improve software health and security
Which leaves us with some simple truths:
- Enterprises building applications on top of the modern open source digital infrastructure need open source they can rely on.
- Open source maintainers should be fully compensated if we expect them to deliver secure, well maintained software built to an enterprise standard.
It might seem kinda obvious, but one solution is to build a model where maintainers get paid to do the work enterprise users need them to do.
How many security issues could be avoided if maintainers were being properly compensated to ensure their components stayed secure and up to date? Even more interesting: how much amazing open source software isn’t even being created today because there aren’t incentives in place for the people who might create and maintain it?
In Rishi Manchanda’s TED Talk about upstream thinking, he tells the story of Veronica, who appeared in his clinic suffering from debilitating migraine headaches and severe allergies. Multiple visits, multiple treatments, nothing could fix what was wrong with her.
Until Dr. Manchanda asked her about where she lived. It turned out that her house was moldy, wet, and roach-infested. Instead of prescribing another medication to treat her symptoms, the doctor referred her to a specialist who could help her improve her housing conditions. As if by magic, once her housing conditions improved, her migraine headaches and allergies disappeared.
If we really want to address the health and security of open source, it’s time to get our house in order.
Let’s pay the maintainers and get a head start on tackling a principal root cause of open source cyber insecurity once and for all.
If you're interested in exploring this topic further, please join us on June 7 for Upstream, a free online one-day celebration of open source, the developers who use it, and the maintainers who create it.
Photo by Michael Niessl on Unsplash
Top comments (3)
Thanks for the post, that was a very interesting and informative read. Two things jumped out at me.
First, this: "Even more interesting: how much amazing open source software isn’t even being created today because there aren’t incentives in place for the people who might create and maintain it?"
The answer to that question is zero or at least trending toward zero. The kind of programmer that makes open source content doesn't do it for incentives and doesn't avoid doing it due to a lack of incentive. Just as a painter is compelled to paint and a writer is compelled to write, a coder simply must code. Every programmer that isn't just punching the clock knows this; we code because coding is a passion. Suits worry about returns on investment and cost/benefit analysis, but coders conceive of an idea and create it for no reason other than that "lonely impulse of delight", as the saying goes.
Second, "If we really want to address the health and security of open source, it’s time to get our house in order."
That is the most important point any of you suits out there can take away. Lately we hear a lot about what the corporate world needs from the open source community. Well, to be frank, the open source world isn't here to serve the interests of suits and their bottom lines. We don't work for you and, to be complete honest, we really don't care about your profits.
Never forget, it wasn't the open source community that came to the corporate world after our software strategy failed, it was the other way around, you came to us because free and open source is better than proprietary and closed source; because passion and freedom of ideas is better than paychecks and demands handed down from on high.
By all means, buy maintainers a coffee, fill their tip jars, donate to their paypals and patreons, I love to see open source devs prosper, but when big corporations talk about incentivizing open source development, all I hear is that they want us to be their unofficial freelancers, coding what they want coded when they want it and how they want it. In other words, they want us to get their houses in order by getting us to dance for dollars.
Instead of coming in as bosses throwing around money and making demands, I would much prefer that the suits pay their own programmers to improve the software and submit that code back to the project. Don't leech, don't try to control, just contribute to the community like the rest of us do.
To say it as simply as possible, if Billy Bob in Nebraska gets bored and abandons his project, causing AmaMicroZonBook's multi-billion dollar platform to crash... Oh, well. I guess they should have been actively contributing code and giving back to the community, instead of just leeching or throwing money around. Like you said, get your own houses in order, instead of trying to turn our community into your new unofficial dev department.
I disagree with the first point. I'm sure there are lots of good ideas that go unimplemented (or at least unreleased) due to the demands of making a living and other responsibilities (e.g. single parent, sole caregiver). A day has 24 hours regardless of how much "passion" for coding you have.
There are companies that include (paid!) open source maintenance and development as explicit team goals; we just need more of them.
That's almost true ;)