BY NOW, most of the internet knows about the famous Log4Shell exploit, and if you don't, it's easy to get a sense of how disastrous it's been. To drive the point home: the US Department of Homeland Security is warning people about it.
There's been a lot of hand-wringing about how open source software, the lifeblood of many businesses today, is often totally unpaid and unthanked work, with some hot takes like 'Open source needs to grow the hell up.' and 'Open source' is broken.
What I want to touch on is something that's been bothering me for the past few days, and solidified after seeing Bloomberg's piece–the fact that the log4j developers had this massive security issue dumped in their laps, with the expectation that they were supposed to fix it. How did that happen? How did a group of smart, hard-working people get roped into a thankless, high-pressure situation with absolutely no upside for themselves?
Rough idea of the timeline:
At 2:51 p.m. on Nov. 24, members of an open-source software project received an alarming email.... “I want to report a security bug,” wrote Chen Zhaojun, an employee on Alibaba Group Holding Ltd.’s cloud-security team, adding “the vulnerability has a major impact.”
This triggered a frantic rush in the log4j team to fix the issue:
He described the conversations among the Log4j group as dispassionate and earnest. “I know these people -- they all have families and things they have to do. But they put everything aside and just sat down for the whole weekend and worked on that,” he said.
They worked liked a professional team of security-focused engineers, except with zero pay or recognition. They gave up a weekend with their families to do this for almost the entire internet. They were criticized and harassed, and their every action scrutinized:
And after all that, they delivered a fix that people have been swiftly upgrading to and breathing a sigh of relief.
Now again I ask, how did these unpaid open source maintainers get roped into this high-pressure, exploitative situation? How is it that the entire internet treated them like their personal paid security engineering department, only with no pay and no thanks? What went wrong that they got from the explicit license terms:
7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
...to slaving away to fix it for everyone like their very lives depended on it?
The great open source brainwash
Let's look at the Open Source Institute's Open Source Definition:
Open source doesn't just mean access to the source code. The distribution terms of open-source software must comply with the following criteria:...(bunch of criteria that the license must satisfy to be considered open source)
Let's read Rich Hickey's (creator of Clojure) small note about being an open source maintainers after people in the Clojure community wrote posts like 'Fuck Clojure':
Open source is a licensing and delivery mechanism, period. It means you get the source for software and the right to use and modify it. All social impositions associated with it, including the idea of 'community-driven-development' are part of a recently-invented mythology with little basis in how things actually work, a mythology that embodies, cult-like, both a lack of support for diversity in the ways things can work and a pervasive sense of communal entitlement.
It is this communal mythology I want to talk about, this great open source brainwashing that makes maintainers feel like they need to go above and beyond publishing source code under an open source license–that they need to manage and grow a community, accept contributions, fix issues, follow vulnerability disclosure best practices, and many other things.
The Alibaba software engineer who reported the issue to the log4j team subsequently (on Dec. 8th) followed up with:
“Some WeChat security chat groups are already discussing the details of the vulnerability, and some security researchers already have the vulnerability,” Chen wrote. “We promise to keep it secret until your official release version comes out. Please hurry up.”
At this point everyone is fully bought in to the idea that the log4j team must urgently fix this because everyone is relying on them. There's not even a question about doing otherwise. Why? Again–because we have this pervasive open source mythology that open source is about open community, governance, security, and all those nice-sounding ideas. I've had discussions where people have told me with a straight face:
Well I'm pretty sure there is an entire industry of open source developers that would stand behind me in saying that open source is about more than a license, but feel free to stick your fingers in your ears and ignore all of us if that makes you happy.
They are incredibly bought into this mythology, and you can't argue against it. It's like a religion.
The wake-up call
In reality what is happening, is that open source maintainers are effectively unpaid outsourcing teams for giant corporations. The Alibaba engineer told the log4j team: 'Please hurry up'. Meanwhile, let's remember that Alibaba has a market cap of $348 billion (that's USD).
So what's the answer here? Seems like a lot of people are saying that corporations should fund open source. Others are pointing out that it's not that simple, because apparently corporations don't want to do that (for some reason). Yet they are totally fine with 'embracing open source' and continuing to use the software and pressure the maintainers. There must be a middle ground.
The suggestion
Here's my suggestion–let's imagine what happens if the scenario plays out like this: the Alibaba engineer sends the warning to the log4j team, and then nothing happens. The log4j team does nothing about it, because they have lives and families, because they're busy, they're not feeling productive, whatever. Alibaba follows up and requests the fix urgently. The team then sends a quote for the fix (to be made open source, of course), for $50,000. That's peanuts to Alibaba. Their market cap alone is nearly 7 million times that quote. They stand to lose untold amounts of money with a vulnerability like Log4Shell.
I invite you to think on how this would play out. Let's break the brainwashing and reveal the mythology as a hoax and false idol. Open source does not equate free vulnerability fixes, security best practices, open community, or any of the other nice-sounding layers that people add on top of it. Maintainers' lives don't depend on doing free labour for megacorps. It's time corporations (and everybody else) took a step back, dug their claws out of the backs of maintainers, and accepted the risks and responsibilities (and yes, expenses) that come with using open source.
Top comments (26)
Here's another suggestion:
What if the Alibaba engineers fixed it and sent the patch to the log4j engineers?
I've submitted patches to open source libraries that my company depended on, and did so during working hours, because it was critical to the work.
Why didn't those paid Alibaba engineers fix it themselves?
Yes, very good point. But also, what if the log4j maintainers charged money to merge the Alibaba patch? After all, properly reviewing a patch, ensuring it fixes the issue, and doesn't introduce a worse one, then publishing the fixed version, and all the work that entails–this is still hard work. Nothing in an OSS license prevents charging for it.
Absolutely. With something this critical, if Alibaba discovered, they definitely should be putting in whatever resources is necessary to getting it fixed quickly.
There have been a lot of people coming out of the woodwork to make hyperbolic claims about the open source philosophy (like the ones you link to) and for the most part they've all been really bad takes that showed a distinct lack of understanding what open source actually is.
However, you basically nailed it with the quote from Rich Hickey, though I would quibble that open source is a philosophy about everyone having the freedom to use and modify source code, and the licenses are just tools to achieve and enforce that freedom.
Otherwise, you and Rich are dead right; open source doesn't give anything beyond free as in freedom and a loose implication of free as in beer, though the latter isn't absolute or all encompassing.
All that said, I would like to address the part about log4j:
Firstly, nothing was dumped in the log4j dev's lap. It's their project, their (frankly stupid) feature, and their vulnerability, so it was already in their lap. Why were they expected to fix it? Because it's their software and it's their boneheaded mistake that caused all this.
In most other cases I would completely agree with you, but in this case it's a feature that never should have been in the library in the first place, being in there it never should have been enabled by default, and being in there and enabled by default it should have been fixed a long time ago.
The log4j devs crapped on the floor, so yes, if they want to keep credibility with their peers, the community, and the industry as a whole, then they need to clean it up. Though, I agree with you, had it been me I would have asked Alibaba for a bunch of money or, barring that, at least asked them to submit a pull request with the required fixes.
As I've said plenty of times around here and other places, the open source community is not here to serve corporate interests. The exact opposite, in fact, we are here to take away as much power as possible from corporations and put it into the hands of everyone.
I agree with almost everything you are saying. But:
To me this attitude is part of the problem. It's part of the insidious culture we have in our industry of 'you must prove yourself good enough to be recognized as one of us' and oddly enough that social proof is done by having them do unpaid labour for others. What happened to the credibility they've had all these years by having their software be used by mega-corporations? It's quite funny how it's now gone with the snap of a finger and they are back to being perceived as n00bs, because of one stupid mistake that all the omniscient godlike engineers criticizing it now, never once brought up since the Year of our Lord 2013 when it was merged.
This is part of the problem, the mindset that leads to the brainwashing, that 'we have to fix this, it's on us'. This is exactly what I'm arguing against here.
I understand what you're saying, but I don't see it from that perspective and wasn't intending to portray my criticisms in that way.
How I see it, they have proven that they are "one of us". They made a functional library that is useful for many other developers, they've given to the open source community, they have credibility and they are by no means "n00bs", and, like all of us, they've made mistakes; they're developers, they are one of us and they qualify as much as any of us to pass any gate kept by coders. And because they are one of us, they have to fix their mistakes.
They don't have to fix it because other people rely on the software; they don't have to fix it because other people want them to; they don't have to fix it because of any misperceived notion of obligation to the open source community; and they definitely don't need to fix it because some suits treat their library as part of a revenue stream.
They have to fix this mistake because it isn't some bug that anyone could have overlooked (even if a lot of people did overlook it for too long), it is a fundamentally incorrect design decision that any self respecting developer would want to fix. They have to fix this mistake because not doing so would be like a chef not caring that the food they made you was burnt. They have to fix this mistake because they are professionals with integrity and cleaning up one's own messes is what a responsible, caring person does.
They don't have to fix it to prove they are competent developers, they have to fix because they are competent developers.
It's easy to say in hindsight that this is a 'fundamentally incorrect design', now that we know all this. My question is, where were all these brilliant insights a month, a year, five years ago, while the library was in production use throughout the Java ecosystem? Where were the critics who are crawling out of the woodwork now? Sorry, but imho, y'all don't get to show up eight years after the fact and criticize and dictate and try to pull mind games like this appeal to the sense of pride thing that you're trying to do. ('Where's your pride as a developer? Don't you want to provide this free patch for a library that you've been maintaining for years, to us who are profiting off it, on our urgent timeline, working through the weekend? Don't you feel any shame?')
And if they didn't want to, they're not competent developers? Slightly different words, same gatekeeper-y meaning.
Look, my argument is not that bugs/design flaws should not be fixed. It's that they should not be fixed on the high-pressure timelines of corporations which are benefiting off their unpaid labour. I say chew on it for one, two, three months. Don't let the urgency of others become your own. Don't let them dictate that you spend evenings and weekends away from your family. They'll be criticized no matter what they do–your comments prove that. So might as well preserve their mental health while they go through that. Or even let others contribute and try to fix this supposedly urgent bug!
That it is and I'm not claiming to speak with anything other than hindsight. But that doesn't change the situation. With current modern knowledge and eyes, there is no denying that a logging library executing code from strings is a major violation of basic design principles.
You're not wrong. Present this scenario in any design class and the conclusion will be that a ridiculous number of eyes saw this problem and did nothing about it.
But we do get to do that. This is the topic of the day, it's both important to the industry and useful as a learning experience; developers are going to comment on it. Que Sera, Sera.
You keep bringing up gatekeeping. Well, telling developers when and where they can have an opinion on the work of their peers sounds to me like the bad kind of gatekeeping.
I've dictated nothing. I didn't even imply that they had to follow anyone's time table, urgent or otherwise. I also didn't say anyone should feel shame about anything.
And it isn't a mind game. Maybe it doesn't apply to you, but a lot of people from all professions take pride in their work, the log4j devs among them it seems and rightly so.
No, that's not what I was saying. The implication is that, as competent developers that take pride in their work, they would want to fix the problem.
And judging people on their actions isn't gatekeeping. It's how one is supposed to judge other people.
On that we completely agree. Open source is volunteer work and no one can tell you what you have to do or when you have to do it. If one doesn't like how a dev runs their project, then they can always fork the repo and do the work themselves. That's the beauty of open source.
Pointing out a mistake isn't criticism in the negative sense of the word. No one I've seen is saying they are bad people or should be punished because they made a bad design decision, but them not being bad people also doesn't make the design decision any less bad.
doesn't mean they have to fix it over the weekend - if it is that urgent to someone who has spotted the issue then they can work over the weekend and submit a patch for the issue.
See, that's exactly what I mean by 'mind game', this 'maybe it doesn't apply to you, but other people take pride in their work'. This is the kind of snide commentary, the mentality that 'if you don't agree with me then you must not be good at your job'. This is exactly gatekeeping.
It's also supremely unhelpful. Nothing constructive about it. You think this needs to be pointed out to the log4j maintainers? You think they aren't intensely aware of the problem? Which part of 'Yet nothing is stopping people to bash us, for work we aren't paid for, for a feature we all dislike yet needed to keep due to backward compatibility concerns.', makes it seem like they need to be told that this feature was a mistake?
But that's exactly it. It's not your place to judge unpaid volunteer OSS maintainers. If you actually want to help, then step up with donations or contributions. The peanut gallery can sit back down!
That's the most made-up definition of 'gatekeeping' I've ever heard of, smh.
If I want to say someone isn't good at their job, then I'd say it directly. You are reading maliciousness in my comments where it doesn't exist. I've been treating your comments with good faith, but if you aren't going to do the same to mine, then perhaps this conversation has run its course.
Unless you're on the log4j dev team, then I've never even spoken to them, I haven't pointed anything out to them, and I definitely haven't been bashing them.
Talking about mistakes that devs have made is extremely useful and helpful to future devs to avoid making the same mistakes.
It's not your place to tell me or anyone else who we can or can't judge.
Thanks for the interesting discussion, but since you seem to have taken this personally and are now turning to accusations and assumptions of bad faith, I think it best that we just agree to disagree.
Best of luck and have a nice day.
Well, you came in highly praising Rich Hickey's post about open source and then proceeded to take away the exact opposite of what he said. Best of luck to you too.
For those nodding agreement, here, I'd like to present an alternative idea about how to develop software, more in keeping with contemporary practice.
This practice is called "no blame". Often, an error is the end result of a long chain of events, each of which contributes to the error, and blame usually lands on just the last person in the chain. Placing blame in this situation is helpful to no one. No one understands what led to the mistake, and more importantly, no one really knows how to prevent that same mistake in the future.
Instead, assume that the error derives from the engineer making the best decision possible in a flawed context, and fixing the error involves fixing the context. Perhaps the context is the workflow, which is a team consensus.
In this case, the "boneheaded mistake" and "frankly stupid feature", if I understand correctly, came from at least 2 features that interacted in an unexpected way. These features on their own (again, IIUC) are not unreasonable.
The fix here obviously is not to call people stupid and boneheaded (which, frankly, I find reprehensible, tbh). Have some compassion and empathy, first and foremost. Next, give the team support to find out where the mistake happened, and give them space to find a long-term fix for, not only "fixing the boneheaded mistake", but to fix the flaw in the process that led to the entire class of mistakes.
As an example, if the mistake were "2 features interact in unexpected ways", perhaps one answer might be to turn off each feature by default. I don't know.
But calling people stupid and boneheaded ain't it, chief.
I didn't call any person or people "stupid" or "boneheaded", so right off the bat you can stop those accusations.
Yep, it's a great practice that I use whenever I lead a team, in both tech and non-tech settings. However, I've never blamed any individual person and have been speaking of the project team as a whole, so I'm not sure what your point is.
I disagree on both counts. While I haven't looked super deep into it, the log4j team seems to be pretty aware of what contexts, decisions, and actions led to the mistake. And, more importantly, it should be obvious to any professional programmer how to prevent the same mistake being made in the future: enforce separation of concerns and don't make obviously dangerous features opt-out.
I don't think you understand correctly. The library has a feature that executes arbitrary strings. That is an unreasonable thing for a logging library to do. It is doubly unreasonable that such a feature is on by default.
This isn't a case of random features interacting in some unforeseeable way, this is a case of a bad feature that is obviously bad on its face. Sure, there are reasons it is in the library and as a dev I can understand why they put it in the library, but that doesn't change the fact that it was a mistake and shouldn't have happened.
That is great advice for fixing this specific issue, but it misses that this is also a great object lesson for the entire dev community for why design principles exist and how they can help avoid problems like this in the future.
log4j is not broken
Stop blaming log4j
Yes, you read it right. It works as documented. It is not a bug, it is a feature, really.
What you can blame on log4j is feature creep, that made log4j handling tasks that go beyond the duty of a logging library.
A logging library should do one thing: take a string, together with a "severity" and save it somewhere. The string should be an opaque, binary blob and no processing should be done by the library. Stop.
Who actually is to blame for the problem is who used this library
Conclusion? Should I assign blame in this matter I would do as follows
That's an interesting comment. As I explain in my post Log4j vulnerability: What the FAQ happened?, the feature was there and intentional (so yes, not broken in that sense), but was never properly documented.
And regarding sanitation/sanitization, OWASP is one of the best authorities on best practices, and they specifically did not recommend sanitization for log4j 2. You're cordially invited to provide a citation for Good Practices 101 that advises otherwise.
Yes very good point. Not really related to the point I am trying to make in my post, which is that people are expecting log4j maintainers to automatically jump into action during nights and weekends to save their hides–for free.
$50,000 for a world-wide impact? $10,000,000 to fix all the world's servers seems more like it. That's a ransom, of course, but there has to be a middle road.
I know, right? I think I was being extremely conservative.
Open source means you can make modifications. Can't code - help with debugging. Can't debug - write documentation. I would understand if said Alibaba engineer would urge to REVIEW and MERGE his urgent fix. But no, someone else must find it, fix it and test it. Open source is not broken, some people are.
Yeah the thing is, if there had been no patch, we would all still be able to do the mitigation of deleting the
JndiLookup.classfile from production JARs to stop this attack. So what did all this pressure on the maintainers achieve? A bunch of people upgrading, and many complaining. 🤷♂️I've done open source for the last 30-odd years. I'm happy to fix your bugs in code I've written for free. But on my timeline. If you want an expedited timeline, then that's done "one the clock" for a fee. And if you'd like me to spend time away from my family during the holidays, the fee can be quite large. Granted, it took 10-15 years of that to arrive at this point. In this situation, I'd have requested a patch from Alibaba since that would expedite the timeline of a fix. Especially since the language used sounds like a veiled threat (people already know about this, so we'll try to keep a lid on this, but because of our sloppy disclosure policies we've put you in a pickle. Please fix it soon). Not sure of a good way around this, though...
Dev.to should pin this article at the top of their listing, well at the least it should be in their "posts of the week" newsletter - you've put this so eloquently and powerfully ... this is a wake-up call if there ever was one ... legendary stuff!
Is it so impossible to believe that a company the size of AliBaba didn't have paid staff that could sit down, open the source, and fix it themselves? They could have submitted their fix to the maintainers along with other fixes to see if theirs be became a permanent patch.
This is what you get when you bleat "code wants to be freeeeeeeeeeeeeeeeee!" Release stuff open source? Don't expect payment, acknowledgement, or anything in return.
Sorry but you've completely misunderstood Free Software/open source.