Law of leaky abstraction and on learning the internal a.k.a the hard way

Update #1 2021 - Someone pointed me to this article about cargo cult programming is killing (Sri Lankan) software industry. I think it's quite related to what I wrote here so you might want to read that as well.

I bumped into a tweet sometimes ago, which attracted me to response. The tweet read:-

Since I’ve developed Tinkerwell I wonder why most of us developers try to enforce CLI tools to other (especially new) developers.
We say things like:
“You aren’t a proper developer if you don’t use git in your terminal!”

“You should use vim instead of PHPStorm”

That’s nonsense

To which I responded as:

using to learn basic concept and how thing works underneath vs using it to gain efficiency is 2 diff things. No one argue for the later. By all means, use what make u the most efficient. But being stuck with the tools bcoz u don't know beyond the abstraction,that's not effective.

Few days ago I bumped into this video on devops and the person being interviewed was talking about calculator analogy on the topic of learning fundamentals first, which I tweeted as well under the same thread:-

it's like calculator. You learn to add up numbers using pen and pencil first. Once you master that, you start using calculator.

Today I was reading article Everyone is not Ops and it mentioned a quote from old Joel's article (which I have read before but simply forgetting about it) on leaky abstraction. The quote reminded back to above discussion.

… all abstractions, leak, and the only way to deal with the leaks competently is to learn about how the abstractions work and what they are abstracting. So the abstractions save us time working, but they don’t save us time learning.

More interesting related quote from the article:-

The law of leaky abstractions means that whenever somebody comes up with a wizzy new code-generation tool that is supposed to make us all ever-so-efficient, you hear a lot of people saying “learn how to do it manually first, then use the wizzy tool to save time.” Code generation tools which pretend to abstract out something, like all abstractions, leak, and the only way to deal with the leaks competently is to learn about how the abstractions work and what they are abstracting. So the abstractions save us time working, but they don’t save us time learning.

And all this means that paradoxically, even as we have higher and higher level programming tools with better and better abstractions, becoming a proficient programmer is getting harder and harder.

I have a discussion with my boss in the past on the topic of learning and skills acquisition and whether bottom-up or top-down approach is more suited.

At that time, I was training our new hire and as part of the syllabus I asked them to write something similar to Django from scratch. So they started building python web app from a simple wsgi and slowly expanding the application to have request dispatcher, views/controller etc, until it will end up similar to Django. This allow them to understand each Django component better. This is basically a bottom-up approach.

For top-down approach, you basically would start with the tutorial trying to build new Django application, and slowly exploring other components such as models, forms, views etc. What matters here is you know how to use all the components. How they're built and work under the hood is not important (yet). Proponent of top-down approach say this is faster and more practical. People can start building application right away, which would lead to better productivity, instead of meddling with low level little stuff that might not even needed in real application.

I'm a fan of bottom-up approach because to me, understanding how things work at the low level allow us to have better insight how to structure our application at the higher level. We know exactly why things in certain place and know when to break from common practices, whenever that common practice seem not aligned with our business objectives. But in the end, we take a middle ground that there must be a balance between knowing low level details and being up to speed in knowing how to use existing things in order to be productive. Shipping the app out as fast as we can so we can sell it to customers so that we can pay our bill is always top priority over everything else.

Fast forward to present time, where we right now fighting our way into the uncharted land of VOIP and SIP, I tend to think again about the old debate. Trying to master these tech, I think it's the best example where the bottom-up approach shine up. Top-down approach is great when the thing you're trying to learn is mature already, like Django. You can depend on the documentation and countless other best practices, to the point that what happen under the hood are not relevant. It's like driving a car. Car technology has enjoyed more than 100 years of refinement. Complex and sophisticated physic that happened in the engine manage to be abstracted out into gear lever that only need to be moved forward and backward, and acceleration and deceleration is just a pedal press on your foot. You don't even need a manual (docs) to drive a car.

But for voip and SIP, building application on top of it is far from being matured. Documentation is just mere reflection of one experiences, after going through all the pains. Anecdotal at best, plain wrong at worst. If you just rely on how to do this and that, you'll face a dead end when some obscure bug happened. The only to master at this moment is to start from the bottom of it. At the socket level, the threads that handling each and one operations, the codecs, the signalling and media protocol etc. You can't just depend on simple example that show how to make or answer call. That won't lead you to anywhere.