Your description of “engineering done right”, where engineers understand the whole of the problem in advance and evaluate multiple competing proofs-of-concept before starting the “real“ work, is the myth that brought us waterfall.
As an aside, in the early 2000s I worked on such a project: an attempt to replace a financial institution's ageing mainframe system with something that would allow the company to branch out into new products. Use cases were defined, specifications were specified, technologies were trialled and chosen for all the right reasons, and eventually engineers started engineering.
Two years and a few million dollars in, the project owners made the inevitable discovery that they were building the wrong thing with the wrong technology. The only part of the system that was delivering value? The quick-and-dirty interim hack for scraping the old mainframe’s green-screen UI and turning it into web forms.
So it turns out, sometimes the solution is writing a front-end framework in COBOL.
A major theme of evolution in software engineering over the last twenty years has been accepting that we don't understand the problem in advance (and in many circumstances can't understand the problem in advance); reimagining development as a process of iterating towards a better understanding of what we're trying to build, while we're building it.
The only thing more certain than us not really understanding the problem we're solving today, is that whatever problem we do solve today, we'll be solving a completely different one tomorrow. In this world, the idea of picking the “best tool for the job“ falls apart because we don't really know what the job is.
PHP looked like the right tool for Facebook up until it wasn't, and they had to write an entirely new language and runtime to replace it. Rails looked like the right tool for Twitter until it wasn't, and they had to sink years of development and lost feature momentum into replacing it.
I'd go as far as saying (in an idea parallel to Tom DeMarco's 2009 essay, Software Engineering: An Idea Whose Time Has Come and Gone?) that the more we can with any certainty decide in advance what the best tool for the job is for any given project, the less important that project is, and therefore the more likely the right choice for that project is whatever tool we know already.
So for important projects, the ones where we have to make decisions about how to solve problems we don't even know we have yet, it is our job as engineers to have an informed opinion about which technology and tools are just better for their general problem domain. The tools that will build upon the successes, and help us avoid the failures of the teams that came before us.
Certainly there are good and bad reasons to prefer one tool or another, and falling in love with some tool so much that you can not see its flaws is a too-common disease, but saying that having any preference at all is “bad engineering“ is an abdication of our responsibility to improve our craft.
Hey Charles, thansk for your awesome reply, some good points there.
You're right- what we're building is increasingly unknown in with agile software dev, and with startups and technology innovations in general. And your bank example is a classic story we've all heard too many times. However, I'd encourage readers not to just apply the above methodology not to whole projects, but any subtask, no matter how small. We do it all the time, Integer or Float here, Enum or String?, Right up to class, to package, to framework, to OS etc. I'd even argue (over a beer, my fingers only have so many key presses in them haha) that Agile looks like mini waterfall sometimes, and that's not necessarily a bad thing, agile is adaptable to the problem and the people.
I definitely think that my assertion of any preference is perhaps too strong of language, so I'll add some subtlety, I think it's strongly advised to proceed with caution when considering your preferences. Are they engineering reasons or other reasons. Engineering reasons are the best way of course :)
Thanks again for your time!
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Your description of “engineering done right”, where engineers understand the whole of the problem in advance and evaluate multiple competing proofs-of-concept before starting the “real“ work, is the myth that brought us waterfall.
A major theme of evolution in software engineering over the last twenty years has been accepting that we don't understand the problem in advance (and in many circumstances can't understand the problem in advance); reimagining development as a process of iterating towards a better understanding of what we're trying to build, while we're building it.
The only thing more certain than us not really understanding the problem we're solving today, is that whatever problem we do solve today, we'll be solving a completely different one tomorrow. In this world, the idea of picking the “best tool for the job“ falls apart because we don't really know what the job is.
PHP looked like the right tool for Facebook up until it wasn't, and they had to write an entirely new language and runtime to replace it. Rails looked like the right tool for Twitter until it wasn't, and they had to sink years of development and lost feature momentum into replacing it.
I'd go as far as saying (in an idea parallel to Tom DeMarco's 2009 essay, Software Engineering: An Idea Whose Time Has Come and Gone?) that the more we can with any certainty decide in advance what the best tool for the job is for any given project, the less important that project is, and therefore the more likely the right choice for that project is whatever tool we know already.
So for important projects, the ones where we have to make decisions about how to solve problems we don't even know we have yet, it is our job as engineers to have an informed opinion about which technology and tools are just better for their general problem domain. The tools that will build upon the successes, and help us avoid the failures of the teams that came before us.
Certainly there are good and bad reasons to prefer one tool or another, and falling in love with some tool so much that you can not see its flaws is a too-common disease, but saying that having any preference at all is “bad engineering“ is an abdication of our responsibility to improve our craft.
Hey Charles, thansk for your awesome reply, some good points there.
You're right- what we're building is increasingly unknown in with agile software dev, and with startups and technology innovations in general. And your bank example is a classic story we've all heard too many times. However, I'd encourage readers not to just apply the above methodology not to whole projects, but any subtask, no matter how small. We do it all the time, Integer or Float here, Enum or String?, Right up to class, to package, to framework, to OS etc. I'd even argue (over a beer, my fingers only have so many key presses in them haha) that Agile looks like mini waterfall sometimes, and that's not necessarily a bad thing, agile is adaptable to the problem and the people.
I definitely think that my assertion of any preference is perhaps too strong of language, so I'll add some subtlety, I think it's strongly advised to proceed with caution when considering your preferences. Are they engineering reasons or other reasons. Engineering reasons are the best way of course :)
Thanks again for your time!