DEV Community

Discussion on: How important is stats knowledge in software development?

lepinekong profile image
lepinekong • Edited on

MVC for statistician: M=Variable under Observation, V=Projection of M on several Viewpoints, Controller=holds the transformation rules

Apart from concepts, I won't use so much stats because of the huge Heteroscedasticity (~unpredictable variance) in this field ;)

Nevertheless, as I used to be a specialist of Deming/Shewhart Statistical Process Control (see my review of one of my favorite books, it is less the numbers by themselves that matter than the spirit: stats are not an end by themselves, what matters is to improve the capability of the system and so Deming did always advocate Visual Tools for visualizing flow to improve it and so reduce that dreaded Heteroscedasticity.

But Heteroscedasticity is not so awfull depending on the context: if you're working on an innovative project or if you're learning, it is expected and wanted otherwise it means you don't advance by leap frogging, but if you're doing a usual one and it's so chaotic, then you should worry and look for the root causes ;)

In any cases, even in the learning and/or innovative context, it's all about finding the causes and effects. For artist like Leonardo Da Vinci, that's how you can perfect a great art. He said many people do things too quickly without trying to discover the fundamental laws behind, that's the reason he himself studied Science to understand the science of water or light to be able to paint them magnificently. The research for causes and effects, Deming calls this SOPK (System of Profound Knowledge) and stats methods like SPC are just a mean to SOPK not the other way round, not for just making stats about how many defects you have detected in testing. As a matter of fact he hates this "after-the-fact" mentality that is so widespread in traditional industry, he would be horrified I guess also by software industry. Scrum is based on PDCA according to Jeff Sutherland the inventor of Scrum himself and PDCA (Plan Do CHECK Act) is supposedly from Deming, he ignores Deming hated PDCA spreaded by Toyota (he is the one who taught Toyota as first Toyota CEO says it in a tribute on their website) so much he created an other acronym PDSA (Plan Do STUDY Act) once again to assert it's about scientific process not about checking numbers. Other important concept introduced by Deming to the Japanese was Visual thinking, like for visualizing the flow, that's where Kanban originated from, it could be something visual that is different than just a board if it helps you to DESIGN and to MODEL ;)

So in software development, the mentality should be the same: to perfect one's craft, one should also look for superior principles instead of just stats or even ...Design Patterns (though they can be very usefull for OOP that's not the point). They should not be the ultimate causes for good design, according to Alan Kay, they may somehow be holding back the software industry so one should reflect upon all our current (not so good though many think they are) software practices like fascination for pure stats (measurements without acting upon them) and other habits and hypes without understanding what's really behind: are they really so good or are there something better/simpler ;)