I look back to my early days of Six Sigma training with great fondness… Descriptive statistics, central limit theorem, normality, statistical process control, graphical analysis, capability studies, regression, hypothesis testing… a veritable cranial smörgåsbord. Oh, yes… let’s not forget the catapults and the factorial design of experiments! If my memory serves me, the engineers were eating this up while a large number of us were experiencing what might best be described as “chronic” indigestion.
One certainly has to give credit to our brave, pioneering instructors and coaches including Forrest Breyfogle III and Mikel Harry. They certainly had their work cut out for them… Six Sigma still relatively new and materials for the various industry sectors and client companies often being delivered “just in time” with pages still warm from the presses. Thankfully, we (and they) survived and many, including yours truly, even went on to make a “career” out of Lean Six Sigma. We were “bitten by the bug” in a big way!
The (Lean Six Sigma) light bulbs go on a little less often for me these days. Although I realize that the “dots” still need to occasionally connect, I am thankful they are less isolated and fragmented than they were twenty years ago. “If I only knew then what I know now…”
I do wonder which important concepts I would have appreciated “getting” sooner than later in those far off and far out days in the classroom. One that comes to mind is that of the difference between “statistical significance” and “practical significance”. I am one of those students who became so focused on the p-values, “If the p is low, the Ho must go!”, that I lost sight of the forest for the trees. I am sure the instructors, including the other “Forrest”, addressed this more than adequately but I somehow “missed it”. It is this early experience that causes me to “hammer” on this very concept every time I am in front of a class.
The headline associated with the horserace depicted in the image at the top of this article reads, “Australia’s most famous horse race witnessed a dramatic photo finish as 40-1 chance Viewed stayed a nose ahead of Bauer to become Bart Cummings’ 12th Melbourne Cup winner”. This is a case where an extremely small difference was of practical significance… especially if you were lucky enough to bet on that 40-1 longshot!
The key concept: Statistical significance itself doesn’t imply that your results have practical or meaningful consequence. A typical student of Six Sigma might learn later in training that high “power” values may allow us to detect small differences even when that difference might be trivial in light of the decision being taken. Note: The power of a study is the likelihood that it will distinguish an effect of a certain size from pure luck (chance).
One case I like to cite as an example of this is the study of the effect of Aspirin described in the Physicians’ Health Study where a randomized, double-blind, placebo-controlled trial was undertaken to determine whether low-dose aspirin decreases cardiovascular mortality. 0.94% (104/11037) of the people in the Aspirin group and 1.71% (189/11,034) of those in the control group suffered a heart attack during the study. In this example, the risk difference is 1.71% – 0.94% = 0.77%. “This may not appear to be large, but keep in mind that we are talking about heart attacks, which are serious.1” Another way to state this would be to say that the risk of having a heart attack is 1.8 times higher for those not taking Aspirin! Now that does sound like a lot! The treatment? One of them is to prescribe a daily low dosage of Aspirin to certain higher-risk patients.
“Fundamental Statistics for the Behavioral Sciences”, David Howell ©2011, 2014. ISBN-13: 978-1285076911
So, where does practicality fit into this story? The cost of an Aspirin today is approximately $0.05 per tablet, so the cost to the patient of this treatment would be $1.50 per month or $18.00 per year. In this case, a small but statistically significant difference is of practical import and appropriate action taken.
What if the cost of Aspirin were $50.00 per tablet and the resulting treatment cost $1,500.00 per month of $18,000.00 per year? One might assume that the prescribing of Aspiring (or other similar cost treatments) would be forgone… the difference no longer of practical significance when it comes to available treatments.
In quality improvement, we are encouraged to view the case for change from two perspectives: “The cost of doing something” and the “cost of doing nothing at all”. Perhaps this is another way of looking at statistical versus practical significance.
Yes, “the p is low and the Ho must go!” But… don’t lose sight of the forest for the trees. Practical significance is an integral part of statistical analysis and decision-making and the sooner Lean Six Sigma students understand that in training, the better off they will be in applying these tools and methods in the future.