Image source: https://www.pickpik.com/
I recently taught a session on Logistic Regression to a group of Lean Six Sigma Black Belts. It’s one of those seldom-used tools that many Black belt programs don’t cover and, perhaps, with good reason.
The case study of the space shuttle “Challenger”, NASA’s ill-fated 25th space shuttle flight, is a common reference when presenting logistic regression (both binary and ordinal) and a reminder of the fallibility of the human condition. How much did launch delays and political pressure influence the decision-making that day?
We have the luxury of viewing the Challenger case in hindsight. NASA provides a detailed, fraction by fraction of a second account of the harrowing, almost numbing experience for those who witnessed the event. I can’t imagine the horror and pain for family members who suffered that day on January 28, 1986. The clinical analysis provided in “Report of the Presidential Commission on the Space Shuttle Challenger Accident” doesn’t capture that very human element.
So, what was known beforehand that might have led to the postponement of that fated mission? Allan McDonald, the director of the Space Shuttle Solid Rocket Motor Project for the engineering contractor Morton Thiokol, was concerned that below-freezing temperatures might impact the integrity of the solid rockets’ O-rings.
While McDonald witnessed the Challenger breaking up over the Atlantic Ocean 73 seconds into its flight, he saw his worst fears come to fruition. McDonald had refused to sign the launch recommendation the night before over safety concerns he had for the mission. His concern was with the very O-Rings that were cited by the subsequent Commission of investigation: “… the Commission concluded that the cause of the Challenger accident was the failure of the pressure seal in the aft field joint of the right Solid Rocket Motor”!
In his book, “Truth, Lies and O-rings: Inside the Space Shuttle Challenger Disaster”, (2012, ISBN 13: 978-0-8130-3326-6), he recalls, “I was absolutely amazed that the NASA people I argued with against the launch didn’t even mention to the other members of the mission management team that there was a concern…”
McDonald’s refusal to sign that launch recommendation, and his subsequent testimony before the Presidential commission and to Congress, have made his story an example in engineering ethics classes around the country. Lending power to that story is the fact that McDonald was demoted and nearly fired by Morton Thiokol after he testified. (http://www.montana.edu/news/7173/engineer-who-warned-of-trouble-before-challenger-disaster-to-sign-books-today)
Of all the conclusions, the simplest can be taken directly from NASA’s site: “The explosion 73 seconds after liftoff claimed crew and vehicle. The cause of explosion was determined to be an o-ring failure in the right solid rocket booster. Cold weather was determined to be a contributing factor.”
Logistic Regression
Ordinal logistic regression, (aka “ordinal regression”), is used to predict an ordinal dependent variable given one or more independent variables. O-Ring “incidents” (number of field joints observed to have suffered erosion or blow by in post-flight inspection) and temperature (degrees Fahrenheit) are the variables of interest for this study. We have used SigmaXL™ as our software of choice for this analysis.
The pair of booster rockets provides six opportunities for failure. After each of the twenty-three preceding missions, each booster is carefully examined and issues document. The O-ring incidents are summarized as on the table to the right;
The resulting analysis (see image below) yields the following, indicating that temperature is a statistically significant factor with the regression model correctly predicting 82% of the incidents, (0, 1 or 2 occurrences).
Like all models, there is some unexplained error, but SigmaXL™ provides the following summary;
The Commission report cited, “Of 21 launches with ambient temperatures of 61 degrees Fahrenheit or greater, only four showed signs of O-ring thermal distress; i.e., erosion or blow-by and soot. Each of the launches below 61 degrees Fahrenheit resulted in one or more O-rings showing signs of thermal distress.” So, what was the temperature for the Challenger launch?
Documented in the same report, it is recorded that “The ambient temperature at time of launch was 36 degrees Fahrenheit, or 15 degrees lower than the next coldest previous launch… Temperature on the opposite side of the right Solid Rocket Booster facing the sun was estimated to be about 50 degrees Fahrenheit.
SigmaXL™ offers a “Response Outcome Probability” tool that we can use.
When we enter the average temperature observed over all of the previous missions, 70 degrees Fahrenheit, we obtain the following prediction… a 78% chance of no incidents and a 19% chance of 1 incident and a 3 % chance of two incidents.
Entering the temperature of 36 degrees, we see the following…
We observe that there is a 98% chance of two incidents. We can also graph the probabilities of 0, 1 and 2 incidents using simple scatter plots.
The chart on the far left demonstrates the effect on probability of 0 incidents is positively correlated with temperature. As temperature increases, the likelihood of 0 incidents increases. Conversely, as temperature decreases, the probability of 1 or 2 incidents increases as previously demonstrated using our “Response Outcome Probability” tool.
The “challenge”, (granted, a poor choice of words for this article), with our model is that there is no historical data for;
More than 2 incidents observed
No flight data for temperatures below 53 degrees Fahrenheit
At this point, l we can only “project” into that space and speculate as to the behaviour of that system. We will never know how many joints failed on that ill-fated mission as the evidence is limited to photographic analysis for the portion of the flight, leading up to the catastrophic explosion and loss of vehicle and crew.
McDonald’s concerns were shown to be valid and it is only in hindsight that this analysis begs the question, “Why?” Of all of the lessons learned, Allan McDonald remarked, “”I really want to influence, in a positive vein, engineers and impress on them that they take on a very professional ethical requirement when they receive their degrees and go out into the world,” he said. “They need to stand on their feet and defend their opinions and not be afraid to do that.”
There are many resources available on the Internet for those who are not familiar with the Challenger mission disaster. A thirty minute excerpt from a NASA documentary can be found at https://www.youtube.com/watch?v=oaQbfMQCHZM.
The crew of STS-51-L: Front row from left, Mike Smith, Dick Scobee, Ron McNair. Back row from left, Ellison Onizuka, Christa McAuliffe, Greg Jarvis, Judith Resnik. Image Credit: NASA