What to do When Things Go Sideways (And How to Recover)

We would all like for product development projects and manufacturing operations to go smoothly and predictably, but in the world of complex healthcare technologies, sometimes “things go sideways.” What the specific scenario looks like depends on the precipitating event which could be:

• Failures in design verification or validation.
• Failures of a device in the field.
• Product recall.
• High rates of customer complaints.
• Excessive manufacturing rejects or low production yields.
• Operational or process improvement initiatives.

Resolving situations like these can be a real challenge. Quality systems will provide procedures for corrective and preventative action (CAPA), but they will not provide a roadmap to getting to the bottom of the actual problem so that reliable solutions can be developed. Below we share some best practices for recovering when things go sideways.

Bring in a Tiger Team

Often, groups are hesitant to bring in an independent perspective in the midst of an on-going crisis. But getting outside help can be the most effective way to get back on track. A “tiger team” approach leverages an independent team of experts to question assumptions, validate (or refute) previous findings, and identify new paths to investigate. The key is to quickly get to the root of the problem so that it becomes clear what needs to be done to recover.

Albert Einstein is reported to have put it this way: “If I had an hour to solve a problem, I’d spend 55 minutes thinking about the problem and 5 minutes thinking about solutions.” Once the problem is known, the solution is more easily revealed. Too often, people jump to unsubstantiated conclusions in search of quick fixes. This usually leads to a prolonged period of trial and error iterations; rarely is it a trial and solution. An independent team can help avoid this.

Coming into a problem situation is a little like looking at a crime scene. Things that a tiger team might do to begin their investigation include:

• Observe the environment involved in the situation. Are there aspects of the environment that might contribute to the problem?
• Meet with the people involved. What are their skill sets, perspectives, and biases? Are there any gaps with regard to working with the relevant technologies?
• Construct a timeline of events leading up to problems that have occurred. There are often subtleties in the sequence of events that can cause unexpected interactions and issues.
• Examine the evidence by collecting relevant data.

Trust but Verify

Often information is collected from a wide variety of sources – engineering design data, test data, production data, supplier information, complaints, etc. Another important source of information is from interviews. Interviews can be very effective right after a critical event (for firsthand knowledge) and later in the investigation process to confirm facts or hypotheses. It is important, however, to keep the idea of “trust but verify” in mind:

• Validate responses to assure yourself of facts (separating fact versus fiction).
• Understand the position/role of interviewee considering the potential for bias.
• Review information obtained and compare it to other information developed.
• Follow up to resolve apparent inconsistencies or “holes,” if necessary.

Use the Tools to Analyze Data

There are many useful tools that can be employed to analyze and evaluate the data and help with the problem investigation process. Many of these are embodied in well-known processes like Six-Sigma.

• The “five-whys” technique involves a series of questions that are posed to determine the cause and effect of situations that ultimately lead to the problem. The goal is to dig deep to find the underlying reason as well as the sequence of events. With such insight, solutions become apparent.
• A fishbone or Ishikawa diagram which is a visual way to depict issues that potentially could lead to the problem. Typically, branches of the diagram cover several main categories of causal factors which are each broken down into further details:
• Failure Modes and Effects Analysis (FMEA) or fault tree analysis to systematically identify potential causes of failure in a process or system.
• Engineering analysis to determine safety factors and margin to failure in a design.
• Scenario evaluation or “what if” analysis to walk through different sequences of events and postulate consequences, considering the potential cascading effect (often a series of individually unlikely events can ultimately lead to real trouble).
• Testing of known good and faulty devices to characterize behaviors.

Good engineering analysis is also critical to understand why a problem occurred and to gain the insights needed to identify changes that will be effective in preventing further occurrences. An example is the investigation into the 1986 disaster with the Space Shuttle Challenger. Extensive engineering analysis of an o-ring seal in the solid rocket boosters revealed that its effectiveness was highly sensitive to the o-ring’s elasticity. And further analysis of the o-ring material showed that when exposed to the cold temperatures that were experienced on the launch pad, it would have hardened and therefore not been able to function as needed.

Getting to the Answer

Ultimately, getting back on track requires fact-based decision making – using evidence to understand the likely root cause of the observed problem. It is a good practice to tabulate the potential causes along with the data that supports the potential cause and any data that would refute it. The key is to ensure that the supporting and refuting evidence is factual and not opinion.

Once the suspected root cause or causes are identified on the basis of all the data, the next step is to plan and execute a series of tests to confirm the conclusion. For example, if a component failure on a circuit board is suspected of causing the problem, a known good board can be modified to simulate the component failure and then tested to determine if it replicates the behavior of known faulty devices.

Finally, as noted above, once the problem is well understood, the solution usually becomes clear. Implementing the solution deserves thoughtful planning though. Of particular note is the situation where the problem is multi-faceted and the solution therefore involves making multiple changes. In this scenario, it is wise (if possible) to implement each change incrementally so that each one can be tested and proven to provide positive results. The alternative, making many changes all at once, can lead to serious difficulty if the end result is not as expected.