Gaps in Quality Audits: 5 Things Every Auditor Should Do

The Situation and the Problem

Quality audits routinely require an examination of the organization’s processes, standard operating procedures (SOPs), and systems that ensure it meets the requirements from regulators and applicable quality management system (QMS) standards. These include the procedures and actions an organization needs to take with regards to data and its analytics. Use of statistical methods is also indicated with qualifying phrases like “as appropriate.”

Since many crucial decisions about a product or service are based on data, errors and omissions in this area have a significant impact on the users and customers. This article illuminates the need for more rigor on auditing of data, its analytics, and application of statistical methods. The need is universal across industry or service sectors, and regardless of the type of audit – certification, surveillance, re-certification, supplier audits, etc.

Much remains to be done.

A Case Study on Customer Impact: Recall of Medical Device Instruments

A well-established and reputed medical device firm had to recall surgical instruments due to out of spec hardness. Their testing lab indicated certain lots of instruments had higher hardness and these were already in the field. It was discovered as part of random checks during quality compliance audits the company conducted periodically.

This created a risk of instrument breakage during use and the potential for ensuing complications of retrieving the broken pieces. The instruments were procured from various suppliers on their approved supplier list.

An examination of the supplier-provided certificates of conformance (CoCs) from the suspect batches indicated the hardness values were within specs. Now the question was if this was a false alarm that triggered an unnecessary recall. Or was the recall justified, and did the firm really avoid a potentially reputation-damaging outcome?

The leadership rightfully wanted to know the truth, what remediation would be needed, and how such errors could be avoided in future. Was there an issue with manufacturing data and analytics? Was it due to manufacturing data analytics software?

The Investigation and Quality Audit Results

A cross-functional team undertook the investigation by examining internal processes for evaluating the hardness of the instruments and recording the same. Some members of the team visited the related suppliers to study their practices for manufacturing, testing, and documenting results. Below is a summary of what they found:

1. The methods of hardness measurement were not identical between the suppliers, nor between suppliers and the organization.
2. CoCs claimed to report correct values, however, measurement errors were systemic and endemic across the board, creating doubts surrounding the trustworthiness of their data.
3. All instruments used (by the suppliers, and the organization) were calibrated and in good working order, based on available records.
4. Internal and supplier audits had been conducted per schedule earlier, and no major nonconformances were found. The few minor ones related to good documentation practice (GDP), and to training records only.

The results befuddled many in the leadership team. If measuring instruments were in decent shape, why did we have systemic and endemic errors? And if such errors were present, how were they missed in prior quality audits? It was time to peel the onion some more!

Peeling the Quality Audit Onion

It was imperative to ask tough questions about the auditing process itself and to take a closer look at:

• The tools/elements used to gather information during compliance audits.
• What was examined during such audits.
• What was included in reports, what was omitted, and why.
• How an internal audit was different from a supplier audit.

Invariably, compliance audits (and other types of audits) use checklists and questions based on the applicable requirements to determine compliance. In this case, there were bare-bones questions pertaining to data, manufacturing data analytics, and statistical methods. Questions like “Do you have a process for collecting, analyzing, and processing data to create quality metrics?” required a Yes/No response. They are of little help and not atypical.

Do the standards and regulations provide enough information in this regard to create a solid quality compliance audit checklist? Let us examine their requirements.

The Requirements from Quality Standards and Regulations

Since the case study is about a medical device, controls defined in 21 CFR Part 820 and ISO 13485 apply.

21 CFR Part 820 requires, where appropriate, each manufacturer shall:

• Establish and maintain procedures for identifying valid statistical techniques required for:
  • Establishing, controlling, and verifying the acceptability of process capability.
  • Product characteristics.

Various sections of ISO 13485 require documented plans and/or procedures for use of appropriate statistical techniques in:

• Design and development verification and validation.
• Validation of processes and services.
• Measurement and analysis.
• Analysis of data.

When it comes to data, the verbiage often used is “appropriate” or “where appropriate” as qualifiers. Further explanation is provided which indicates the default is to be taken as appropriate unless the manufacturer can document justification otherwise, or non-implementation could result in product related issues. This allows some room for judgement and opinion, both on the part of the auditor and the auditee.

What else?

Equally if not more important than manufacturing data analytics is data validation and data integrity. Lack of checks and balances in these areas increase subsurface risks – akin to the proverbial iceberg – where 90% of the risk lies below the waterline.

Couple this with the prevalent situation where a vast majority of auditors do not even know enough to ask the right questions about data, and its analysis, and you have the perfect recipe for “very weak” compliance audit.

Is there a solution? YES!

Smart Compliance Audits and Data-Smart Quality Auditors

When it comes to focusing on data and analytics, the intent of standards and regulations is right –although there is room for improvement. Since auditors are at the frontlines – examining firsthand how data gets used, processed, and presented – can we equip them with better tools (questions and techniques) on this topic? Sure! While a comprehensive treatment of this topic will require a daylong workshop and practice, here are five questions that every auditor can use, and the rationale for each.

1. How did you establish what quality data needs to be gathered for this product (or service)?
   Addresses if the appropriate data is collected for analysis.
2. What statistical methods were used to analyze quality data associated with “this” product (or service)?
   A check on use of statistical methods.
3. How did you determine that this method was appropriate?
   Addresses if the statistical method was appropriate.
4. If no statistical methods were used, can we see the justification for the same?
   Focuses on a justification if statistical methods were deemed “not appropriate.”
5. How is the education, background, experience, and training of the analyst who works with such data ascertained to be sufficient?
   Check to see if the right personnel are performing such analyses (Ref: 21 CFR 820.25).

So, there you have it – Five things every auditor should do regarding data analysis.

Copyrights to the Author – 2023 Rai Chowdhary, The KPI System