The Quality Metrics Journey

One of the most discussed and debated topics on today’s pharmaceutical landscape is the issue of quality metrics. Establishing, maintaining, and interpreting quality metrics to determine the suitability of pharmaceutical products has become a high priority for the FDA. To understand the issue of quality metrics it is important to start at the beginning. The first stop on this journey took place in 2012 when Congress passed the Food Drug Administration Safety and Innovation Act (FDASIA) which enhanced FDA’s capability to proactively react to, prevent, and alleviate drug shortages. This direction is embedded in the language contained in Title VII—Drug Supply Chain and Title X—Drug Shortages. Specifically, Title VII Section 705 of the Act states FDA “shall inspect establishments described in paragraph [1] that are engaged in the manufacture, preparation, propagation, compounding, or processing of a drug or drugs (referred to in this subsection as ‘drug establishments’) in accordance with a risk-based schedule established by the Secretary” (1).

This section also describes the risk factors considered in establishing the inspection schedule. One of the risk factors listed in this section is “Any other criteria deemed necessary and appropriate by the Secretary for purposes of allocating inspection resources.” Section 706 of the same act authorizes FDA to request certain company information in advance of or in lieu of inspections by stating “Any records or other information that the Secretary may inspect under this section from a person that owns or operates an establishment that is engaged in the manufacture, preparation, propagation, compounding, or processing of a drug shall, upon the request of the Secretary, be provided to the Secretary by such person, in advance of or in lieu of an inspection…” In addition, Title X section 506C-1 (Annual Reporting on Drug Shortages). Title X section 506C-1 requires FDA to annually provide Congress “a report on drug shortages…”The second part of the journey occurred in the Federal Register

Notice of February 12, 2013. In that notice, FDA asked the industry to “assist the Food and Drug Administration in drafting a strategic plan on drug shortages as required by the Food and Drug Administration Safety and Innovation Act…” This notice asked a series of thought-provoking questions including “What metrics do manufacturers currently use to monitor production quality?” and “How frequently would such metrics need to be updated to be meaningful?”(2).

The industry reaction to this information was varied. Many trade organizations responded to the questions in the Federal Register

Some prepared white papers while others held meetings to discuss the issue with their members. The general consensus was that industry needed to seriously engage with FDA to define which quality metrics would provide information to the agency supporting efforts to eliminate drug shortages and to be used in establishing a risk-based approach to inspections.The third leg of the journey was the issuance of a draft guidance in July 2015 titled Request for Quality Metrics (3).

In between the issuance of the Federal Register and the release of this original draft guidance there were many industry comments submitted to the agency suggesting various metrics that might be employed by the agency. In addition, industry trade organizations held conferences and seminars to discuss the issue with the stakeholders and agency representatives. A number of the trade organizations published white papers based on the proceedings from these conferences and put forth the position that the need to submit metrics to the agency was redundant because of the requirement to submit annual product reviews. The agency listened to industry and some of the feedback was incorporated into the guidance. The metrics that were chosen for the 2015 guidance were Lot Acceptance Rate, Product Quality Complaint Rate, Invalidated OOS Rate and Annual Product Review or Product Quality Review on Time Rate. The guidance also contained three optional metrics intended to measure quality culture: Measuring Senior Management Engagement, CAPA Effectiveness and Process Capability/Performance. This version of the quality metrics guidance generated 83 comments from industry submitted to the docket. A number of these comments focused on the practicality of submitting the requested metrics. In response to this concern the agency issued in June 2016 the Quality Metrics Technical Conformance Guide: Technical Specifications Document (4) that explained how companies would be expected to submit the data to the agency.

The fourth part of our journey came in November 2016 when the second version of the guidance, titled Submission of Quality Metrics Data Guidance for Industry (5) was released for comment. The docket for commenting on this most recent version of the guidance closed in March of 2017. There were 25 comments submitted. 12 from trade associations, 10 from individual firms, one from a hospital group, one from an academic institution and one from a private citizen. The guidance clearly stated the “…metrics can be also useful to FDA: to help develop compliance and inspection policies and practices…. improve the agency’s ability to predict, and therefore, possibly mitigate future drug shortages; and to encourage the pharmaceutical industry to implement state-of-the-art, innovative quality management systems for pharmaceutical manufacturing.” The FDA also clarified the goal of the metrics program by stating “As described in this guidance, FDA is initiating a voluntary reporting phase of the FDA quality metrics reporting program.” This current version of the document requires companies to report on only three metrics: Lot Acceptance Rate, Product Quality Complaint Rate and Invalidated OOS Rate.

The optional metrics intended to measure quality culture were removed. These proposed metrics are not new to the pharmaceutical industry. Many of them are currently being used by companies to measure their performance. In order to be successful, companies need to review and analyze the information the FDA is asking for, as well as other metrics they are collecting, and identify potential problem signals so that they can solve issues and self-correct before regulatory inspections occur. The problem with the proposed metrics, however, is that they are lagging indicators of performance. There is no set requirement on which metrics a company should track to measure their overall performance. Each company should determine which metrics to track based on their operations, number of facilities they operate and where they are located, what types of products they manufacture, and what type of culture exists in their places of business. Any metrics chosen must be meaningful and written to provide a clear analysis of ongoing activities. It is important for operations and quality to agree on the metrics and how to report them to management to avoid overreaction to the data. It is not sufficient to simply report the data. The interpretation of the data is a crucial element because it may include a root-cause analysis of its own and may help to promote continuous process improvements.

When choosing a metric, it is important that the architects of the metric are aware of unintended consequences that may inadvertently drive negative behavior. Management attempting to incentivize achievement of the goal such as offering a financial award if the goal is achieved may lead to inappropriate behaviors that do not address the real issue. In these cases, it is generally not the metric that will drive the behavior but rather use of behavioral rewards. Reward for achievement rather than analysis of the real underlying causes will not lead to sustainable positive change. When managed properly, metrics are an important tool to help drive positive change and quality process improvements. Regardless of what metrics a company chooses to measure their performance, achieving a quality culture is important in assuring the reported metrics are accurate and reliable.

A quality culture requires management and employees to establish an environment where responsibility, accountability, and reliability are paramount, and to understand the role each person performs in delivering a high-quality product to the customer and sustaining that performance on a continual basis. Management must educate employees and provide the tools and environment where they can perform their functions in an atmosphere that encourages excellence and continuous improvement. Continuous improvement programs are, in fact, reliable indicators of the presence of a quality culture. Where the industry’s and the agency’s destination on this journey ends is yet to be seen. What is evident is that companies must have a robust quality metrics program so they may continue to supply quality products without interruptions to patients. Companies need to keep in mind that when establishing a metrics program, they should evaluate numerous data input points including, but not limited to, product-quality attributes, manufacturing site performance, people metrics, and quality-system metrics. For product-quality metrics, companies should consider reporting on batch-specific data such as trending drug product, drug substance, and stability test results against customer complaint rates.

Indirect product-quality metrics could include environmental monitoring, water trend results, and yield rates. When establishing site metrics, the company could look at inspection history including internal audit findings and maintenance history such as equipment age versus defect-failure rates. People metrics should consider ongoing job-specific training and education, skills and experience assessments, and employee turnover rate by job function and site. Quality systems metrics might look at change control, investigation root-cause trends, and release-testing cycle times. Companies that can successfully establish a robust metrics program that helps them continuously improve should have no trouble meeting the FDA metrics requirements.

References:

• FDA, Request for Quality Metrics Guidance for Industry, (July 2015)
• FDA, Quality Metrics Technical Conformance Guide: Technical Specifications Document, (June 2016)
• FDA, Submission of Quality Metrics Data Guidance for Industry, (Nov. 29, 2016).
• FDASIA
• FDA, Federal Register, Vol. 78, No. 29 (Feb 12, 2013) 9928-9929.