Back in 2013, following the ICH Q10(1) , the FDA started the Quality Metrics initiative. Two years later, the Agency released a draft guide as a baseline for discussion with the industry. The objectives were to assess feasibility, and to evaluate the effort it would take to bootstrap the project. Despite the release of the revised softened edition of the guide in 2016(2), it is most unlikely that industry will adhere to this Quality Metrics initiative.
This article develops the known limits of the currently proposed approach to clarify the cautious industry position as posted by the Cross-Industry Quality Metrics Collaboration Group on March 27th(3) this year.
Key Performance Indicator: for what purpose and use?
A Quality Key Performance Indicator (KPI) is defined as a measurable value used to quantify quality objectives reflecting the performance of an organization, a process or a system. Choosing the right KPIs relies upon a good understanding of the organization context and the principle goals they are following and must be linked to the quality of the product or the process efficiency.
Relevant KPIs are used at site and global levels to monitor products and processes in a sustained objective of continuous improvement with a view in the end of maintaining the right level of efficiency. At site level, they guide improvement measures aligned with the specific activity and context. At a global level, they are used to compare sites to spur on and to prioritize improvement efforts.
The Health Authority intentions
For the FDA, the main goal is to prevent and mitigate future drug shortages often linked to quality defects revealed at the finished product level. Thus, FDA intends to improve inspection policies and practices and to drive inspections based on the risk for customers and for public health.
The second goal consists in encouraging companies to implement a state of the art quality system & supporting innovation in the quality management system field. In order to achieve this goal, exploratory & descriptive studies based on collected data are forecasted & furthermore monitoring the quality of products & sites to detect signals such as common causes (trends or seasonal effects) or special ones (assignable special event), using the Lean Six Sigma language.
Which KPIs for Quality Metrics?
FDA has defined 3 global KPIs to analyze the quality level of sites and groups to prioritize inspections:
- Lot Acceptance Rate (LAR) as an indicator of manufacturing process performance. It corresponds to the number of accepted batches in a timeframe compared to the number of batches produced in the same period.
- Product Quality Complaint Rate (PQCR) as an indicator of patient or customer satisfaction.
- Invalidated Out-of-Specification (OOS) Rate (IOOSR) as an indicator of robustness of laboratory operations. It is the ratio of invalidated OOS attributed to the laboratory related to the total OOS.
Why a high risk of a misuse of data?
At the outcome of the discussions between the Health Authorities & the industry, different special scenarios were analyzed and illustrate the difficulties for companies to define and calculate indicators that meet the Health Authority needs. As examples, the Agency shared some specific cases in November 2016 during a Technical Webinar(4). The following examples highlight particularly the complexity & risks associated with these indicators.
If 5 customers report the same type of complaint, 5 complaints are counted. If 5 different departments from the same customer (a hospital for example) scenario 2 considers only 1 batch impacted. What if it appears that 2 batches are impacted by the 5 first complaints? Are the 5 complaints grouped in 2 complaints after investigation?
“Your product is grape flavoured and I prefer cherry favouring”. In first intention, this complaint is not counted. “Your grape avoured product doesn’t taste right”, this one is counted, we suspect a quality defect. What if the first complaint reveals, after investigation, a mix up?
In the case of a saleable lot obtained after a pool of many batches, we can count only the accepted saleable batch or the lot acceptance rate can consider all in process and packaging batches. What is considered as a lot?
For a batch, OOS can be considered at time of release or after long-term stability studies some product nonconformance can be only observed after a long-time storage. Nonconformities or OOS are often of different nature from release test and give evidence of nonconformity of a quality attribute of the product (i.e. container closure integrity defect). Should those OOS be considered?
The OOS occurrence depends on the definition of the specifications and the acceptance ranges. On the contrary, one shouldn’t miss out on a result Out of Trend.
There are many other specific scenarios and operational situations not considered by the FDA & will be revealed by degrees.
For example, a cosmetic defect will trigger a complaint for Japanese customers whereas it will not be considered in many other countries. Should this type of complaint be included into the KPI?
All these considerations lead to the question: how to compare indicators that are not calculated in the same way or in comparable contexts?
Based only on global indicators, data could conduct to misinterpretation. Out of the context, without explanation on the calculation rules, & on what the root causes are, it could be difficult to give logical analysis of the current situation. For instance, if a group acquires another company within the year, the global indicator may compromise the interpretation of trends
In the end, given the FDA intent, working at macroscopic level, & considering the challenge set for industry to supply representative indicators, there is a risk of prioritizing inspections in the wrong place.
Despite the description of the data expected by the FDA within the last Guidance draft, standardized calculation rules and aligned KPIs definitions need further considerations. Moreover, granularity & statistic use of data needs to be addressed. To be relevant, a KPI cannot be calculated with only 2 data: 2 saleable lots including 1 rejected lot = LAR 50%!
Feedback from our experience
In addition to the risks highlighted above, these are some additional issues that need to be considered:
- Submission of Quality Metrics data to a Health Authority requires a high level of control from the senior management; efforts may not be compensated with benefits.
- Sharing data, linked to specific products & manufacturing processes, raise an issue about confidentiality (depending on the use of data), although the FDA proposed to exclude the Quality Metrics initiative from the applicability scope of FOI Act (Freedom of Information).
- Some key indicators used to evaluate the Quality System performance are difficult to measure, such as the Quality Mindset. However, the Quality Mindset strongly contributes to the confidence level to grant a company.
- Getting expected performance/result may lead to shortcuts (i.e. under reporting deviations) & potentially data integrity issues.
Relevant KPIs linked to SMART objectives give a site a direction. In general terms, there is a real difficulty to compare sites even within the same group, or companies even doing the same activities. Each site has a specific context, given its specific activities, its own organization & distinguishing features such as its level of automation, regional culture, staff turnover, products portfolio, production capacity…
Beyond the need of a complementary FDA effort to further clarify the metrics definition & calculation rules, the difficulties encountered highlight that mutual confidence & transparency are a condition for Quality Metrics. The Quality Metrics cannot be the foundation of confidence. Whether within a pharmaceutical group, between sites or with the Health Authority, the mutual confidence is a continual construction process but a prerequisite for any continuous improvement program.
Amélie DIEBOLD – AKTEHOM
Since 2007, after an engineering degree from Ecole de Biologie Industrielle, Amélie Diebold has been working on product and process understanding in the biotechnological industry for Aktehom. She now is a successful Project Leader and takes an active part in Paris agency in parallel to her consultant role. She is focusing her activities on the implementation of the Pharmaceutical Quality System.
 FDA, Submission of Quality Metrics Data, Guidance for Industry, draft Revision 1, November 2016
 FDA CDER-CBER Stakeholder Technical Webinar, November 2016