Training Effectiveness – A Quality by Design Approach

In the role of a consultant, you have the benefit of seeing what works well across your client base and their shared common pain points. Because of this perspective, I’m often invited onto a project to provide guidance on best practices and practical advice on how to avoid the common pain points.  As a case in point, recently a client asked me, “What’s the best practice for training effectiveness?” My response, in typical consultant fashion was, “How are you defining training effectiveness?” Their response was, “We’re not sure, but we were cited for it in a number of client audits.”

What is training effectiveness?

So let’s begin by breaking the term down and defining it. If we think about “effective training,” we know that training is effective when the employee demonstrates the desired behavior change (new skill, new knowledge, etc.) they learned during the training. Training effectiveness is essentially a measure that examines the degree to which training improved the employee’s knowledge, skill and behavioral pattern within the organization as a result of the training. Simply put – did the training do what it was supposed to do? Did employees learn what they were supposed to learn? Were the employees who attended training able to do what they should be able to do once they left their classroom? Training effectiveness seems to be a topic that is percolating, especially in GMP environments.

Yet, it’s shrouded in mystery – i.e., no one is clear on what it is, how to implement it, and what the regulators are looking for when it comes to effective training. If you look at the regulations (see regulatory references below), they are very vague. Summed up they basically say something along the lines of – an organization should have in place a program to periodically evaluate the effectiveness of training. Hmmm…a lot of grey area in there that is left up for interpretation. Couple this with the fact that there is limited information out there on the practical aspect of implementing this concept and you’ve got a murky recipe for audit citations.

What We Know about Training Effectiveness 

The trends from a regulatory standpoint

Regulatory authorities are paying more attention to training effectiveness. However, the FDA does not commonly cite the effectiveness of training except in the case where an organization frequently has demonstrated that retraining (typically as a result of a finding and associated with a CAPA) did not work. You probably know the cycle all too well:  

1. Audit findings cited the root cause of an issue to be human error.
2. The humans are retrained.
3. The deviation reoccurs.
4. The root cause is determined to be human error.
5. Retraining reoccurs.
6. The deviation reoccurs... and so the cycle continues.  We could speculate that a proper root cause analysis is typically not performed in these cases, but that’s a topic for another article.  I recently asked a former FDA employee this question, “What would you look for and be satisfied with that provides evidence that a pharmaceutical company has effective training?” His response was, “Reduction in human error as a root cause to a very low and stable rate.” What this tells me is that their focus is outcomes based, not in determining if you have a super set of processes built around training. 

The trends from a quality standpoint

Earlier versions of ISO9000 focused on identifying training needs, delivering training, and keeping records (as part of employee qualifications).  However, the most recent version has additional wording regarding competence and effectiveness of training. In section 6.2.2 Competence, Training and Awareness, it states (among other things), “The organization shall:

• determine the necessary competence for personnel performing work affecting conformity to product requirements, 
• where applicable, provide training or take other actions to achieve the necessary competence,
• evaluate the effectiveness of the actions taken…”

This language hints that similar to regulatory authorities, ISO9000 has shifted its focus from process to outcomes. 

Practically speaking…a shift from process to outcomes

It is no longer acceptable to show that you have the processes and documentation in place that ensure the right training for the right employees took place. And as an aside, many organizations struggle with getting just those basics down. Ask anyone who has been required to read a large number of SOPs that had no relevance to their job or role responsibilities and they will agree with this statement. But getting back to my point, you are expected to “periodically assess the effectiveness” of your training. But what does that really mean in practical terms? What type of guidance is out there to help you interpret these regulatory requirements?

There isn’t any. Of course, we can use the standard Kirkpatrick Model to measure if our training is effective. That’s a good model, but how do we translate that into an environment where we need to ensure we are consistent not only in process but in outcomes, we’ve documented our consistency, and we can readily provide evidence of that consistency? How do we look across training at a site and across large organizations to be able to say, yes – we consistently produce high quality training which results in having skilled, qualified employees, which results in a higher quality outcome (i.e., clinical data, safer patients, medicinal product) for our business?

If you’re someone who is not sure where to begin, I’d like to present you with an approach that represents both quality and process excellence best practices and can easily apply to training effectiveness – in fact, you might be doing some of this already.

A two-pronged approach to training effectiveness

1. Ensure training effectiveness - An organization can ensure training effectiveness through key activities in its best practices training design, development, and delivery methodology. This is truly a Quality by Design approach and it happens before any employee participates in the training.

• How is it done? An organization embeds best practices in adult learning and training design, development and delivery into its standard training methodology (yes – that’s right, you need a methodology) enabling the organization to achieve the goal of “Right Training delivered by the Right Trainers to the Right Persons at the Right Time to achieve the Right Outcomes.” 

• What does it look like? This Quality by Design approach would look something like this*:

*Note: this is not a comprehensive look at all training processes. 

2.  Assess training effectiveness - An organization can assess training effectiveness through periodic review of current trainings after the trainings have been completed.

• How is it done? Periodic assessment is likely to happen at an individual site level but can be a coordinated effort to look across a larger global organization that has multiple sites. Assessing the effectiveness of training occurs after employees participate in the training and should be part of an identified periodic review.

• What does it look like? Conducting a periodic review might look like this:

• Audit a select number of training

• Document the review results

• Continuously improve and feed to ensure training effectiveness

Key input: defining minimum measures

Prior to conducting a periodic review, your organization, especially if it has more than one location, should identify minimum measures. For example, at a minimum, sites/locations/organizations that want to determine the effectiveness of training may decide to use the following measures:

From my perspective, these minimum measures would give you a very good feel for whether the training was effective or not. However, there are many additional measurements (follow-up evaluations or employee demonstration of skills, annual performance reviews, etc.) you could use to provide more data regarding whether your training is effective or not. Be cautious - don’t over measure!

Audit a select number of trainings

Once you’ve identified your measures and are ready to conduct a periodic review, which trainings will you assess – one training, all trainings? I recommend taking an audit approach.  An audit approach will ensure you have a fairly accurate feel for the effectiveness of your training without expending high amounts of effort during your periodic review. For example, during your periodic review you might:

1. Select a percentage or minimum number of trainings that makes sense for your organization to evaluate for effectiveness. The training should be a mix (at least one instructor-led training for a high risk or high impact topic, at least one self-study*, etc.)

Footnote: * Read and Understand (AKA Read & Sign) “Training” is not considered a self-study.

1. 2.  Measure the effectiveness of the trainings by assessing your minimum measurements. Let’s take the second minimum measurement of “Employee Successfully Completes Assessment. Here is an example of an approach to take:

• Gather the assessment scores for 25% (or whatever percentage or number makes sense for your organization) of the total number of participants across all sessions of training for each identified training.
• Review the assessment scores. 
• Ask yourself the following questions:

• What percentage of the reviewed employee assessment scores did not meet the minimum completion expectation requirements? This assumes your organization keeps failed scores.
• Is this percentage 25% or higher?
• If the percentage of failed assessments for an identified training is 25% or higher, you need to determine why. 
• Conduct a root cause analysis and document the root cause, indicating how the site will correct this, who will drive correcting it, and by when.

You don’t have to follow this exactly. It’s just an example, but this example can give you a better feel for what a part of an assessment of training effectiveness might look like. Next steps include documenting the audit and any results and/or follow up actions and using these as input into your continuous improvement process.

Training effectiveness requires training maturity

The reality is – training effectiveness takes training maturity. Remember back in the “ensure training effectiveness” paragraph, I mentioned having a standard training methodology (yes that’s right, you need one)? The methodology doesn’t have to be perfect, it can be flexible, but it needs to be consistent and carry standards (e.g., retention standard - limiting the number of read and understand procedures you can expect an employee to take, limiting the number of training hours). An organization needs maturity in its training processes. If you do not have a mature training function that follows best practices, why bother putting loads of effort into creating a periodic review of training? The result will always be the same: the training is not effective. 

Bottom line

Invest the time upfront in ensuring the design, development, and delivery of your training is setting your employees up for achieving the outcomes. One of the best demonstrations of training effectiveness is the reduction in deviations which in the long run pays huge dividends in reducing the cost of quality. Without an outcomes based approach where you are ensuring training effectiveness, your employee’s performance will continue to suffer and so will your quality. 

Regulatory References - GMP

Australia

Therapeutic Good Administration (TGA) - Australian code of good manufacturing practice for human blood and blood components, human tissues and human cellular therapy products Section 208.

“Continuing training should also be given, and its practical effectiveness should be periodically assessed.”  https://www.tga.gov.au/book/personnel-and-training 

European Union

The Rules Governing Medicinal Products in the European Union Volume 4 – EU Guidelines for Good Manufacturing Practice for Medicinal Products for Human and Veterinary Use, Chapter 2: Personnel  Section 2.11 Training “Besides the basic training on the theory and practice of Good Manufacturing Practice, newly recruited personnel should receive training appropriate to the duties assigned to them. Continuing training should also be given, and its practical effectiveness should be periodically assessed. Training programmes should be available, approved by either the head of Production or the head of Quality Control, as appropriate. Training records should be kept.” https://ec.europa.eu/health/files/eudralex/vol-4/pdfs-en/cap2en200408_en.pdf  

European Medicines Agency – EMA

COMMISSION DIRECTIVE 2003/94/EC, Article 7  Personnel “The personnel shall receive initial and ongoing training, the effectiveness of which shall be verified, covering in particular the theory and application of the concept of quality assurance and good manufacturing practice, and, where appropriate, the particular requirements for the manufacture of investigational medicinal products.” https://ec.europa.eu/health/files/eudralex/vol-1/dir_2003_94/dir_2003_94_en.pdf 

Health Canada - GMP

C.02.06 Personnel 1.2 “The effectiveness of continuing training is periodically assessed.” https://www.hc-sc.gc.ca/dhp-mps/compli-conform/gmp-bpf/docs/gui-0001-eng.php#Personnel 

World Health Organization (WHO)

WHO – Annex 2 - WHO good manufacturing practices for pharmaceutical products: main principles Section 10: Training “10.2 Besides basic training on the theory and practice of GMP, newly recruited personnel should receive training appropriate to the duties assigned to them. Continuing training should also be given, and its practical effectiveness periodically assessed. Approved training programmes should be available. Training records should be kept.” https://www.who.int/medicines/areas/quality_safety/quality_assurance/TRS986annex2.pdf