Why Do CAPAs Fail Their Effectiveness Checks?

When we start talking about deviations and CAPAs, we can’t help having a sidebar discussion about root causes and more specifically the rant about finding the true root cause.  I intentionally skipped that content in the previous blog.  It was my intention to kick off the new Deviation and CAPAs blog series by first looking at deviations by themselves.  And the learning opportunities deviations can provide us about the state of control for our quality systems.  From those deviations and ensuing CAPA investigations, I ask you this: are we improving anything for the long term (aka prevention).  Are we making any progress towards sustaining those improvements?

Corrective Actions Preventive Actions (CAPA) Steps

Let’s step back a moment and quickly review typical steps for CAPAs:

CAPA Components

The purpose of an Effectiveness Check (EC) is for verifying or validating that actions taken were effective and do not adversely affect product, device or process.  It goes beyond the statement in the investigation form to include a follow-up activity that closes the loop on the specific CAPA.  If an effectiveness check fails meaning the CA/PA was not effective or another deviation /nonconforming incident has occurred, we go back to the beginning and either start again or in most cases, we re-open the investigation.  The pressing question is why did the EC fail?  Almost instinctively, we believe that we did not find the true root cause.  Perhaps.  Was there a rush to close the investigation?  Probably.  Did the investigation team grab the first probable cause as the root cause because the “problem” felt familiar?  Maybe. Or is it a case of a fix that backfired into unintended consequences? Possibly. I will also suggest that the CA/PA may not have been aligned properly.

Ask these 3 questions about CA/PAs

  • Is the CA/PA Appropriate? The focus of this question is about the affected people.  What is the size of this audience? Is it mainly one person or groups of people?

Can the CA/PA be executed efficiently?  Is it for one site or multiple sites?

  • Is the CA/PA Economical? What budget is available?

Is it a “cheap” fix or a 3 – 6 month project? Or an expensive solution of more than 6 months and will need capital expenditure funding?

  • Is the CA/PA Feasible? The real question is about the timeline.

            Need it fast – within 3 months or

            Have time – don’t need until more than 3 months from now.

And then there is the unspoken 4th question – is the CA/PA “political”?  I experienced first hand what happens to CAPAs that are politically oriented.  Most of them failed their ECs.  Request “Can You Stay a Little While Longer”. The best CAPAs are the ones that map back to the root cause.

Introducing the HPISC CAPA Performance Chain

On the left hand side, you will recognize the 3 traditional tasks to complete.  After the EC is written, trace upwards to ensure that the EC maps back to the CA/PA and that the CA/PA maps back to the root cause; hence, the bottom up arrow.  On the right hand side are performance improvement activities that I use as a Performance Consultant (PC) to bring another dimension to the CAPA investigation, namely, Human Performance Improvement (HPI). 

Before I can write the root cause statement, I examine the “problem” also known as a Performance Discrepancy or an incident and I conduct a Cause Analysis that forces me to take a three tiered approach (the worker, the work tasks, the workplace) for the possible causes and not get bogged down in observable symptoms only.  The Performance Solution is more appropriately matched to the identified gap. In theory, this is what the corrective action(s) is supposed to do as well. During the performance solution planning, determination of success and what forms of evidence will be used happens with key stakeholders.  So that collecting the data happens as planned, not as an after thought, and the effectiveness is evaluated as discussed.    

What can we really control?

In RCA/CAPA meetings, I often hear about what management should do to fix the working conditions or how most of the operator errors are really managements’ fault for not taking the culture factor seriously enough.  While there may be some evidence to back that up, can we really control, reduce or eliminate the human factor?  Perhaps a future blog on understanding human errors will be released.

Management Can:

  • Design work situations that are compatible with human needs, capabilities and limitations
  • Carefully match employees with job requirements
  • Reward positive behaviors
  • Create conditions that optimize performance
  • Create opportunities to learn and grow professionally.

Clues for Failed Effectiveness Checks

One of the first activities to perform for a failed EC is to evaluate the effectiveness check statement.  I have read some pretty bizarre statements that challenge whether the EC was realistic to achieve at all. The conditions under which we expect people to perform must be the same as the conditions we evaluate them during an EC review.  So why would we set ourselves up to fail by writing ECs that don’t match normal workplace routines? What, because it looked good in the investigation report and it got the CAPA approved quicker?

Next, trace back each of the CAPA tasks to identify where to begin the re-investigation.  I also suggest that a different root cause analysis tool be used. And this is exactly what we did while I was coaching a cohort of Deviations Investigators.  Future blogs will discuss RCA tools in more detail. -VB

The Big Why for Deviations

As part of my #intentionsfor2019, I conducted a review of the past 10 years of HPIS Consulting.  Yes, HPISC turned 10 in August of 2018, and I was knee deep in PAI activities.  So there was no time for celebrations or any kind of reflections until January 2019, when I could realistically evaluate HPISC: vision, mission, and the big strategic stuff.  My best reflection exercise had me remembering the moment I created HPIS Consulting in my mind.

Human Performance Improvement (HPI) and Quality Systems

One of the phases for HPI work is a cause analysis for performance discrepancies.  The more I learned how the HPI methodology manages this phase the more I remarked on how similar it is to the Deviation /CAPA Quality System requirements.  And I found the first touch point between the two methodologies.  My formal education background and my current quality systems work finally united.  And HPIS Consulting (HPISC) became an INC.  

In my role of Performance Consultant (PC), I leverage the best techniques and tools from both methodologies.  Not just for deviations but for implementing the corrective actions sometimes known as HPI solutions.  In this new HPISC blog series about deviations, CAPAs, and HPI, I will be sharing more thoughts about HPISC touch points within the Quality Systems. For now, lets get back to Big Why for deviations.

Why are so many deviations still occurring? Have our revisions to SOPs and processes brought us farther from a “State of Control”? I don’t believe that is the intention. As a Performance Consultant, I consider deviations and the ensuing investigations rich learning opportunities to find out what’s really going on with our Quality Systems.

The 4 cross functional quality systems

At the core of the “HPISC Quality Systems Integration Triangle” is the Change Control system.  It is the heartbeat of the Quality Management System providing direction, guidance and establishing the boundaries for our processes.  The Internal Auditing System is the health check similar to our annual physicals; the read outs indicate the health of the systems.  Deviations/CAPAs are analogous to a pulse check where we check in at the current moment and determine whether we are within acceptable ranges or reaching action levels requiring corrections to bring us back into “a state of control”.  And then there is the Training Quality System, which in my opinion is the most cross-functional system of all.  It interfaces with all employees; not just the Quality Management System.  And so, it functions like food nourishing our systems and fueling sustainability for corrections and new programs.

Whether you are following 21CFR211.192 (Production Record Review) or ICHQ7 Section 2 or  820.100 (Corrective and Preventive Action), thou shall investigate any unexplained discrepancy and a written record of the investigation shall be made that includes the conclusion and the follow up. Really good investigations tell the story of what happen and include a solid root cause analysis revealing the true root cause(s) for which the corrective actions map back to nicely.  Thus, making the effectiveness checks credible. In theory, all these components flow together smoothly.  However, with the continual rise of deviations and CAPAs, the application of the Deviation /CAPA Management system is a bit more challenging for all of us.  

Remember the PA in C-A-P-A?

Are we so focused on the corrective part and the looming due dates we’ve committed to, that we are losing sight of the preventive actions? Are we rushing through the process to meet imposed time intervals and due dates that we kind of “cross our fingers and hope” that the corrective actions fix the problem without really tracing the impact of the proposed corrective solutions on the other integrated systems? Allison Rossett, author of First Things Fast: a handbook for performance analysis, explains that performance occurs within organizational systems and the ability to achieve, improve and maintain excellent performance, depends on integrated components of other systems that involve people. 

Are we likewise convincing ourselves that those fixes should also prevent re-occurrence? Well, that is until a repeat deviation occurs and we’re sitting in another root cause analysis meeting searching for the real root cause.  Thomas Gilbert, in his groundbreaking book, Human Competence: engineering worthy performance tells us, that it’s about creating valuable results without using excessive cost.  In other words, “worthy performance” happens when the value of business outcomes exceeds the cost of doing the tasks.  The ROI of a 3-tiered approach to solving the problem the first time, happens when employees achieve their assigned outcomes that produce results greater than the cost of “the fix”. 

Performance occurs within three tiers

So, donning my Performance Consulting “glasses”, I cross back over to the HPI methodology and open up the HPI solutions toolbox.  One of those tools is called a Performance Analysis (PA). This tool points us in the direction of what’s not working for the employee, the job tasks a/or the workplace. The outcome of a performance analysis produces a 3 tiered picture of what’s encouraging or blocking performance for the worker, work tasks, and/or the work environment and what must be done about it at these same three levels.  

Root cause analysis (RCA) helps us understand why the issues are occurring and provides the specific gaps that need fixing.  Hence, if PA recognizes that performance occurs within a system, then performance solutions need to be developed within those same “systems” in order to ensure sustainable performance improvement.  Otherwise, you have a fragment of the solution with high expectations for solving “the problem”.  You might achieve short-term value initially, but suffer a long-term loss when performance does not change or worsens. Confused between PA, Cause Analysis and RCA? Read the blog – analysis du jour.

Thank goodness Training is not the only tool in the HPI toolbox!   With corrective actions /HPI solutions designed with input from the 3 tiered PA approach, the focus shifts away from the need to automatically re-train the individual(s), to implementing a solution targeted for workers, the work processes and the workplace environment that will ultimately allow a successful user adoption for the changes/improvements.   What a richer learning opportunity than just re-reading the SOP! -VB

  • Allison Rossett, First Things Fast: a handbook for Performance Analysis; 2nd edition 
  • Thomas F. Gilbert, Human Competence: Engineering Worthy Performance
You might want to also read:

Are you doing the Training 2 Step?

Has this situation ever happened to you? You are in a root cause meeting and the *CAPA investigator is conducting an interview with the primary individual involved with the discrepancy. When questioned why did this happen, he shrugs first and then quietly mumbles I don’t know. When pushed further, he very slowly says I just kind of went brain dead for a moment.  And then silence.

While that may be the honest truth, the investigator must resist the temptation to label it as Operator Error and explore possible causes. One of my favorite root cause analysis tools for “I Don’t Know Why” response is to use the Fish Bone Diagram also known as the 4 M’s diagram. This tool provides a structured focus to explore many possibilities and not just stop at the first plausible cause; such as Operator Error. Aptly nicknamed, the 4 M’s are Man, Machine, Methods and Materials.   When the results of this exercise points to a training or operator related issue, don’t stop at “operator error –> retrain”.

Consider for a moment, what this retraining session would look like. Will re-reading the procedure be enough to “jog his memory”? Will repeating the procedure be a good use of precious time when s/he already knows what to do? More than likely it won’t prevent “going brain dead” from happening again. Instead, do the HPISC Training 2 Step:

Step 1 – confirm the results of the gap analysis

What task, what step(s) or actions are in question?

Step 2 – address why the original training did not transfer back to the job.

Using the 4 M’s diagram as the framework, explore Man, Machine, Methods and Materials questions with regards to the training this operator receives. See diagram below. The full set of questions can be found in the eBook Training Cause Analysis.

4msfortrca

Is this really worth it?

I think it is. Conducting these 2 steps will accomplish two objectives.   It will provide further evidence that some kind of training is needed. And it will highlight what areas are in need of revising either for the performer, the training program or course materials. Yet, there are some who will resist this added work because it’s easier to find blame than to uncover the cause. Fixing the true root cause could trigger a re-validation of the process or an FDA filing if it’s a major process change.   Why create more work? Isn’t it easier to just retrain ‘em? No, not really. Finding the true root cause is the only effective way of eliminating many of the costly, recurring problems that can plague manufacturers.

But what if

Some folks will push back with the excuse – “this never caused a problem until now”, so it must be the operator’s fault! This may be the first time it was discovered but that does not mean the procedure is 100% accurate. Often, experienced operators know how to work around an incorrect step and don’t always report a misstep in the procedure while a less savvy operator follows the procedure and causes the non-conformance to occur. See Sidebar SOP Logic Rules. Is the procedure difficult, lengthy or requires weeks to become proficient let alone qualified? Was the qualification routine or performed as a simulation? Was the procedure written with support from a lead operator or qualified trainer? Did the draft version undergo some kind of field test or dry run prior to release? And the classic situation, are proposed changes hung up in change control awaiting effective release?screen-shot-2016-10-19-at-4-33-39-pm

Understanding Why Human Errors Occur

Industry practice is evolving to explore why people make the decisions they do by looking at the Organization’s systems. It’s usually a poor decision made somewhere in the error chain. We must believe that the person who made the poor decision did not intend for the error to occur. As part of CAPA investigations, we need to explore their physical environment as well; the conditions under which they make those decisions. The Training Program Improvement Checklist can be requested using this link to capture your findings.

If you are going to spend time and money on training, at least identify what the gap is; fix that and then “train” or provide awareness on what was corrected to prevent the issue from re-occurring again.   That is after all, the intention of *Corrective Action Preventive Action investigations. -VB

You may to explore these other library gems:

BandAidsPicture1
Why the Band Aids Keep Falling Off

OpErrorPicture1

With a little help from my Validation Colleagues – The Training Protocol

In the 17 June 2014 issue: http://hpiscblog.hpisconsulting.com/2014/06/learning-on-the-fly/ I blogged about an urgent learning need requiring instructor led classroom delivery that needed to be facilitated among a group of talented SMEs.  During the needs assessment portion, I hit a huge barrier.

“I teach GMP Basics and conduct Annual GMP Refreshers several times a year and preach to audiences that you must follow the procedure otherwise it’s a deviation.  And in less than two weeks, I am expected to teach a process that is changing daily!   Yet on the other hand, how could I teach a work instruction that is known to be broken; is being re-designed and not yet finalized?”

My dilemma challenged the essence of my “learned” compliance belief system about following the 1st basic GMP principle – “thou shall follow written procedures”!  The instructional designer side of me screamed – how can you teach flawed content?  That’s wasted training that results in scrap learning. How is that training going to be effective beyond a check in the box?

And then it hit me – validation engineers use protocols to capture their “change in process” work.  Whether it’s experimental batches, 3 batches for process validation or *IQ-OQ-PQ protocols for equipment qualifications.  They are validating the procedure or the new process before it can become the standard operating procedure by developing the plan, developing acceptance criteria, managing the unexpected deviations and capturing the results.  So why couldn’t I borrow the concept and adapt it to my situation?

While it was the intention of the business unit leader to deviate from the approved set of work instructions, a planned deviation would not be appropriate in this case.  The purpose of the training sessions was to test the new sequence of steps and confirm the robustness of the criteria to make correct decisions where needed.  The learners would still be in compliance with the quality policy document and would still meet the intention of the quality system regulation.  They were essentially testing the future “how-to steps” for the proposed new work instructions.

Now before you fire off a rant of emails to me, I did not copy and paste the validation protocol template.  I did however, include a “please pardon our appearance while we are under construction” paragraph in the training plan to document the departure from the current set of work instructions.  This protocol like section also included our intentions for the outcomes of the sessions and stipulated required SOP training of all affected users once the finalized set of work instructions were approved and went into effect.

Sometimes the very solution can be found around the next cubicle.  –VB

*Installation Qualification, Operational Qualification, Performance Qualification

“Learning on the fly” or is this what they meant by the Agile Learning Model?

When Rapid Design for E Learning found its way into my vocabulary, I loved it and all the derivatives like rapid prototyping.  And soon, I starting seeing Agile this and Agile that.  It seemed that Agile was everywhere I looked.  When Michael Allen published his book, LEAVING ADDIE for SAM, I was intrigued and participated in an ATD (formerly known as ASTD) sponsored webinar.  It made a lot of sense to me and “I bought into the concept”.  Or so I thought …

 

A few weeks back, I joined a project that was already in-progress and had to “hit the ground running to get caught up to speed”.  The element of urgency was the anticipation of a post FDA visit following a consent decree.   If you’ve experienced this “scene” before, you can relate to the notion of expedited time.   As part of remediation efforts, training events needed to be conducted.  I learned during a meeting sometime my first week, I was to be the trainer.  Okay, given my background and experience, that made sense.  Sure, in a few weeks when we have the new procedure in place, I’d be happy to put the training materials together, is what I was thinking.  Wait – in two weeks?  Are you kidding me?  I’m not the SME and I don’t even have the software loaded on my laptop yet.  Well, some cleaned up version of those words was my response.

 

But what about all that buzz for rapid design and prototyping I’ve been reading about?  In theory, I totally bought it.  But, this is different I argued with myself.  This is compliance with a quality system for a company who is undergoing transformative change as a result of a consent decree!  I teach GMP Basics and conduct Annual GMP Refreshers several times a year and preach to audiences that you must follow the procedure otherwise it’s a deviation.  And in less than two weeks, I am expected to teach a process that is changing daily!   Yet on the other hand, how could I teach a work instruction that is known to be broken; is being re-designed and not yet finalized?  Stay tuned for a future blog about how I overcame this dilemma.

 

My bigger issue was to get out of my own design way.  I’m classically schooled in *ADDIE and with 25+ years as an instructional designer, very comfortable with how to design, develop and deliver training.  All I needed was more time and it hit me!  I was so focused on what I needed, that I was missing the urgency of the learners’ needs.  It was time to put theory into practice and take the agile plunge into the domain of the unknown.

 

By shifting the prioritization away from perfectly designed classes with pristine training materials, I was able to diagnose that the need was to get the learners into a live classroom.   They needed to see the database software in action and “play in the sandbox”; the training materials could follow afterwards.  I shifted my role to facilitator and found the true SMEs to navigate the software screens and explain how to complete field transactions.  To my surprise and delight, trainer-wannabes volunteered to paste screen shots into participant worksheets so they could take notes.  I became a scribe and worked on sequencing these pages for the next round of attendees.  Together, we all collaborated to meet the urgent need of the learners.   And we documented it!  Once they had the tour and sand-box time, the learners were paired up with a buddy for guided instruction of real entry into the live system.  The following week, the department was able to go live with a project plan that focused on a series of interim roles, changed roles and transitioning responsibilities within established roles.  The project launched on time to meet commitments promised to the agency.

 

It was energizing and empowering for the learners. A truly collaborative experience for the SMEs and the biggest surprise of all was that they thanked me.  Me?  I did not deliver the training; I was not the SME nor did I provide perfect training materials.   If I had pursued my classically trained ADDIE technique, we’d still be waiting to deliver those sessions.  However, I’m not ready to throw ADDIE over board yet.  She has served me well and continues to be an appropriate technique in most of my training needed situations.

 

My lesson learned was this: when the need is for speed and the design is not the key focus, I need to give up control to the SMEs and Learners and focus on facilitating the best learning experience given the daily change challenges and system constraints.   Is this “learning on the fly” or agile learning in practice?  You decide.

 

*NOTE: ADDIE = Analyze, Design, Develop, Implement, Deliver – classic phases of Instructional Systems Design (ISD) Technique.