Please Tell Me There Are More Quality Tools Than Fish Bone and the 5 Whys!

Don’t get me wrong.  I am a huge fan of both these methods especially for training related errors.  However, there are a lot of other quality tools in the problem solving toolbox that aren’t as popularly used.  In this blog, I will continue part two of the previous blog, RCA Meetings vs. Problem Solving Discussions in which I presented an alternative problem-solving model that helps investigators take a deeper dive into understanding the problem first (#1), then asking a series of questions to generate possible causes (#2) that move from the event trigger to patterns to systems glitches.

So many tools, so little time!

Before Google became a verb, the best way to learn about a problem-solving tool was to ask someone, ask to borrow his or her “The Memory Jogger” pocket guide, or wait patiently to take your course.  Getting information at your fingertips instantly is no longer a barrier.  But the consequence of too many available tools can lead to confusion and indecision resulting in sticking to what is familiar instead. One way to mitigate the bewilderment is to choose these tools by which stage the investigation is in and whether the team needs to analyze the process or the collected data.

Where are you in the investigation?

Are you exploring and just learning more about the triggering event?  Are you generating possible causes and need an organized decision making technique?  Or are you verifying and eliminating causes?  Refreshing on the purpose of the tool helps the Lead Investigator make the right tool choice.  Consider the Job Aid: Which Tool to Use When?

WHAT IS THE ASSIGNABLE CAUSE?

Just because the team has generated more than one possible cause does not mean that one of them has to be the root cause by default.  Has the team asked the “second right answer”?  Charles “Chic” Thompson, in his book, What a Great Idea, explains that we’ve been trained to look for the right answer, to seek the one and only answer, ever since we started school.  Instead, he suggests that we ease “into a new frame of mind” by asking a host of questions in search of potential solutions.  Included in this perspective, is asking, “dumb” but penetrating questions that lead to finding “the true source of bottlenecks buried deep in habitual routines of the firm”.  

When the team has really explored all possible causes, then it’s time to switch tools and use data analysis tools such as histograms, run/control charts, and/or scatter diagrams.

“A fishbone represents opinions.  Opinions must be verified with facts before action to change is taken.”  

Waste Chasers: A Pocket Companion to Quality and Productivity, p.43.

Pareto charts can be very helpful when you need to display the relative importance of all of the conditions in order to identify the basic cause of the problem.  It helps the team to focus on those causes that will have the greatest impact if solved.  The team has to agree on the most probable cause aka the root cause.  One of the best decision making tools I’ve experienced is Nominal Group Technique.  NGT provides a way to give everyone in the group/team an equal voice in the selection.  Team members rank the items without being pressured by others.  Therefore, it builds commitment to the team’s choice through equal participation in the process.

What is the solution?

One of the most versatile tools in the arsenal is brainstorming.  It’s quick; it’s easy to administer and almost everyone has experienced a session or two.  We like it because it is an efficient way to generate a lot of ideas that are free of criticism and judgment when facilitated properly. 

At this point in the investigation, some members on the team finally get to share their ideas on how to solve the problem.  For teams pressed for time, having already made ideas sounds like a gift ready to be opened.  And yet, there can be a tendency to allow the eager problem solvers to dominate the discussion and solution path forward. 

A nice warm up exercise prior to the solution brainstorm, is to assign an individual idea map first.  By mapping individuals’ own thoughts first, they are not lost to the group’s brainstorming results when the time comes to participate in the session. The original ideas, thoughts, and even impulses can be easily shared since they have already been captured in their own idea map.

For teams stuck in the “same old way” mindset or wrestling with a frustrating repeat problem, try switching up the brainstorm trigger word.  For example, the trigger word “when solved” can help us work backwards.  Or use the opposite trigger word and brainstorm on ideas to make it true.  Try a random trigger word or a metaphorical trigger phrase such as (the problem …) is like … the day after a vacation when … .  I’ve seen amazing idea maps resulting from stepping up the brainstorming trigger words.  If the team is still stuck, compare all of the maps to look for commonalities or common themes.  And sometimes you just have to change your shoes or take a walk outside to shift your thoughts.

What is the best way to implement the solution?

Now, the team is responsible for bringing their ideas into reality aka the corrective actions / CAPA Plan.  One of my favorite tools is the Force Field Analysis when the team is working on corrective actions.  The tool helps team members to identify forces and factors that will support or work against the solution of an issue so that the positives can be reinforced and /or the negatives eliminated or reduced. It forces the team to think through all the aspects of making the desired change a permanent one. 

“After falling in love with your idea, you must also be prepared to fall out of love.”

Charles Thompson, What A Great Idea, p.161.

Now Just Do It

Implementing the corrective action(s) may be the longest duration depending upon the scope of the project and therefore needs to be managed using project management techniques.  And that could be the basis of another blog series. 

Close the Investigation

Before the investigation can be closed, the effectiveness check needs to be performed and evaluated as described in the CAPA plan. Over the course of the incident and its CAPA, the story evolves.  This is normal and expected especially as we learn more.  To an outsider reviewing the investigation report, the story and all the supporting evidence has to be understandable, even to someone internal to the organization but not involved in the investigation, the CAPA or the project. 

The Lead Investigator must review the entire file one more time to finalize the problem statement and confirm the cause statement given what is known at this final point of the investigation.  The following is a list of questions to ask:

  • Is the report easy to follow? Is there a logical flow?
  • Is the report free of unnecessary documentation? Is it uncluttered?
  • Is there sufficient information to back up the investigation, results and the outcome?
  • Does the report support decisions about product disposition?

Conclusion

Part One of this blog suggested the use of a more creative problem-solving approach to help us see our problems using systems thinking perspective.  Systems Thinkers understand complex relationships and their interdependencies. They step up and take responsibility to fix the problem.

In part two, I suggested different problem solving tools to use depending on the investigation stage and the team’s task.  As a Performance Consultant, the problem solving approach and the plethora of “quality” tools has me excited about identifying the true root cause and implementing systematic solutions as the corrective actions because we have to get better at solving our problems.  System Thinking,

  • puts a problem into a context of the larger whole with the objective of finding the most effective place to make an appropriate change and
  • it can help us identify and respond to a series of changes before those changes lead to more unwanted deviations and CAPAs. 

Isn’t that the purpose of the PA in CAPA, the preventive actions? -VB

Root Cause Analysis and Qualifying Investigators Coaching Briefs
  • Charles “Chic” Thompson, What A Great Idea: Key Steps Creative People Take, HarperPerennial, 1992.
  • The Memory Jogger: A Pocket Guide of Tools For Continuous Improvement, GOAL/QPC, 2nd 1988.
  • Waste Chasers: A Pocket Companion to Quality and Productivity, Conway Quality, Inc.

Conducting Root Cause Investigation Meetings vs. Facilitating Problem Solving Discussions

I began this Deviation and CAPA series with the intent to share what I call the human performance improvement (HPI) touch points; where HPI and the CAPA quality system have cross -over connections.  See The Big Why of Deviations and Why Do CAPAs Fail Their ECs.  In this two-part blog, I will reveal another noteworthy HPI crossover connection.

If you perform a Google search on CAPA process and the ensuing steps, the results are varied but in general, industry follows a routine process that looks like the following:

  • Compliant “CAPA” Program includes procedures for:
    • Discovery and Notification
    • Containment
    • Impact Assessment
    • Product Investigation
    • RC Investigation
    • Corrective Actions (CA)
    • Prevention Actions (PA)
    • Effectiveness Checks (EC)
    • Close

If you are satisfied with the results you are getting, great, more success to you!  One of the misconceptions about today’s CAPA process is that it suggests the steps start and end cleanly and the process moves in a linear fashion. I have one of those theory vs. practice moments where what is supposed to happen “if you correctly follow this process” does not happen in reality.  That somehow it’s the fault of the lead investigator if it’s not progressing as described. 

Deviations and CAPA investigations are not theoretical. As the investigation unfolds and the true story is discovered, we learn quickly that the process is really iterative.  Often we find ourselves back at the beginning, rewriting the event details and fine tuning the problem statement or re-examining the root cause because a repeat deviation occurred.  Frustrated that the clock is ticking and the pressure to get ‘em closed is always in the forefront of the investigation, there is no time allowed to think outside this process; no room for creatively solving the problems.  The mere mention of creativity makes some compliance professionals break out in hives. 

I get it.  The intention of the process is to provide a structured approach to conduct the investigation and manage the related CAPA when assigned.  Otherwise, it’s too overwhelming and unwieldy.  We could go down a rabbit hole and never close an investigation or come up with totally out of the question corrections.  However, it seems to me that we’ve ingrained these steps into routine habit, that our root cause investigation meetings have also become check the boxes to get the CAPA closed on time!  Hence, our industry is still getting “inadequate investigation” observations.  Could there be a causal relationship?

So, if you seek something else, I am proposing a problem-solving model below that mirrors very closely with CAPA.  The following problem-solving steps give us the freedom to consider new ideas and new possibilities for solving the same repeated problems without violating our Deviation and CAPA procedures.

DEFINING THE PROBLEM

It sounds simple enough to do.  Collect the event details.  Answer the universal questions.  And lo behold, the problem should be evident.  SMEs unwittingly suffer from the curse of too much knowledge and as they learn more about the event, they tend to form early hypothesis about the problem especially if they’ve seen a similar event.  They tend to collect evidence about problems they are familiar with and understand.  Unfortunately for most, they seldom go beyond their current knowledge base. 

In the Qualifying SMEs as Investigators Program, nominees are asked to discuss and prepare their response to the following question: Is the problem different from the cause?  Initially, everyone nods his or her head yes with a baffled look as to why further discussion is needed.  In their break out groups, I ask them to prepare their group rationale. And then it happens, a few will inadvertently slip in the cause with the problem explanation and to their amazement they are stunned at hearing these words.  Alas, that’s the point.  Before you can assign the root cause, you need to know what the problem really is.

WHAT DO I ASK TO FIND THE CAUSES?

“To find the answer, ask a lot of questions first”.  The discovery of a non-conformance, deviation or discrepancy is usually triggered by an event and thus the investigation begins here.  In the event stage, not only are investigators collecting details they also interview the individual performers who are involved.  However, there are three levels of interaction that can influence performance:

  • The individual performer level
  • The process level
  • The systems level.

To identify the possible causes not just the obvious one, the investigation team needs to move beyond just the event triggers.

Moving from Events to Patterns and Systems Influences

If we keep the scope limited to the event and performers involved “as an isolated incident”, we often find that the easy way out (aka the fastest way to close the investigation) will often lead back in with more repeat deviations.  So, we need to expand the investigation to include other individuals and /or other similar incidents to look for a pattern or a reoccurring theme.

(c) HPIS Consulting, Inc.

As investigators, we need to keep asking questions that enable information sharing to drive a deeper understanding of what the individuals were/are experiencing when performing their part of the tasks within the process. 

Mapping the task/process helps SMEs to re-trace the steps or sub-tasks rather then rely on their memory (their expertise reservoir).  Many SMEs think they know the procedure and will often state a step from an older version of the SOP or will be surprised to see new changes in the most current version of the procedure.  In this group review, SMEs can look for weaknesses and offer their insights about the point of discovery. 

Within this process analysis/review, ask are there other procedures that have been implicated in other deviations?  What about the hand offs; the linkages between the procedures and other quality systems?  Are these clearly defined and understood or are these the basis of “miscommunication” causes between other groups?  The big question to ask is what would happen if we left it alone? What other systems would be impacted?  When investigation teams are given the time to explore (investigate) enough, they often find these system glitches just waiting to contribute to another deviation.

Problem Solving ala Systems Thinking

Peter Senge in his 1990 ground breaking book, The Fifth Discipline: The Art and Practice of the Learning Organization, describes a system as being perceived whole whose elements “hang together” because they continually affect each other over time and operate toward a common purpose.  There’s a pattern of interrelationships among the key components of the system, including the ways decisions are made.  Often invisible until someone points them out.

Systems thinking allows problem solvers to see the event in the context of the whole system not just individual performers or procedure.  They see patterns of recurring problems that deepen their understanding of the situation and leads them to reframe the problem, thus providing a new framework from which to generate an additional set of possible causes.

The next blog will continue describing the problem-solving model and explore other quality tools investigation teams can use. – VB

Subscribe (at the left side bar menu) now so you don’t miss part two of this blog.

(c) HPIS Consulting, Inc.

Investigations 101: Welcome Newbies

So the event description is clarified and updated. The assigned investigator is up to speed on the details of “the story”.  What happens next? What is supposed to happen?  In most organizations, there is rush to find the root cause and get on with the investigation.  A novice investigator will be anxious to conduct the root cause analysis (RCA).  S/he can easily make early root cause mistakes like grabbing the first contributing factor as the root cause without being disciplined to explore all possible causes first.

Thus it makes sense to get the Investigators trained in root cause analysis. Unfortunately for many, this is the ONLY training they receive and it is not nearly enough. RCA is a subset of the investigation process and the training agenda is heavy on the tools, which is perfectly appropriate.  But when do they receive training on the rest of the investigation stages like determining CAPA significance and writing the report?  Given the amount of FD-483 observations and warning letter citations for inadequate investigations that continue to be captured, I’d say we need more training beyond RCA tools.  As a result, we are starting to see FDA “recommendations” for trained and QUALIFIED Investigators.  This means not only in how to conduct a root cause analysis, but also the Deviation and CAPA Process. 

This goes beyond e-sign the Read and Understand procedures in your LMS

E-Doc systems are a great repository for storing controlled documents.   Searching for SOPs has become very efficient.  In terms of documenting “I’ve read the procedure”, very proficient and there’s no lost paperwork anymore!  But learning isn’t complete if we’ve merely read through the steps.  We also need to remember it.  At best, we remember that we read it and we know where to find it when we need to look something up.  Does that translate to understood?  Maybe for some. 

To help us remember the actual steps, we need to do something with the knowledge gained.  This is where the responsibilities section of the procedure tells us who is to do what and when.  But the LMS doesn’t include structured and guided practice as part of the assigned curricula.  Unless your equipment and complex procedures are also flagged for Structured OJT and possible Qualification Events as in most Operations groups, practice happens incidentally as part of on the job experience.  Feedback is typically provided when there’s a discrepancy or a deviation.  This is reactionary learning and not deliberate practice. 

If we want Deviation Investigators to understand and remember their tasks (procedures) so they can conduct investigations and write reports that get approved quickly, then we need to design learning experiences that build those skills and ensures accurate execution of assigned roles and responsibilities for Deviations and CAPAs. They need an interactive facilitated learner centered qualification program.

More than just studying a set of procedures and filling out related forms

It’s about putting the learners; the assigned SMEs as Investigators and QA Reviewers, at the center of the whole learning experience.  It’s about empowering them to take charge of their own learning by enabling them to experience real work deviations / CAPA investigations and to deliberately practice new skills in a safe environment with the assistance of adult learning facilitator(s) and coaches.  Thereby bridging the “R & U Only Knowledge Gap”.

The look and feel of the program follows a Learn By Doing approach with customized learning content, using interactive techniques and offering more hands-on opportunities for them to engage with real work application that ensures learners are immediately using the knowledge and tools in class and for their homework assignments thus increasing the connections for knowledge transfer.

This requires a shift from the traditional mindset of a classroom course where the emphasis is on the expertise of the instructor and the content. The learners and their learning experience becomes the priority.  The instructor’s task isn’t to deliver the content, it’s to help their learners acquire knowledge and skill.

Shifting the priority to a more engaging Learning Experience

Qualifying SMEs as Deviation Investigators Program

This unique curriculum uses a variety of teaching methods fostering more balanced and meaningful instruction over the duration of the program.  It is not a single course or 2-day training event.  It is delivered in modules, with weekly “homework” assignments consisting of active deviations and open investigations.

“Spaced learning works, in part, because the brain needs resting time to process information, create pathways to related information, and finally place the new information into long-term memory – the main objective of learning.” (Singleton, Feb 2018, p.71). 

Each module revisits the Investigation Stages and builds on the prior lessons by reviewing and debriefing the homework.  Then, expanding on that content and including new lessons with increasing intensity of the activities and assignments.

By design, the program provides time and space to interact with the content as opposed to delivering content dumps and overwhelming the newbies; short-term memory gets maxed out and learning shuts down. The collaborative participation and contributions from the Investigators and Program Facilitator(s) result in better overall engagement. Everyone is focused on accomplishing the goal of the program; not just checking the box for root cause analysis tools.

The goal of the program is to prepare subject matter experts to conduct, write and defend investigations for deviations and CAPAs.  The program also includes QA reviewers who will review, provide consistent critique and approve deviations, investigations, CAPAs. Attending together establishes relationships with peers and mutual agreement of the content.  The learning objectives describe what the learners need from the Deviation and CAPA quality system procedures while the exercises and assignments verify comprehension and appropriate application.

“Learning happens when learners fire their neurons, not when the trainer gives a presentation or shows a set of Power-Point slides.” (Halls, Feb 2019, p.71).

Qualified, really? Isn’t the training enough?

Achieving “Qualified status is the ultimate measure of the training program effectiveness.  For newly assigned Investigators, it means the company is providing support with a program that builds their skills and confidence and possible optional career paths.   Being QUALIFIED means that Investigators have undergone the rigor of an intensely focused investigations curriculum that aligns with the task and site challenges.  That after completing additional qualification activities, Investigators have experienced a range of investigations and are now deemed competent to conduct proper investigations.

For the organization, this means two things.  Yes, someone gets to check the FDA commitment box.  And it also means strategically solving the issues.  Better investigations lead to CAPAs that don’t fail their effectiveness checks.  Now that’s significant performance improvement worthy of qualifying Investigators!  -VB

References:

  • Campos,J. The Learner Centered Classroom. TD@Work, August, 2014, Issue 1408.
  • Chopra,P. “give them what they WANT”, TD, May, 2016, p.36 – 40.
  • Halls,J. “Move Beyond Words to Experience”, TD, February, 2019, p. 69 – 72 DL.
  • Parker, A. “Built to Last: Interview with Mary Slaughter”, TD, May, 2016, p. 57.
  • Singleton, K. “Incorporating a Spiral Curriculum Into L&D”, TD, February, 2018, 70 – 71.

HPISC Coaching Brief available here.

The Investigation Hand-off: Is it a handshake or missed step?

Upon discovery and containment of a deviation or non-conformity, the investigation begins.  Whose responsibility is it, the one who discovers it or the one who gets assigned to investigate it?  In most organizations, it is not the one who discovers the performance discrepancy.  It usually gets assigned to an SME or a dedicated investigator if QA has the resources. 

The hand-off between the discoverer and the assigned investigator is not always clearly defined and is often vaguely described in the CAPA procedures. And here is where the first misstep has potential to delay the start of the investigation clock. In this third blog of the Deviation and CAPA series, I intend to argue for the importance of the “hand-off” especially if investigations are assigned to SMEs and staff who were not present during initial discovery, containment and notification activities.

Making the Hand-off Formal

The purpose of the hand-off is to acknowledge responsibility for the investigation and review all actions taken up to the current point in time.  Given the nature of the event, the significance and urgency to move forward, there can be a lot of confusion, misinformation and misunderstanding of roles, responsibilities and authority.

One misconception that can occur with novice investigators is the lack of awareness that the investigation details can change and should be revised as the investigation unfolds especially the event description.

Please Tell Me the Story (again)

There are 3 main elements in every “story”.  Usually a cause and effect flow to the story unfolds and becomes the basis of the narrative that is told or captured in the investigation form.   Next are the key variables that reveal the impact.  The trick is to recognize and capture them and not get bogged down or distracted in the extra details.  And then, there are the two sides of every story.  The investigators job is to find the similarities among the interviews especially if there has been a long chain of events occurring before the hand-off.

A second misconception with new investigators is to park all the information in the investigation form as possible drafts and then revise and revise and revise again until they are finished.  Alternatively, they could use a tool that is designed to capture and review the details for the event.

The Investigators Hand-Off Tool – Tell Me the Story

The tool is essentially a worksheet to capture, review and collate the answers to the “Universal Problem Solving Questions”. 

  • What Happened?
  • Where did it happen?
  • Who was involved?
  • When did it happen?
  • How did it happen?
  • How big is the scope of the problem?

Notice that “why did it happen” is not included. How it happened can easily morph into why it happened if not carefully checked.  At this stage of the investigation, root cause analysis is way too early and can misguide the investigation.  Investigators in a rush to get their assignments done, need to stay true to the CAPA process and confirm the event details and then finalize the problem as they know it; more about problem statements in a future blog. 

“How it happened” is more about how it was found and how it was discovered?  “Where it happened” is seeking clues about a breakdown in the process, the procedures, paperwork and/or batch records.  Something happened somewhere and the investigation has to first identify what and where before the why.  Having these questions answered accurately allows for a richer probable cause identification step and eventual determination of the true root cause(s). 

Until the next blog is released, I have a homework assignment for you.  Reread your Deviation and CAPA SOPs specifically looking for the roles and responsibilities for the Initiator and the Investigator.  Are they clearly written? Ask your assigned Investigators (if this applies at your site) if they follow a hand-off guide or use any kind of fact-finding/event description worksheet.  If you are interested in the HPISC Tell Me the Story worksheet, you can find it here.  May all your CAPAs map back to the true root causes.  -VB

Why Do CAPAs Fail Their Effectiveness Checks?

When we start talking about deviations and CAPAs, we can’t help having a sidebar discussion about root causes and more specifically the rant about finding the true root cause.  I intentionally skipped that content in the previous blog.  It was my intention to kick off the new Deviation and CAPAs blog series by first looking at deviations by themselves.  And the learning opportunities deviations can provide us about the state of control for our quality systems.  From those deviations and ensuing CAPA investigations, I ask you this: are we improving anything for the long term (aka prevention).  Are we making any progress towards sustaining those improvements?

Corrective Actions Preventive Actions (CAPA) Steps

Let’s step back a moment and quickly review typical steps for CAPAs:

CAPA Components

The purpose of an Effectiveness Check (EC) is for verifying or validating that actions taken were effective and do not adversely affect product, device or process.  It goes beyond the statement in the investigation form to include a follow-up activity that closes the loop on the specific CAPA.  If an effectiveness check fails meaning the CA/PA was not effective or another deviation /nonconforming incident has occurred, we go back to the beginning and either start again or in most cases, we re-open the investigation.  The pressing question is why did the EC fail?  Almost instinctively, we believe that we did not find the true root cause.  Perhaps.  Was there a rush to close the investigation?  Probably.  Did the investigation team grab the first probable cause as the root cause because the “problem” felt familiar?  Maybe. Or is it a case of a fix that backfired into unintended consequences? Possibly. I will also suggest that the CA/PA may not have been aligned properly.

Ask these 3 questions about CA/PAs

  • Is the CA/PA Appropriate? The focus of this question is about the affected people.  What is the size of this audience? Is it mainly one person or groups of people?

Can the CA/PA be executed efficiently?  Is it for one site or multiple sites?

  • Is the CA/PA Economical? What budget is available?

Is it a “cheap” fix or a 3 – 6 month project? Or an expensive solution of more than 6 months and will need capital expenditure funding?

  • Is the CA/PA Feasible? The real question is about the timeline.

            Need it fast – within 3 months or

            Have time – don’t need until more than 3 months from now.

And then there is the unspoken 4th question – is the CA/PA “political”?  I experienced first hand what happens to CAPAs that are politically oriented.  Most of them failed their ECs.  Request “Can You Stay a Little While Longer”. The best CAPAs are the ones that map back to the root cause.

Introducing the HPISC CAPA Performance Chain

On the left hand side, you will recognize the 3 traditional tasks to complete.  After the EC is written, trace upwards to ensure that the EC maps back to the CA/PA and that the CA/PA maps back to the root cause; hence, the bottom up arrow.  On the right hand side are performance improvement activities that I use as a Performance Consultant (PC) to bring another dimension to the CAPA investigation, namely, Human Performance Improvement (HPI). 

Before I can write the root cause statement, I examine the “problem” also known as a Performance Discrepancy or an incident and I conduct a Cause Analysis that forces me to take a three tiered approach (the worker, the work tasks, the workplace) for the possible causes and not get bogged down in observable symptoms only.  The Performance Solution is more appropriately matched to the identified gap. In theory, this is what the corrective action(s) is supposed to do as well. During the performance solution planning, determination of success and what forms of evidence will be used happens with key stakeholders.  So that collecting the data happens as planned, not as an after thought, and the effectiveness is evaluated as discussed.    

What can we really control?

In RCA/CAPA meetings, I often hear about what management should do to fix the working conditions or how most of the operator errors are really managements’ fault for not taking the culture factor seriously enough.  While there may be some evidence to back that up, can we really control, reduce or eliminate the human factor?  Perhaps a future blog on understanding human errors will be released.

Management Can:

  • Design work situations that are compatible with human needs, capabilities and limitations
  • Carefully match employees with job requirements
  • Reward positive behaviors
  • Create conditions that optimize performance
  • Create opportunities to learn and grow professionally.

Clues for Failed Effectiveness Checks

One of the first activities to perform for a failed EC is to evaluate the effectiveness check statement.  I have read some pretty bizarre statements that challenge whether the EC was realistic to achieve at all. The conditions under which we expect people to perform must be the same as the conditions we evaluate them during an EC review.  So why would we set ourselves up to fail by writing ECs that don’t match normal workplace routines? What, because it looked good in the investigation report and it got the CAPA approved quicker?

Next, trace back each of the CAPA tasks to identify where to begin the re-investigation.  I also suggest that a different root cause analysis tool be used. And this is exactly what we did while I was coaching a cohort of Deviations Investigators.  Future blogs will discuss RCA tools in more detail. -VB

The Big Why for Deviations

As part of my #intentionsfor2019, I conducted a review of the past 10 years of HPIS Consulting.  Yes, HPISC turned 10 in August of 2018, and I was knee deep in PAI activities.  So there was no time for celebrations or any kind of reflections until January 2019, when I could realistically evaluate HPISC: vision, mission, and the big strategic stuff.  My best reflection exercise had me remembering the moment I created HPIS Consulting in my mind.

Human Performance Improvement (HPI) and Quality Systems

One of the phases for HPI work is a cause analysis for performance discrepancies.  The more I learned how the HPI methodology manages this phase the more I remarked on how similar it is to the Deviation /CAPA Quality System requirements.  And I found the first touch point between the two methodologies.  My formal education background and my current quality systems work finally united.  And HPIS Consulting (HPISC) became an INC.  

In my role of Performance Consultant (PC), I leverage the best techniques and tools from both methodologies.  Not just for deviations but for implementing the corrective actions sometimes known as HPI solutions.  In this new HPISC blog series about deviations, CAPAs, and HPI, I will be sharing more thoughts about HPISC touch points within the Quality Systems. For now, lets get back to Big Why for deviations.

Why are so many deviations still occurring? Have our revisions to SOPs and processes brought us farther from a “State of Control”? I don’t believe that is the intention. As a Performance Consultant, I consider deviations and the ensuing investigations rich learning opportunities to find out what’s really going on with our Quality Systems.

The 4 cross functional quality systems

At the core of the “HPISC Quality Systems Integration Triangle” is the Change Control system.  It is the heartbeat of the Quality Management System providing direction, guidance and establishing the boundaries for our processes.  The Internal Auditing System is the health check similar to our annual physicals; the read outs indicate the health of the systems.  Deviations/CAPAs are analogous to a pulse check where we check in at the current moment and determine whether we are within acceptable ranges or reaching action levels requiring corrections to bring us back into “a state of control”.  And then there is the Training Quality System, which in my opinion is the most cross-functional system of all.  It interfaces with all employees; not just the Quality Management System.  And so, it functions like food nourishing our systems and fueling sustainability for corrections and new programs.

Whether you are following 21CFR211.192 (Production Record Review) or ICHQ7 Section 2 or  820.100 (Corrective and Preventive Action), thou shall investigate any unexplained discrepancy and a written record of the investigation shall be made that includes the conclusion and the follow up. Really good investigations tell the story of what happen and include a solid root cause analysis revealing the true root cause(s) for which the corrective actions map back to nicely.  Thus, making the effectiveness checks credible. In theory, all these components flow together smoothly.  However, with the continual rise of deviations and CAPAs, the application of the Deviation /CAPA Management system is a bit more challenging for all of us.  

Remember the PA in C-A-P-A?

Are we so focused on the corrective part and the looming due dates we’ve committed to, that we are losing sight of the preventive actions? Are we rushing through the process to meet imposed time intervals and due dates that we kind of “cross our fingers and hope” that the corrective actions fix the problem without really tracing the impact of the proposed corrective solutions on the other integrated systems? Allison Rossett, author of First Things Fast: a handbook for performance analysis, explains that performance occurs within organizational systems and the ability to achieve, improve and maintain excellent performance, depends on integrated components of other systems that involve people. 

Are we likewise convincing ourselves that those fixes should also prevent re-occurrence? Well, that is until a repeat deviation occurs and we’re sitting in another root cause analysis meeting searching for the real root cause.  Thomas Gilbert, in his groundbreaking book, Human Competence: engineering worthy performance tells us, that it’s about creating valuable results without using excessive cost.  In other words, “worthy performance” happens when the value of business outcomes exceeds the cost of doing the tasks.  The ROI of a 3-tiered approach to solving the problem the first time, happens when employees achieve their assigned outcomes that produce results greater than the cost of “the fix”. 

Performance occurs within three tiers

So, donning my Performance Consulting “glasses”, I cross back over to the HPI methodology and open up the HPI solutions toolbox.  One of those tools is called a Performance Analysis (PA). This tool points us in the direction of what’s not working for the employee, the job tasks a/or the workplace. The outcome of a performance analysis produces a 3 tiered picture of what’s encouraging or blocking performance for the worker, work tasks, and/or the work environment and what must be done about it at these same three levels.  

Root cause analysis (RCA) helps us understand why the issues are occurring and provides the specific gaps that need fixing.  Hence, if PA recognizes that performance occurs within a system, then performance solutions need to be developed within those same “systems” in order to ensure sustainable performance improvement.  Otherwise, you have a fragment of the solution with high expectations for solving “the problem”.  You might achieve short-term value initially, but suffer a long-term loss when performance does not change or worsens. Confused between PA, Cause Analysis and RCA? Read the blog – analysis du jour.

Thank goodness Training is not the only tool in the HPI toolbox!   With corrective actions /HPI solutions designed with input from the 3 tiered PA approach, the focus shifts away from the need to automatically re-train the individual(s), to implementing a solution targeted for workers, the work processes and the workplace environment that will ultimately allow a successful user adoption for the changes/improvements.   What a richer learning opportunity than just re-reading the SOP! -VB

  • Allison Rossett, First Things Fast: a handbook for Performance Analysis; 2nd edition 
  • Thomas F. Gilbert, Human Competence: Engineering Worthy Performance
You might want to also read:

How committed to SOJT is your organization?

In this 2019 refreshed SOJT blog series, I’ve explored why curricula alone is not enough to call your OJT structured. And I shared what else needs to be incorporated into your OJT program to move from traditional OJT to SOJT.  These are all necessary components to provide direction and tools for the Qualified Trainer to deliver required training.  But what about management support for the QT’s and for delivering SOJT?  If only we were required to have a procedure for that! 

ICH Q10 – Pharmaceutical Quality System

While not mandatory, management needs to seriously take notice of ICHQ10 guidance document released in April 2009 (1).  In particular to the following:

  • MANAGEMENT RESPONSIBILITY 2.3 Quality Planning 

“(d) Management should provide the appropriate resources and training to achieve the quality objectives”.

  • “CONTINUAL IMPROVEMENT OF THE PHARMACEUTICAL QUALITY SYSTEM

4.3 Outcomes of Management Review and Monitoring

The outcome of management review of the pharmaceutical quality system and monitoring of internal and external factors can include:

(b) Allocation or reallocation of resources and/ or personnel training.”

Under a Quality System: Managers Expectations for Training

Referencing the Sept. 2006 issue of Guidance Document for Quality Systems: IV. Management Responsibilities | B. Resources (2), we see alignment with the training CGMPs for continued training so “as to remain proficient in their operational functions and in their understanding of CGMP regulations.”  “Typical quality systems training should address the policies, processes, procedures, and written instructions related to operational activities, the product/service, the quality system, and the desired work culture (e.g., team building, communication, change, behavior).”

And my personal favorite, “When operating in a robust quality system environment, it is important that managers verify that skills gained from training are implemented in day-to-day performance.”  The responsibility for training under a quality system is not assigned to just one person or one function.  It is a shared responsibility across the entire organization.

Goals of the Train-the-Trainer Program (TTT) vs. OJT Program vs. Employee Qualification Program

In order to have qualified employees, they need to receive structured on the job training delivered by a qualified trainer who is content qualified and training process qualified via a TTT course.   Each of the three “programs” has defined outcomes that are dependent upon each other.  Unfortunately, the term program has been a bit overused throughout the years and can have a variety of meanings for folks.  For purposes of this blog, TTT program means OJT Qualified Trainers workshop, OJT Program means OJT methodology / procedure and Qualified Employee program means having a robust quality training system beyond the newest Learning Management System (LMS).

As you can see from the diagram, Qualified Trainers are at the core for all three “programs”.   I’ve previously blogged and written several resources on what’s involved.  The OJT QT Workshop is designed to prepare the QT’s for the realities of life as a Qualified Trainer. It includes the pre-work of familiarizing themselves with the training procedures and then in class: exploring basic learning theory and committing to what the QT signature means. The OJT Program is about delivering structuring on the job training consistently following the approved methodology as discussed in the previous blog.  An Employee Qualification Program is the ultimate validation of training effectiveness of the OJT methodology and the clarity of the underlying procedures.

Readiness Factors for SOJT and an Employee Qualification Program

Let’s start with a written purpose statement for having qualified employees beyond it’s required.  What is the company’s philosophy on achieving qualified status?  Is there agreement among the leadership for the level of rigor required to demonstrate performance and achieve a pass rating?  Where are the OJT program goals written?  Does a schedule exist for OJT and qualification events other than a LMS printout with required due dates.  That is not a schedule for OJT.  Do you have clearly defined objectives for the QT workshop captured in a document or perhaps a procedure?  Is there a single owner for all three programs or is responsibility and accountability assigned accordingly? Check out the HPISC Readiness Factors for SOJT/RTS/Employee Qualification Checklist.

Ronald Jacobs and Michael Jones, in their 1995 ground breaking book, Structuring on-the-Job Training, inform us that SOJT as a system functions within a larger context, namely the organization.  SOJT is not a standalone program.  Conflicts, competing priorities and mixed messages can influence the success of your SOJT program.  What else is going on in the organization that will compete for the same set of QTs? Remember they are also your most experienced and technical subject matter experts.  How is the overall Employee Qualification program aligned with the other quality systems? 

Recognition for QTs and Qualified Status

Most QTs are not fully dedicated to delivering training for departments.   There are pros and cons for this decision.  For now, I will leave them out.  Suffice it to say, they are tasked with both their “day” job and the responsibility for delivering training when needed.  They are at times, doing two jobs.  Whether or not they are compensated additionally for delivering OJT, acknowledging their contribution to the department and the organization is part of management support.   It takes more than “you are doing a good job, keep it up”. 

Often supervisors and managers don’t know what else they are supposed to do to show their support, other than allow them to attend the QT workshop.  The interested ones will “pop” in during lunch and chat with their direct reports.  Others will show up at the end for the poster activity (equivalent to a written test) and some will come to learn about the parking lot issues that need follow up.  The energy in the room when this happens is amazing. 

To help ease the knowledge gap between a manager and their now Qualified Trainer, I started delivering the Leadership Briefing module prior to the QT workshop delivery.  The purpose is to provide an overview of the content highlights, alignment with initiatives / CAPAs/ agency commitments and more importantly to secure agreement for the following:

  • criteria for nominating a QT
  • roles and responsibilities of QT
  • scope of work QT’s can be assigned
  • expectations for QT’s post launch
  • what happens day one after workshop is done
  • what is the status of the SOJT checklists
  • scheduling and budget concerns.

If the organization says they support the qualification program, then what happens when employees achieve qualification status?  Nothing?  A nonevent? Or is it announced in newsletters, plasma screens and other company announcements?  Is it a big deal to be able to perform independently and free up a much-needed QT for another learner?  I keep hearing over and over again about how there aren’t enough QT’s to deliver OJT the right way.  One would think qualification status on SOPs, especially big complex processes deserves SOME kind of recognition.   Just how committed are the managers and supervisors?  QTs and employees draw their own conclusions about the level of real management support for the programs.  

If truth be told, after launching many supervisors privately don’t support the program.  They lose their top performers for the workshop participation and during the hours it takes to train someone. Forget about giving QTs adequate time to complete the paperwork properly! And then leadership wonders why good documentation practice (GDP) issues continue to be a problem? The non-distracted performance observations that QT’s are expected to conduct for the qualification demonstration drive supervisors and line managers crazy the most – what, they can’t do anything else but observe?  Hence, many QT’s are asked to multi task just to get the work done: not enough resources they are told.  For supervisors, productivity and the workload will always trump OJT and qualification events, until their bonuses include completion of training and qualification events.

What Real Support Is Supposed To Look Like

My key take away message is that attending the TTT program is not the end of the OJT program or the Employee Qualification Program but rather the launching point.  Management support needs to go beyond just nominating QTs and allowing them to participate in the workshop.  The real support is in the alignment of goals, clarifying expectations continuously, allocating resources for training and budgeting time to deliver OJT using an approved OJT methodology that includes qualification events.  This commitment of time and sponsorship for qualified employees is a culture shift for many line managers and site leaders.  But actions do speak louder than words.  -VB

Alas, we have come to the end of this SOJT blog series. For a recap, visit the individual blog links below:

(1) Guidance for Industry Q10 Pharmaceutical Quality System | US Department of HHS | FDA
| CDER | CBER April2009 ICH

(2) Guidance for Industry Quality Systems Approach to Pharmaceutical CGMP RegulationsU.S.Department of HHS | FDA | CDER | CBER | CVM | ORA| September 2006 Pharmaceutical CGMPs

Jacobs RL, Jones MJ. Structured on-the-job training: Unleashing employee expertise in the workplace.  San Francisco: Berrett-Koehler: 1995.

Is Documenting Your OJT Methodology Worth It?

The short answer is yes!  In this blog post, I intend to share with you how I came to this answer and to make the case for you to also say yes.  I will also explore the challenges of documenting OJT as promised in a previous blog post.

Years ago, during the Qualified Trainers (QT) workshop, I would ask QTs the following two questions:

  • What is OJT?
  • How do you deliver OJT?

Invariably, all would answer the first question the same way: on the job training.  And then I would ask the attendees to form into groups to discuss the second question among fellow peers.  I purposely mixed the groups so that there was equal representation of manufacturing trainers and QC analytical laboratory trainers and a fascinating exchange occurred between the attendees.  During the debriefing activity, we learned that there was a lot of variability in how the trainers were conducting OJT.  How can that be when they all answered the first question so consistently? “Well”, one of them said, “we don’t have a procedure on it so I just go about it the way I think it should be done.  I’ve had success so far, so I keep on doing what I’ve done in the past.”

How many ways is there to train on this procedure?

In the blog post, “When SMEs have too much secret sauce”, I share the story about a Director of Operations who had to find out from an FDA Investigator, that his SMEs were teaching techniques for a critical process procedure that 1) were not written down nor were they approved (aka their secret sauce) and 2) were not at all consistent with each other.  Which lead to a FD-483 observation, a high visibility corrective action project with global impact and a phone call to HPISC.

In order to get consistent OJT, you need to define the process and you need to approve the content from which QTs will be using to deliver OJT.  I’m not proposing a cookie cutter approach for QTs to become all the same.  I am advocating a clear distinction between each step / stage / phase so that both the learner and the QT know exactly where they are in the process, what is expected of them in that step and why it is needed.  This is no longer just go follow Joe or Jane around which is how traditional OJT happened in the past.  

Declaring your OJT Model

I’m less focused on what you call these steps or how many there are.   I am looking to see how these steps move a new hire along the journey to becoming qualified prior to being released to task.  For me, this is what makes OJT really structured.  And that model needs to be captured in a standard operating procedure or embedded in a training procedure so that all employees are informed and aware of how they will receive their OJT.  The last step has to include the final evaluation of performance, not to be confused with demonstrating proficiency as in practice sessions. 

How many times does a learner have to practice an SOP before s/he is ready for the qualification event (Q-Event)? 

The nature of the SOP or the complexity of the task at hand determines this.  But, how do I proceduralize that, you ask?  It starts by not arbitrarily picking the magic number 3.  I have engaged in countless discussions regarding the exhaustive list of exceptions to forcing the rule of 3 times to practice.  And some QT’s will argue for more than 3 sessions especially when the procedure is so infrequently performed.  It’s not about the number of times, folks. 

Documenting OJT sessions presents a challenge for many trainers and document control staff.  Are we required to capture every OJT session or just one?  What is considered an OJT session?  My favorite lament is – “Do you know what that will do to our database not to mention the amount of paperwork that would create!” A workaround to all these questions and concerns is to capture at least one session along the progression of each OJT step as per the OJT Model, thus documenting adherence to the procedure.  For example, the first step is to Read SOP 123456.  As mentioned in other HPISC blogs and white papers, we are pretty good at this already.  Then, the next step is to discuss / introduce the SOP, so capture when that discussion occurred if it’s different from Step 1 READ.  The “trainer demonstrates” portion can also be captured.  Where it gets tricky is when we ask the learner to demonstrate and practice.  Why not capture the last instance when it is confirmed the learner is ready to qualify?  If we keep it simple and document that our learners have experienced each step /stage, then we are complying with our OJT methodology and minimally documenting their OJT progression.

Is one qualification session enough to pass?

At some point in these documentation discussions, we have to let the QT evaluate the outcome of the learner’s demonstration(s).  Does the performance meet “business as usual” expectations?  If it does, the learner is ready to qualify in order to perform independently.  If not, feedback is provided and the learner undergoes more practice.  How many times is enough? Until both the learner and the QT are confident that s/he is not going to have an operator error deviation a week after going solo.  The QT is ultimately the one who has to assess progress and determine that “with a few more sessions”, my learner will get this or no, s/he may never get it and it’s time to have a discussion with the supervisor.

How do you know if someone is qualified to perform task?

Ideally, the answer would be because we can look it up in our LMS history.  And that of course depends on how well the critical steps and behaviors are captured in the documentation tool. The tool is supposed to help the QT be as objective as possible and consistently evaluate performance as demonstrated. In the article, “A Better Way to Measure Soft Skills”, author Judith Hale explains the difference between a checklist and a rubric.

            “Checklists only record whether a behavior occurred, though, and not the quality of the behavior.  Rubrics, on the other hand, measure how well the learner executed the behavior.”  p. 62.

Yes, but was it tolerable, adequate or exemplary?

What I typically see are checklists with varying levels of tasks, steps and/or behaviors with a column for Yes, No and Comments.  What I don’t see is a column to mark how well the learner performed!  Is it enough to mark Yes or No for each item since most Q Events are Pass or “needs more practice”?  Maybe.  But consider the following situation.  A human error deviation has occurred and the LMS indicates the technician has earned qualified status.  The document used to qualify this individual shows all Yeses.  Yes s/he was able to demonstrate the step, critical task, and/or behavior, but what we don’t know is how well?  Are we to assume that No means “No, not at all” and Yes means performed “Well” or it is “As Expected” or “Adequate” or maybe in this case it was “Sort of”? 

An additional column describing what it means to score low, medium, high or in our situation: Poor, Adequate, As Expected, and even Exemplar could provide much needed insight for the root cause analysis and investigation that will follow this deviation.  It provides a level of detail about the performance that goes beyond Yes, No, or Comment.  In most checklists I’ve reviewed, the comments column is hardly ever used.

In future posts, I will blog about what the QT signature means.  Until then, is documenting your OJT methodology worth it?  What is your answer? – VB  

Hale,JA Better Way to Measure Soft Skills, TD, ATD, August, 2018, pps. 61-64.

Curricula Creation and Maintenance are NOT a “one and done event”!

Over the last few weeks, I’ve been blogging about structured on the job training (SOJT) with refreshed insights from my Life Sciences consulting projects.  Allow me to have a sidebar blog about the need for keeping curricula up to date.  I realize the work is tedious even painful at times.  That’s why donuts show up for meetings scheduled in the morning, pizza bribes if it’s lunchtime and quite possibly even cookies for a late afternoon discussion.  So I get it when folks don’t want to look at their curricula again or even have a conversation about them. 

Once a year curricula upkeep?

It’s like having a fichus hedge on your property.  If you keep it trimmed, pruning is easier than hacking off the major overgrowth that’s gone awry a year later.  And yet, I continue to get push back when I recommend quarterly curricula updates.  Even semi-annual intervals are met with disdain.  In the end we settle for once a year and I cringe on the inside.  Why?  Because once a year review can be like starting all over again.

Don’t all databases know the difference between new and revised SOPs?

Consider for a moment the number of revisions your procedures go through in a year.  If your learning management system (LMS) is mature enough to manage revisions with a click to revise and auto-update all affected curricula, then once a year may be the right time span for your company. 

Others in our industry don’t have that functionality within their training database.  For these administrators, revisions mean manual creation into the “course catalog” each time with a deactivation/retirement of the previous version; some may be able to perform batch uploads with a confirmation activity post submission.  And then, the manual search for all curricula so that the old SOP number can be removed and replaced with the next revision.  Followed by a manual notification to all employees assigned to either that SOP or to the curricula depending on how the database is configured.  I’m exhausted just thinking about this workload.  

Over the course of a year, how many corrective actions have resulted in major SOP revisions that require a new OJT session and quite possibly a new qualification event?  What impact do all these changes have on the accuracy of your curricula? Can your administrator click the revision button for these as well?   And then there’s the periodic review of SOPs, which in most companies is two years.  What is the impact of SOP’s that become deleted as a result of the review?  Can your LMS / training database search for affected curricula and automatically remove these SOPs as well? 

The Real Purpose for Curricula

Let’s not lose sight of why we have curricula in the first place.  So that folks are trained in the “particular operations that the employee performs” (21CFR§211.25).  And “each manufacturer shall establish procedures for identifying training needs and ensure that all personnel are trained to adequately perform their assigned responsibilities” (21CFR§820.25).  Today’s LMSes perform reconciliation of training completion against curricula requirements.  So I’m grateful that this task is now automated.  But it depends on the level of functionality of the database in use.  Imagine having to manually reconcile each individual in your company against their curricula requirements.  There are not enough hours in a normal workday for one person to keep this up to date!  And yet in some organizations, this is the only way they know who is trained.  Their database is woefully limited in functionality.

The quality system regulation for training is quite clear regarding a procedure for identifying training needs.  To meet that expectation, industry practice is to have a process for creating curricula and maintaining the accuracy and completeness of curricula requirements.  Yes, it feels like a lot of paperwork.  §820.25 also states “Training shall be documented”.   For me, it’s not just the completion of the Read & Understood for SOPs.  It includes the OJT process, the qualification event AND the ownership for curricula creation and maintenance. (Stay tuned for a future blog on documenting OJT.)

Whose responsibility is it, anyway?

Who owns curricula in your company?  Who has the responsibility to ensure that curricula are accurate and up to date?  What does your procedure include?  Interestingly enough, I have seen companies who get cited with training observations often have outdated and inaccurate curricula!  Their documentation for curricula frequently shows reviews overdue by 2 – 3 years, not performed since original creation and in some places, no specialized curricula at all!   “They were set up wrong.”  “The system doesn’t allow us to differentiate enough.”  “Oh, we were in the process of redoing them, but then the project was put on the back burner.”  Are you waiting to be cited by an agency investigator during biennial GMP inspection or Pre-Approval Inspection?

The longer we wait to conduct a curricula review, the bigger the training gap becomes.  And that can snowball into missing training requirements, which leads to employees performing duties without being trained and qualified.  Next thing you know, you have a bunch of Training CAPA notifications sitting in your inbox.  Not to mention a FD-483 and quite possibly a warning letter.  How sophisticated is your training database?  Will once a year result in a “light trim” of curricula requirements or a “hack job” of removing outdated requirements and inaccurate revision numbers? Will you be rebuilding curricula all over again?  Better bring on the donuts and coffee!  -VB

How many procedures does it take to describe a training program?