Why we love and hate Training Curricula and our LMS (Learning Management System)

Let’s begin with why we have curricula in the first place.  So that folks are trained in the “particular operations that the employee performs” (§21CFR 211.25).  Having this “list” is a huge improvement from the previous practice years ago. 

Remember when SOPs sat in binders on a shelf?

I was once told, “If an SOP is sitting in a binder in my department, then everybody is going to train on it”.  He had the entire SOP binder collection sitting on a shelf in his office.  He routinely required that all his direct reports sign the training form to back up his statement.  I congratulated him on documenting the READ part.  But when I asked him if his folks were trained on how to conduct the preventative maintenance procedure for cleaning the HVAC air ducts on the roof, he extolled, “that’s not our responsibility!” “Yes, you are right, but you all signed for the UNDERSTOOD part of those SOPs as well!” I replied.  He didn’t take my comment very well. 

The quality system regulation for training is quite clear regarding a procedure for also identifying training needs.  And “each manufacturer shall establish procedures for identifying training needs and ensure that all personnel are trained to adequately perform their assigned responsibilities” (§21CFR820.25). To meet that expectation, industry practice is to have a process for creating curricula, maintaining the accuracy and completeness of curricula requirements. 

Once I explained the purpose of curricula and how this manager could determine which procedures his folks should train on rather than using the SOP Index, he stopped the unrequired R & U for SOPs that were outside of their responsibilities.  Then I jokingly said, “Now you have time to send them to Annual GMP Training!” He frowned and walked away.

§820.25 also states “Training shall be documented”.  Yes, it feels like a lot of paperwork.  Thank goodness today’s LMSes can perform reconciliation of training completion against curricula requirements.  So, I’m grateful that this task can be executed with a click for simple reports. And with just a few key search words, more advanced reports can be compiled. No more excel spreadsheets filled with matrix like set ups cross referencing SOPs, attendance and employees! But there’s more to training curricula than setting them up and earning 100% Training Complete.

Who is Really Responsible for Employee Training?

Almost every training SOP I read will include a statement that Department Managers are responsible for training of their direct reports.  Managing SOP changes and ensuring affected employees are up to date is a constant challenge for everyone. Especially for companies whose weekly revision churn rate is anywhere from 10 to 150+ revisions.

First, we rely on the LMS to automatically revise curricula or at least expect the LMS Admin to click [Revised] function.  Then we wait for the notifications either automatically from the LMS or the Admin.  And then we begin the huddle brow beating to “get ‘em done” before or by the effective date. The end goal for most people is the proverbial check mark (√) and lots of line items claiming “Read and Understood” the revised procedure.  This is when we hate the LMS because the notifications won’t go away until we sign off.  Btw, this isn’t training at all when the real outcome is a rushed end-user who has less than 15 minutes to “train on the changes”.

What Do They Really Need to Know About the Revised SOP?

Training is about closing gaps in knowledge and skills. Consider analyzing what precipitated the SOP change. Was is it a significant CAPA issue or FDA commitment that requires a solution more than “Read and Understood”/ e-sign for the revised SOP? How significant is the change in the procedure?

The SOP Author has listed what changed. Call him/her and ask why the change was made. Perhaps it is related to a deviation or audit observation that other employees need to know about. Find out when the changes will go into effect and how will folks be notified. Take advantage of the revision history page if available. And then incorporate the what, the where and the when into the training session:

  • Will the changes be upon an effective date?
  • Or a trigger condition that now has a different decision path?
  • Or is it a completely overhauled FORM to be used?

Include these notes as part of your preparation for end-users to recognize when to make the changes in their tasks.  PLEASE NOTE: Revision training is not to be used as a substitute for planned On Job Training or new hire SOP Training.

SOP Revisions Needing Real OJT: Will Management Agree with the Recommendation?

But what if the SOP revision requires more than a “tell and forget” discussion?  Are the changes significant enough to require a demonstration?  Will a one-time demo by the Qualified Trainer (QT) be enough to call it OJT?   Does the revision require some practice to unlearn the old way and learn the new steps?  How confident are the Qualified Trainers in redirecting the discussion with management regarding the most effective method to use, despite that it will take longer to satisfy the LMS requirement?

First and foremost, the QT signature means that employees were trained and assessed as having learned the content of what was delivered.  Can the QT confidently say their colleagues are trained in the revision and can perform the revision without errors?    Can the department / site / organization incur waste as in mistakes, scrap work, delayed release of product, and/or re-work as a result of “lack of proper training for the revision”? It comes down to how critical to quality is the new change.

So, Who Owns Curricula In Your Company? 

In those same Training SOPs that I mentioned earlier, I usually find a responsibility statement about determining training requirements as well.  Who has the responsibility to ensure that curricula are accurate and up to date as well as no overdue training for SOP revisions? Is it QA L&D because they manage the LMS or is it Dept Management as per the Training SOP?

What is the frequency for curricula review in your procedure? I realize that curricula requirements work is tedious even painful at times.  That’s why donuts show up for meetings scheduled in the morning, pizza bribes if it’s lunchtime and quite possibly even cookies for a late afternoon discussion.  So, I get it when folks don’t want to look at their curricula again or even have a conversation about them. 

Once a year curricula upkeep?

It’s like having a fichus hedge on your property.  If you keep it trimmed, pruning is easier than hacking off the major overgrowth that’s gone awry a year later.  And yet, I continue to get push back when I recommend quarterly curricula updates.  Even semi-annual intervals are met with disdain.  In the end we settle for once a year and I cringe on the inside.  Why? 

Because once a year review can be like starting all over again. If your LMS is mature enough to manage revisions via auto-update all affected curricula, then once a year may be the right time span for your company.  Others in our industry don’t have that functionality within their training database.  For these administrators, revisions mean manual creation into the “course catalog” each time with a deactivation/retirement of the previous version; some may be able to perform batch uploads with a confirmation activity post submission.  And then, the manual search for all curricula so that the old SOP number can be removed and replaced with the next revision.  Followed by a manual notification to all employees assigned to either that SOP or to the curricula depending on how the database is configured.  I’m exhausted just thinking about this workload.  

What about the periodic review of SOPs, which in most companies is two years.  What is the impact of SOP’s that become deleted as a result of the review?  Can your LMS / training database search for affected curricula and automatically remove these SOPs without intervention from the LMS administrator as well? 

And then there’s the corrective actions that have resulted in major SOP revisions that require a new OJT session and quite possibly a new qualification event?  What impact do all these changes have on the accuracy of your curricula? Can your administrator click the revision button for these as well?

Interestingly enough, I have seen companies who get cited with training observations, often have outdated and inaccurate curricula!  Their documentation for curricula frequently shows reviews overdue by 2 – 3 years or not performed since original creation, and in some places, no specialized curricula at all! 

  • “They were set up wrong.” 
  • “The system doesn’t allow us to differentiate enough.” 
  • “Oh, we were in the process of redoing them, but then the project was put on the back burner.” 

Are you waiting to be cited by an agency investigator during biennial GMP inspection or Pre-Approval Inspection? The longer we wait to conduct a curricula review, the bigger the training gap can become.  And that gap snowballs into missing training requirements, which leads to employees performing duties without being trained and qualified.  Next thing you know, you have a bunch of Training CAPA notifications sitting in your inbox.  Not to mention a FD-483 inspection report and quite possibly a warning letter.  So why does QA L&D continue to experience pushback for this required activity? Maybe folks really like morning bagels or the pizza bribes?

Why do Curricula Owners pushback on curricula reviews?

Will once a year result in a “light trim” of curricula requirements or a “hack job” of removing outdated requirements and inaccurate revision numbers? Will you be rebuilding curricula all over again?  It’s a self-fulfilling prophecy. Can we please change this already? You just might be surprised at how many SOPs are no longer required and that you may reduce the time needed to train on SOPs.  In the meantime, better bring the donuts and coffee!  -VB

Tired of bloated and over-allocated curricula?

Who is Vivian Bringslimark?

(c) HPIS Consulting, Inc.

Employee Qualification is the Ultimate Level 3 Training Evaluation

On the Job Training is as old as some of the original apprentice-style forms of learning and ranges from very informal like follow Joe around to structured OJT that is formally documented and includes a qualification event observed by a Qualified Trainer.  While OJT means on the job training, the steps for OJT can also vary from trainer to trainer and from company to company unless the methodology is captured in an approved written procedure. 

Multiple Performance Demonstrations Occur

One of the first instances of a demonstration occurs from the trainer himself.  S/he shows the learner how to perform the technique, task, or process.  The learner observes and asks questions.  Then the roles reserve and the learner performs a mimicked rendition of what s/he observed.  The trainer provides feedback and sometimes will ask questions intended to assess the knowledge gained as well. 

Is one demonstration enough to determine OJT is done?  Sometimes it is.  When the task is simple, one time is all that most learners need.  When the task or process is complicated, it will take more than one demonstration to get the SOP steps right.  The nature of the SOP or the complexity of the task at hand determines this. 

But, how do I proceduralize that, you ask?  It starts by not arbitrarily picking the magic number 3. I have engaged in countless discussions regarding the exhaustive list of exceptions to forcing the rule of 3 times to practice.  And some QT’s will argue for more than 3 sessions especially when the procedure is so infrequently performed.  It’s not strictly about the number of times. We recognize that multiple sessions become practice sessions when the learner is still demonstrating the procedure under the supervision of his/her trainer.  But documenting the number of demonstrations and/or practice sessions is still a challenge for the Life Sciences Industry. 

At what point, is the learner going to be qualified to perform independently? As an industry, there is no standard number of times.  There are no standard learners either.  There is a range of “quick studys” to typical to slow learners.  The caveat to this is monitoring both the quick study and the slow learner.  In the QT workshops, this topic is explored using scenarios with tips and techniques that are shared during the debriefings.  Qualified Trainers know what is typical and they are empowered to evaluate the outcome of the learner’s demonstration(s).  Is the procedure being performed according to the SOP or is the learner still a bit hesitant about the next step? Is s/he relying on the QT for the assurance that the step is right?  While the steps may be performed correctly, is it also the confidence of both the QT and learner that we are assessing as well.

How many times is enough? Until both the learner and the QT are confident that s/he is not going to have an operator error deviation a week after going solo.  The QT is ultimately the one who has to assess progress and determine that “with a few more sessions”, my learner will get this or no, s/he may never get it and it’s time to have a discussion with the manager.

BTW, what does “Qualified Employee” mean?

Being SOP Qualified is the demonstrated ability of an employee to accurately perform a task or SOP independent of his OJT Qualified Trainer with consistency to meet acceptable quality standards. It satisfies the CFR ξ 211.25 (c ) regulation, “there shall be an adequate number of qualified employees to perform”.

Don’t be tempted to take the Performance Demo short-cut!

The end goal of OJT and the Qualification Event is for the employee to perform independently of his/her QT.  In order to be “released to task”, a final performance demonstration is scheduled, observed, and documented by an OJT Qualified Trainer. But don’t be fooled into taking the performance demo short cut!  The last step in the training portion of OJT is a performance demonstration to show the OJT-QT that the employee can perform the steps AND perform at the same level of proficiency as his/her peer group. If s/he can’t perform at this level, then the learner is not ready to “go solo”.

He may need more encouragement to build up confidence, correct paperwork documentation errors, and time to become proficient with his/her speed while maintaining accuracy.  That’s what practice sessions are for; time to master confidence with the steps and increase speed.  When his/her performance is on par with “business as usual” performance levels, then the employee is ready to perform the final demonstration aka the Qualification Event.  While the “readiness indicator” may not be documented, the (Q-Event) must be formally captured, assessed with the outcome being documented and communicated to both the learner and his/her supervision.  It is a separate event from the OJT demonstrations

Final Performance Demo = Qualification Event

During the final performance demonstration, the QT observes the learner’s performance.  When feedback is provided, it is evaluative and the rating result is formally documented.  Granted, when someone is watching us, we tend to follow the rules.  With enough repeated practice sessions,  learners tend to perform procedures as “business as usual”.  It’s how they learn the ebb and flow from their peers.  This is the optimum moment to determine if s/he is truly ready to perform without coaching or supervision from his QT.  If a QT has to interrupt to correct a misstep or remind the employee that his step is out of sequence, the event is terminated and documented as requires more review. 

More training practice is then scheduled until readiness is once again achieved.  And this also means the learner cannot sign for his work without his trainer’s co-signature or initials.  Do not misinterpret this as signing for the verification entry aka the second check.  In this situation, the Qualified Trainer cannot be both the co-signer and the second check person verification/reviewer.  You will need three sets of initials to properly document the supervision of a learner requiring more practice.  Otherwise you violate data integrity rules around independent verification.

Qualification events are not intended to be a rushed get ‘er done / one and done paperwork exercise.  Sufficient time for proficiency and expected department productivity levels is required to ensure knowledge has been retained and skill can be accurately repeated.  OJT demonstrations are not to be misused as the Q-Event.  This distinction is critical to ensuring a successful qualification event and the confidence of consistently performing the SOP tomorrow, next week, etc.  And not creating a deviation one day or one week after declaring the learner qualified.

It happens when QT’s are urged to “get’em done” by impatient or overly anxious supervisors consumed with productivity and not quality metrics.  With the qualification event being so recent, the QT will most certainly be interviewed as part of the investigation.  The checklist will also be examinedThis tool is supposed to help the QT be as objective as possible and consistently evaluate performance as demonstrated.  But typically, the checklist used to qualify individuals shows all Yeses; otherwise, they wouldn’t be qualified status.  And that, of course, depends on how well the critical steps and behaviors are captured in the OJT Checklist.  Yes, he was able to demonstrate the step, critical task, and/or behavior, but what we don’t know is how well?  Are we to assume that No means “No, not at all” and Yes means performed “Well” or it is “As Expected” or “Adequate” or maybe, in this case, it was “Sort of”?  The comments column would have been the ideal place to record observations and enter comments.

Validating Your SOP Effectiveness

Meeting FDA expectations for qualified employees is paramount.  But the “100% Trained on Curricula Requirements” printouts aren’t winning favor with FDA.   In the March 2015 article, Moving Beyond Read & Understand SOP Training”, I asserted that the current 100% trained reports and SOP quizzes would not be enough to satisfy the performance challenge for training effectiveness.  Are your employees qualified? How do you know? has become the training effectiveness question asked at every inspection.  The use of “100% completed” reports is a metric for completeness only; a commonly used data point from the LMS.  It does not address the transfer of learning into performance back on the job.  Neither does the 5-question multiple-choice “SOP Quiz”. The true measure of effective OJT is an observed performance demonstration of the SOP; aka the qualification event.

Employee Qualification is the ultimate Level 3 Training Effectiveness Strategy

Level 3 Behavior Change –> Transfer of Training/Learning

The focus of Employee Qualification is about the employee’s ability to apply knowledge and skill learned during OJT back on the job or in the workplace setting.  I call this Transfer of Training.  Others in the training industry refer to this as Level 3 – Behavior Change.   Actual performance is the ultimate assessment of learning transfer.   If an employee is performing the job task correctly during the final  performance demonstration (Q-Event), his performance meets the expectation for successful “OJT Required SOP”.

Yet, according to the 2009 ATD research study “The Value of Evaluation”, only 54.6% of respondents indicated that their organization conducts Level 3 evaluations.   The top technique used is follow-up surveys with participants (31%), while observation of the job was fourth (23.9%).    

If on the job assessment is the “ultimate” measure of transfer, then why isn’t it being used more frequently? “Post-training” assessments are time and labor-intensive.   But for organizations that have to meet compliance requirements (46.9% of survey respondents), documenting training effectiveness is now on FDA performance radar.

Not all SOPs require a Qualification Event

SOPs generally fall into two categories: FYI-type and OJT Required.  The more complex an SOP is, the more likely errors will occur.  Observing “critical to quality” steps is a key focus during the final performance demonstration.  However, a 1-1 documentation path for every OJT related SOP may not be needed.  Instead, batch SOPs a/o multiple SOPs of similar processes can be grouped into a “module” with documentation supporting similarity.  Where there are differences in these SOPs, then the Q Event would also require observation of these unique CTQ differences.

Two Types of SOPs
Two Types of SOPs: Only Critical Task-Based SOPs Required OJT and Qualification Events

An active Employee Qualification Program also verifies that the training content in this case the SOP, accurately describes how to execute the steps for the task at hand.  If the SOP is not correct or the qualifying documentation (checklist) is too confusing, a cause analysis needs to be conducted. Successful qualification events also validate the OJT methodology is effective. That Qualified OJT Trainers are consistently delivering OJT sessions for “OJT Required SOPs”. 

What does “Qualified Employee” mean for a company?

Qualified Employee status is not only a compliance imperative but a business driver as well. A qualified workforce means a team of well-trained employees who know how to execute their tasks accurately and with compliance in mind, own, and document their work properly.  When anyone in the organization can emphatically answer “Yes, my employees are qualified and yes, I have the OJT checklists to back that up”, then the Employee Qualification Program is not only working but is also effective at producing approved products or devices fit for use. The bonus is a renewed level of confidence in the ability of employees to deliver on performance outcomes for an organization.

*The Value of Evaluation: Making Training Evaluations More Effective. An ASTD Research Study, 2009, ASTD.

What happens when the performance demonstration becomes more of a "this is how I do it discussion" instead of an actual demonstration? Read the Impact Story - I've Fired My Vendor - to learn more.

Who is Vivian Bringslimark?

(c) HPIS Consulting, Inc.

Is Documenting Your OJT Methodology Worth It?

The short answer is yes! 

Years ago, during the Qualified Trainers (QT) workshop, I would ask QTs the following two questions:

  • What is OJT?
  • How do you deliver OJT?

Invariably, all would answer the first question the same way: on the job training.  And then I would ask the attendees to form into groups to discuss the second question among fellow peers.  I purposely mixed the groups so that there was equal representation of manufacturing trainers and QC analytical laboratory trainers and a fascinating exchange occurred between the attendees.  During the debriefing activity, we learned that there was a lot of variability in how the trainers were conducting OJT.  How can that be when they all answered the first question so consistently? “Well”, one of them said, “we don’t have a procedure on it so I just go about it the way I think it should be done.  I’ve had success so far, so I keep on doing what I’ve done in the past.”

Declaring your OJT Model

In order to get consistent OJT, you need to define your OJT steps and you need to ensure that only approved content (i.e. SOJT checklists)  will be used to deliver OJT; not the SME’s secret sauce. I’m not proposing a cookie-cutter approach for QTs to become all the same.  But rather, I am advocating a clear distinction between each step/stage / phase so that both the learner and the QT know exactly where they are in the OJT process, what is expected of them in that step and why the OJT step is needed.  This is no longer just go follow Joe or Jane around; this is structuring the OJT sessions.  Defining your OJT steps in a methodology ensures that all QT’s consistently deliver their 1-1 sessions and document the progression through the steps.

I’m less focused on what you call these steps or how many there are.   I am looking to see how these steps move a new hire along the journey to becoming qualified prior to being released to task.  For me, this is what makes OJT really structured.  And that model needs to be captured in a standard operating procedure or embedded in a training procedure so that all employees are informed and aware of how they will receive their OJT.   

The Assess Step

What is your purpose for this step?  Is it to evaluate the learners’ knowledge and skill via a performance demonstration, the effectiveness of the OJT sessions, or to determine qualified status?  The answer matters, because the QT will be providing feedback that impacts very different outcomes.

Be clear on the purpose of the performance demonstration

 The visual on the right indicates the main difference between how QTs feedback is used during a performance demonstration for OJT vs. feedback for the Qualification Event. Despite that, the learner is asked to perform the procedure the same way.  Is it clear to the learner what the performance demonstration means?  Does your methodology articulate the difference between a practice demo, a readiness demo, a performance demo and/or a Qualification Event demonstration?

Documenting OJT sessions presents a challenge for many trainers and document control staff.  What is considered an OJT session?  My favorite lament is – “Do you know what that will do to our database not to mention the amount of paperwork that would create!” A workaround to all these questions and concerns is to capture at least one session along the progression of each OJT step as per the OJT Model, thus documenting adherence to the procedure. 

For example, the first OJT step may be READ.  It means Read the SOP first, if not already completed. We are pretty good at documenting R & U for SOPs.  Then, the next step may be DISCUSS / INTRODUCE the SOP.  Capture when that discussion occurred if it’s different from Step 1- READ.  The “trainer demonstrates” portion can also be documented.  Where it gets tricky is when we ask the learner to demonstrate and practice.  Are we required to capture every OJT session or just one? 

Why not capture the last instance when it is confirmed the learner is ready to qualify?  If we keep it simple and document that our learners have experienced at least one instance of each step /stage, then we are complying with our OJT methodology and minimally documenting their OJT progression.   It is important to describe how to document the OJT progression in the SOP.  Don’t leave that up to the QT’s to figure out.  It is in our documentation, that we also need to be consistent.

How do you know if someone is qualified to perform this SOP?

Ideally, the answer would be because:

1.) we have a structured OJT process that includes a task specific OJT Checklist and

2.) we can look up the date of completion (Qualification Event) in our LMS history. 

And that, of course, depends on how well the critical steps and behaviors are captured in the SOJT checklist. The checklist is a tool to help the QT be as objective as possible and consistently evaluate performance as demonstrated. In the article, “A Better Way to Measure Soft Skills”, author Judith Hale explains the difference between a checklist and a rubric.

            “Checklists only record whether a behavior occurred, though, and not the quality of the behavior.  Rubrics, on the other hand, measure how well the learner executed the behavior.”  p. 62.

Yes, but was it tolerable, adequate or exemplary?

What I typically see are checklists with varying levels of tasks, steps and/or behaviors with a column for Yes, No and Comments.  What I don’t see is a column to mark how well the learner performed!  Is it enough to mark Yes or No for each item since most Q Events are Pass or “needs more practice”?  Maybe. 

But consider the following situation.  A human error deviation has occurred and the LMS indicates the technician has earned qualified status.  The document used to qualify this individual shows all Yeses.  Yes s/he was able to demonstrate the step, critical task, and/or behavior, but what we don’t know is how well?  Are we to assume that No means “No, not at all” and Yes means performed “Well” or it is “As Expected” or “Adequate” or maybe in this case it was “Sort of”? 

An additional column describing what it means to score low, medium, high or in our situation: Poor, Adequate, As Expected, and even Exemplar could provide much needed insight for the root cause analysis and investigation that will follow this deviation.  It provides a level of detail about the performance that goes beyond Yes, No, or Comment.  In most checklists I’ve reviewed, the comments column is hardly ever used.

What does the QT Signature Mean?

What the signature means on the document used to qualify an employee performing the task, technique or procedure as defined in the tool is whether or not the performance matched criteria (Y/N) or to what degree if using a qualitative rubric.

  • It does not mean that said employee completed all his curricula requirements.
  • It does not mean that said employee explained how to execute the procedure without performing it.
  • It does not mean the QT is responsible for the future performance of said employee.

In fact, it means just the opposite.  It documents that on this date, said employee was capable of performing the procedure as expected and that from this date forward, said employee owns his/her own work including deviations.  The employee is no longer being supervised by the QT for this SOP.  Without this understanding and agreement, the integrity of the whole program is put into question, not just the effectiveness of SOJT.  Be sure to explain this in the QT workshop and in the Robust Training System SOPs.  – VB

Hale,JA Better Way to Measure Soft Skills, TD, ATD, August, 2018, pps. 61-64.

Who is Vivian Bringslimark?

Is your OJT Methodology being documented consistently?

(c) HPIS Consulting, Inc.