Employee Qualification is the Ultimate Level 3 Training Evaluation

On the Job Training is as old as some of the original apprentice-style forms of learning and ranges from very informal like follow Joe around to structured OJT that is formally documented and includes a qualification event observed by a Qualified Trainer.  While OJT means on the job training, the steps for OJT can also vary from trainer to trainer and from company to company unless the methodology is captured in an approved written procedure. 

Multiple Performance Demonstrations Occur

One of the first instances of a demonstration occurs from the trainer himself.  S/he shows the learner how to perform the technique, task, or process.  The learner observes and asks questions.  Then the roles reserve and the learner performs a mimicked rendition of what s/he observed.  The trainer provides feedback and sometimes will ask questions intended to assess the knowledge gained as well. 

Is one demonstration enough to determine OJT is done?  Sometimes it is.  When the task is simple, one time is all that most learners need.  When the task or process is complicated, it will take more than one demonstration to get the SOP steps right.  The nature of the SOP or the complexity of the task at hand determines this. 

But, how do I proceduralize that, you ask?  It starts by not arbitrarily picking the magic number 3. I have engaged in countless discussions regarding the exhaustive list of exceptions to forcing the rule of 3 times to practice.  And some QT’s will argue for more than 3 sessions especially when the procedure is so infrequently performed.  It’s not strictly about the number of times. We recognize that multiple sessions become practice sessions when the learner is still demonstrating the procedure under the supervision of his/her trainer.  But documenting the number of demonstrations and/or practice sessions is still a challenge for the Life Sciences Industry. 

At what point, is the learner going to be qualified to perform independently? As an industry, there is no standard number of times.  There are no standard learners either.  There is a range of “quick studys” to typical to slow learners.  The caveat to this is monitoring both the quick study and the slow learner.  In the QT workshops, this topic is explored using scenarios with tips and techniques that are shared during the debriefings.  Qualified Trainers know what is typical and they are empowered to evaluate the outcome of the learner’s demonstration(s).  Is the procedure being performed according to the SOP or is the learner still a bit hesitant about the next step? Is s/he relying on the QT for the assurance that the step is right?  While the steps may be performed correctly, is it also the confidence of both the QT and learner that we are assessing as well.

How many times is enough? Until both the learner and the QT are confident that s/he is not going to have an operator error deviation a week after going solo.  The QT is ultimately the one who has to assess progress and determine that “with a few more sessions”, my learner will get this or no, s/he may never get it and it’s time to have a discussion with the manager.

BTW, what does “Qualified Employee” mean?

Being SOP Qualified is the demonstrated ability of an employee to accurately perform a task or SOP independent of his OJT Qualified Trainer with consistency to meet acceptable quality standards. It satisfies the CFR ξ 211.25 (c ) regulation, “there shall be an adequate number of qualified employees to perform”.

Don’t be tempted to take the Performance Demo short-cut!

The end goal of OJT and the Qualification Event is for the employee to perform independently of his/her QT.  In order to be “released to task”, a final performance demonstration is scheduled, observed, and documented by an OJT Qualified Trainer. But don’t be fooled into taking the performance demo short cut!  The last step in the training portion of OJT is a performance demonstration to show the OJT-QT that the employee can perform the steps AND perform at the same level of proficiency as his/her peer group. If s/he can’t perform at this level, then the learner is not ready to “go solo”.

He may need more encouragement to build up confidence, correct paperwork documentation errors, and time to become proficient with his/her speed while maintaining accuracy.  That’s what practice sessions are for; time to master confidence with the steps and increase speed.  When his/her performance is on par with “business as usual” performance levels, then the employee is ready to perform the final demonstration aka the Qualification Event.  While the “readiness indicator” may not be documented, the (Q-Event) must be formally captured, assessed with the outcome being documented and communicated to both the learner and his/her supervision.  It is a separate event from the OJT demonstrations

Final Performance Demo = Qualification Event

During the final performance demonstration, the QT observes the learner’s performance.  When feedback is provided, it is evaluative and the rating result is formally documented.  Granted, when someone is watching us, we tend to follow the rules.  With enough repeated practice sessions,  learners tend to perform procedures as “business as usual”.  It’s how they learn the ebb and flow from their peers.  This is the optimum moment to determine if s/he is truly ready to perform without coaching or supervision from his QT.  If a QT has to interrupt to correct a misstep or remind the employee that his step is out of sequence, the event is terminated and documented as requires more review. 

More training practice is then scheduled until readiness is once again achieved.  And this also means the learner cannot sign for his work without his trainer’s co-signature or initials.  Do not misinterpret this as signing for the verification entry aka the second check.  In this situation, the Qualified Trainer cannot be both the co-signer and the second check person verification/reviewer.  You will need three sets of initials to properly document the supervision of a learner requiring more practice.  Otherwise you violate data integrity rules around independent verification.

Qualification events are not intended to be a rushed get ‘er done / one and done paperwork exercise.  Sufficient time for proficiency and expected department productivity levels is required to ensure knowledge has been retained and skill can be accurately repeated.  OJT demonstrations are not to be misused as the Q-Event.  This distinction is critical to ensuring a successful qualification event and the confidence of consistently performing the SOP tomorrow, next week, etc.  And not creating a deviation one day or one week after declaring the learner qualified.

It happens when QT’s are urged to “get’em done” by impatient or overly anxious supervisors consumed with productivity and not quality metrics.  With the qualification event being so recent, the QT will most certainly be interviewed as part of the investigation.  The checklist will also be examinedThis tool is supposed to help the QT be as objective as possible and consistently evaluate performance as demonstrated.  But typically, the checklist used to qualify individuals shows all Yeses; otherwise, they wouldn’t be qualified status.  And that, of course, depends on how well the critical steps and behaviors are captured in the OJT Checklist.  Yes, he was able to demonstrate the step, critical task, and/or behavior, but what we don’t know is how well?  Are we to assume that No means “No, not at all” and Yes means performed “Well” or it is “As Expected” or “Adequate” or maybe, in this case, it was “Sort of”?  The comments column would have been the ideal place to record observations and enter comments.

Validating Your SOP Effectiveness

Meeting FDA expectations for qualified employees is paramount.  But the “100% Trained on Curricula Requirements” printouts aren’t winning favor with FDA.   In the March 2015 article, Moving Beyond Read & Understand SOP Training”, I asserted that the current 100% trained reports and SOP quizzes would not be enough to satisfy the performance challenge for training effectiveness.  Are your employees qualified? How do you know? has become the training effectiveness question asked at every inspection.  The use of “100% completed” reports is a metric for completeness only; a commonly used data point from the LMS.  It does not address the transfer of learning into performance back on the job.  Neither does the 5-question multiple-choice “SOP Quiz”. The true measure of effective OJT is an observed performance demonstration of the SOP; aka the qualification event.

Employee Qualification is the ultimate Level 3 Training Effectiveness Strategy

Level 3 Behavior Change –> Transfer of Training/Learning

The focus of Employee Qualification is about the employee’s ability to apply knowledge and skill learned during OJT back on the job or in the workplace setting.  I call this Transfer of Training.  Others in the training industry refer to this as Level 3 – Behavior Change.   Actual performance is the ultimate assessment of learning transfer.   If an employee is performing the job task correctly during the final  performance demonstration (Q-Event), his performance meets the expectation for successful “OJT Required SOP”.

Yet, according to the 2009 ATD research study “The Value of Evaluation”, only 54.6% of respondents indicated that their organization conducts Level 3 evaluations.   The top technique used is follow-up surveys with participants (31%), while observation of the job was fourth (23.9%).    

If on the job assessment is the “ultimate” measure of transfer, then why isn’t it being used more frequently? “Post-training” assessments are time and labor-intensive.   But for organizations that have to meet compliance requirements (46.9% of survey respondents), documenting training effectiveness is now on FDA performance radar.

Not all SOPs require a Qualification Event

SOPs generally fall into two categories: FYI-type and OJT Required.  The more complex an SOP is, the more likely errors will occur.  Observing “critical to quality” steps is a key focus during the final performance demonstration.  However, a 1-1 documentation path for every OJT related SOP may not be needed.  Instead, batch SOPs a/o multiple SOPs of similar processes can be grouped into a “module” with documentation supporting similarity.  Where there are differences in these SOPs, then the Q Event would also require observation of these unique CTQ differences.

Two Types of SOPs
Two Types of SOPs: Only Critical Task-Based SOPs Required OJT and Qualification Events

An active Employee Qualification Program also verifies that the training content in this case the SOP, accurately describes how to execute the steps for the task at hand.  If the SOP is not correct or the qualifying documentation (checklist) is too confusing, a cause analysis needs to be conducted. Successful qualification events also validate the OJT methodology is effective. That Qualified OJT Trainers are consistently delivering OJT sessions for “OJT Required SOPs”. 

What does “Qualified Employee” mean for a company?

Qualified Employee status is not only a compliance imperative but a business driver as well. A qualified workforce means a team of well-trained employees who know how to execute their tasks accurately and with compliance in mind, own, and document their work properly.  When anyone in the organization can emphatically answer “Yes, my employees are qualified and yes, I have the OJT checklists to back that up”, then the Employee Qualification Program is not only working but is also effective at producing approved products or devices fit for use. The bonus is a renewed level of confidence in the ability of employees to deliver on performance outcomes for an organization.

*The Value of Evaluation: Making Training Evaluations More Effective. An ASTD Research Study, 2009, ASTD.

What happens when the performance demonstration becomes more of a "this is how I do it discussion" instead of an actual demonstration? Read the Impact Story - I've Fired My Vendor - to learn more.

Who is Vivian Bringslimark?

(c) HPIS Consulting, Inc.

Is Documenting Your OJT Methodology Worth It?

The short answer is yes! 

Years ago, during the Qualified Trainers (QT) workshop, I would ask QTs the following two questions:

  • What is OJT?
  • How do you deliver OJT?

Invariably, all would answer the first question the same way: on the job training.  And then I would ask the attendees to form into groups to discuss the second question among fellow peers.  I purposely mixed the groups so that there was equal representation of manufacturing trainers and QC analytical laboratory trainers and a fascinating exchange occurred between the attendees.  During the debriefing activity, we learned that there was a lot of variability in how the trainers were conducting OJT.  How can that be when they all answered the first question so consistently? “Well”, one of them said, “we don’t have a procedure on it so I just go about it the way I think it should be done.  I’ve had success so far, so I keep on doing what I’ve done in the past.”

Declaring your OJT Model

In order to get consistent OJT, you need to define your OJT steps and you need to ensure that only approved content (i.e. SOJT checklists)  will be used to deliver OJT; not the SME’s secret sauce. I’m not proposing a cookie-cutter approach for QTs to become all the same.  But rather, I am advocating a clear distinction between each step/stage / phase so that both the learner and the QT know exactly where they are in the OJT process, what is expected of them in that step and why the OJT step is needed.  This is no longer just go follow Joe or Jane around; this is structuring the OJT sessions.  Defining your OJT steps in a methodology ensures that all QT’s consistently deliver their 1-1 sessions and document the progression through the steps.

I’m less focused on what you call these steps or how many there are.   I am looking to see how these steps move a new hire along the journey to becoming qualified prior to being released to task.  For me, this is what makes OJT really structured.  And that model needs to be captured in a standard operating procedure or embedded in a training procedure so that all employees are informed and aware of how they will receive their OJT.   

The Assess Step

What is your purpose for this step?  Is it to evaluate the learners’ knowledge and skill via a performance demonstration, the effectiveness of the OJT sessions, or to determine qualified status?  The answer matters, because the QT will be providing feedback that impacts very different outcomes.

Be clear on the purpose of the performance demonstration

 The visual on the right indicates the main difference between how QTs feedback is used during a performance demonstration for OJT vs. feedback for the Qualification Event. Despite that, the learner is asked to perform the procedure the same way.  Is it clear to the learner what the performance demonstration means?  Does your methodology articulate the difference between a practice demo, a readiness demo, a performance demo and/or a Qualification Event demonstration?

Documenting OJT sessions presents a challenge for many trainers and document control staff.  What is considered an OJT session?  My favorite lament is – “Do you know what that will do to our database not to mention the amount of paperwork that would create!” A workaround to all these questions and concerns is to capture at least one session along the progression of each OJT step as per the OJT Model, thus documenting adherence to the procedure. 

For example, the first OJT step may be READ.  It means Read the SOP first, if not already completed. We are pretty good at documenting R & U for SOPs.  Then, the next step may be DISCUSS / INTRODUCE the SOP.  Capture when that discussion occurred if it’s different from Step 1- READ.  The “trainer demonstrates” portion can also be documented.  Where it gets tricky is when we ask the learner to demonstrate and practice.  Are we required to capture every OJT session or just one? 

Why not capture the last instance when it is confirmed the learner is ready to qualify?  If we keep it simple and document that our learners have experienced at least one instance of each step /stage, then we are complying with our OJT methodology and minimally documenting their OJT progression.   It is important to describe how to document the OJT progression in the SOP.  Don’t leave that up to the QT’s to figure out.  It is in our documentation, that we also need to be consistent.

How do you know if someone is qualified to perform this SOP?

Ideally, the answer would be because:

1.) we have a structured OJT process that includes a task specific OJT Checklist and

2.) we can look up the date of completion (Qualification Event) in our LMS history. 

And that, of course, depends on how well the critical steps and behaviors are captured in the SOJT checklist. The checklist is a tool to help the QT be as objective as possible and consistently evaluate performance as demonstrated. In the article, “A Better Way to Measure Soft Skills”, author Judith Hale explains the difference between a checklist and a rubric.

            “Checklists only record whether a behavior occurred, though, and not the quality of the behavior.  Rubrics, on the other hand, measure how well the learner executed the behavior.”  p. 62.

Yes, but was it tolerable, adequate or exemplary?

What I typically see are checklists with varying levels of tasks, steps and/or behaviors with a column for Yes, No and Comments.  What I don’t see is a column to mark how well the learner performed!  Is it enough to mark Yes or No for each item since most Q Events are Pass or “needs more practice”?  Maybe. 

But consider the following situation.  A human error deviation has occurred and the LMS indicates the technician has earned qualified status.  The document used to qualify this individual shows all Yeses.  Yes s/he was able to demonstrate the step, critical task, and/or behavior, but what we don’t know is how well?  Are we to assume that No means “No, not at all” and Yes means performed “Well” or it is “As Expected” or “Adequate” or maybe in this case it was “Sort of”? 

An additional column describing what it means to score low, medium, high or in our situation: Poor, Adequate, As Expected, and even Exemplar could provide much needed insight for the root cause analysis and investigation that will follow this deviation.  It provides a level of detail about the performance that goes beyond Yes, No, or Comment.  In most checklists I’ve reviewed, the comments column is hardly ever used.

What does the QT Signature Mean?

What the signature means on the document used to qualify an employee performing the task, technique or procedure as defined in the tool is whether or not the performance matched criteria (Y/N) or to what degree if using a qualitative rubric.

  • It does not mean that said employee completed all his curricula requirements.
  • It does not mean that said employee explained how to execute the procedure without performing it.
  • It does not mean the QT is responsible for the future performance of said employee.

In fact, it means just the opposite.  It documents that on this date, said employee was capable of performing the procedure as expected and that from this date forward, said employee owns his/her own work including deviations.  The employee is no longer being supervised by the QT for this SOP.  Without this understanding and agreement, the integrity of the whole program is put into question, not just the effectiveness of SOJT.  Be sure to explain this in the QT workshop and in the Robust Training System SOPs.  – VB

Hale,JA Better Way to Measure Soft Skills, TD, ATD, August, 2018, pps. 61-64.

Who is Vivian Bringslimark?

Is your OJT Methodology being documented consistently?

(c) HPIS Consulting, Inc.

The Journey of a New Hire to Qualified Employee: What really happens at your company?

After weeks if not months of waiting for your new hire, she is finally here, finishing up 1st day orientation. Day 2, she’s all yours. Are you excited or anxious? The LMS printout of training requirements is overwhelming; even for you. Bottom line question running through your mind — when can she be released to task? Isn’t there a faster way to expedite this training, you ask? There is, it is called SOJT.

[SOJT] is the planned process of developing task level expertise by having an experienced employee train a novice employee at our near the actual work setting.” Jacobs & Jones, 1995

This blog has been moved to “What Happens After New Hires Complete R & U SOPs?”.

Jacobs RL, Jones MJ. Structured on – the – job training: Unleashing employee expertise in the workplace. San Francisco: Berrett – Koehler,1995.

(c) HPIS Consulting, Inc.