Is your response, “Yes!” or is it “Yea, I think so. How can I be sure?”
Are you trying to explain what you dofor training or describe how your site handles the Quality Training System?
Who is your audience?
Who are the procedures being written for?
There shall be written procedures even for training!
Step back and take a big view of training from start to finish. There are three distinct segments: (1) preparation, (2) delivery, and (3) measuring effectiveness. To some leaders, the idea of creating more than one training procedure for these three segments is baffling. Training isn’t that hard. Why are you making this so complicated, they ask? The simpler the process is, the easier it is to follow and administer, is what they are really thinking.
Yet, training root cause analyses involving operator or human error often yields some contributing factors around how the individual was trained. On the Job Training is not that simple after all and requires more than a check on someone’s to-do list. Find out why in the e-Book, Training Root Cause Analysis. In spite of this, the level of detail in Training SOPs is always a concern for both SOP Author and Approver.
But SMEs like KISS for Training
From an end-user perspective, I need enough detail so I can accomplish what I need to do without over-complicating my task, stay within compliance, and be effective as a qualified trainer. Since most Qualified Trainers are not full-time training professionals, the training procedures become their how-to tools not just a bunch of QA or HR rules about training. Without defining what your program includes and how to execute it, Qualified Trainers and other SMEs will conduct their training delivery assignments in a manner that is as simple as possible and may not be 100% compliant. Administrative procedures are not for them, they argue.
It takes 6 elements to make a training system robust
The big view of training may have 3 Big Blocks (preparation, delivery, and measuring effectiveness), but it takes 6 elements to execute it well. They are:
GxP and Training Curricula
Planned OJT for Procedures
Use of Qualified Trainers
Employee Qualification and Training Effectiveness Measures
On-Going GxP Refresher Series
Training Documentation Process
For each of these elements, there needs to be a set of “how-to-execute” instructions. However, it is not enough to describe what these elements are. That leaves a lot of room for interpretation and inconsistent documentation or no documentation at all. For example:
Does your organization complain about over-allocated curricula?
Do they understand the documentation requirements for your LMS?
Does your training document explain how to conduct OJT? You’d be surprised just how many different techniques SMEs have for conducting OJT on their watch.
Can anyone who has a signed training record be a department trainer? If yes, you need to upgrade your criteria and create a process around nominating and qualifying SMEs for OJT. Training is on the top 10 list for inspections.
So how is this not complicating Training?
The debate for the 1 All-Encompassing SOP (standard operating procedure) vs. multiple tailored work instructions will always bring comments after the latest version goes into effect. This includes non-QT end-user feedback criticizing how complicated and confusing the procedures are! When pressed further for specifics, most admit that they just want to train themselves and not deal with the documentation requirements. So, addressing one set of concerns can actually create more complaints around the very changes. See the table below for pros and cons.
Remind me again who these procedures are for?
The real question is to what level of detail is appropriate. And the answer lies in the culture of the department and the company’s history with the FDA. I once worked in an organization where the procedures were stripped to bare essentials, even the definitions were removed! A recent warning letter left battle scars for the General Manager.
But seriously, how much is enough? Another way to answer this question is by looking at your document control hierarchy of documents. If your site differentiates between SOPs and Work Instructions, the need for detail can be relocated into a suitable work instruction that is designed to provide detailed how-to steps. AKA Level 3 type documents. Refer to the diagram below. The SOP (Level 2) can then provide a broader high-level overview addressing “the what” without getting encumbered into the “how-to execute” steps. Some document pyramids include a Level 4 in which much detail can be used for training purposes.
Back to the opening question
While addressing ease of navigation and use concerns is laudable, it is not always realistic to satisfy all users. You need to explain what your Training “Program” contains, a high level policy that answers “the what”. But you also need to describe how your training is conducted. Ask yourself the following two questions:
Does the “Training SOP” contain many pages detailing steps for each of the 6 elements? This is an indicator that you need to consider some Level 3 Work Instructions.
Do you have too many standalone task-focused SOPs (or Level 3 Work Instructions) that could be grouped into a few larger processes? Recall the 3 Big Blocks of Training.
From experience working with SME teams, three “how-to” procedures are usually sufficient: preparation process, delivery and documentation process, and measuring training effectiveness process. So the next time you find yourself in the level of detail discussion, consider which of the levels (1,2,3,4) is the best place to park your “detailed” content. – VB
I got a phone call from my Lead SME’s boss one morning. “How many more sessions do you need”, I asked him. I had already delivered 4 back-to-back workshops with class sizes of 25-30 SMEs; which was beyond optimal. So I asked him why. I needed to find out what was driving the surge in identified Qualified Trainers (QTs). I learned that a retrospective qualification needed to take place in order to close out an inspection observation. The total number of SMEs needing “proper paperwork” was well over 700. Since the redesigned training system was now in effect, these undocumented SMEs as Trainers would have to follow the new procedure. Or would they? Our discussion shifted to what type of training these SMEs will be delivering.
I then shared a related story with him. Several years prior, I got entangled with a “CAPA crisis” that involved QTs. No sooner did we launched the QT program and put the new procedure into effect, the CAPA quality system temporarily shut down shipping over a weekend. Upon return to the site, I was summoned to an emergency meeting from the security gate. Amazingly, a new practice/rule that only a Qualified Trainer can conduct training evolved from “only OJT QT’s can deliver OJT and perform Qualification Events” as per the SOP! This was clearly a case of misunderstood scope.
Does every SME need to be qualified as a Trainer?
In the Life Sciences arena, there are 5 recurring situations that require training: Self, Corrective Actions, Classroom (ILT), Structured OJT, and Qualification Events (Final Performance Demos).
Self can be achieved by the individual reading the procedure and signing the training record. This is also known as Read & Understand (R & U) for SOPs. I personally don’t think of it as training, it is reading. Yet, in some situations, reading is all that is required to gather the SOP information. If on the other hand, you need to execute the steps of the SOP and complete required forms, then additional training with the SOP Author or a QT is the appropriate next level of training.
Deviations/ Corrective Actions stemming from a Corrective Action Preventive Action incident. Minimally an SME or the SOP Author is needed to ensure the credibility of the content. These types of training sessions have become known as Corrective Actions “Awareness” Training. And more and more SMEs are now being required to deliver this training in a classroom setting. They need to be qualified to deliver classroom sessions especially if the event is related to a significant CAPA or regulatory inspection observation.
Classroom (Instructor-Led Training) is preferred for knowledge-based content affecting a wide range of employees. The skillset needed is facilitation / managing the classroom and delivering content as designed by the instructional designer. Think of GMP Refresher sessions in the Training Room.
Years ago, it was a lot clearer to distinguish between classroom trainers and SMEs as OJT Trainers. OJT was delivered 1-1 by “following Joe/Jane” around. Classroom Trainers delivered their content in a classroom of many learners using slides, flipcharts, and handouts. They were usually full-time dedicated training staff. Instructor-led training requires training in learning theory design and practice in what used to be referred to as platform skills. Today, it is more commonly known as “Running a Classroom” or “Basic Facilitation Skills”.
Many of today’s OJT QTs are also being requested to deliver “Group Training” sessions on content found within their SOPs. While the target audience may be the same set of peers, the scope, objectives, and tools used to deliver instructor-led training is vastly different from the OJT train the trainer course.
Structured OJT is On the Job Training delivered by a Qualified OJT SME using the approved OJT Methodology. OJT QT’s attend the Qualified Trainers Workshop which focuses on the OJT Steps Model, how to perform the equipment, and complex SOPs via hands-on and the challenges of Life as a Trainer. Should every seasoned employee become a QT based on their seniority and subject matter expertise? No, not necessarily. Because there are some SMEs that don’t want to share their knowledge and therefore, may not make an effective OJT Trainer. Establishing a set of nominating criteria provides an objective rationale for additional interpersonal qualities that help define a more well- rounded SME.
Qualification Events (the Final Performance Demonstration) are formally documented observations of learners performing the procedure/task at hand in front of a Qualified OJT SME using an approved SOJT Checklist or rubric. It is these events that set apart a Technical SME from a Qualified Trainer. The QT workshop includes a dedicated lesson on what to look for during Q-Events and what the QT signature means for the integrity of the Employee Qualification Program.
Can having too many QTs be a problem?
It can be when there is no one else to train; to deliver OJT steps. While many of you may be wishing for this situation, it can eventually happen if staffing levels are adequate, shifts are normalized and SOPs revisions are managed via R & U only with the LMS. How do you keep your QTs engaged and fresh if there are no opportunities for OJT sessions? I have some ideas for you to explore.
Re-examine the practice of online R & U only for SOP revisions. I bet some of those revisions were significant enough for a face to face discussion (aka Group Training) and there is probably at least one SOP revision in the past year that should have required a demonstration of task for optimum transfer of learning back on the job. *Just because all employees are now qualified, doesn’t mean the program sits in hiatus waiting for new hires to join the company.
When you have too many QTs who may be underutilized, it is also an appropriate time to administer the Trainer Mojo Assessment. Based on the QTs scores, it might be time to say thank you for a job well done for the low scoring QTs. You may be pleasantly surprised by who is ready to walk away from the training role? Or you may have a cadre of QTs who legitimately need more training and hence, the need for some new modules is now justified. Many of your excess SMEs were identified long before criteria was put into a place or the SOP was established. If the Trainer Mojo Assessment doesn’t bring any discussion, perhaps it’s time to “re-nominate” them using the criteria within the SOP and offer a refresher series on the QT Workshop content. Or arrange for developmental assignments that expand their subject matter expertise or advances their training repertoire into a classroom facilitators?
What is exciting for me is that many OJT-QTs are stepping up and volunteering to attend the SMEs as Classroom Facilitators workshop as part of expanding their QT’s toolkit. Many of them want to learn more about teaching peers and working with adults. A few have now become promoted to full-time trainer for L&D /QA departments. Which of your OJT QTS are ready to step up and move into the classroom? It’s time to find out and be part of the current trend. -VB
Are you looking for management support for the Qualified Trainers and the time needed to deliver SOJT? If only we were required to have a procedure for that! It may not be an SOP or even a policy document, but industry guidance documents provide a lot of references to management involvement.
ICH Q10 – Pharmaceutical
While not mandatory, management needs to seriously take notice of ICHQ10 guidance document released in April
2009 (1). In particular to the following:
MANAGEMENT RESPONSIBILITY 2.3 Quality Planning
“(d) Management should provide the appropriate resources and training to achieve the quality objectives”.
“CONTINUAL IMPROVEMENT OF THE PHARMACEUTICAL QUALITY SYSTEM
4.3 Outcomes of Management Review and Monitoring
The outcome of management review of the pharmaceutical quality system
and monitoring of internal and external factors can include:
(b) Allocation or reallocation of resources and/ or personnel training.”
Under a Quality System: Managers Expectations for Training
Referencing the Sept. 2006 issue of Guidance Document for Quality Systems: IV. Management Responsibilities | B. Resources (2), we see alignment with the training CGMPs for continued training so “as to remain proficient in their operational functions and in their understanding of CGMP regulations.” “Typical quality systems training should address the policies, processes, procedures, and written instructions related to operational activities, the product/service, the quality system, and the desired work culture (e.g., team building, communication, change, behavior).”
And my personal favorite, “When operating in a robust quality system environment, it is important that managers verify that skills gained from training are implemented in day-to-day performance.” The responsibility for training under a quality system is not assigned to just one person or one function. It is a shared responsibility across the entire organization.
Goals of the
Train-the-Trainer Program (TTT) vs. OJT Program vs. Employee Qualification Program
In order to have qualified employees, they need to receive structured on the job training delivered by a qualified trainer who is content qualified and training process qualified via a Qualified Trainers’ workshop. Each of the three “programs” has defined outcomes that are dependent upon each other. Unfortunately, the term program has been a bit overused throughout the years and can have a variety of meanings for folks. For purposes of this blog, TTT program means OJT Qualified Trainers workshop, OJT Program means OJT methodology/procedureand Qualified Employee program means having a robust quality training system beyond the newest Learning Management System (LMS).
As you can see from the diagram, Qualified Trainers are at the core of all three “programs”. The OJT QT Workshop is designed to prepare the QT’s for the realities of life as a Qualified Trainer. It includes the pre-work of familiarizing themselves with the Quality Training System procedures and then in class: exploring basic learning theory and committing to what the QT signature means. The OJT Program is about delivering structuring on the job training consistently following the approved methodology. An Employee Qualification Program is the validation of training effectiveness of the OJT methodology and the clarity of the underlying procedures.
Readiness Factors for SOJT
and an Employee Qualification Program
Let’s start with a written purpose statement for having qualified employees beyond it’s required. What is the company’s philosophy on achieving qualified status? Is there agreement among the leadership for the level of rigor required to demonstrate performance and achieve a pass rating? Where are the SOJT program goals written? Does a schedule exist for SOJT and qualification events other than an LMS printout with required due dates. That is not a schedule for SOJT. Do you have clearly defined objectives for the QT workshop captured in a document or perhaps a procedure? Is there a single owner for all three programs or is responsibility and accountability assigned accordingly?
Ronald Jacobs and Michael Jones, in their 1995 ground breaking book, Structuring on-the-Job Training, inform us that SOJT as a system functions within a larger context, namely the organization. SOJT is not a standalone program. Conflicts, competing priorities and mixed messages can influence the success of your SOJT program. What else is going on in the organization that will compete for the same set of QTs? Remember they are also your most experienced and technical subject matter experts. How is the overall Employee Qualification program aligned with the other quality systems?
Recognition for QTs and
Most QTs are not fully dedicated to delivering training for departments. There are pros and cons for this decision. For now, I will leave them out. Suffice it to say, they are tasked with both their “day” job and the responsibility for delivering training when needed. They are at times, doing two jobs. Whether or not they are compensated additionally for delivering SOJT, acknowledging their contribution to the department and the organization is part of management support. It takes more than “you are doing a good job, keep it up”.
Often supervisors and managers don’t know what else they are supposed to
do to show their support, other than allow them to attend the QT workshop. The interested ones will “pop” in during lunch
and chat with their direct reports.
Others will show up at the end for the poster activity (equivalent to a
written test) and some will come to learn about the parking lot issues that
need follow up. The energy in the room
when this happens is amazing.
“When operating in a robust quality system environment, it is important that managers verify that skills gained from training are implemented in day-to-day performance.”
Guidance Document for Quality Systems, Sept 2006
To help ease the knowledge gap between a manager and their now Qualified Trainer, I started delivering the Leadership Briefing module prior to the QT workshop delivery. The purpose is to provide an overview of the content highlights, alignment with initiatives / CAPAs/ agency commitments and more importantly to secure agreement for the following:
criteria for nominating a QT
roles and responsibilities of QT
scope of work QT’s can be assigned
expectations for QT’s post launch
what happens day one after workshop is done
what is the status of the SOJT checklists
scheduling and budget concerns.
If the organization says they support the qualification program, then what happens when employees achieve qualification status? Nothing? A non-event? Or is it announced in newsletters, plasma screens and other company announcements? Is it a big deal to be able to perform independently and free up a much-needed QT for another learner? I keep hearing over and over again about how there aren’t enough QT’s to deliver SOJT the right way. One would think qualification status on SOPs, especially big complex processes deserves SOME kind of recognition. Just how committed are the managers and supervisors? QTs and employees draw their own conclusions about the level of real management support for the programs.
If truth be told, after launching the QT workshop, many supervisors privately don’t support the program. They lose their top performers during the workshop and the hours it takes to train someone. Forget about giving QTs adequate time to complete the paperwork properly! And then leadership wonders why good documentation practice (GDP) issues continue to be a problem? The non-distracted performance observations that QT’s are expected to conduct for the qualification demonstration drive supervisors and line managers crazy the most – what, they can’t do anything else but observe? Hence, many QT’s are asked to multi-task just to get the work done: not enough resources they are told. For supervisors, productivity and the workload will always trump SOJT and qualification events, until their bonuses include completion of training and qualification events.
What Real Support Is
Supposed To Look Like
My key take away message is that attending the TTT program/ QT workshop is not the end of the OJT program or the Employee Qualification Program but rather the launching point. Management support needs to go beyond just nominating QTs and allowing them to participate in the workshop. The real support is in the alignment of goals, clarifying expectations continuously, allocating resources for training and budgeting time to deliver OJT using an approved OJT methodology that includes qualification events. This commitment of time and sponsorship for qualified employees is a culture shift for many line managers and site leaders. But actions do speak louder than words. -VB
(1) Guidance for Industry Q10 Pharmaceutical Quality System | US Department
of HHS | FDA | CDER | CBER April2009 ICH
(2) Guidance for Industry Quality Systems Approach to Pharmaceutical CGMP Regulations U.S.Department of HHS | FDA | CDER | CBER | CVM | ORA| September 2006 Pharmaceutical CGMPs
Jacobs RL, Jones MJ. Structured on-the-job training: Unleashing employee expertise in the workplace. San Francisco: Berrett-Koehler: 1995.
How can you tell if your site is ready for SOJT?
HPISC has created a 2 part checklist of questions and attributes to explore. The checklist is available gratis when you become a HPISC mailing list member. Just be sure to include that are you interested in receiving the SOJT Readiness Checklist.
during the Qualified Trainers (QT) workshop, I would ask QTs the following two
do you deliver OJT?
all would answer the first question the same way: on the job training. And then I would ask the attendees to form
into groups to discuss the second question among fellow peers. I purposely mixed the groups so that there
was equal representation of manufacturing trainers and QC analytical laboratory
trainers and a fascinating exchange occurred between the attendees. During the debriefing activity, we learned
that there was a lot of variability in how the trainers were conducting
OJT. How can that be when they all
answered the first question so consistently? “Well”, one of them said, “we don’t have a procedure on it so I just go
about it the way I think it should be done.
I’ve had success so far, so I keep on doing what I’ve done in the past.”
Declaring your OJT Model
In order to get consistent OJT, you need to define your OJT steps and you need to ensure that only approved content (i.e. SOJT checklists) will be used to deliver OJT; not the SME’s secret sauce. I’m not proposing a cookie-cutter approach for QTs to become all the same. But rather, I am advocating a clear distinction between each step/stage / phase so that both the learner and the QT know exactly where they are in the OJT process, what is expected of them in that step and why the OJT step is needed. This is no longer just go follow Joe or Jane around; this is structuring the OJT sessions. Defining your OJT steps in a methodology ensures that all QT’s consistently deliver their 1-1 sessions and document the progression through the steps.
I’m less focused on what you call these steps or how many there are. I am looking to see how these steps move a new hire along the journeyto becoming qualified prior to being released to task. For me, this is what makes OJT really structured. And that model needs to be captured in a standard operating procedure or embedded in a training procedure so that all employees are informed and aware of how they will receive their OJT.
The Assess Step
What is your purpose for this step? Is it to evaluate the learners’ knowledge and skill via a performance demonstration, the effectiveness of the OJT sessions, or to determine qualified status? The answer matters, because the QT will be providing feedback that impacts very different outcomes.
The visual on the right indicates the main difference between how QTs feedback is used during a performance demonstration for OJT vs. feedback for the Qualification Event. Despite that, the learner is asked to perform the procedure the same way. Is it clear to the learner what the performance demonstration means? Does your methodology articulate the difference between a practice demo, a readiness demo, a performance demo and/or a Qualification Event demonstration?
Documenting OJT sessions presents a challenge for many trainers and document control staff. What is considered an OJT session? My favorite lament is – “Do you know what that will do to our database not to mention the amount of paperwork that would create!” A workaround to all these questions and concerns is to capture at least one session along the progression of each OJT step as per the OJT Model, thus documenting adherence to the procedure.
For example, the first OJT step may be READ. It means Read the SOP first, if not already completed. We are pretty good at documenting R & U for SOPs. Then, the next step may be DISCUSS / INTRODUCE the SOP. Capture when that discussion occurred if it’s different from Step 1- READ. The “trainer demonstrates” portion can also be documented. Where it gets tricky is when we ask the learner to demonstrate and practice. Are we required to capture every OJT session or just one?
Why not capture the last instance when it is confirmed the learner is ready to qualify? If we keep it simple and document that our learners have experienced at least one instance of each step /stage, then we are complying with our OJT methodology and minimally documenting their OJT progression. It is important to describe how to document the OJT progression in the SOP. Don’t leave that up to the QT’s to figure out. It is in our documentation, that we also need to be consistent.
How do you know if someone is qualified to perform this SOP?
Ideally, the answer would be because:
1.) we have a structured OJT process that includes a task specific OJT Checklist and
2.) we can look up the date of completion (Qualification Event) in our LMS history.
And that, of course, depends on how well the critical steps and behaviors are captured in the SOJT checklist. The checklist is a tool to help the QT be as objective as possible and consistently evaluate performance as demonstrated. In the article, “A Better Way to Measure Soft Skills”, author Judith Hale explains the difference between a checklist and a rubric.
“Checklists only record whether a
behavior occurred, though, and not the quality of the behavior. Rubrics, on the other hand, measure how well
the learner executed the behavior.” p.
What I typically see are checklists with varying levels of tasks, steps and/or behaviors with a column for Yes, No and Comments. What I don’t see is a column to mark how well the learner performed! Is it enough to mark Yes or No for each item since most Q Events are Pass or “needs more practice”? Maybe.
But consider the following situation. A human error deviation has occurred and the LMS indicates the technician has earned qualified status. The document used to qualify this individual shows all Yeses. Yes s/he was able to demonstrate the step, critical task, and/or behavior, but what we don’t know is how well? Are we to assume that No means “No, not at all” and Yes means performed “Well” or it is “As Expected” or “Adequate” or maybe in this case it was “Sort of”?
An additional column describing what it means to score low, medium, high or in our situation: Poor, Adequate, As Expected, and even Exemplar could provide much needed insight for the root cause analysis and investigation that will follow this deviation. It provides a level of detail about the performance that goes beyond Yes, No, or Comment. In most checklists I’ve reviewed, the comments column is hardly ever used.
What does the QT Signature Mean?
What the signature means on the document used to qualify an employee performing the task, technique or procedure as defined in the tool is whether or not the performance matched criteria (Y/N) or to what degree if using a qualitative rubric.
It does not mean that said employee completed all his curricula requirements.
It does not mean that said employee explained how to execute the procedure without performing it.
It does not mean the QT is responsible for the future performance of said employee.
In fact, it means just the opposite. It documents that on this date, said employee was capable of performing the procedure as expected and that from this date forward, said employee owns his/her own work including deviations. The employee is no longer being supervised by the QT for this SOP. Without this understanding and agreement, the integrity of the whole program is put into question, not just the effectiveness of SOJT. Be sure to explain this in the QT workshop and in the Robust Training System SOPs. – VB
Hale,J. A Better Way to Measure Soft Skills, TD, ATD, August, 2018, pps. 61-64.
Many QA/ L&D Training Managers are tasked with improving their “training program”. An integral component of a robust quality training system is the Qualified Trainers (QT). Having a cadre of existing department Subject Matter Experts (SMEs) as Trainers can be very helpful when implementing the rollout of the new quality redesign to meet regulatory commitments and expected timelines. But, sometimes it can also lead to sustainability issues after the launch is over and the next big project becomes the new site priority.
During my on-site response to an urgent performance problem, the Head of Operations expressed deep concerns about inconsistent OJT being delivered by his trainers. A series of significant non-conformances occurred in his area. As part of the CAPA (Corrective Action Preventive Action) investigation, trainers were interviewed to uncover how they trained the identified employee(s) and what was said specifically for each step of the procedure. Their responses revealed a lack of consistent process and the use of varied content; despite having an OJT checklist, the procedure, and approved Training SOPs.
Once a Trainer; Forever a Trainer
I was then invited into a conversation with the Training Operations Manager (My Performer), regarding her desire to upgrade the existing department SMEs as Trainers. Responsible for the effectiveness check of the CAPA corrective action and the overall quality of Operations OJT sessions, she complained that many of the trainers should no longer be considered Dept. Trainers. While she had position title influence, she was frustrated by the lack of support for her “improvement suggestion”. I became her catalyst to help her push through the fixed barrier regarding SMEs.
The site followed a cultural assumption regarding department SMEs: once a Trainer; always a Trainer; regardless of feedback and informal impressions of their ability. Without any tangible criteria and lack of assessment tools, my Performer had no authority to remove the underperforming Dept. Trainers. Granted these SMEs were long ago chosen when the widely accepted practice of being proficient as a technician after a year earned them the designation of subject matter expert and automatically, a Dept. Trainer. Today, the Life Sciences Industry, with FDA investigators’ observations, has evolved their understanding that it takes more than seniority and R & U SOP training to become an OJT QT. Unfortunately, the environment where my Performer worked, the mindset about acquired expertise still held.
Significant CAPAs can be Drivers for Change
Undaunted, my Performer seized the CAPA as an opportunity for change. Leveraging suggested criteria and the use a form to document justification for each Dept. Trainer, she now had a process (SOP with form) that she could “educate” her colleagues on what it takes to become a Qualified Trainer. The focus of her message dramatically changed. She became strategic in her communications, using the effectiveness check portion of the CAPA as her “Why / WIIFM for Operations Managers”. In order to close out the CAPA, Managers had to complete their portion of the form.
The long-term success of my Performer depended on her owning her solution. She never lost of her original desire; she was patient and waited for her colleagues to accept today’s best practices for OJT QTs. In the meantime, we brainstormed on a variety of feedback options that could be used to evaluate the current status of each SME at the same time the Managers completed the new form. My Performer chose a rating system and arranged for a 1-1 sessions with Operations Managers to discuss what rating they would use for each criteria if they got challenged during a CAPA investigation or a regulatory inspection.
While the results were not formally documented, my Performer was effective with the assessment rating exercise. The Managers reconsidered who they wanted to nominate based on the new formal criteria and the informal ratings discussions. They did not automatically submit the form for all existing Dept. Trainers. A constructive dialogue then ensued regarding skills remediation support for those SMEs deemed as potentials. At last, my Performer achieved her desired outcome. “As catalysts, we build a bridge, light the path, and give [ ] our hand to help [ ] demolish or jump over obstacles”, (Haneberg, 2010, p.96). I was privileged to be part of a dramatic shift in their training culture.
An alternate alignment exercise
For many, adding ratings suggests a formal performance assessment and this can raise HR issues if not fully supported by the organization. In addition, many Operations Managers do not have the luxury of “weeding out undesirables”. They simply do not have enough SMEs to complete the training curricula generated requirements. Yet, there needs to be mutual consent between manager and identified SME in order to effectively deliver the OJT Methodology and to ensure a successful learner experience.
For those situations where automatically re-nominating existing SMEs is raising a red flag, I created the Trainer Mojo Assessment. Nominated SMEs and existing SMEs as Trainers rate themselves on 10 attributes that align with the characteristics of an effective OJT Trainer. Low scoring SMEs/QTs are encouraged to have a discussion with their management regarding continuation in the program and possible action steps. For SMEs/QT’s that score in the On-Target range, this is both validation of the nomination and confirmation that manager and QT are in sync. For high scoring QTs, this is also confirmation and an early indicator for potential QT Rock Stars!
Haneberg, L. Coaching up and down the generations. Alexandria, Virgina:ASTD, 2010.
You might be interested in the Impact Story – From Dept. SME to QT.
The Client Request –”Can you help us upgrade our Trainer Qualification process?”
Hint: It’s more than 100% complete with your curricula requirements!
First there was “Just go follow Joe around” training …
And then came “and it shall be documented” …
Next the follow up question: “Are they trained in everything they need to know?”
So, line managers used the SOP Binder Index and “Read and Understand SOP” became a training method…
But alas, they complained that it was much too much training and errors were still occurring …
So, training requirements were created, and curricula were born.
Soon afterwards, LMS vendors showed up in our lobbies and promised us with a click and a report, we could have a training system!
But upper management called forth for METRICS! So, dashboards became a visible tool. Leader boards helped create friendly competition among colleagues while “walls of shame” made folks hang their heads and ask for leniency, exemptions and extensions …
Why Traditional OJT Takes So Long
While the introduction to this blog poked fun at how we’ve evolved over the last 30 + years, allow me to step back in time for a brief refresher about On the Job Training. Traditional OJT (TOJT) is not planned so for the new hire it comes across as informal almost incidental. Monikers like “follow Joe or Jane around” or “Sink or Swim” accurately described how TOJT occurred. My personal favorite is “trial by fire”. It can certainly feel like that when the “trainer” albeit a SME, is not qualified to train resulting in inaccurate steps and possible bad habits like unsafe short cuts. It causes anxiety for the new hire especially when the daily schedule dictates what will be “taught” by the technician assigned to that task. Sometimes, over-anxious new hires quit before on boarding is finished because of their chaotic OJT sessions and lack of confidence in their trainers.
Structured OJT Defined (SOJT)
SOJT is the process in which a qualified employee (trainer) passes job knowledge and skills to another employee in an organized, personalized and thoughtfully planned manner. It involves both learning and doing at the same time.
For first time managers, SOJT appears to take a lot of work and time. They would rather focus on the production schedule convincing themselves that new hires are being trained while shadowing their training buddy. Managers are evaluated on how productive they are, not on how many new hires are being trained effectively. They also tell themselves, that it takes 6 – 12 months to get fully up to speed, so why spend even more time planning and scheduling a new hire’s learning journey. In spite of that impression, the Quality of OJT directly impacts:
time it takes to “bring someone up to speed”
how Employees perceive the training they receive
how satisfied Employees feel about their jobs and working conditions
how quickly Employees can be re-trained on new SOP revisions.
But Managers Find the Time to Train Twice!
Yet that is exactly what happens when the root cause of a deviation or a mistake is deemed as Human/Operator Error. The corrective action is often “repeat training”. This training involves unlearning the first sessions and then relearning the proper way, causing more strain on the new hire’s space for learning capacity (cognitive overload). This is why TOJT takes so long. If, SOJT had been executed instead, the error more than likely would not have happened. It’s an argument that I have been defending for over 30 years with Senior Leaders. Why punish the New Hire for management’s decision to make production more important than effective training?
DOES HAVING CURRICULA MAKE OJT STRUCTURED?
Just having curricula doesn’t necessarily make OJT structured. SOJT includes a deliberate review of regulatory, departmental and positional /functional requirements. It also includes a TRAINING SCHEDULE for the curricula requirements and an individualized learning plan for new hires; not just a matrix with their name highlighted in yellow marker.
Isn’t The Training Matrix Enough?
Just ask an OJT Qualified Trainer that question and watch the reaction you get. Maybe a shrug and a half smile if you’re lucky. The matrix is only a listing of the requirements usually generated from the LMS or an excel spreadsheet for those who haven’t migrated to an LMS yet. In more sophisticated systems, it will include due dates. But most printouts don’t include the Qualified Trainers assigned to deliver the training or any additional information about the learner.
From this list, QTs are expected to manage the completion of the requirements, deliver effective training sessions, provide feedback in a safe and nurturing learning environment and qualify learners with qualification events / performance assessments. All that from a training matrix? Wow.
During the HPISC Qualified Trainers workshop, I present the difference between TOJT (traditional) and SOJT (structured). When I ask the QT’s where they feel their organization is, most of them will say still in the TOJT box but closer to the middle of the range. Why I ask? Invariably, they’ll tell me OJT is not scheduled. “Just because I have the list of curricula requirements doesn’t mean the training gets scheduled or that qualification events get conducted”. But rather, it happens when someone makes it a priority, an inspection is coming or a CAPA includes it as part of corrective actions.
There Is Still A Huge Misunderstanding Regarding R&U For Sops
For years, we have been documenting that we’ve read and understood our required SOPs. I can still recall my Employee Blue Card that listed every Monthly GMP and SOP meeting/discussion I attended. When I was asked about logging every SOP that I read during an interview, I responded that it was not a practice where I worked at the time. After I accepted the new position, I realized why that was such a watershed moment for me.
Oh, But Now We Have Curricula!
As an industry, we became obsessed about signing for our SOPs. Then LMSes were developed to help us not only record our R & U SOP dates but to give us a mechanism for tracking what SOPs were assigned to us, reporting 100% Trained metrics and overdue requirements. And for a short while those records were sufficient.
But over the years, FDA Investigators began seeing a disconnect between the R & U training records and the actual performance of employees who signed that they understood the SOP. Upon the FDA site tour, they observed departures from the written procedures. They uncovered deviations involving Operator /Human Error, repeat deviations and even CAPAs for training fixes. So, is it falsification of records? Is it poorly written SOPs? Is the training ineffective?
FDA stepped up their expectations and began asking deeper questions. How do you know if your training is effective? This question applies to both GxP Training sessions as well as SOPs.
Who is qualified to train employees and how do they get qualified?
Are your employees qualified to perform their job functions and how do you know?
What does that documentation look like?
SOPs Are Now Online, But It’s Still Read and Understood for SOP!
Well, I’m told we are now more compliant with ensuring that only the most current version of the SOP is used for training. LMSes and eDoc platforms have given us efficient mechanisms to document that employees have completed their training curricula. But do they understand their procedures enough to perform them correctly back on the job?
It’s still Read and Understand SOPs! Whether we access our SOPs through an eDoc system or the LMS portal, we are still only reading them. To call this eLearning is a bit of a stretch especially when compared to the design of today’s eLearning courses. Nonetheless, some new hires are still being provided with a long list of required SOPs and trained on where to find them in the LMS. “Oh, we’ve made it easier for them to manage their SOP list. They’re online now!”
I’m excited that as an industry, we are evolving with our training practices to keep up with regulators’ expectations regarding GMP and SOP Training. But have we changed the training culture yet? Are we just replacing attendance forms for e-signatures or are we delivering effective training? Can we confidently say, “Yes, our employees are qualified prior to release to task”?
FD-483 Observation: “Your site has numerous instances of R & U for SOPs”
So, when there is a large number of documented R & U events for SOPs, very little OJT documentation and they are still finding repeat errors, HE/OE deviations and a host of CAPAs, FDA Investigators are going to examine what is your training process and how effective is it.
SO, WHAT ELSE MAKES OJT STRUCTURED?
SOJT is formal and it’s documented. Without the use of approved training materials, department SMEs often use different methods each time they deliver training and the training content can vary from “trainer to trainer”. This causes confusion for the new hire during his/her onboarding phase which can then lead to Operator Error or non- compliant performance later.
During an on-site response to an urgent performance problem, the Head of Operations expressed deep concerns about inconsistent OJT being delivered by his trainers. During an FDA inspection, he was shown numerous examples from the Investigator, that his SMEs were teaching techniques for a critical process procedure that
Naturally, this led to a FD-483 observation, a high visibility corrective action project with global impact. As part of the CAPA (Corrective Action Preventive Action) investigation, trainers were interviewed. Their responses revealed the use of varied content; despite having an OJT checklist, the actual procedure, and knowledge of the site training SOPs.
QT’s need to use “Quality Control Unit” approved written procedures (aka SOPs) as the main document to train with and the proper documentation to record an OJT session. But documenting OJT sessions has been a bit of a challenge for Trainers, LMS Administrators, and QA Doc Control Staff.
What Is Considered An OJT Session?
The hardest question to resolve is “Are we required to capture every OJT session or just one?” My favorite lament is – “Do you know what that will do to our database not to mention the amount of paperwork that would create!” A workaround to all these questions and concerns is to capture at least one session along the progression of each OJT step as per your OJT Steps Model, thus documenting adherence to the procedure. If we keep it simple and document that our learners have experienced each step, then we are complying with our OJT process and minimally documenting their OJT progression.
Even more challenging is adding the required SOJT steps to curricula. Unfortunately, it generates almost double the requirements. So for companies who focus solely on the number of requirements, this is daunting. What helps is differentiating between R & U step and completion of the actual OJT events. Some organizations go one step further and also add the Qualification Event as a final requirement.
SOJT IS ALSO DELIVERED BY A DEPARTMENT SME QUALIFIED TO DELIVER OJT. The old adage for selecting department trainers based documented R & U SOP paperwork no longer meets regulatory expectations. The expectation today is to have a QA approved process for qualifying SMEs. I’m not talking about a mere mention in your training policy. But a standalone process that depicts steps from start to finish. Yes, it is necessary if you want to answer “Yes, I have a procedure for Qualifying SMEs as Department Trainers”. Being able to present a SOP that addresses how your SMEs are qualified to sign the training records is priceless in the middle of an ongoing regulatory inspection.
What Belongs In The Qualified Trainer SOP?
Ideally, the procedure needs to include these 4 parts:
II. QT Workshop/TTT Equivalent
IV. Performance Demonstration
In some organizations there is no difference between SMEs and Trainers. And this is precisely why FDA has asked for clarification. Can anyone be a Qualified Trainer (QT)? Establishing criteria is the best way to reduce favoritism and the tendency to pick the most senior person. SMEs need to become QTs through a formal process. The selection criteria should be part of the qualifying documentation along with any supporting statements for eligibility selection (for example, nominated by supervisor or responded to the call for volunteers).
In this definition, everyone who is a QT is an SME in some specified area. All nominees need to be content qualified on the subject matter they will be teaching. This means being able to produce proper documentation confirming nominee’s eligibility. This is not always the case. I have seen this assumption backfire horribly and cause major ripple effects on project milestones. Find out now before an inspection, please.
II. QT Workshop/ TTT Equivalent
Sometimes known as Train-the-Trainer. The course needs to include learning theory, training and coaching adult learning peers and agreeing to use the proper documentation. Another key component of the OJT TTT workshop is exploring the challenges of “Life as a Qualifed Trainer”. Learning how to facilitate a live classroom event that can come later. In fact, many QT’s are stepping up and want to expand their trainer’s toolkit for “Basic Facilitation Skills / Running a Live Classroom Event”.
III. QT Workshop Evaluation:
Simply attending this course isn’t enough. But whether a written test is the best measure to use is open for debate. Let’s start with the question: Why do we need an evaluation in the first place? If you’re tempted to say because FDA wants it, I suggest re-reading CFR x211.25 again. If you are anticipating the training effectiveness question, then you are in sync with industry practice. But it is less about “the test” and more about how you determine the effectiveness of the training event. Benchmarking from other certification courses, a written test usually follows the course. So, having a written evaluation is not unreasonable. The debate is about what format you use.
The Written Test
Most folks are familiar with taking a written test. When informed upfront, QT’s expect the test to come at the end of the workshop. But what is the outcome of test? What does it really measure? Is it a measure of their retention or comprehension? Since SOPs are not supposed to be memorized, how can we dictate memorization of the course content? Open book is allowed in some organizations. What then does the paper-n-pencil test accomplish? Having the knowledge doesn’t mean that they will use the concepts “in the moment of choice”.
Consider the Action Planner
If learning transfer is what we really want and expect to achieve, then wouldn’t some kind of post workshop action planner be a more appropriate measure of effectiveness? “Oh, but we can’t control what happens after the workshop”, you say? “Using an action planner requires buy-in from their managers. And commitment to follow through to host the 1-1 follow up meetings with their QTs” is what you might be thinking right now.
Let us not lose focus on the ultimate goal of the workshop. It’s a 3-way partnership between QTs, L&D/QA Training, and Managers. The real work happens AFTER the workshop is over. What better way to use classroom time to discuss strategies for barriers and challenges and document their commitment for applying the concepts and procedures than a post workshop action planner? Can the written test do all this?
IV. Performance Demonstrations
This is after all a qualification of SMEs as Trainers program and therefore, a performance demonstration is required. What type depends on where your site is with respect to employee qualifications. The two types are demonstrating a live OJT session in the classroom and conducting an employee qualification at the workstation.
RECAP: WHAT IT TAKES TO MAKE OJT STRUCTURED
Rigorous Curricula that includes SOJT and Qualification Events not just SOPs
Methodology for OJT Steps that includes documentation
Qualified Trainers who deliver SOJT curricula requirements
Individualized Learning Plans and schedules for New Hire’s Learning Journey
Ronald Jacobs and Michael Jones, in their 1995 groundbreaking book, Structuring On-the-Job Training, inform us that SOJT as a system functions within a larger context, namely the organization. SOJT is not a standalone program. Conflicts, competing priorities and mixed messages can influence the success of your SOJT program.
Do these sound familiar?
Business priorities and organizational change initiatives
Upper managements real perception of the value of OJT
Union shop and potential violation to use SMEs as Qualified Trainers
Alignment of goals for training and goals for other quality systems
Willingness of line leaders and staff functions to manage and maintain SOJT after launch
The Training Quality System, in my opinion, is THE most cross functional system. It has to harmonize with other quality systems AND organizational systems in order to deliver performance improvement.
So, management support has to be more than lip service. The real support is in the alignment of goals, clarifying expectations, allocating resources and budgeting time to deliver OJT using an approved OJT methodology. This is a culture shift for many organizations but well worth the effort if management really believes in SOJT. The “proof is in the pudding”. Are your leaders “walking their talk”? – VB
For companies who are required to have standard operating procedures (SOPs), managing changes and ensuring affected employees are up to date is a constant challenge. Especially for companies whose weekly revision churn rate is from 10 to 150+ revisions. The standard approach is to “get ‘em done” in the fastest amount of time. The end goal for most people is the proverbial (√) and lots of records claiming “read and understood” the procedure.
What’s a Trainer now Performance Consultant to do? With all this focus on training effectiveness and return on stakeholder expectations, how does a Training / Performance Consultant demonstrate his/her worth when the real outcome is a rushed end-user who has less than 20 minutes to train on the changes?
According to Rummler and Morrill*, it comes down to how serious do the changes need to be? Performance consulting is about closing gapsin results, training is about closing gaps in knowledgeand skill. So, consider analyzing what precipitated the SOP change. Was is it a significant CAPA issue or FDA commitment that requires a solution more than “Read and Understood”/ e-sign for the revised SOP?
In other words, isolate the underlying issue that prompted the change. How significant is the change in the procedure? Is it vital enough to improve performance results? Then take a performance consulting approach for these changes. Don’t fret about the others requiring only knowledge. Those SOP revisions can be managed using routine training techniques whose goal is numbers trained.
What Do They Really Need to
Know About the Revised SOP?
The HPI/HPT approach ties the results back to the business outcomes. If the end goal is truly a √ for SOP revision training, you have met the expectations. But as Trainer/ Performance Consultants, we know this is not an acceptable measure for any type of formal review. (Human Performance Improvement / Human Performance Technology).
What we can do is use a modified root-cause approach to address why the procedure changed and then incorporate the what, the where and the when into the training session. Take advantage of the revision history page if available. The SOP Author has listed what changed. Call him/her and ask why the change was made. Perhaps it is related to a deviation or audit observation that other employees need to know about. Find out when the changes will go into effect and how will folks be notified.
Will it be upon an effective date?
Or a trigger condition that now has a different decision path?
Or is it a completely overhauled FORM to be used?
Include these notes as part of your
preparation for end-users to recognize when to make the changes in their tasks.
PLEASE NOTE: Revision
training is not to be used as a substitute for planned OJT or new hire SOP
SOP Revisions Needing Real OJT: Will Management Agree with Your Recommendation?
But what if the SOP revision requires more than a “tell and forget” discussion?
Are the changes significant enough to require a demonstration?
Will a one-time demo by the Qualified Trainer (QT) be enough to call it OJT?
Does the revision require some practice to unlearn the old way and learn the new steps?
How confident are your Qualified Trainers in redirecting this discussion with management regarding the most effective method to use, despite that it will take longer to complete the training notification flags?
First and foremost, the QT signature means that learners were assessed as having learned the content of what was delivered. Can the QT confidently say their colleagues are trained in the revision and can perform the revision without errors? Again, it comes down to how critical-to-quality is the new change? Can the department / site / organization incur waste as in mistakes, scrap work, delayed release of product, and/or re-work as a result of “lack of proper training for the revision”? After the fact, many managers have “a case of amnesia” about their original directive regarding the SOP revisions. “Just tell them what the changes are, better yet, have them R & U and sign the darn form already!” With more deviations being opened, these same managers now question why they have QTs in the first place. Funny how there’s time to do the training properly a second time?
I realize this is easier said than done especially for new QT graduates. A directive is a directive, I understand. In these cases, documentation matters and needs to be explicit that demonstration requirements were waived by “Management Decision”. When the investigation points back to a QT, the documentation will provide what content was covered and to what level of demonstration and practice was included; if any. If none are required, then the risk for mistakes is minimal. These questions, as painful as they are can be excellent references for future events where OJT is really needed for a significant SOP revision. -VB
*Rummler, G. and Morrill, K. “The Results Chain”. T &
D, Feb 05; 26—37.
After weeks if not months of waiting for your new hire, she is finally here, finishing up 1st day orientation. Day 2, she’s all yours. Are you excited or anxious? The LMS printout of training requirements is overwhelming; even for you. Bottom line question running through your mind — when can she be released to task? Isn’t there a faster way to expedite this training, you ask? There is, it is called SOJT.
Structured on the job training (SOJT) is an organized and planned approach for completing training requirements. Yet for many line managers, they want their trainees now. Ironically, the faster you “push” trainees through their training matrix, the slower the learning curve. This in turn often leads to more errors, deviations, and quite possibly CAPA investigations for numerous training incidents. It’s a classic case of pay now or pay later.
This proactive vs. reactive dilemma is not new. Traditional OJT aka “follow Joe around” looks like a win-win for everyone on the surface. The new hire gets OJT experience, a SME is “supervising” for mistakes, and supervisors are keeping up with the production schedule. So what’s wrong, you ask?
[SOJT] is the planned process of developing task level expertise by having an experienced employee train a novice employee at our near the actual work setting.” Jacobs & Jones, 1995
After 6 months or so, the trainee isn’t new anymore and everyone “expects”
your new employee to be fully qualified by then with no performance issues and
no deviations resulting from operator error. Without attentive monitoring of
the trainee’s progress, the trainee is at the mercy of the daily schedule. S/he is expected to dive right in to whatever
process or part of the process is running that day without taking into account
where the trainee is on their learning curve.
The assigned SME or perhaps the “buddy” for the day is tasked with not
only trying to perform the procedure correctly but explain what he’s doing and
why it may be out of sequence in some cases.
The burden of the learning gap falls to the SME who does his best to
answer the why.
The structured approach puts the trainee’s needs center stage. What makes sense for him/her to learn what and when? The result is a learning plan individualized for this new hire that includes realistic time frames. Added to the plan is a Qualified Trainer who can monitor the progression towards more complex procedures and increase success for first time qualification success. Still too much time to execute? How many hours will you spend investigating errors, counseling the employee and repeating the training? Seems worth it to me. – VB
When I got introduced to the flipped classroom back in 2012, I fell in love with the concept immediately. But I was stymied on how to sell the mind shift to management. And then it occurred to me that I was already delivering the flipped classroom for my Qualified OJT Trainers Workshop and have been for quite some time.
In the academic model, students study the concept at home using video instructions and other learning technologies. Then they come to class to do the homework in a collaborative and mentor style environment. The corporate approach can be tweaked if you rethink when and where learners (aka employees) access the content. If the content exists elsewhere, why waste valuable classroom time lecturing on it when you can create a facilitative and experiential learning lab?
But what content are we talking about?
As part of a Robust Training System, procedures are established that describe how to execute the many elements of a training system including how to quality a department SME as trainer and the OJT Methodology. These standard operating procedures (SOPs) become the basis of the Qualified Trainer (QT) curriculum, for LMS tracking purposes. Upon release (or by the effective date), flagged employees read the required Training SOPs and become aware of implementation timelines. The “Read and Understood” portion is completed in advance and outside of the classroom. Hence, the content of the QT workshop is not a slide deck repeat of the curriculum content.
OJT TTT Workshop is NOT SOP Training
Recently I was challenged about why I was working with the company SMEs at all. If I didn’t work at this company, wasn’t engaged with day to day processing activities or involved in writing their SOPs, how could I possibly teach the SMEs anything about teaching their processes, my contender demanded. Furthermore, he marveled at my audacity to assist with rebuilding their quality training system.
Seemingly, there is still some confusion about the purpose of an OJT TTT workshop. This stakeholder’s frame of reference was entirely research oriented and analytical in execution. In his experience, train the trainer meant true expert trains others on the subject matter (technical SOPs) via a knowledge transfer session. See graphic below. It took multiple conversations and an invitation to join the training design team for him to embrace the notion that SMEs should and could be taught learning theory. Also see The Real Meaning of TTT.
The “flip” is in the instructional design
If I am asking for 8 hours of participation from SMEs in a classroom setting, then it MUST BE value added. The focus of the workshop design is “Life as a Qualified Trainer” and the realities of delivering OJT on the shop floor and/or in the analytical QC lab. Attendees will not find this content in their curriculum or in the e-DOC system for SOPs. And this is precisely why the classroom is the most effective environment to come together in a structured, guided and facilitated learning experience.
It means that the instructor-trainer is no longer the sage on the stage, but becomes a guide on the side, where the QTs are doing most of the talking. This switch in learning design reinforces collaboration among the QTs and better transfers the knowledge building so that scrap learning is significantly reduced. Activities are designed to be active and participatory thus promoting “learning by doing” practice and honing their learning for each step of the OJT Methodology while recognizing robust training system key concepts in action. The final activity requires engagement and participation of each nominated QT in order to complete the course.
A community of internal QT graduates
During the experiential activities, QT’s will share anecdotes or a “war story” from their past. Listening to those stories creates a connection and often the insights gained forms a bond with each other. A wonderful consequence of the TTT flipped classroom design is the community of internal QT graduates that grows after the workshop is over. QTs leave the classroom able to articulate and share what they learned and experienced together. This does not happen when the course is delivered as eLearning/CBT or self advanced power point slides.
Today’s classroom is still viable
The modern learner needs a modern learning experience. And while modern tech tools are fast on the rise, let’s not dismiss what a flipped classroom can produce – confident, competent and valued Qualified Trainers. Are you ready to flip your learning design to meet today’s modern learners? – VB