Why Your Evaluation Strategy Cannot Be an Afterthought

In March of 2017, one of my proteges, Stephanie, interviewed me for a school assignment regarding the 4 Levels of Evaluation Model. The following Question and Response exchange has been slightly edited for clarity.

Vivian Bringslimark, President and Owner, HPIS Consulting, Inc., started with a very influential quote from a client of hers that spoke of the importance of evaluation: “When you look at the ADDIE model, without the A, you die. And then without the E, you’ve only just added another course to the curriculum.”

She feels it’s very insightful because the evaluation strategy is not an afterthought. It has to be part of the design so that learners can engage with the material and practice in order to be successful with what they’ll be assessed on. Evaluation, while having the potential to be controversial, problematic, and open to a variety of opinions, has to be added to the agenda of the course so that time can be allotted to implement it. 

QUESTION 1: How do you evaluate a learners’ Reaction (or Level 1) to the training, such as [a] Participants’ satisfaction with the training [b] Participants’ level of involvement (or engagement) in the learning experience [c] Level to which participants’ found value (or relevance) to what they learned?

RESPONSE: While there are problems and flaws with smile sheets, Vivian states that they are most popular, easiest to administer, and results can be quantifiable to generate reports. However, while participants can love the course, smile sheets cannot guarantee effectiveness. And participants may not be able to apply what they learned back on the job, even though they liked a course. Another problem with smile sheets is that the emphasis tends to be on the presenter, whether or not they were on target and delivered the right course for the audience.

QUESTION 2: How do you evaluate Learning (or Level 2), such as [a] Degree to which participants acquired knowledge and skill [b] Participants’ beliefs (attitude) that it will be worthwhile to implement what is learned on the job [c] Participants’ level of confidence in doing what they learned on the job

RESPONSE: Tests or quizzes can measure comprehension, knowledge, and recall, but they cannot measure attitude. Ideally, knowledge assessments should be more situation- or scenario-based and problem-solving-oriented, and they should be written to assess higher levels of understanding on Bloom’s Taxonomy. In reality, tests are not ideal because it takes too much time to develop a proper test. SMEs have a lack of experience in writing what they think are knowledge checks but are actually only measuring recall and comprehension. 

QUESTION 3: How do you evaluate Behavior ( or Level 3), such as [a] Degree to which participants apply what they learned during training when they are back on the job [b] Organizational support which reinforces, monitors, encourages, and rewards the performance of critical behaviors on the job [c] Organizational support for on-the-job learning opportunities

RESPONSE: Vivian said, “I’m in love with Level 3!” She calls this level “transfer” and notes a difference between changing behavior and changing performance, which Thomas Gilbert said are not the same. When you focus only on behavior, she explained, it’s implied that it will show itself observably on the job. The focus has to be on performance (with behavior being an aspect of it). One shouldn’t focus only on behavior and think that you’ll get change in performance on the job.

In the pharmaceutical industry, the ultimate measurement of effectiveness is whether you can do what you learned on the job without error.  The concern is not necessarily what you learned or how you learned it – what matters most is if you can perform your job without making a mistake. 

Vivian mentioned a book written by Donald and Jim Kirkpatrick called “Transfer of Learning” which expanded on organizational support and accountability.  She emphasized something very prevalent when you get to level 3 that is, “most failures of transfer have nothing to do with training.” There are “barriers to the transfer of training” that have very little to do with the quality of training or the quality of the trainer, and prevent the ability for employees to transfer learning on the job. Management doesn’t often acknowledge these barriers but blames them on bad training. [See Article – Training Does Not Stand Alone | Transfer Paragraphs]

QUESTION 4: How do you evaluate Results (or Level 4) or the impact of training on organizational performance?

RESPONSE: Vivian mentioned Jack and Patti Phillips, who happened to write Chapter 30 in our ASTD Handbook.  When I told her that I read about the ROI model, she responded that most business managers (not necessarily executives) are not really looking for the bottom dollar figure because it’s a tremendous amount of work. 

Vivian said that there is a concept called Return On Expectation (ROE), which she found very appealing. Demonstration that objectives were achieved could be a measure of effectiveness. She has become a huge fan of Robert Brinkerhoff’s “Success Case Method” that suggests Level 4 can be measured with qualitative data. 

QUESTION 5: How do you evaluate Return on Investment (or Level 5)?

RESPONSE: Jack and Patti Phillips did a lot of research pointing to a lack of support for evaluation studies and that not everything you deliver is worthy of a level 4 and 5 evaluation. The reason is that it’s very expensive and companies do not have the finances, people, etc. to commit to it. Jack Phillips said only 5% of courses and programs are really worthy of an ROI.

QUESTION 6: Please provide an example of the kind of data you would collect for each level?

RESPONSE: For clients desperate for metrics, the first and easy metric is completion against training requirements. But, she notes it’s a false metric because simply being at 100% in satisfying requirements doesn’t mean you won’t have deviations and errors on the job.

Another metric can be knowledge checks (targeting level 2) such as verbal or paper-and-pencil tracking scores. Metrics can also be obtained through a performance review or performance demonstration (targeting level 3).  Vivian has not seen a solid level 4 package in her specific industry but hopes to see one in the future.

QUESTION 7: Which level(s) is the most important to you? 

LEVEL 3 (see Question 3 Response)

QUESTION 8: Which level(s) is the least important to you? 

RESPONSE: Vivian has seen the use of the Level 2 written assessment abused by serving as a metric for terminating employees. This is not what tests were created for. Level 2 written assessments can lose their educational intention and value when turned into a performance issue under the guise that employees are not getting the knowledge. 

Remember, the goal of the training is for employees to perform their procedures without human errors after they return to work. It’s less about receiving 100% on a knowledge recall immediately after the session is over.

Vivian Bringslimark, HPIS Consulting, Inc.

QUESTION 9: Is there ever a time when training does not require evaluation?

RESPONSE: Yes, evaluation is not supported by employers when they only need documentation that employees attended the training. She has heard an HR director say, “I just need a checkmark. I need to produce training rosters to show the [agency] that we delivered what we promised”.

Vivian implements activities more indicative of transfer (of learning) than a written evaluation such as practice with simulated scenarios. One unique activity has learners develop marketing posters for future attendees and present them in class as their final activity, which is extremely engaging for learners and serves as a measure of effectiveness for the trainer. 

QUESTION 10: What do you think about The Kirkpatrick Model or the New World Kirkpatrick Model? What do you like or dislike and why?

RESPONSE: Vivian admits to mixed feelings about the Kirkpatrick Model. She noted that Donald Kirkpatrick, whom she has met, was intending to write a research paper (for his dissertation), but the model went rogue because people were grasping for anything to make evaluation tangible.  She feels the Kirkpatrick model can be a very plausible solution when management is demanding proof of learning. The Kirkpatrick Model works well when you’re in a classroom setting and you have a course.

Vivian explains the model’s limitations: it cannot be solely relied upon when expanding beyond the course into larger-scale program evaluations for performance improvement. The Kirkpatrick Model becomes challenging to scale up when enlarging the scope of the training programs to include site-wide initiatives.

However, Vivian likes the New World Model’s critical behaviors and leading indicators that were added because it makes the model more practical and even useful in program initiatives. By identifying the critical behaviors, the organization can likely sustain the new behaviors and therefore improve the performance outcome(s).  She said that leading indicators can serve as post points or check-in intervals that are more advantageous than waiting for the end when the ROI study is done.

RE: Assignment 6: Training Manager Interview Comments

1.) Wow, Stephanie, you did a great job with this interview on a very interesting interviewee. So interesting that Jack and Patti Phillips feel that not every course is worthy of a Level 4 or 5 evaluation, and that Jack said only 5% are really worth a level 5. I have never done or designed even a level 4 so I don’t have first-hand experience with the effort and expense involved but this is eye-opening – so it sounds part of the analysis might be to assess whether a course is worth going beyond level 3, making it a conscious, careful decision. This would be somewhat determined by the organization too – it would interesting to know what typical decision points are to conduct (finance) the more advanced assessments.

I really enjoyed this – in fact, I am about to read it the 3rd time!

2.) There is so much great content in your interview summary. I particularly liked the section on how Ms. Bringslimark is “in love with Level 3.” The distinction between changing behavior and changing performance is an important one. It really is all about how you can perform your job, not how you learned it. Over the last couple of weeks, I have also gained a measurable appreciation for Level 3 evaluations. I have been developing training for a workshop and feeling stressed over how to perform an effective level 2 evaluation. The idea of focusing more on the transfer of information and performance in the workplace has allowed me to concentrate my efforts on Level 3 evaluations.

Looking at performance evaluations for performance demonstration also resonates. In a new competency-based training (being developed) both performance evaluations and training need analyses based on the individual competencies will be used to determine transfer of learning and behavioral changes in the workplace. By using both the performance evaluations and the training needs analysis, both non-training and training issues can be determined.

Amazing job, Stephanie. Truly educational.

3.) Excellent thorough interview and post.  It looks like Vivian has built very strong evaluation processes in her design and execution.  

A few comments:

Level One – I agree “smile sheets” have many downsides, however, if they only identify a logistic item (the room was too small for effective learning) or as you indicated about the facilitator (the facilitator wasn’t boring) both of these bring value in the set up for the next training event.  

Level Three – Absolutely agree with her assessment of performance versus behavior.  I thought her take on the barriers to training was incredibly insightful.

Q # 8 – Her answer surprised me about level 2 being the least important. It is very hard to see which ones are least important because they all have value if used correctly.  I have found level 2 is critical to assess whether learning is actually taking place and hard to move on to 3 without a strong level 2 execution. 

Q # 10 – Great way to end the interview.  It was great to hear her perspective on the new world model. 

Again an excellent post.  Thank you.

Do you have an evaluation procedure or a training effectiveness process? Yes, you need to make this formal. Allow me to explain why.

(c) HPIS Consulting, Inc.

How committed to SOJT is your organization?

Are you looking for management support for the Qualified Trainers and the time needed to deliver SOJT?  If only we were required to have a procedure for that!  It may not be an SOP or even a policy document, but industry guidance documents provide a lot of references to management involvement.

ICH Q10 – Pharmaceutical Quality System

While not mandatory, management needs to seriously take notice of ICHQ10 guidance document released in April 2009 (1).  In particular to the following:

  • MANAGEMENT RESPONSIBILITY 2.3 Quality Planning 

“(d) Management should provide the appropriate resources and training to achieve the quality objectives”.

  • CONTINUAL IMPROVEMENT OF THE PHARMACEUTICAL QUALITY SYSTEM

4.3 Outcomes of Management Review and Monitoring

The outcome of management review of the pharmaceutical quality system and monitoring of internal and external factors can include:

(b) Allocation or reallocation of resources and/ or personnel training.”

Under a Quality System: Managers Expectations for Training

Referencing the Sept. 2006 issue of Guidance Document for Quality Systems: IV. Management Responsibilities | B. Resources (2), we see alignment with the training CGMPs for continued training so “as to remain proficient in their operational functions and in their understanding of CGMP regulations.”  “Typical quality systems training should address the policies, processes, procedures, and written instructions related to operational activities, the product/service, the quality system, and the desired work culture (e.g., team building, communication, change, behavior).”

And my personal favorite, “When operating in a robust quality system environment, it is important that managers verify that skills gained from training are implemented in day-to-day performance.”  The responsibility for training under a quality system is not assigned to just one person or one function.  It is a shared responsibility across the entire organization.

Goals of the Train-the-Trainer Program (TTT) vs. OJT Program vs. Employee Qualification Program

In order to have qualified employees, they need to receive structured on the job training delivered by a qualified trainer who is content qualified and training process qualified via a Qualified Trainers’ workshop.   Each of the three “programs” has defined outcomes that are dependent upon each other.  Unfortunately, the term program has been a bit overused throughout the years and can have a variety of meanings for folks.  For purposes of this blog, TTT program means OJT Qualified Trainers workshop, OJT Program means OJT methodology/procedure and Qualified Employee program means having a robust quality training system beyond the newest Learning Management System (LMS).

As you can see from the diagram, Qualified Trainers are at the core of all three “programs”.  The OJT QT Workshop is designed to prepare the QT’s for the realities of life as a Qualified Trainer. It includes the pre-work of familiarizing themselves with the Quality Training System procedures and then in class: exploring basic learning theory and committing to what the QT signature means. The OJT Program is about delivering structuring on the job training consistently following the approved methodology. An Employee Qualification Program is the validation of training effectiveness of the OJT methodology and the clarity of the underlying procedures.

Readiness Factors for SOJT and an Employee Qualification Program

Let’s start with a written purpose statement for having qualified employees beyond it’s required.  What is the company’s philosophy on achieving qualified status?  Is there agreement among the leadership for the level of rigor required to demonstrate performance and achieve a pass rating?  Where are the SOJT program goals written?  Does a schedule exist for SOJT and qualification events other than an LMS printout with required due dates.  That is not a schedule for SOJT.  Do you have clearly defined objectives for the QT workshop captured in a document or perhaps a procedure?  Is there a single owner for all three programs or is responsibility and accountability assigned accordingly?

Ronald Jacobs and Michael Jones, in their 1995 ground breaking book, Structuring on-the-Job Training, inform us that SOJT as a system functions within a larger context, namely the organization.  SOJT is not a standalone program.  Conflicts, competing priorities and mixed messages can influence the success of your SOJT program.  What else is going on in the organization that will compete for the same set of QTs? Remember they are also your most experienced and technical subject matter experts.  How is the overall Employee Qualification program aligned with the other quality systems? 

Recognition for QTs and Qualified Status

Most QTs are not fully dedicated to delivering training for departments.   There are pros and cons for this decision.  For now, I will leave them out.  Suffice it to say, they are tasked with both their “day” job and the responsibility for delivering training when needed.  They are at times, doing two jobs.  Whether or not they are compensated additionally for delivering SOJT, acknowledging their contribution to the department and the organization is part of management support.   It takes more than “you are doing a good job, keep it up”. 

Often supervisors and managers don’t know what else they are supposed to do to show their support, other than allow them to attend the QT workshop.  The interested ones will “pop” in during lunch and chat with their direct reports.  Others will show up at the end for the poster activity (equivalent to a written test) and some will come to learn about the parking lot issues that need follow up.  The energy in the room when this happens is amazing. 

“When operating in a robust quality system environment, it is important that managers verify that skills gained from training are implemented in day-to-day performance.

Guidance Document for Quality Systems, Sept 2006

To help ease the knowledge gap between a manager and their now Qualified Trainer, I started delivering the Leadership Briefing module prior to the QT workshop delivery.  The purpose is to provide an overview of the content highlights, alignment with initiatives / CAPAs/ agency commitments and more importantly to secure agreement for the following:

  • criteria for nominating a QT
  • roles and responsibilities of QT
  • scope of work QT’s can be assigned
  • expectations for QT’s post launch
  • what happens day one after workshop is done
  • what is the status of the SOJT checklists
  • scheduling and budget concerns.

If the organization says they support the qualification program, then what happens when employees achieve qualification status?  Nothing?  A non-event? Or is it announced in newsletters, plasma screens and other company announcements?  Is it a big deal to be able to perform independently and free up a much-needed QT for another learner?  I keep hearing over and over again about how there aren’t enough QT’s to deliver SOJT the right way.  One would think qualification status on SOPs, especially big complex processes deserves SOME kind of recognition.   Just how committed are the managers and supervisors?  QTs and employees draw their own conclusions about the level of real management support for the programs.  

Supporting QTs is more than participation in QT Workshop

If truth be told, after launching the QT workshop, many supervisors privately don’t support the program.  They lose their top performers during the workshop and the hours it takes to train someone. Forget about giving QTs adequate time to complete the paperwork properly! And then leadership wonders why good documentation practice (GDP) issues continue to be a problem? The non-distracted performance observations that QT’s are expected to conduct for the qualification demonstration drive supervisors and line managers crazy the most – what, they can’t do anything else but observe?  Hence, many QT’s are asked to multi-task just to get the work done: not enough resources they are told.  For supervisors, productivity and the workload will always trump SOJT and qualification events, until their bonuses include completion of training and qualification events.

What Real Support Is Supposed To Look Like

My key take away message is that attending the TTT program/ QT workshop is not the end of the OJT program or the Employee Qualification Program but rather the launching point.  Management support needs to go beyond just nominating QTs and allowing them to participate in the workshop.  The real support is in the alignment of goals, clarifying expectations continuously, allocating resources for training and budgeting time to deliver OJT using an approved OJT methodology that includes qualification events.  This commitment of time and sponsorship for qualified employees is a culture shift for many line managers and site leaders.  But actions do speak louder than words.  -VB

(1) Guidance for Industry Q10 Pharmaceutical Quality System | US Department of HHS | FDA
| CDER | CBER April2009 ICH

(2) Guidance for Industry Quality Systems Approach to Pharmaceutical CGMP Regulations U.S.Department of HHS | FDA | CDER | CBER | CVM | ORA| September 2006 Pharmaceutical CGMPs

Jacobs RL, Jones MJ. Structured on-the-job training: Unleashing employee expertise in the workplace.  San Francisco: Berrett-Koehler: 1995.

How can you tell if your site is ready for SOJT?

HPISC has created a 2 part checklist of questions and attributes to explore. The checklist is available gratis when you become a HPISC mailing list member. Just be sure to include that are you interested in receiving the SOJT Readiness Checklist.

Mailing List Subscribers automatically receive newest assets and tools directly into the inbox.

Who is Vivian Bringslimark?

(c) HPIS Consulting, Inc.