Why Knowledge Checks are Measuring the Wrong Thing

When I taught middle school math, tests were used to assess knowledge comprehension and some application with word problems and a few complex questions requiring logic proofs. Results were captured via a score; a metric if you will as to how well you answered the questions and very appropriate in academia.

In our quest for training evaluation metrics, we have borrowed the idea of testing someone’s knowledge as a measure of effectiveness. This implies that a corporate classroom mirrors an educational classroom and testing means the same thing – a measure of knowledge comprehension. However, professors, colleges, universities and academic institutions are not held to the same results oriented standard. In the business world, results need to be performance oriented, not knowledge gained.

So why are we still using tests?

Call it a quiz, a knowledge check or any other name it is still assessing some form of knowledge comprehension. In training effectiveness parlance, it is also known as a level 2 evaluation. Having the knowledge is no guarantee that it will be used correctly back on the job. Two very common situations occur in the life science arena where “the quiz” and knowledge checks are heavily used: Annual GMP Refresher and Read & Understand Approach for SOPs.

Life sciences companies are required by law to conduct annual regulations training (GMP Refreshers) so as to remain current. To address the training effectiveness challenge, a quiz / questionnaire / knowledge assessment (KA) is added to the event. But what is the KA measuring? Is it mapped to the course /session objectives or are the questions so general that they can be answered correctly without having to attend the sessions? Or worse yet, are the questions being recycled from year to year / event-to-event? What does it mean for the employee to pass the knowledge check or receive 80% or better? When does s/he learn of the results? In most sessions, there is no more time left to debrief the answers. This is a lost opportunity to leverage feedback into a learning activity. How do employees know if they are leaving the session with the “correct information”?

The other common practice is to include a 5 multiple choice as a knowledge check for Read & Understood (R & U) SOPs especially for revisions. What does it mean if employees get all 5 questions right? That they will not make a mistake? That the R & U method of SOP training is effective? The search function in most e-doc systems is really good at finding the answers. It doesn’t necessarily mean that they read the entire procedure and retained the information correctly. What does it mean for the organization if human errors and deviations from procedures are still occurring? Does it really mean the training is ineffective?

What should we be measuring?

The conditions under which employees are expected to perform need to be the same conditions under which we “test” them. So it makes sense to train ‘em under those same conditions as well. What do you want/need your employees (learners) to do after the instruction is finished? What do you want them to remember and use from the instruction in the heat of their work moments? Both the design and assessment need to mirror these expectations. And that means developing objectives that guide the instruction and form the basis of the assessment. (See Performance Objectives are not the same as Learning Objectives.)

So ask yourself, when in their day to day activities will employees need to use this GMP concept? Or, where in the employees’ workflow will this procedure change need to be applied? Isn’t this what we are training them for? Your knowledge checks need to ensure that employees have the knowledge, confidence and capability to perform as trained. It’s time to re-think what knowledge checks are supposed to do for you. – VB

Need to write better Knowledge Check questions?  Need to advise peers and colleagues on the Do’s and Don’ts for writing test questions?

Calling ALL User Generated Tools Home

What do I mean?  You know, job aids, tools users have created, and SME cheat sheets.  I’ve even seen task instruction sheets, quick reference guides for completing forms, and process flow diagrams.  But I’m not talking about posters on the wall describing how to turn on the projector in the conference room.  In this 3rd issue of Making It Work for Compliance Trainers series, I blog about why creating and openly sharing user-generated tools may not be a good thing in a regulated environment.

The Dilemma

As a Performance Consultant (PC) or HPT specialist, works with SMEs, Key Performers, or STAR employees, s/he invariably uncovers or discovers that their SMEs have “other” tools they’ve developed that help them be so good at what they do.  While these are helpful to the key performers, it presents a dilemma for the PC who is also a Compliance Trainer or a QA Manager.  “If I expose the source of their secret sauce, will I break trust and create a barrier to the relationship?  On the other hand, if I don’t speak up about this tool, what assurance do I have that the content is approved by the Quality Control Unit (per GMP) and is version controlled?

Why create them in the first place?

To get grounded, the PC/Compliance Trainer needs to perform a quick cause analysis upon the discovery of the tool.  Why was it created in the first place?  Is there information or steps missing from the standard operating procedure (SOP)?  Was this tool created to “chunk up” the steps or create bite-sized training materials that evolved into a job aid?  Or is it a maneuver to bypass the change control system?  The answers to the questions could provide the basis for a more user-friendly revision or at least be officially approved as a supporting tool to the SOP upon the next version release.

What’s the big deal?

Rejection of product, deviation from approved written instruction that could result in adulterated product, additional follow up testing, and rework are all forms of waste to the organization.  Not to mention that consistency is the key to compliance and assuring public confidence in approved marketed products.  If folks are not using the approved procedure, then there’s an issue somewhere.

To what level of control is needed?

That is the most sought after question regarding job aids and user tools.  The answer lies in each company’s level of risk and its document hierarchy.  I’ve seen extreme cases where “NO Paper” on the floor means not even an SOP is allowed to be in hand.  I do believe that some level of control is needed to ensure that the content is valid, is in sync with the current procedure and users have the most current version of the tool.  Can your organization defend the level of control?  Are you sure about that?  Or do you use a “don’t tell and we won’t ask policy”?  Are folks making errors because they followed an uncontrolled worksheet vs. the approved procedure?

Tips for Establishing Level of Control

  • If the tool /job aid is tied to a procedure, it needs to become a required tool and included in the SOP.
  • Job aids should not be a standalone orphan.  It needs to have a procedure that it supports.  The use of the job aid is included in the hands-on training so folks know how to use it properly and where to access it if it is not “attached” to the SOP.

o   For example, some companies have a separate numbering system for these exhibits and the storage location may not be in the same folder directory as the parent SOP.

  • If more than one tool / job aid / worksheet exists per procedure, then an appendix or reference section needs to highlight the existence of these “tools in use”.
  • Establish an amnesty initiative to raise awareness for the quality and compliance consequences of using uncontrolled tools.

Calling all User Generated Tools Home

The purpose of the initiative is to allow users to admit that they have these tools and that no performance consequences will follow when they surrender them.  The second focus of the program is to find a proper home for these tools once they are deemed valuable.  They need proper care and nourishment.  In other words, content is valid, accurate, up to date and approved for use.  The PC/Compliance Trainer is the ideal conduit to make this happen.

One company that I visited did just that and more.  Once it was discovered that a series of mistakes were coming from an old tool that had been downloaded and copied to their desktop, a team of auditors was dispatched to observe the removal of all tools from employee’s desktops.  The 2nd phase of their program was the identification of an owner for the share-point site who now manages access and content revisions.  The 3rd phase includes a content/tool submission process that is vetted by a designated users group of SMEs.

Is it time for a Job Aid/Users Tool Amnesty Project where you work? – VB