Taking the Agile Learning Plunge

When Rapid Design for eLearning found its way into my vocabulary, I loved it and all the derivatives like rapid prototyping, etc.  And soon, I started seeing agile this and agile that.  It seemed that agile was everywhere I looked.  When Michael Allen published his book, LEAVING ADDIE for SAM, I was intrigued and participated in an ATD sponsored webinar.  It made a lot of sense to me and “I bought into the concept”.  Or so I thought …

I joined a project that was already in-progress and had to “hit the ground running to get caught up to speed”.  The element of urgency was the anticipation of a post FDA visit following a consent decree.   If you’ve experienced this “scene” before, you can relate to the notion of expedited time.   As part of remediation efforts, training events needed to be conducted.  I learned during a meeting sometime my first week, I was to be the trainer.  Okay, given my instructional design background and classroom facilitation experience, that made sense.  Sure, in a few weeks when we have the new procedure in place, I’d be happy to put the training materials together, is what I was thinking.  Wait, what, in two weeks?  Are you kidding me?  I’m not the SME and I don’t even have the software loaded on my laptop yet.  Well, some cleaned up version of those words was my response.

My biggest challenge was to get out of my own design way

I’m classically schooled in *ADDIE with 30+ years as an instructional designer and very comfortable with how to design, develop and deliver training.  All I needed was more time; more than two weeks, for a process that was changing daily!   And then I found myself thinking about all the buzz for rapid design and prototyping I had been reading about.  

*ADDIE = Analysis, Design, Develop, Implement, Evaluate: a project management approach to training projects.

In theory, I totally bought into it. But this is different I argued with myself.  This is compliance with a quality system for a company that is undergoing transformative change as a result of a consent decree!  Furthermore, I teach GMP Basics and conduct Annual GMP Refreshers several times a year. My GMP dilemma challenged the very essence of my “learned” compliance beliefs about following the 1st basic GMP Work Habit – “thou shall follow written procedures” otherwise, it’s a deviation. 

Are we really planning to deviate from the SOP while under a consent decree?

While it was the intention of the business unit leader to deviate from the approved set of work instructions, a planned deviation would not be appropriate in this case.  I mean we were talking about a corrective action for a consent decree item.  Were we really considering a PLANNED DEVIATION to intentionally teach unapproved procedures and then submit the documentation as a completed corrective action for the CAPA to the agency?  I was truly baffled by how I was going to pull this off in two weeks.  I’m not a magician, I can’t pull this rabbit out of my laptop is what I was thinking when I left the VP’s office.

Yet on the other hand, how could I teach a work instruction that was known to be broken; was being re-designed and not yet finalized?  The instructional designer side of me screamed – how can you teach flawed content?  That’s wasted training that results in scrap learning. How is that training going to be effective not to mention having to explain a failed effectiveness check during the post inspection?

And then, it hit me!  I was so focused on WHAT I NEEDED, that I was missing the urgency of the learners’ needs. Julia Lewis Satov refers to this situation as ‘agility by fire’ – “the ability to move quickly but not easily, and still excel”, (p. 50, 2020). It was time to put theory into practice and take the agile learning plunge into the realm of the unknown.  If I could come up with a way to document what we were doing and get it approved, then I could reconcile my GMP dilemma and satisfy my instructional designer. 

 With a little help from my validation colleagues – the training implementation plan

Validation engineers use protocols to capture their “change in process” work.  Whether it’s experimental batches, 3 batches for process validation or **IQ-OQ-PQ protocols for equipment qualifications.  They are validating the procedure or the new process before it can become the standard operating procedure by developing the plan, developing acceptance criteria, managing deviations and capturing the results.  So why couldn’t I borrow the concept and adapt it to my situation?

**Installation Qualification, Operational Qualification, Performance Qualification

The purpose of the initial training session was to test the new sequence of steps and confirm the robustness of the software responses for each field entry and then make correct decisions where needed.  The learners were still in compliance with the quality policy for complaint handling and were still meeting the intention for Medical Device Reporting requirements.  They were essentially testing the future “how-to steps” for the proposed new work instructions.

Agile QT’s processing their learning experience

I did not copy and paste the validation protocol template. I did, however, include a please pardon our appearance while we are under construction” paragraph in the training plan to document the departure from the current set of work instructions.  This protocol-like section also included our intentions for the outcomes of the sessions. We also stipulated that required SOP training of all affected users including the Qualified Trainers, would be mandatory once the finalized set of work instructions were approved.

Anybody want to play in the sand-box?

By shifting the prioritization away from perfectly designed classes with pristine training materials, I was able to diagnose that the need was to get the learners into a live classroom. But first I needed a small group of super users who wanted to see the database software in action and “play in the sandbox”; the training materials could follow afterwards. 

It didn’t take long for them to find me.  These “learning-agile individuals” wanted the challenge of not only learning something new but seemed to thrive on the idea that they would be managing their part of the training implementation plan.  They were not at all worried about the lack of available training materials for themselves.  They allowed the learning experience to occur spontaneously.  Their ability to learn new knowledge and skills did not get in the way of previously learned skills. They embraced the changes rather than resist them.

A new breed of SMEs as Agile Qualified Trainers?

I shifted my role to facilitator and allowed these learning agile SMEs to navigate the software screens and then work out the explanation of how to complete field transactions.  In the Center for Creative Leadership “Learning Agility” white paper, authors Adam Mitchinson and Robert Morris explain that learning-agile individuals understand that experience alone does not guarantee learning; they take time to reflect, seeking to understand why things happen, in addition to what happened”, p. 2.

“SMEs are true front-line and onsite educators” says Satov.  Every organization has employees who are brimming with intelligent and diverse ideas and are eager to share their talent producing work deliverables. “[…] Our focus must shift to finding and developing individuals who are continually able to give up skills, perspectives, and ideas that are no longer relevant, and learn new ones that are”, (Mitchinson and Morris, 2014, p.1).

We documented these sessions as training because we all learned how to navigate the screens; albeit it was learning on the fly.  We recognized that learning the software was the goal.  Developing the process steps and eventually the work instructions was the secondary goal.  This training documentation became the qualifying evidence for their train-the-trainer knowledge transfer.  And collectively they decided what choices end users were to pick from the drop down tables.  

Is this “learning on the fly” or agile learning in practice? You decide.

1 + 1+ 1 is more than 3

I shifted my role again to become a scribe and worked on sequencing these pages for the next round of end-users. To my surprise and delight, my new breed of Agile QTs volunteered to paste screen shots into participant worksheets so their “students” could take additional notes.  Together, we all collaborated to meet the urgent need of the end-users. Each of us in our niche roles experienced first-hand the value the others brought with them to that room.  And in that time away from our regular job tasks, we became more valuable to the organization.

The learners were paired up with their Agile QT for guided instruction of real entry into the live system.  The following week, the department was able to go live with a project plan that focused on a series of interim roles, changed roles and transitioning responsibilities within established roles.  The project launched on time to meet commitments promised to the agency.

Why are they thanking me?

It was an energizing and empowering learning experience for the super-users. A truly collaborative experience for the SMEs and the biggest surprise of all was that they thanked me.  Me?  I did not deliver the training; I was not the SME, nor did I provide perfect training materials.   If I had pursued my classically trained ADDIE approach, we would have waited for the perfect SOP to deliver those sessions and woefully miss FDA committed timelines. While I’m not ready to throw ADDIE overboard yet, Satov makes a compelling plea, “move aside elite and long-standing establishments of formal education”. 

My lesson learned was this: when the demand is for speed and the content design is not the key focus, I need to give up control to the true onsite educators and focus on facilitating the best learning experience given the daily change challenges and system constraints. Satov would agree, “the role of learning is to capitalize and create the architecture of the hybrid-mind”.  Is this “learning on the fly” or agile learning in practice?  You decide. But agile instructional design is here to stay if QA L&D is going to keep up with the fast-paced, often reactive, and regulated world of the Life Sciences Industries. – VB

  • Allen, M. Leaving ADDIE for SAM: An Agile Model for Developing the Best Learning Experiences. ASTD, 2012.
  • Mitchinson, A & Morris, R. Learning Agility. Center for Creative Leadership white paper, 2014.
  • Satov, JML. “Agile by Fire”, Chief Learning Office, July/ August, 2020, p. 50.
Need to expedite a CAPA remediation project? |Looking for a facilitator/ quality systems project manager to align your SMEs for collaborative deliverables?

Who is the Author, Vivian Bringslimark?

(c) HPIS Consulting, Inc.

Please Pardon Our Appearance: We Are Remodeling Our Quality Training System!

An urgent request to fix a broken training system is typically the point of entry for my Robust Training System (RTS) makeovers.  When I join the conversations, I discover that a lot of “work” has already begun.  So, I take a deep dive into the documentation to “hit the ground running and get caught up to speed” as quickly as I can. By this time, the agency response letter has already been sent with commitments to correct observations and promises to fix system issues.  And immediately, CAPAs are opened to track these activities. Sigh.  [CAPA = Corrective Action Preventive Action]

So often, immediate fix CAPAs are generated so quickly and only focused on the fast fix of the observation, that the short-term benefit becomes a major barrier for the long-term prevention of a systems upgrade.  The advice that is given to a CAPA owner is “don’t fixate on the big picture right now, we can always revise it later…”. “We’ve only got 15 days to respond to the agency”.  While revisions are part of our SOP management system, revising the CAPA fix to something else in the next revision can appear to look like it was only for the FD-483 observation or the Warning Letter response. 

And a lot of the time, those CAPA fast-fixes conflict with a proposed future state.  It can become a huge obstacle to redesigning an effective and efficient quality training system.  There is a belief that it takes too long to map out the proposed future state and pinpoint exactly where the observation or citation gets corrected.  Given how some redesign projects are managed, there is a lot of truth to that belief.  So, the rule of thumb is to correct first, then come back and “improve” later when site employees have a moment to catch their breath.  But closing out those fast-fix CAPAs in record time prevents redesign projects from moving forward because available SMEs are busy executing the corrective actions.  What’s really frustrating is realizing that the fast fix is incompatible with the proposed future state. Sigh again.

Little CAPAs and Big CAPAs

By little CAPAs I am referring to the immediate fix corrective actions and in no way am I minimizing their importance.  Their scopes tend to be single focused and short term.  By comparison, big CAPAs are created when many of the little CAPAs are rolled into one big CAPA.  The scope is expanded to include the related little CAPAs because the corrective actions relate to each other, impact each other, and may produce repetitive paperwork.  So, all the CAPAs get bundled together in an overall CAPA aka, the big CAPA.

Little CAPAs and Big CAPAs

When this occurs, be careful that little CAPA commitments to the agency via response letters don’t get lost in translation.  By that I mean, the fast fixes need to remain visible or traceable in the deliverables of the project.  If by chance those steps were removed or changed, be prepared with heavily documented justification with appropriate approved signatures like Department Heads.  Take note, merging little CAPAs into one big overarching CAPA usually changes the completion timeline given that the scope of the big CAPA is bigger.  For most projects, this is a welcomed relief.  It can also change the timelines of the little CAPAs.  Be mindful of the commitments made to the agency. 

But We’ll Be Out Of Compliance The Second The SOPs Go Into Effect!

One way to keep multiple CAPAs visible is to create subprojects that are also managed within the overall future state project.  Sometimes these projects have completion dates projected to be 6 – 9 months out.  Yet, the future state quality system procedures need to be operational before these subprojects can come to fruition.  In essence, the day the SOPs go into effect, the site would automatically be out of compliance and numerous deviations would have to be initiated.  To manage the SOP change gap, an overarching CAPA (the Big CAPA) can be generated to allow the organization to meet agency milestones and move forward with the new process while working on the subprojects.  The key to this approach is ensuring that the implementation plan is written, approved, up to date, and indicating progress forward.

During the development of the last set of redesigned SOPs for a client, FDA conducted a follow-up visit for the original FD-483 inspection.  One of the little training CAPAs was not yet completed. It was waiting for the redesigned set of procedures to go into effect.  The new version would encompass this observation and essentially close it out.  

As part of their Go-Live strategy, a decision was made to schedule the release of the revised procedures all at once when the final set of SOPs was ready to go to Change Control.  It was determined that too many little changes for the Training System would result in confusion, retention issues, and a paperwork nightmare controlling which version of the form was correct.  This is a popular option for design teams because the coordination of new changes happens on the same day and is much easier to track. Albeit the learning curve can be high as the site works on closing the size of their change gap.

Unfortunately, this incomplete CAPA created a hold for product to be released.  As a result, the priority to complete a retrospective qualification of 700(+) SMEs as Trainers derailed the Training Quality System project for close to 4 months.  Painfully, additional immediate fix CAPAs were also generated.  Ironically, these CAPAs could have been avoided if the design team chose to release each SOP as it was ready instead of waiting for all of the SOPs to go into effect all at once.

Be Mindful Of The Quest For The Perfect SOP

The most typical corrective actions involve revising procedures to include a changed step, a form revision, or in some cases, a completely overhauled process that includes multiple work instructions or SOPs.  Standard Operating Procedures mean a description of the approved steps for the task at hand to be followed repeatedly without exception or deviation.  But SOPs are not static, they change frequently which is a good thing for folks who seek improvement and a source of exasperation for the end users assigned to keep current with their procedure revisions.  

Change fatigue is real and can often be the root cause of SOP deviations.  CAPA owners need to be made aware of the learning curve for major revisions. Hence, awareness training and primary end-user workshops are integral for implementation strategies.  However, the antidote for SOP change fatigue is not waiting for the perfect SOP that will never change again!  But knowing in advance that a procedure is going to revise multiple times is also frustrating.  Just when end users get used to the change(s), the SOP changes again and quite possibly re-introduces a host of deviations almost suggesting that the changes are a waste of time and energy. 

There Is No Such Thing As The Perfect SOP; Don’t Delay CAPAs Unnecessarily

What then is the remedy? Conducting impact assessments of the change(s) and then managing the go-live strategy for awareness training, triage support, and FAQ type communications depending on how big the change gap is.  And this is exactly the approach that one Quality System L & D Manager implemented as part of the go-live strategy for a major overhaul of her training system. 

With careful examination of what tasks, forms, and steps were actually changing, the Project Lead was able to realistically determine how much time it would take to mitigate the impact.  Subtasks such as final form revisions and revisions to other documents were monitored and project tracked as well.  This information shaped a pragmatic and timed sequence of events that lead to the conclusion that releasing one SOP at a time was the most effective way to transition into future state with minimal disruption despite that the overall quality system policy document would have to be revised multiple times during the transition period.  The project lead volunteered to be the task master on those subsequent policy changes and updates for the stakeholder briefings.

Agile Approach Enables SOP Change Resilience

The timed sequencing of these revised SOPs addressed potential version confusion and change fatigue.  By managing smaller rollouts, there was less impact on the organization.  The Training Manager was able to introduce the employees to SOP change resilience by combining awareness training, project briefings and end user feedback for the “next SOP revision”.  Rather than dread the next release, end users discovered that their feedback was taken into consideration.  And in most SOPs, the next release didn’t really change anything but rather added another component or element of the future state that they were now ready to implement. 

The iterative design and roll out expedited a continuous improvement cycle that kept the project moving forward to meet FDA commitments, milestones, and stakeholder expectations.  -VB

There is no such thing as the perfect SOP!
Is your quality System redesign project not progressing as you expected? let’s have a conversation!

Like the blog? Subscribe today. Sign up in the left sidebar menu.

Who is the Author, Vivian Bringslimark?

(c) HPIS Consulting, Inc.

Will You Please Just Sign the Training Form!

For companies who are required to have standard operating procedures (SOPs), managing changes and ensuring affected employees are up to date is a constant challenge. Especially for companies whose weekly revision churn rate is from 10 to 150+ revisions. The standard approach is to “get ‘em done” in the fastest amount of time. The end goal for most people is the proverbial (√) and lots of records claiming “read and understood” the procedure.

This blog has been merged into blog – “Why We Love and Hate Training Curricula and our LMS”.

Who is Vivian Bringslimark?

(c) HPIS Consulting, Inc. All rights reserved.

May I See Your Implementation Plan for this Change Control?

In Part 1, we find Cara, a performance consultant has been hired to help a former client with implementing a robust training system. After waiting 3 months for the executive leadership group to get aligned around the priority for Miguel’s RTS project, Cara finally got to debrief her assessment findings. But a new development surfaced that was unexpected.

In part 2, we observe how Cara brings her inexperienced design team up to speed on how to be a team.

In part 3, we learned how Cara facilitated the design team of SMEs through various stages of working as a team to manage internal politics and a team member’s personal agenda.

In this Final Part of the Change Readiness Gap Impact Story, the design team launches and gets a surprise visit from the agency.

“But future state is being designed on the assumption that change control will be redesigned first.  We still have a lot of preparation work to do before we are even close to submitting these for change control.  And that is why change control is out of scope for this team.  We will not delay our deliverables because we decided mid-stream to go fix change control first.   There are plans for a change control project team to begin and some of you may be tapped to participate.”

After the change control rant from their distressed team mate, whenever anyone even mentioned the word change control, they joked and said: “we’re not allowed to discuss change control anymore, remember?”

Robust Training System SOPs: IMPLEMENTATION: The GO-LIVE STRATEGY

Through the efforts of Miguel negotiating behind the scenes, a new quality manager, Stuart, was hired right as the team began to work on the implementation strategy.  The timing was ideal because the team was ready to present their recommendations on how to go live and this was Stuart’s first priority.

Meet your new project leader

While Stuart got caught up to speed and completed his onboarding tasks, Cara transitioned out of the project manager role and back into external consultant mode.  The team had successfully designed their process flows and together decided the number of procedures that made sense for the organization as well as where to park the content.  They collaborated on the design of forms while leaving room for flexibility given the nature of work for each department.  The team had two proposals that competed with each other and Stuart, now fully up to speed weighed in with his decision.

Critical vs Important Focus

Critical vs. important: OJT documentation or curricula accuracy

The first proposal mapped a path forward based on OJT as the priority.  This was clearly identified in the gap assessment report and what appealed the most to the executives given their business objectives.  The second proposal was logical and made more sense to start from an overhaul of the curricula; ensuring that the training and qualifications were the right requirements for the right roles.  At the next meeting, Stuart took the lead and announced that he chose the curricula proposal and would defend this choice to Miguel and the executives as his first major task assignment.

Stuart was successful in his curricula proposal pitch with the executives.  He was able to make a compelling argument for both efficiency and effectiveness.  His next task was to finalize the implementation plan.  So he asked the team to meet once again to refine the “Go-Live Strategy”. 

“Please Pardon Our Appearance”

In order to move forward with the necessary tasks, the team needed the authorization to complete the work using the newly designed forms and process without approved Standard Operating Procedures (SOPs).   Releasing the new design before the sub-tasks were complete, would create an “out of compliance” situation at the onset of launch.  And yet, going forward without an approved procedure also put them in SOP violation status.

GMP Redesign SOP Dilemma

As part of the team’s drafted implementation plan, Stuart generated a change control package that documented the project, articulated the necessary steps, included the process flow diagrams as well as the forms.  Included in this packet, was the timeline for execution.  The effect of this documentation was to communicate that all these changes were not out of control but rather a controlled and planned change in the current procedures in order to make them standard once the subtasks were in place such as the updated position curricula.

Delivering the Qualified Trainers Workshop was a critical implementation task that needed to be timed with the SOP roll out.  The day before the Qualified Trainer’s workshop was to be delivered by Cara, Stuart called Cara and announced that a regulatory inspection was to begin on the same day.  While disappointed with the delay, Cara was optimistic that the training portion of the inspection would be favorable.   One month later, Stuart phoned her back.

“Can you deliver the workshop next week?” Stuart inquired.

“Yes, I’ll clear my calendar.  But how did training do?”

All in all, it went well.  We have some issues of course,” Stuart added.

“What about the training implementation plan? Was it accepted or challenged?” Cara asked.

“Well, we didn’t get cited for being ‘out of compliance’ so that was good.  But it was clearly stated that the plan MUST be executed ASAP! Hence, the reason for my call, today,” he answered.

Cara was pleased to hear that the Robust Training System (RTS) training project finally became a “Top Site Priority”.

END RESULT: Future State is now Current State

A few months later, Cara also delivered a GMP Basics course and qualified the site trainer to deliver it routinely as per the new training procedures. The RTS project was now officially closed and life as new normal began.   Their next follow up inspection was favorable.  A few minor issues and some verbal comments for training.  With this earned “regulatory approval”, the company was able to move forward with plans for launching their new product.

LESSONS LEARNED: Breaking down silos one meeting at a time

While the project team of SMEs learned how to collaborate in order to achieve project charter deliverables, the business units were very much still entrenched in their functional silos, defending current practices.  Deviating from approved procedures, even with a regulatory recommendation to prioritize the execution of the training plan, was not well received.  Stuart and his staff faced resistance from front-line supervision with right-sizing their curricula.  Through determination and persistent “working meetings”, the curricula sub-project finished.

Given the curricula “battle”, Stuart initially backed off from communicating the big project picture in the hopes that early accomplishments would inspire the front line to continue with the tasks and not overwhelm them with too much change at once.  

Instead, the OJT checklist sub-project was also slow, tedious, and a struggle.  Incumbent subject matter experts (SMEs)were reluctant to share their expertise or participate in the generation of the OJT Checklists, let alone be required to use them and not change the content without following the change control process.  Ironically, as new SMEs were vetted, the quality of the content improved and the checklists became a non-issue.

KEY TAKEAWAYS:

  1. The decision to delay project launch until Miguel felt confident that the executive leaders would sponsor the project and approve resources for the design team was paramount for keeping the momentum going forward after the initial launch meeting.
  2. Miguel recognized early on that the identified design team SMEs needed a specially developed curriculum to prepare them for the challenges that lay ahead.  The first four meetings as introductory lessons provided context as well as content and established the project lexicon while reinforcing team ground rules.
  3. Being prepared to defend the change control packet and explain the “Go-Live” Implementation Plan with FDA investigators not only gave credence to the project but it also elevated the importance of plan execution and made completion an urgent priority. -VB

Who is Vivian Bringslimark?

HPISC has articles, impact stories, and white papers.

Visit the Robust Training System Resource Library

(c) HPIS Consulting, Inc.

When SMEs have too much “secret sauce”

Many QA/ L&D Training Managers are tasked with improving their “training program”. An integral component of a robust quality training system is the Qualified Trainers (QT). Having a cadre of existing department Subject Matter Experts (SMEs) as Trainers can be very helpful when implementing the rollout of the new quality redesign to meet regulatory commitments and expected timelines. But, sometimes it can also lead to sustainability issues after the launch is over and the next big project becomes the new site priority.

During my on-site response to an urgent performance problem, the Head of Operations expressed deep concerns about inconsistent OJT being delivered by his trainers. A series of significant non-conformances occurred in his area. As part of the CAPA (Corrective Action Preventive Action) investigation, trainers were interviewed to uncover how they trained the identified employee(s) and what was said specifically for each step of the procedure. Their responses revealed a lack of consistent process and the use of varied content; despite having an OJT checklist, the procedure, and approved Training SOPs.

Once a Trainer; Forever a Trainer

I was then invited into a conversation with the Training Operations Manager (My Performer), regarding her desire to upgrade the existing department SMEs as Trainers. Responsible for the effectiveness check of the CAPA corrective action and the overall quality of Operations OJT sessions, she complained that many of the trainers should no longer be considered Dept. Trainers. While she had position title influence, she was frustrated by the lack of support for her “improvement suggestion”. I became her catalyst to help her push through the fixed barrier regarding SMEs.

The site followed a cultural assumption regarding department SMEs: once a Trainer; always a Trainer; regardless of feedback and informal impressions of their ability. Without any tangible criteria and lack of assessment tools, my Performer had no authority to remove the underperforming Dept. Trainers. Granted these SMEs were long ago chosen when the widely accepted practice of being proficient as a technician after a year earned them the designation of subject matter expert and automatically, a Dept. Trainer. Today, the Life Sciences Industry, with FDA investigators’ observations, has evolved their understanding that it takes more than seniority and R & U SOP training to become an OJT QT. Unfortunately, the environment where my Performer worked, the mindset about acquired expertise still held.

Significant CAPAs can be Drivers for Change

Undaunted, my Performer seized the CAPA as an opportunity for change. Leveraging suggested criteria and the use a form to document justification for each Dept. Trainer, she now had a process (SOP with form) that she could “educate” her colleagues on what it takes to become a Qualified Trainer. The focus of her message dramatically changed. She became strategic in her communications, using the effectiveness check portion of the CAPA as her “Why / WIIFM for Operations Managers”. In order to close out the CAPA, Managers had to complete their portion of the form.

The long-term success of my Performer depended on her owning her solution. She never lost of her original desire; she was patient and waited for her colleagues to accept today’s best practices for OJT QTs.   In the meantime, we brainstormed on a variety of feedback options that could be used to evaluate the current status of each SME at the same time the Managers completed the new form. My Performer chose a rating system and arranged for a 1-1 sessions with Operations Managers to discuss what rating they would use for each criteria if they got challenged during a CAPA investigation or a regulatory inspection.

While the results were not formally documented, my Performer was effective with the assessment rating exercise.   The Managers reconsidered who they wanted to nominate based on the new formal criteria and the informal ratings discussions. They did not automatically submit the form for all existing Dept. Trainers. A constructive dialogue then ensued regarding skills remediation support for those SMEs deemed as potentials. At last, my Performer achieved her desired outcome. “As catalysts, we build a bridge, light the path, and give [ ] our hand to help [ ] demolish or jump over obstacles”, (Haneberg, 2010, p.96). I was privileged to be part of a dramatic shift in their training culture.

An alternate alignment exercise

For many, adding ratings suggests a formal performance assessment and this can raise HR issues if not fully supported by the organization. In addition, many Operations Managers do not have the luxury of “weeding out undesirables”. They simply do not have enough SMEs to complete the training curricula generated requirements. Yet, there needs to be mutual consent between manager and identified SME in order to effectively deliver the OJT Methodology and to ensure a successful learner experience.

For those situations where automatically re-nominating existing SMEs is raising a red flag, I created the Trainer Mojo Assessment.  Nominated SMEs and existing SMEs as Trainers rate themselves on 10 attributes that align with the characteristics of an effective OJT Trainer.   Low scoring SMEs/QTs are encouraged to have a discussion with their management regarding continuation in the program and possible action steps. For SMEs/QT’s that score in the On-Target range, this is both validation of the nomination and confirmation that manager and QT are in sync. For high scoring QTs, this is also confirmation and an early indicator for potential QT Rock Stars!

Haneberg, L. Coaching up and down the generations. Alexandria, Virgina:ASTD, 2010.

You might be interested in the Impact Story – From Dept. SME to QT.

From Dept. SME to QT

The Client Request –”Can you help us upgrade our Trainer Qualification process?”

Who is Vivian Bringslimark?

(c) HPIS Consulting, Inc.

So, we went LIVE, what happens next?

Blog # 9 is the final blog in this Redesigning Quality Systems series.

Refer to blog # 7 for an overview of the 5 Steps for GO-Live Strategy.

Go-Live: Step 4 – Triage Impacted Documents

The purpose of the triage step is to manage the impact of in-process system documents the day the new design goes live.  What happens to forms that were started and not yet completed?  Employees will need to know how to move forward and still be in compliance.   The easiest path on Day 1 is to use the new form(s) to begin a new record.  But what about the documents that were already started?  The ones that were initiated.  They will reflect the previous version although in effect at the time of initiation. Anticipate these questions from affected users:

  • Do we process the old form?
  • Do we start over with the new form and add the original started document with EOD notation for missing information?
  • Do we suspend and pick up with the new form?

A good example of this situation is when paper change control requests have SOP drafts attached. The Project Manager with the guidance of the Head of Quality needs to meet prior to the launch date to carve out the decision paths based on the amount of initiated documentation stages each document contains.  These decisions and individual directions are the main tenets for daily guidance from the Project Manager and/or Quality System owner during the start and end period for the transition phase.  Revisit Step 2. 

In addition, s/he or a small team needs to answer questions that are sent via emails, phone calls, drop by’s and drop-ins for direction on how to close the loop on initiated previous versioned documents. Anticipate additional 1-1 coaching through the new paperwork until the end of the transition period. 

“What happens to forms that were started and not yet completed? ”

Vivian Bringslimark, HPIS Consulting, Inc.

When Will We Be Fully Operational with The New Design?

When this design team began putting together its go-live strategy, their impact assessment (Step 1) mirrored an “End in Mind” approach.  The project manager asked, “what needs to be in place in order for us to be fully compliant with this procedure”.  The list of items was long.  The second question – “What is the best way to accomplish this” was then asked and detailed steps were captured.  The third question focused on the estimated time to complete and so on.  In essence, the team was generating an implementation plan to guide its launch decisions.  It became clear that they needed to launch in high priority clusters

However, two cluster sets were competing for first place to launch.  One focused on the business objective while the other focused on establishing foundational principles for the quality system.  In a stunning presentation, the system owner made her compelling appeal to the site leadership team and was given the approval to begin with foundational principles despite the number of tasks and time to complete those tasks. 

In order to move forward with identified tasks from the implementation plan, the team needed the high-priority procedures to be in effect in order to gain cooperation from department managers.  This created a compliance dilemma of being out of compliance immediately upon launch.  To mitigate this gap and not create unnecessary deviations, the system owner generated a change control packet that also included the implementation plan tracker.  It gave the design team the necessary authority to invoke the changes without becoming bogged down in the compliance dilemma dialogue which derails progress. 

When FDA made an unannounced visit to the site weeks later, the change control packet of documentation was not only accepted as an appropriate way to cover the obvious compliance gaps in the new procedures, but the implementation dates were moved up and execution of the plan became a site priority. 

The project manager asked, “what needs to be in place in order for us to be fully compliant with this procedure”

Vivian Bringslimark, HPIS Consulting, Inc.

Go-Live Step 5: Monitor Performance and Collect Feedback.

The project ends for the consultants when the handoff is completed, and they disengage. The handoff usually includes a formal document describing the current status of the system at the point in time including deliverables, suggestions for next steps and sustainment. 

Can the team celebrate that the project is over? Or should they wait until someone determines whether it is a successful adoption? Recall Step 2 of the Go-Live Strategy defined the end date of the transition period as well.  Did this include the expected date for being fully operational? By the way, who is responsible for declaring success or needs improvement?

Phase Three – Successful User Adoption

One more component of the overall project plan is User Adoption aka Phase 3.  In my experience, this is perhaps the most overlooked phase in redesigning quality systems.  Once training is delivered and the consultants leave, “life as normal” sort of returns with pockets of progress, “learning curve” deviations and loads of suggestions for “fixing” the new design. 

In this phase, the project plan expands Go-Live Step 5 into two main activities, initial user adoption results and sustaining the redesigned quality system through controlled continuous improvement feedback loops.

User Adoption: Did we succeed?

What kind of data/evidence will you use to confirm the success of the Redesigned Quality System?  Will you be relying on the CAPA Effectiveness Check statements to determine User Adoption success?  One word of caution here. These may be too narrowly focused on particular steps as it maps back to a single observation and does not provide enough evidence to tell a complete story just yet.  

What the System Owner really needs to know is:

  • Who is following the new process?  Who is still dragging their feet? Who is confused about how to proceed? Who needs help remembering what the changes are?
  • Do we have any early examples of how or where the new process is helping to produce positive results?
  • What is the value of those results?
  • How long do I have before I am forced to release the next set of “promised” revisions?

You Need a User Adoption Study

The purpose of the study is to get data both quantitative and qualitative to address the system owner’s questions.  This means the study needs to focus on identifying the specific changes (removed, added, and modified) and then collecting the evidence to evaluate how well those changes have been transferred into this “life as normal” stage.   Whether this means reconvening the design team or initiating a new subgroup of early adopters, user input into the tools, existing KPIs, new metrics, surveys, and who to interview is another excellent opportunity for users to be involved.

It also means keeping key stakeholders and the sponsor informed and up to date.  The “project may be over” however, the adoption has only just begun.  Feedback is essential to not only determine successful adoption but to also gather detailed information for continuous improvements that feed revisions.

“In my experience, [phase 3] is perhaps the most overlooked phase in redesigning quality systems.”

Vivian Bringslimark, HPIS Consulting, Inc.

Managing “New Normal”

Life as new normal has begun. The quality system is fully operational and Affected Users are now performing their tasks.  New normal is really about managing the new workflows and the status of the current resources to sustain the new system.  Are you able to sustain the new system or are requisitions pending for more full-time employees?

A screenshot of a cell phone

Description automatically generated
why redesigned SOPs are more than a set of revised procedures

What existing performance metrics can you leverage? Perhaps it is only a matter a slight tweaking here or there to accommodate your redesigned quality system.  In some systems that have gone digital, brand new metrics and KPIs (key performing indicators) need to be established.  Most vendors will partner with their clients and recommend industry best practices around popular metrics in use.

Anticipate periodic revisions that come from corrective actions and from the Affected User groups.  Hold focus group meetings quarterly for the first year and then semi-annually the second year to stay on top of trends and possible performance challenges to prevent unexpected deviations.

Continuously Improve the Quality System

The redesigned quality system has to feed itself.  Identify and continuously monitor the key linkages (handoffs) between one system interface and the other.  Are they working or are they becoming the source of miscommunication between users? 

How many CAPA investigations have been initiated where the handoff may have been a contributing factor?  How many small or incremental changes has the quality system undergone in the course of a year?  When will it be time to remap the process and identify all the tweaks that have occurred? Would anyone recognize the differences?  I’m betting on savvy new hires to ring the change bell and raise our awareness.  We forget just how many changes can happen to a quality system over 1 – 2 years.

Summary

  • Any change we introduce into an organization must be aligned to fit the existing system or modify the system to accept the changes.
  • Quality Improvement Projects/ CAPA teams are no stranger to change and all that comes with changing old ways.  But managing change fatigue can wear us down and make us slow to outwardly embracing major change efforts. Include change management strategies that go beyond Awareness Training only.
  • This blog series outlined why some post-implementation issues can thwart our best designs for users and provided tips and suggestions to deploy during the project phases rather than waiting until post-launch to ensure User Adoption success. 

Here’s a recap of the series:

  • Blog # 1 – Redesigning Quality Systems: Achieving User Adoption
  • Blog # 2 – Manage Your Stakeholders and Users Expectations
  • Blog # 3 – Gap Assessments are Necessary for Redesign Projects but so is the right level of support
  • Blog # 4 – What to Expect When Processing Map with SMEs
  • Blog # 5 – Field Testing Your Final Draft of Redesigned SOPs and Forms: Helpful or Seen as unnecessary?
  • Blog # 6 – Change Management and It’s Little Cousin Training
  • Blog # 7 – Do I Really Need a GO-Live Strategy?
  • Blog # 8 – Is an Awareness Training Only Session Enough for Successful User Adoption?

Who is Vivian Bringslimark?

© HPIS Consulting, Inc.

Do I really need a Go-Live Strategy for Redesigned SOPs?

Develop Your Go-Live Strategy Before Opening up a Change Control Request

Don’t underestimate what Go-Live will look like the day the new SOPs go into effect.  The overall project plan may reflect one lined task – [Go-Live] and the date, but who is responsible for making it happen?  Doc Control administration hits a button and the eDoc system sends out a notification or perhaps folks find out about the revisions and effectiveness date from their LMS notification. What else needs to be done, you ask? 

Actually, a lot of things and without a detailed plan of assigned tasks or a set of instructions, the first week of Go-Live could be disastrous if not managed well.  Whether it is broken down further in the project plan or contained in the communication strategy, the plan needs to drill down into specifying how tasks and activities will be managed, by whom, and by when. 

5 Basic Steps for a GO-LIve Plan
Steps to take BEFORE draft SOPs are sent to Doc / Change Control

Go-Live: Step 1 – Transition Options

Most quality system redesigns include more than one SOP.  Sometimes the project includes multiple work instructions that are designed to work in tandem.  Releasing one without the other automatically creates a departure from the new SOP.  One approach design teams take is to release the package all at once.  Other teams decide to release serially, one after the other with timed intervals to manage the learning curve. Other approaches include pairs or high priority clusters similar to a wave or a phased tactic.  All have their advantages and disadvantages. 

And all need an impact assessment that examines the details about what will change, how that change will be handled and how many users will be involved.  In other words, how many tasks could be out of compliance when the procedure(s) goes into effect?  Will the release shut down the organization on Day 1?

Now vs. Later – Timing is Everything!

A few years ago, a client received a pretty involved FD-483.  One of the corrective actions was to enhance the Training Quality System.  So, a project team was deployed, process maps verbally approved, and the revised SOPs were in various stages of being ready for a field test.  The design team was concerned that SOP decisions for in-process drafts could impact the final drafted SOPs significantly. The team needed to be able to update a final draft SOP immediately without getting bogged down in their change control process.  Not to mention managing a new round of awareness training that could change a week later!

So, as part of the Go-Live strategy, a decision was made to schedule the release of the procedures all at once when the final set of SOPs were ready to go to Change Control.  It was determined (assessed), that too many little changes for the Training System would result in confusion, retention issues, and a paperwork nightmare controlling which version of the form was correct.  This is a popular option for design teams because the coordination of new changes happens on the same day and is much easier to track. Albeit the learning curve can be high as the site works on closing the size of their change gap.

During the development of the last set of SOPs, FDA conducted a follow-up visit for the FD-483.  One of the training CAPAs was not yet completed. It was waiting for the redesigned set of procedures to go into effect.  The new version would encompass this observation and essentially close it out.   Unfortunately, this incomplete CAPA created a hold for product to be released.  As a result, the priority to complete a retrospective qualification of 700(+) “trainers” derailed the Training Quality System project for close to 4 months.  Additional immediate fix CAPAs were also generated.  Ironically, these CAPAs could have been avoided if the design team chose the serial release option. 

Go-Live: Step 2 – Who Will Do What, When?

Establish roles, responsibilities, and priorities for the transition period.  This means identifying the beginning and end date for the transition from what we used to do to what we will do by what date. When do we anticipate being fully operational with the new procedures?  Is the end date automatically the day the procedures go into effect? Or is there a sequence of tasks that need to be executed before the Go-Live date? Have those tasks been assigned to individuals with clear expectations and due dates?

We’re Still Open During Our Construction Phase!

One client began their Go-Live Strategy with Step 2 to help assess the impact of the changes to her organization which dictated the decision to release each SOP serially; one at a time.  Using a worksheet, each targeted SOP was listed vertically while impact criteria were listed horizontally.  The team need to “see” associated form numbers, cross-referenced SOPs, the actual impact, mitigation steps, how to transition, other impacted documents, disposition status and dates of those other documents and of course next steps. 

With careful examination of what tasks, forms, and steps were actually changing, the Project Lead was able to realistically determine how much time it would take to mitigate the impact.  Subtasks such as final form revisions and revisions to other documents were being monitored and project tracked as well. 

This information shaped a pragmatic and timed sequence of events. Resulting in the conclusion that releasing one SOP at a time was the most effective way to transition into the future state with minimal disruption despite that the overall policy document would have to be revised multiple times during the transition period.  The project lead volunteered to be the taskmaster on those subsequent policy changes and updates for the stakeholder briefings.

What else needs to be done, you ask? Actually, a lot of things and without a detailed plan of assigned tasks or a set of instructions, the first week of Go-Live could be disastrous if not managed well.

Vivian Bringslimark, HPIS Consulting, Inc.

Included in this impact assessment were numerous immediate response CAPA corrective actions.  It was vital to ensure that future state procedures did not eliminate commitments that were made to the agency.  These CAPAs resulted in subprojects that also needed to be managed within the overall quality system redesign project.  Given the scope of these projects, completion dates were projected to be 6 – 9 months out.  Yet, the future state quality system procedures were expected to be operational way before these subprojects came to fruition. 

In essence, the day the SOPs went into effect, the site would automatically be out of compliance and numerous deviations would have to be initiated.  To manage the SOP change gap, an overarching CAPA was approved to allow the organization to meet agency milestones and move forward with the new process while working on the subprojects.   With each planned release of a future state SOP, the impact assessment worksheet was updated with feedback including future release notes.

Catch up on the previous redesigned Quality Systems blogs below:

  • Blog # 1 – Redesigning Quality Systems: Achieving User Adoption
  • Blog # 2 – Manage Your Stakeholders and Users Expectations
  • Blog # 3 – Gap Assessments are Necessary for Redesign Projects but so is the right level of support
  • Blog # 4 – What to Expect When Processing Map with SMEs
  • Blog # 5 – Field Test Your SOPs before they go into effect
  • Blog # 6 – Change Management and It’s Little Cousin Training
  • Blog # 8 – Is an Awareness Training Only Session Enough for Successful User Adoption?
  • Blog # 9 – So, We Went Live, What Happens Next?

Who is Vivian Bringslimark?

© HPIS Consulting, Inc.  All rights reserved.

Gap Assessments are necessary for QS redesign projects but so is the right level of support

This is the 3rd blog in the on-going Redesign Quality Systems: Achieving Successful User Adoption series. Visit Blog 1 and 2 here.

Assessments – “Truth Be Told”

Assessing the current state of the targeted quality system is analogous to collecting the “voice of the customer” input.  Discovering what is and is not working is vital to ensuring a successful redesign and ultimately user adoption. This is more than a review of the SOPs and forms; however.  Interviews are key to this phase of the project.  From the system owner to primary users, QA staff and even the LMS or the eDoc administrator are essential.  The more cross functional the input, the better the confirmation of the “data” and the “evidence”.  Which in turn, provides a truer gap assessment.

Most interviewees know the system’s weaknesses and they know how to fix it.  But they are frustrated with ignored attempts to change it.  Many simply give up trying to get it changed on their own or within their circle of responsibility.  So, when quality systems interviews are being conducted with an external consultant, some are eager to share the “dirt” on just how broken the system really is.  And some of the “fixes” are political and can only be handled when trust has been earned or when the sponsor deems it so.

We recognize Training is important but …

A small vitamin and supplements company set out to become a commercial manufacturer of pharmaceutical drug products.  Naturally, their quality systems were not robust enough to meet 21CFR 211 code of regulations. So, assessments were conducted for the major quality systems.  The Training Quality System assessment revealed several positive activities that were enabling current success and a lot of gaps that would need to be addressed to meet regulatory expectations for today’s pharmaceuticals companies. 

The sponsor explained that without top leadership support (meaning physically show up and attend this debriefing meeting), no one else would show up nor take the project seriously.

The sponsor routed the Training Gap Assessment Report to his peers and the CEO with an invitation to attend the first in-person briefing.  No one showed up except the sponsor, his direct reports and the consultant.  The meeting was rescheduled for a month later when all executives’ schedules showed availability.  One month later, the sponsor learned that key executives would be out of office during the meeting, so he postponed the briefing indefinitely.

The sponsor then arranged for a one-on-one follow-up session with the CEO and explained that without top leadership support (meaning physically show up and attend this debriefing meeting), no one else would show up nor take the project seriously. He also reiterated that in order to grow the business as intended, resources needed to be hired and the quality systems including Training needed immediate prioritization.  Or else the company would not receive approval to manufacture and could further impact its current relationship with the agency.

Three months later the meeting was scheduled.  However, the agenda took on a different focus. Particularly, the collaboration benefits of working cross-functionally with department managers and the shared ownership for the Training Quality System.  More specifically, the training project was not just a QA program, but a robust training system that impacts all employees.  This time the meeting occurred and was fully attended by all invitees.  

Change Readiness for the Design Team

With the briefing finally conducted, the sponsor explained to the consultant, that the “Design Team of SMEs” had very limited experience working cross-functionally or as a team. They knew even less about project management concepts like scope and project charters and they lacked fundamental concepts like quality systems and systems-based inspections.  And they did not know how to process map. They had been silo’d far too long.

SME Readiness Curriculum
Design Team of SMEs Readiness Curriculum

The consultant’s background included curricula building, instructional design, and quality systems redesign experience. She sketched a brief outline of the mini-curriculum for the Design Team of SMEs and the Sponsor agreed.   Over the next few weeks, they used their meeting time to work through the 4 introductory lessons.  And as a team, they used the “live” aspects of the project work to illustrate the concept and apply its principles to the progress and development of their team.   In essence, they were taught just what they needed to know at the moment the project needed it, similar to learning in the flow of work.  Four weeks later, the team was ready to begin their process mapping sessions together.

How Big is the Change Readiness Gap?

Who is Vivian Bringslimark?

The rest of the blog series can be found below:
  • Blog # 1 – Redesigning Quality Systems: Achieving User Adoption
  • Blog # 2 – Manage Your Stakeholders and Users Expectations
  • Blog # 4 – What to Expect When Processing Map with SMEs
  • Blog # 5 – Field Testing Your Final Draft of Redesigned SOPs and Forms: Helpful or Seen as unnecessary?
  • Blog # 6 – Change Management and It’s Little Cousin Training
  • Blog # 7 – Do I Really Need a GO-Live Strategy?
  • Blog # 8 – Is an Awareness Training Only Session Enough for Successful User Adoption?
  • Blog # 9 – So, We Went Live, What Happens Next?

(c) HPIS Consulting, Inc. All rights reserved.

Manage Your Stakeholders and Users Expectations thoughout the Entire QS Redesign Project

Engage Users from the Beginning and Throughout the Redesign Project

When Quality System redesign CAPAs are project managed using the 5 typical phases (see diagram), the progression of the project is easier to track and report status updates to stakeholders.

Involve Users throughout all phases and stages

Tapping into the different types of users for their input from the start of the project advances their engagement with the project along with the phases.

For example, during the Initiate Phase, an assessment of the current procedures, processes, and practices can include a variety of users, not necessarily the SMEs or system owners only. 

But without an organized and approved communication plan for each of these phases /stages, rumors evolve, misinformation circulates and the success of the new system being adopted is put into jeopardy.

Managing Stakeholders Expectations

A popular activity from the project management toolbox is to conduct a stakeholder analysis whereby a list of affected folks is captured.  And their role with the project team is clarified.  A handy acronym called PARIS can be followed: Participant, Accountable/Owner, Reviewer, Input Required, Sign-off required before proceeding. Also popular is a RACI matrix chart where;

  • R = Responsible
  • A = Accountable
  • C = Consulted
  • I = Informed.

So who are the stakeholders for the quality systems? In a quality system redesign, the following types of stakeholders are usually present,

  • Sponsor
  • System Owner
  • Managers
  • QA, HR, IT if it includes technology platform or upgrades
  • All affected Users.

Additionally, team members can identify their stakeholders’ relationship to the project which can be very helpful when the project hits barriers and is sent derailing off timelines.

Executive Sponsorship is a Key Success Factor

Quality systems by their nature are interdependent and frequently cross into boundaries of other departments’ workflow. Sponsorship is needed especially if the changes are significant, there are multiple quality system projects happening at the same time and other site initiatives all vying for the same resources.  Every quality system redesign project needs an executive sponsor:

  • To approve resource requests
  • To become aware of the impact of the changes being proposed
  • To be informed with the most current status

Thus, naming the sponsor holds this person accountable for the expected results. Thereby endorsing the project and becoming an advocate for the team’s work.

The Communication Strategy

Once the stakeholders have been identified and categorized, the next activity is to determine what level of involvement is necessary.  Ask them directly.  Ask them how often and in what format do they prefer. If you think one size fits all stakeholders, think again.  Many a project has derailed, missed its goal or became defunct as a result of miscommunication and misunderstandings.  Often stakeholders do not interact directly with the team.  It’s the Project Manager’s responsibility to be the interface.  One way to achieve this is to conduct briefings.

Briefings as a Communication Medium

The purpose of briefings is to provide updates on the project regarding timelines, milestones, deliverables and unresolved issues.  The expected results for a redesigned quality system are not always clearly understood because of confusing priorities and diluted focus.

Not all briefings have to be conducted in person via meeting time and conference room space.  Storyboard updates can be routed electronically.  Weekly updates, interim reports, and even significant project meeting minutes can be useful for briefing the stakeholders.  Make sure you match the type with the frequency as per your stakeholder analysis and communications strategy.  As the project progresses through its phases, stakeholders’ roles can change.  Use these briefings to continuously check-in for feedback and adjustment with frequency and format.

Routine Stakeholder Briefings: An example in practice

One client of mine decided to leverage the cadence of their standing update meetings and used the proposed system changes as the focus of the agenda.  As each procedure progressed to working-draft status, the Project Manager invited the key stakeholders and guest users to the meeting.

She diligently took notes in the drafts, captured their concerns on the parking lot flipchart and brought up issues from the design team for further exploration and discussion among the system owners. Then she invited them back to the next briefing to review the newest SOP drafts and secured a verbal approval for the draft so that the draft did not get hung up in disagreements among the approvers during formal change control. 

They used the briefings to work through their issues and concerns.  She would point to where the previous discussion was now resolved and often, she would provide an explanation as to why a concern or an issue did not get acknowledged in the procedures as discussed.  Participation in these meetings was generally good; a few key leaders were absent from time to time.  So, occasionally one-on-one meetings were scheduled for critical approvers.

In the next blog, I will walk through assessments in more detail. – VB

Redesigning Quality Systems Blog Series
  • Blog # 1 – Redesigning Quality Systems: Achieving User Adoption
  • Blog # 3 – Gap Assessments are Necessary for Redesign Projects but so is the right level of support
  • Blog # 4 – What to Expect When Processing Map with SMEs
  • Blog # 5 – Field Testing Your Final Draft of Redesigned SOPs and Forms: Helpful or Seen as unnecessary?
  • Blog # 6 – Change Management and It’s Little Cousin Training
  • Blog # 7 – Do I Really Need a GO-Live Strategy?
  • Blog # 8 – Is an Awareness Training Only Session Enough for Successful User Adoption?
  • Blog # 9 – So, We Went Live, What Happens Next?

Who is Vivian Bringslimark?

(c) HPIS Consulting, Inc. All rights reserved.