A novel internal training program using Kern’s 6-step approach to curriculum development for medical laboratory scientists training to be international quality assurance/quality control coordinators

Abstract

Objective

Patient Safety Monitoring in International Laboratories (pSMILE) is a resource ensuring quality testing in clinical laboratories performing National Institutes of Health–funded HIV research requiring specific staff training. We demonstrate the development of an online asynchronous training model using Kern’s 6-step approach to support pSMILE functions.

Methods

An existing curriculum was revamped to incorporate Kern’s approach. Metrics for success were described in rubrics with feedback guiding improvements and updates.

Results

Curriculum updates took more than a year. Direct observations of skills informed curriculum changes. Module self-evaluations were reviewed to assess performance and the overall curriculum. The content, curriculum, and training documentation were deemed compliant with International Organization for Standardization (ISO) 9001:2015.

Conclusion

Asynchronous training for highly skilled and self-directed staff is a novel way to deploy training while maintaining productivity of existing staff. Feedback and evaluation allowed for curriculum updates including previously underdeveloped topics. Kern’s approach ensured that the needs of the sponsor, management, laboratories, and learners were met.

Introduction

Patient Safety Monitoring in International Laboratories (pSMILE) is a contract resource between the National Institutes of Health (NIH) Division of AIDS (DAIDS) and Johns Hopkins University with a mission to provide laboratory quality assurance support, guidance, and training to international clinical laboratories performing NIH-funded research originally focused on HIV but now including other infectious diseases. The 10 technical team members of pSMILE are medical laboratory scientists (MLSs) with an average of 20 years of clinical laboratory experience in diverse settings and disciplines. Staff members are credentialed laboratorians, holding MLS certifications from the American Society for Clinical Pathology and American Medical Technologists. They possess a wide range of experience from large university hospital laboratories and commercial laboratories as well as international research and clinical laboratories. Team members also possess a wide range of knowledge and experience gained from working in nearly every department, including chemistry, hematology, immunology, microbiology, and flow cytometry.¹ The international quality assurance/quality control (QA/QC) coordinators are expected to use their professional knowledge of clinical laboratory principles, theories, practices, and methodologies to review and recommend quality improvement, assurance and control programs in international laboratories. This includes extensive work coordinating and monitoring external quality assessment (EQA) , also referred to as proficiency testing, activities.

Although our work encompasses the knowledge and experience gained from working in clinical laboratories, it is unique, requiring specific and detailed training. Each employee works with approximately 20 international laboratories on a variety of issues related to compliance with DAIDS good clinical laboratory practice (GCLP).² We have developed standard operating procedures (SOPs) for all of the types of laboratory assistance that we provide. All new employees need to be thoroughly trained in pSMILE procedures as well as taught how to interact and navigate the collaboration between the laboratories, NIH, and clinical trials networks and investigators that perform the research studies. Additionally, many of our core function processes are automated, driven by 3 databases designed and created by the pSMILE information technology (IT) team. These automated processes are complex and require extensive training to ensure that the deliverables of the contract are fulfilled in compliance with our contractual requirements, GCLP standards and our ISO 9001:2015 certification. Completion of the entire training curriculum for new pSMILE employees takes approximately 12 months and includes an online asynchronous training program combined with 1-on-1 mentorship training with current staff members.

This online asynchronous training program was designed for new pSMILE employees in 2007.³ Over the years, portions of the curriculum were updated to align with changes to the pSMILE project scope and direction. With an evolution in workflow and documentation of processes, the pSMILE training team undertook a complete redesign of the curriculum in 2013 using the Six-Step Approach to Curriculum Development for Medical Education by David Kern.^4,5 The delivery method was also moved from the Moodle open-source software (moodle.org) platform to Blackboard (Blackboard.com) in 2017. The program was thoroughly reviewed and updated again in 2020 to ensure that the modules remained current and relevant. In 2021, the curriculum was once again modified to address a staffing shortage that necessitated more expeditious training, with a reduction in training time from 9 months to 4 months. See FIGURE 1 for timeline summary. Herein we describe our experience with the creation of an online asynchronous training model to ensure evidence-based education that produces International QA/QC Coordinators who are competent and prepared to assist international laboratories.

Figure 1.

Timeline summarizing the evolution of the pSMILE training curriculum.

Kern’s approach identifies 6 progressive steps to evaluate the curriculum’s applicability, appropriateness, and feasibility. Applicability can be defined as the determination that the curriculum meets the goals and objectives of the training program. Appropriateness refers to the notion that the instructional procedures work well for both the targeted learners and the curriculum content. Feasibility can include both budgetary and logistical considerations. It is the process of determining that what may appear to be a great curriculum in theory can actually be practically achieved.

The steps to curriculum development as described by Kern are meant to create a cyclical process in which the steps influence each other, although they are presented in sequential order.^4,5 The 6 steps are as follows: (1) problem identification and general needs assessment, (2) targeted needs assessment, (3) goals and objectives, (4) educational strategies, (5) implementation, and (6) evaluation and feedback. Each of the 6 steps leads the developer through a series of questions designed to help guide the creation or redesign of the curriculum. The steps are described below along with pSMILE’s assessment of each of the successive steps (also see FIGURE 2). The format and the additional questions in each step were completed according to the Kern model.

Figure 2.

Steps for curriculum development as assessed by pSMILE. Thomas, Patricia A., MD, David E. Kern, MD, MPH, Mark T. Hughes, MD, MA, Sean A. Tackett, MD, MPH, and Belinda Y. Chen, MD, eds. Curriculum Development for Medical Education: A Six-Step Approach. pp. 9, Figure 1.1 . © 2022 Johns Hopkins University Press. Figure reprinted with permission of Johns Hopkins University Press.

Step 1: Problem Identification and General Needs Assessment

In step 1 of Kern’s approach, the educator identifies the problem that necessitates a new (or modified) curriculum and compares the current educational approach with the ideal approach. This allows the identification of limitations in the existing curriculum and the proposal of improvement that will address the needed end product of the curriculum.

Problem Identification

The pSMILE staff must be adequately and appropriately trained to ensure the health and safety of research study participants, the integrity of study data, and the safety of the international staff performing laboratory testing. Although the staff are experienced MLSs, the job of an international QA/QC coordinator is complex and requires training to ensure that staff meet contractual and regulatory requirements, in addition to possessing the technical skills to perform the job.

Limitations in Curriculum Prior to Redesign

pSMILE international QA/QC coordinators were previously trained using an asynchronous training curriculum developed in 2006 using the Moodle delivery platform.⁶ As of 2013, the curriculum had not been updated, although many internal pSMILE procedures had changed. In addition, the pSMILE contract had just been renewed for 7 more years. The new pSMILE contract contained many changes in approach, scope, and requirements that needed to be incorporated into an updated curriculum.

Following completion of the Moodle online training modules, new pSMILE staff members were paired with experienced coordinators for 1 to 2 weeks. New employees were rotated through the staff to give them exposure to unique styles and approaches to solving problems and to a wide variety of projects and tasks. Although the mentorship rotations were effective, they lacked a standardized, validated assessment tool to ensure the learners acquired the desired knowledge from the rotation.

Approaches to Improve Curriculum

By evaluating what had been the existing approach and comparing it with an ideal approach, Kern guides us to analyze the gaps between current practice and what we seek to achieve. By delineating between the responsibilities of the sponsor, the contracted organization, and the contract team itself, we were able to identify external resources to assist with the comprehensive training. We were also able to focus our efforts on the finer details of pSMILE coordinator task training and identify ways to improve over the current training model.

Sponsor

• Provides training to all new pSMILE employees on GCLP- and NIH-specific procedures for completion of contract deliverables

Institution

• Provides institutional/department-based mandatory training

• Provides access to basic computer skills training (MS Outlook, MS Excel, MS Word, SharePoint, MS PowerPoint)

pSMILE

• Provides in-depth, hands-on training in all pSMILE processes and procedures

• Provides simulated training working with mock scenarios to build familiarity and comfort level with pSMILE processes and procedures

• Provides up to 2 weeks of 1-on-1 mentorship with each of the experienced pSMILE coordinators, which has clear objectives, requirements for covered topics, and an assessment tool to ensure that the mentorship rotation is effective.

Step 2: Targeted Needs Assessment

A curriculum for the training of new pSMILE coordinators already existed. The program had been highly successful and was used to train 8 coordinators. However, areas for improvement in the program were identified through the feedback of learners and trainers, leading us to identify this as a targeted environment that could benefit from curriculum redesign. In step 2, Kern suggests that the educators focus on the needs of the targeted learners. To determine the needs of the targeted learners, the following methods were used: (1) review of the curriculum with the education/training committee to determine applicability and effectiveness, (2) careful review of the contract to ensure that the training program met the requirements of the new contract, and (3) informal discussions with experienced pSMILE staff, many of whom completed the curriculum, to obtain suggestions for improvement.

Targeted Learners

The targeted learners for this curriculum are new employees of the pSMILE project. The learners are experienced MLSs with extensive backgrounds in all areas of the clinical laboratory. They typically have some background in QA, QC, laboratory audits, and proficiency testing. They may or may not have international experience, which also affects how well they comprehend the nature of the work and how quickly they assimilate certain aspects of the curriculum.

Needs Assessment

Informal feedback from learners and trainers using the Moodle curriculum modules was previously used to make changes and updates to the training modules. However, the need to completely redesign the curriculum was determined based on the response by learners and trainers alike that the modules felt somewhat disjointed and at times lacked continuity. The training committee surveyed recent graduates of the program and the project manager and determined that a simulated work environment would provide the best sense of continuity to the modules.

Survey of Recent Graduates, Trainers and Management

Kern’s model for curriculum development relies heavily on the feedback of all recent participants in the training program to inform improvements. Prior to the initial redesign of the curriculum in 2013, The 3 most recent graduates of the training curriculum were surveyed to determine their feedback on the current program’s strengths and weaknesses. At the time, there had been very little turnover in staff, so the number of recent graduates was small. To further validate the need for change, current and former trainers were also interviewed informally to determine feedback on the program’s effectiveness and difficulties in delivering it.

All felt that the modules were valuable but lacked depth and connection to the actual work at pSMILE. They also felt that the modules stood disconnected from one another, making it difficult to understand how the pieces of the curriculum fit together to make up a typical pSMILE coordinator’s caseload. All recent graduates felt that using a simulated caseload of international laboratories would help add relevance to each module.

The pSMILE management acknowledged gaps in the existing Moodle training, such as handling investigation reports required when a laboratory has unsatisfactory performance on a proficiency testing event. Management agreed that the use of a simulated caseload would improve the training program.

Since the initial redesign, an additional 5 coordinators were trained on the new curriculum. Their feedback allowed for the continued improvement of the program, a key to the Kern model, which is not meant to be static but rather an ongoing, fluid process of improvement. Trainee feedback was reviewed by the training team at the completion of each module. When trainee responses identified problems in training organization or content every effort was made to address the issue in real time.

Step 3: Goals and Objectives

Goals

The goal of the training curriculum is to develop pSMILE international QA/QC coordinators who are competent to perform all tasks required to provide laboratory QA support, training, and guidance to NIH-funded international HIV research laboratories.

Objectives

The module-specific objectives were written to address specific tasks required for job performance. The detailed steps for each of these tasks are outlined in the pSMILE internal SOPs. Trainees are expected to meet the objectives in adherence to the pSMILE Internal SOPs. Detailed curriculum objectives are provided in TABLE 1, which contains a summary of each training module of the newly developed curriculum.

TABLE 1.

pSMILE training curriculum for international QA/QC coordinators

Module number: name (approx. time to complete)	Summary of learning objectives	Summary of measurement of proficiency
1: Introduction (1 week)	Select and access the appropriate tools to begin the training program Schedule and complete required JHU-specific training modules Describe the applicable components of the work environment Review and prepare for the requirements for working remote or virtual	Complete and pass all quizzes for institution -specific modules (with a minimum of 80% score) and required DAIDS IT security training Print out and begin completing items on the pSMILE training checklist Demonstrate ability to work off site productively from laptop through using the current method for connecting to the pSMILE server
2: International Involvement and Diversity (1 week)	Explain the role of the pSMILE contract within the DAIDS International HIV studies Demonstrate knowledge and understanding of the pSMILE contract, workflow, network/NIH partners and international involvement Demonstrate awareness and knowledge of pSMILE’s mission, vision, and core values	Discuss the role of pSMILE and the pSMILE contract Discuss pSMILE’s mission, vision, and core values Discuss the assigned diversity activities
3: Protocols, PALs and Preparing Site Folders (1 week)	Correctly follow approved methods of pSMILE data collection, storage, and development Identify the laboratory safety testing used in NIH supported studies Correctly follow approved methods to set up a new laboratory on the pSMILE internal server	Correctly prepare a laboratory site folder complete with subfolders following pSMILE internal SOPs Review assigned SOPs and network protocols Complete a spreadsheet that relates information from network protocols to the information in the corresponding site PALs
4: GCLP & QM (1 week)	Navigate DAIDS GCLP guidelines Detect DAIDS GCLP failures	Complete and pass the required DAIDS GCLP online course Discuss current DAIDS GCLP Guidelines and other QM resources Discuss the pSMILE quality policy and QM system
5: Site Assessment and Action Plans (1 week)	Interpret DAIDS requirements from an annual Laboratory Assessment report Develop APs using available resources and review the process of working with a site to resolve AP items Discuss best practices for communication with site laboratories via email	Review assigned audits and related APs Prepare a complete AP from an assigned Laboratory Assessment report Prepare an Audit/AP email using pSMILE template
6: EQA Theory (2 weeks)	Explain purpose and identify significant items and attachments in the EQA Summary, Schedule, and EQA Monthly Report List major EQA providers, log into and navigate major provider websites using own accounts Identify qualitative and quantitative peer group information from different providers’ Participant Summary Reports Use HIV package inserts to determine correct information on assay and results	Demonstrate comprehension of the critical sections of each EQA document type as outlined in the topic 6 Assignments section by independently successfully completing 80% of the tasks Use HIV test package inserts to correctly identify test generation, reporting requirements and testing method
7: AutoSMILE (8 weeks) *This includes working with each coordinator for 1 to 2 weeks after completion of topic 8	Navigate the AutoSMILE home page Enter and review EQA results Create EQA reviews, schedules and monthly EQA and AP emails Correctly file analytes according to method	Review and complete assigned EQA Reviews providing appropriate comments and/or suggested actions with 90% accuracy Generate a monthly EQA email including appropriate attachments Successfully complete a 1- or 2-week rotation with each pSMILE coordinator
8: Investigation Reports (1 week)	Describe the investigation process from initiation to completion Analyze the data presented for completeness, accuracy and correct conclusion Recommend additions or corrections to the IR as needed Review the process of investigating EQA failures using the electronic IR	Review and complete assigned IRs, providing appropriate comments and/or suggested actions, with 80% accuracy
9: pSMILE Website (3 days)	Manually upload and download documents to site folders Upload documents by accessing the pSMILE mass upload functions Correctly manage requests for site access, documents and resources Locate and obtain site information from the oversight master list	Correctly upload and download documents to the pSMILE website Successfully upload EQA documents both manually and using the mass upload function Search “Resource” section to access requested documents and successfully download identified documents
10: Preparation for Working with Site Laboratories (1 week)	Review ISO 9001 standards and certification process Demonstrate an awareness of pSMILE’s Quality Key Indicators, core functions, quality objectives, and QMS Take notes during the pSMILE staff meeting Complete the process of reviewing all required SOPs.	Demonstrate an awareness of ISO 9001 standards and certification process Complete reading and signing off on assigned SOPs Successfully take minutes at pSMILE staff meetings for one week Review all the available information on assigned site laboratories

Following the successful completion of the first 10 training topics, new coordinators will be assigned their first set of site laboratories. They will begin working with those laboratories following the completion of topic 10. They should continue working through training topics 11-17, as time allows, with the goal of completing the entire training program by the end of the first 12 months at pSMILE.
11: Validation (2 weeks)	Identify and describe the required elements of qualitative and quantitative validations Explain the concept of total allowable error (TEa) and use the pSMILE TEa table appropriately Describe the requirements for, and evaluation of, precision, accuracy, linearity, measurement and reportable ranges, and reference range verification Demonstrate proficiency with the pSMILE validation tools	Correctly analyze precision, accuracy, and linearity data using pSMILE validation tools Identify causes of failed validation results and suggest methods for troubleshooting Review a completed validation plan and summary Evaluate a site validation summary and develop a response using the pSMILE validation email and review templates
12: QC (2 weeks)	Explain the concept of internal QC and interpret QC charts and data Outline the internal QC differences between quantitative and qualitative methods Describe the usefulness of sensitive QC ranges and how to acquire them Identify causes of QC failures and how to address them	Locate pSMILE QC tools available in the Resources section of the psmile.org website Correctly identify a bias, shift and trend in both quality control and EQA records Accurately calculate SD and coefficient of variation of raw data provided Provide solutions to given QC failure examples
13: Correlation Testing (1 week)	Describe the pSMILE approach to correlation testing and how it can be implemented to meet GCLP guidelines Direct site laboratories to appropriate resources for developing correlation testing policy and SOPs Assist sites with correlation testing action items listed in site audits and APs	Review all assigned materials and discuss application with members of the training team Complete correlation exercises with 80% accuracy Respond to correlation action items on assigned APs with minimum 80% accuracy
14: Parallel Testing and Reagent Lot Verification (1 week)	Outline acceptable methods of parallel testing and communicate this information to laboratories participating in DAIDS clinical trials Assist pSMILE site laboratories with the development of appropriate parallel testing programs for new lot numbers of testing reagents and control materials	Review written materials including SOPs related to reagent lot verification and parallel testing Apply knowledge of reagent lot verification and parallel testing to resolution of action items from selected APs
15: pSMILE EQA Audit (2 weeks)	Develop a working knowledge of the pSMILE EQA audit process Review the process for ordering EQA surveys Complete an EQA audit for assigned site	Locate pSMILE EQA Audit templates Correctly complete all 4 parts of EQA audit for assigned site
16: pSMILE Travel (3 days)	Explain the roles and responsibilities of the pSMILE coordinator while traveling Follow travel policies and procedures while traveling for the contract, from DAIDS travel request to submitting request for reimbursement Promote security awareness through studying relevant SOPs. Access the necessary tools in order to submit a travel request, prepare a coverage plan, complete trip reimbursement, and write a trip report.	Review all travel related SOPs Demonstrate ability to retrieve pSMILE travel references, tools, and resources located on the internal server Review documents generated for a completed site visit
17: Training Completion (2 days)	Complete all training checklists for topics 1-16 Complete the pSMILE Training Completion Form	N/A

AP, action plan; DAIDS, Division of AIDS; EQA, external quality assessment; GCLP, good clinical laboratory practice IT, information technology; IR, investigation report; NIH; National Institutes of Health; pSMILE, Patient Safety Monitoring in International Laboratories; QC, quality control; QM, quality management; QMS, QM system; SOPs, standard operating procedures; PALs, protocol analyte lists; ISO, International Organization for Standardization .

Step 4: Educational Strategies

Kern’s step 4 identifies and selects educational strategies that will allow the curriculum to attain the goals identified in step 3.

The educational strategy developed uses a combination of online asynchronous training modules originally delivered via the Moodle online platform. Once the curriculum was revised, the individual training topics were loaded onto the Blackboard platform. Additional training is provided by 1-on-1 mentoring with experienced coordinators. The online training modules are designed to simulate the work environment of an international QA/QC coordinator by providing the learners with a caseload of simulated international laboratories. The pSMILE skills must be built onto the learners’ prior knowledge, and the Moodle modules allow for a constructivist method of building this expertise. The use of simulation reinforces the construction of new knowledge and incorporates experiential learning by allowing the learner to experience the typical workflow of the pSMILE job.

After completing the online training modules, the learner works side by side with an experienced coordinator on a 7-week rotation. During the mentorship rotation, the learning is at first observational learning. The trainee learns by observing their mentor and is then given a chance to reinforce that learning by reproducing the mentor’s work. By the end of the rotation, the trainee should be completing all the work with the mentor standing by to observe and correct any mistakes, allowing for experiential learning. By spending 1 week with each coordinator, the learner is exposed to a variety of work styles and experiences the daily workflow of the pSMILE project firsthand.

Content

The curriculum content was designed to provide pSMILE international QA/QC coordinators with the specific skills needed to perform all job duties. Completion of the training program serves as the evidence of initial competency of newly hired staff.

Methods

Blackboard asynchronous training modules are followed by 1-on-1 mentorship by experienced pSMILE coordinators. Time for completion of this portion of the training is approximately 6 months. The content outline (TABLE 1) is very similar to the original curriculum. The full curriculum contains 17 topics. Following the successful completion of the first 10 training topics, new coordinators will be assigned their first set of site laboratories. Topics 1 through 10 include orientation and an overview of the project followed by detailed training on the daily operations and job duties of the coordinators. Topics 11 through 17 provide additional specialized supplemental skills, such as validation and travel procedures, which can be completed after the coordinators begin working with their laboratories. The goal is to complete the entire training program, including the supplemental topics, by the end of the first 12 months at pSMILE.

The primary changes to the curriculum involved the use of a simulated work environment that includes providing the learner with a simulated laboratory. The learner is given a background story so that they can understand the laboratory’s culture, language, and any barriers to communication. All modules revolve around the same simulated laboratory so that the learner can build that information into improved understanding of the laboratory. Additionally, improved evaluation tools help to guide the ongoing real-time improvement of the curriculum.

Step 5: Implementation

In Kern’s step 5, the necessary resources to implement the curriculum are identified and an implementation strategy is initiated. Implementation of this curriculum was made possible by the support of the pSMILE project manager, the university, and the sponsor. Proper training of personnel is key to the success of the pSMILE project. Adequate time, resources, equipment, and facilities are provided by the project in support of this curriculum.

Resources

Personnel

Two training coordinators who are certified and experienced MLSs and have extensive international QA/QC experience on the pSMILE project are dedicated to the training team. Both coordinators have a strong desire to teach and have a caring and supportive attitude. When they have an active trainee, approximately 20% of their workday is devoted to teaching. Salary support is provided by the pSMILE contract. The remaining pSMILE staff (who serve as mentors to the learners) provide additional training support during the mentorship phase.

Facilities, Equipment, Materials

When first implemented, the training took place in an office location with provision of all necessary computer equipment, software, and office supplies. The project staff has since moved to telecommuting and remote work. All training activities are now conducted via virtual tools, with computer equipment and software provided for off-site work.

Funding

Budget for all the materials and equipment listed above is provided by the NIH contract that supports the pSMILE project. Additional funding for materials and personnel support is provided by the institution.

Communication

Communication between educators and learners occurs verbally, by phone teleconference, by email, and through the feedback mechanisms included in each module. The learner attends the daily pSMILE staff meeting. Updates are provided to the staff on the progress of training as applicable. The learner then meets with the training coordinators to discuss the day’s activities, goals, and objectives and answer any questions. The learner works independently on the Blackboard modules.

Operations

The training coordinators are responsible for preparation of the Blackboard modules and providing all supporting documents needed to complete each module. They are responsible for directing the daily activities of the learner to ensure that the training progresses efficiently. The trainers also collect all evaluation data and make any necessary curriculum revisions.

Barriers

Primary barriers to the implementation of this curriculum are the demands of the pSMILE project that would necessitate the training coordinators pausing their training duties to handle emergent duties related to the contract. In addition, there are times that the learner may have to be redirected in their training schedule to meet immediate contract demands. These barriers can be overcome by flexibility on the part of the trainee and the trainers.

Additional barriers are occasionally encountered when a mentorship fails due to personnel or personality conflicts. Training coordinators must be vigilant and monitor the mentor–mentee relationship to ensure that personality differences do not affect the learning.

Introducing the Curriculum

The introduction of the original asynchronous Moodle-based curriculum occurred in 2007. Small and subtle changes to the curriculum occurred continuously to adapt to needs and changes of the project. The redesigned curriculum developed using the Kern model was introduced in 2017 when 2 new staff members were hired. The new curriculum was piloted and reevaluated after the pilot period to determine effectiveness.

Step 6: Evaluation

The sixth and final step in the Kern approach to curriculum development is evaluation and feedback. in this step, both the curriculum and its effectiveness and the individual learners are assessed. The evaluation can be both formative (occurring throughout the training process) and summative (occurring at the conclusion of training).

The evaluation of the pSMILE curriculum occurred over a period of several years. This curriculum is typically used by only 1 learner at a time, and over the last 15 years, there have been only 13 coordinators trained using this program. This makes evaluation slow and somewhat unreliable. In the early years of the curriculum, formal evaluation was conducted in the form of a questionnaire at the end of each module. The questionnaire consisted of very general “customer satisfaction” questions. Learner evaluation was conducted by grading written assignments; however, the grading was done by different instructors and an answer key was not used. This led to inconsistent grading of assignments and potential instrument bias and interrater unreliability. Informal evaluation also occurred during the mentorship phase using informal feedback given to the learners by the mentors. As the evaluation methods previously used lacked validity, a revamp of the evaluation at the end of each module was added in 2020.

For the redesigned curriculum, we proposed to improve the evaluation and feedback process. The learner self-evaluations at the end of each module now contain questions that are more targeted to the objectives and competencies of each module. The evaluation also includes questions that address the curriculum components. Additionally, the evaluation of the learners based on written assignments is graded with the use of a standardized answer key and a grading rubric. This not only standardizes the grading of each assignment, providing interrater reliability, but it helps to eliminate instrument bias. The data collected is also used to guide feedback to the learners. The use of the rubric allows for the collection of valid data that will either give evidence of the effectiveness of the training or will guide the improvement of the curriculum. During the mentorship phase, mentors are given an evaluation form that directs and guides the evaluation of their trainees, rather than leaving the feedback to random comments.

Evaluation Questions and Measurement Methods

An example of a module course evaluation form is shown in FIGURE 3 and the mentorship evaluation rubric is provided in TABLE 2. Measurement of the evaluation of learners is conducted by a combination of direct observation of skills, self-evaluation, and evaluation by a mentor. The direct observation of skills takes place at the end of each module. The training coordinator will give direct feedback to the learner. Scores on the direct observation are used to inform changes to the curriculum.

Figure 3.

Example of training module evaluation form. EQA, external quality assessment; pSMILE, Patient Safety Monitoring in International Laboratories

TABLE 2.

Excerpt from pSMILE coordinator training: mentorship evaluation rubric

Criteria for evaluation of coordinator trainee		Does not demonstrate criteria	Adequate demonstration of criteria, but needs improvement	Excellent demonstration of criteria
		1	3	5
1	EQA review	Does not complete EQA review independently following the pSMILE internal SOP. Incorrectly identifies shifts, trends, bias. Incorrectly identifies need for an investigation.	Completes the EQA review independently with few errors or corrections necessary by the mentor.	Completes the EQA review independently, in compliance with pSMILE internal SOPs with no errors or corrections necessary.
2	EQA summary	Does not update EQA summary correctly based on review (e.g., Uses incorrect color-coding, comments are incorrect, missing date reviewed, etc.)	Correctly updates EQA summary based on review. Occasional errors or corrections necessary by the mentor.	Correctly updates EQA summary based on review. No errors or corrections necessary by the mentor.
3	Investigation reports	Does not demonstrate understanding of the investigation process and workflow. Unable to assess whether an investigation is acceptable. Unable to assist laboratory with investigation steps.	Demonstrates understanding of the investigation process and workflow. Able to assess whether an investigation is acceptable with assistance from the mentor. Able to assist laboratory with investigation steps with assistance from the mentor.	Demonstrates understanding of the investigation process and workflow. Able to assess whether an investigation is acceptable without assistance from the mentor. Able to assist laboratory with investigation steps without assistance from the mentor.
4	Email correspondence	Uses incorrect email contact list for laboratory. Does not follow pSMILE template for email format. Does not include appropriate attachments to EQA email.	Requires occasional assistance from mentor to perform the following: uses correct email contact list for laboratory; follows pSMILE template for email format; includes appropriate attachments to EQA email.	Performs the following independently: uses correct email contact list for laboratory; follows pSMILE template for email format;. includes appropriate attachments to EQA email.
5	pSMILE website	Unable to navigate through website. Unable to upload, download or update documents on website.	With occasional assistance is able to navigate website, upload, download, and update documents on website.	Independently is able to navigate website, upload, download, and update documents on website.
6	Laboratory site assessment	Does not demonstrate understanding of GCLP audit report. Unable to distinguish audit findings.	With minimal assistance from mentor, demonstrates understanding of GCLP audit report. Distinguishes audit findings.	Independently reviews and understands GCLP audit report. Distinguishes audit findings.
7	Action plans	Does not develop an action plan from GCLP audit report using the approved template and SOP. Unable to describe audit findings and provide appropriate suggested actions.	Develops an action plan from a GCLP audit report using the approved template and SOP with minimal assistance. Describes audit findings and provides appropriate suggested actions most of the time. Occasionally deviates from the approved format.	Develops an action plan from a GCLP audit report using the approved template and SOP without assistance. Describes audit findings and provides appropriate suggested actions. Does not deviate from the approved format.

AP, action plan; EQA, external quality assessment; pSMILE, Patient Safety Monitoring in International Laboratories; SOP, standard operating procedure.

Analysis of Training Evaluation

The training coordinators collect feedback on the effectiveness of each training module. Trainees are not permitted to move to the next module without completing the postmodule evaluation. The training coordinators also compile cumulative feedback following completion of the entire curriculum.

The training coordinators analyze feedback from the post module self-evaluations and the grading of written assignments is compared to determine whether there is a weakness in the curriculum or a need for the learner to repeat any sections to ensure competency. Feedback from the mentorship evaluation is also compared with the cumulative module evaluations to determine whether improvements to the curriculum are necessary.

Reporting of Results

Collated evaluation results, including the mentorship evaluation, are presented to the project manager. This information is used to determine whether the learner is ready to graduate to full work responsibilities. Evaluation results are also discussed between the training committee and the project manager in an effort to continue ongoing improvement to the curriculum. The effectiveness of the curriculum is further shown by the successful addition of the new coordinators to the pSMILE workflow. Following their first year of employment, all new staff members received annual review ratings at or above expectations.

Discussion

The transition from Moodle to Blackboard was a lengthy process involving selection of a new training platform, learning how to use Blackboard effectively, and determining how to best optimize training materials for the selected platform. A small committee of trainers collaborated on reorganization of existing training materials with the goal of updating them to accommodate changes in pSMILE procedures as well as making training information and assignments more relevant. Blackboard training updates included a requirement for new coordinators to complete an evaluation at the end of each training section. Information obtained during review of these evaluations as well as direct discussion with trainees was invaluable in making meaningful revisions to training materials on an ongoing basis.

Lessons Learned and Future Challenges

Laboratory support for international laboratories participating in NIH HIV, tuberculosis, and other infectious disease clinical trials is not static. The work of pSMILE coordinators is constantly evolving to meet changing requirements generated by new protocols, testing methods, and shifting site laboratory responsibilities. Although these changes are incorporated into the flow of work for pSMILE coordinators on an ongoing basis, the process of updating the training program is often slower, leading to the situation in 2020 where a complete revamp of the training program was required. Revising the training program is time consuming if updates are allowed to accumulate. One of the major challenges is making revisions in real time, avoiding making a large number of modifications prior to preparing the training program for a new learner. This process has already been initiated as shown in TABLE 3 (Examples of Real Time Updates to the Training Program). The table identifies issues with the training program that needed to be addressed and how the training team responded.

TABLE 3.

Examples of real time updates to the training program

Source of change	Issue	Discussion
Learner-generated feedback	Training material failed to successfully explain pSMILE’s working relationships with collaborating organizations.	Learners did not fully understand the complex relationships between pSMILE, NIH, and collaborating networks. Failure to grasp these interactions caused confusion and frustration that affected progress on other training topics. The training team developed visual resources to better explain the relationships between organizations and pSMILE. In addition, more time was allotted in feedback sessions with the training team to ensure that trainees fully understood the most important interactions. Additional considerations: There may be a need for members of the training team to work 1-on-1 with the learner while covering this topic.
	Reorganization of training topics to improve the logical flow of training activities.	Learners made meaningful suggestions that assisted pSMILE trainers with improving the flow of training materials and assignments. One example involved moving the topic on creating APs to follow DAIDS GCLP. This made sense as the AP section is highly dependent on GCLP knowledge. Previously it would have taken many weeks before learners had the opportunity to begin the AP topic.
Initiated by the pSMILE training team	Adapting the training program to accommodate the ability to work virtually and transitioning to a total virtual platform.	The ability to work effectively using a virtual platform gained increased importance over time, especially in response to the COVID-19 pandemic and the decision for pSMILE to move to a totally virtual workplace. The training team moved topics on working virtually from the end of the training program to the beginning. Extensive IT support was initiated for new learners to ensure they had access to equipment and resources needed to effectively work on assigned training modules.
	Implementing changes to the HIV EQA section to improve clarity.	HIV EQA is currently particularly challenging due to the many test methods and generations of each test available. Accurate knowledge of this testing area is critically important to ensure that appropriate EQA is ordered and that results are correctly submitted and evaluated. This section was revised to include different scenarios requiring trainee coordinators to select the appropriate HIV EQA. Additional exercises were provided when necessary to ensure optimal comprehension of HIV testing requirements.
Changes to pSMILE’s internal processes	Updating training program to incorporate the introduction of the online IR	IR review is an important part of the work performed by pSMILE coordinators. Site laboratories are required to complete an IR when they receive a score of less than 100% on any protocol analyte. During the process of making updates to the training program significant changes were made to incorporate changing this process from a manual process to an online process.

AP, action plan; DAIDS, Division of AIDS; EQA, external quality assessment; GCLP, good clinical laboratory practice IT, information technology; IR, investigation report; pSMILE, Patient Safety Monitoring in International Laboratories.

Another area of the training program that could be better supported is the time allocated for new learners to shadow and work with experienced pSMILE coordinators. Two identifiable factors make this area of training challenging for the trainer. First, the lack of in-person contact with the learner and reliance on a now totally virtual platform for communication, and second, the ability to meet the demands of regular work requirements while training at the same time. The workload for coordinators is often difficult to predict, making it challenging to create an optimal training environment where there is time to answer questions, provide guidance, and review work for the trainee coordinator. Future measures that could be introduced to improve the experience for both educator and learner include providing coordinators with opportunities to develop training skills, such as webinars and in-person continuing education programs, if available, and using backup coordinators to help with day-to-day work if more time is needed for the trainer to provide effective training.

The pSMILE training program is long and often stressful for trainee coordinators. New employees are invariably well qualified and highly motivated, but it can still be confusing and difficult to put all the pieces together prior to beginning the process of working with site laboratories. Trainers should be able to provide encouragement as well as help with connecting the dots and explaining how pSMILE interacts with laboratories, networks, and other organizations. The ability of trainers to provide motivation and reassurance will aid the development of a training environment conducive to producing effective, confident, and knowledgeable pSMILE coordinators with the ability to use their skills to support the work of international clinical laboratories.

Conclusion

The use of asynchronous training for highly skilled and self-directed staff is a novel way to deploy training without hampering the productivity of existing staff. For this model to be successful, it is important to ensure that the curriculum is well developed and continually updated for relevance. Using feedback and evaluation of trainees allowed us to update our curriculum to include topics that had previously been lacking or underdeveloped. We were also able to improve the mentorship rotations, which is key to the success of the program. The fact that we have updated the curriculum continuously since the initial 2013 redesign demonstrates one of the key points of the Kern 6-step method, which is meant to be fluid, interactive, and interchangeable. The 6 steps are not intended to be purely sequential, but rather are designed to lead the curriculum developer to return again and again to previous steps to ensure continuous improvement. The use of Kern’s 6-step approach in this manner ensured that the curriculum met the needs of the contract, the sponsor, the project manager, and the learners, thereby validating its appropriateness, applicability, and feasibility.

Glossary

Abbreviations

pSMILE

Patient Safety Monitoring in International Laboratories

NIH

National Institutes of Health

DAIDS

Division of AIDS

MLSs

medical laboratory scientists

QA/QC

quality assurance/quality control

EQA

external quality assessment

GCLP

good clinical laboratory practice

SOPs

standard operating procedures

IT

information technology

ISO

International Organization for Standardization

Funding

This work was supported in whole or in part with federal funds from the National Institute of Allergy and Infectious Diseases, National Institutes of Health, Department of Health and Human Services, under Contract No. 75N93020C00001.