Evaluation of Diverse Health Professionals’ Learning Experience in a Continuing Education Activity for Quality Practices in Molecular Genetic Testing

Abstract

OBJECTIVE:

This study was conducted to evaluate the responses of 3,265 health professionals who took a continuing education (CE) activity during June 2009 - April 2012 for a comprehensive set of good laboratory practice recommendations for molecular genetic testing.

DESIGN:

Participants completed an evaluation questionnaire as part of the CE activity. Responses were summarized to assess the participants’ learning outcomes and commitment to applying the knowledge gained.

PARTICIPANTS:

Participants included nurses (47%), laboratory professionals (18%), physicians (14%), health educators (4%), public health professionals (2%), office staff (1%), and other health professionals (10%).

RESULTS:

Only 32% of all participants correctly answered all 12 open-book knowledge-check questions, ranging from 4 to 42% among the different professional groups (P<0.0001). However, over 80% of all participants expressed confidence in describing the practice recommendations, and 75% indicated the recommendations would improve the quality of their practice. Developing health education materials and local practice guidelines represented the common areas in which participants planned to use the knowledge gained (49% and 18% of all participants, respectively).

CONCLUSION:

Despite perceived self-efficacy in most participants, as high as 68% did not fully use the learning materials provided to answer the knowledge-check questions. These findings suggest the need for improved CE activities that motivate effective learning and address the specific needs of different health professions.

INTRODUCTION

Effective use of genetic laboratory services in health care requires quality practices not only by laboratory professionals who provide the testing services, but also by healthcare professionals and other users of laboratory services who are involved in the pre-analytic activities (e.g., test selection and ordering, appropriate informed consent, specimen collection and handling) and the post-analytic phase of testing (e.g., interpretation and use of test results in patient care, retention of records and tested specimens).^1–6 As applications of molecular genetic testing increasingly impact most disciplines of medical practice, it is critical that all professions involved in the performance, delivery, and use of molecular genetic testing services comprehend the best practices applicable to their roles and responsibilities.

In June 2009, the Centers for Disease Control and Prevention (CDC) published a comprehensive guide of good laboratory practices in molecular genetic testing for heritable diseases and conditions as a Morbidity and Mortality Weekly Report (MMWR) document.⁷ The document clarified applicable requirements of the Clinical Laboratory Improvement Amendments of 1988 (CLIA) regulations,⁸ and provided additional quality management recommendations aimed to improve the practices at the laboratory-clinician interfaces, or the pre- and post-analytic phases of molecular genetic testing.⁷

To encourage learning of the recommended laboratory practices by the broader healthcare community, CDC provided a continuing education (CE) activity for the primarily text-based document through the CDC Training and Continuing Education Online (TCEO) system from June 12, 2009 through April 11, 2012.⁹ CE credits were available for continuing medical education, continuing nursing education, continuing education units, and continuing education contact hours (CECH) for certified health education specialists.

While many resources have been devoted to developing and providing graduate, postgraduate, and residency training in genetics and genomics,^10–13 few studies have specifically evaluated feedback from CE activities addressing quality practices in molecular genetic testing. This study evaluated the learning experiences of health professionals from diverse professional backgrounds, with a specific focus on the CE participants’ ability to correctly answer the knowledge-check questions and their perceived self-efficacy with the recommended practices. The findings are intended to help improve the provision of future CE activities that motivate effective learning and address the specific learning needs of different health professions.

METHODS

Design of the CE Activity

Participants were instructed to review either the print or the PDF version of the MMWR document and complete a 29-question CE evaluation. The questionnaire included 1 question on the participants’ primary professional activities, 12 knowledge-check questions based on the good laboratory practice recommendations and CLIA requirements discussed in the MMWR document, and 16 questions assessing the participants’ confidence in describing what they learned, their perceived value and commitment to applying the knowledge gained, and satisfaction with the learning activity (Table 1). The answer keys for the knowledge-check questions were provided at the end of the questionnaire in both the print and the PDF version of the MMWR document in accord with the standard practice of CE activities at the time. Participants were instructed to answer all questions in the questionnaire to receive the CE certificates in their requested categories. No minimum correct score was required on the knowledge-check questions for completing the CE activity.

Table 1.

Summary of evaluation questions for soliciting feedback on the CE activity¹

Area of Feedback	Summary of Evaluation Questions
Participants’ professional activities	Which best describes your professional activities?
	A. Physician.
	B. Nurse.
	C. Health educator.
	D. Office staff.
	E. Laboratory professional.
	F. Public health professional.
	H. Other.
Knowledge-check questions2	1. Under the Clinical Laboratory Improvement Amendments (CLIA) regulations, laboratories performing molecular genetic testing that they have developed are subject to… (Indicate all that apply.)
	A. the general quality systems requirements for nonwaived testing.
	B. personnel requirements for high-complexity testing.
	C. specialty requirements for molecular genetic testing.
	2. For each molecular genetic test a laboratory performs, the laboratory should provide the following information to the users of its services before test selection and ordering: (Indicate all that apply.)
	A. The intended use of the test.
	B. Information on analytic validity and clinical validity of the test.
	C. Indications for testing.
	D. Information on appropriate collection, handling, and submission of specimens.
	E. The specific types of patient information needed to perform the testing and interpret test results.
	F. Laboratory contact information for consultation and discussion regarding a test being considered.
	3. For testing of mutations associated with human genetic diseases such as cystic fibrosis and Tay-Sachs disease, laboratories need to solicit the patient’s race/ethnicity and family history information with the test request to determine all of the following except…
	A. the type and amount of specimen needed.
	B. whether the specimen should be referred to another laboratory.
	C. the test methods to be used.
	D. the specific mutations to be tested.
	E. interpretation of test results.
	4. CLIA regulations require laboratories to refer a specimen for patient testing only to a CLIA-certified laboratory or a laboratory meeting equivalent requirements as determined by Centers for Medicare & Medicaid Services (CMS). Which of the following does not meet this CLIA requirement?
	A. Laboratories of the Department of Veterans Affairs.
	B. Laboratory facilities of the Department of Defense.
	C. Laboratories approved by the New York State Clinical Laboratory Evaluation Program.
	D. Laboratories approved by the Washington State Laboratory Quality Assurance Program.
	E. Research laboratories that do not have a CLIA certificate.
	5. For molecular amplification procedures, which of the following is considered an effective mechanism for monitoring and detecting cross-contamination of patient specimens?
	A. Inclusion of a positive control that represent the genotype to be detected with each run of patient specimens.
	B. Inclusion of a normal sample as the negative control with each run of patient specimens.
	C. Inclusion of a no-template control sample that contains all components of the amplification reaction except nucleic acid templates with each run of patient specimens.
	D. Inclusion of a spiked-in internal control in each amplification sample.
	6. Test reports of molecular genetic testing for heritable diseases or conditions should be retained for the longest possible time frame for all the following reasons except…
	A. Test results have long-term implications for the patients.
	B. Test results have implications for patients’ families and future generations.
	C. Advances in knowledge and understanding of disease processes might lead to improved interpretation of test results.
	D. Laboratories need to access previous test reports to conduct quality assessment activities.
	E. Laboratories must protect the confidentiality of patient information.
	7. A molecular genetic test report should …
	A. be understood by geneticists only.
	B. be understandable by nongeneticist health professionals and other authorized users of the test results.
	C. always be written in English.
	D. indicate “test result is negative” if no mutation is detected so that the test result can be easily understood.
	8. The director of a laboratory performing molecular genetic testing should… (Indicate all that apply.)
	A. ensure effective policies and procedures are in place for monitoring and maintaining the competency of the laboratory personnel.
	B. be able to perform a molecular genetic test better than anyone else in the laboratory.
	C. ensure available information needed to interpret test results for a patient is documented for each molecular genetic test the laboratory performs.
	9. When considering whether a new molecular genetic test should be introduced to the patient testing offered by a laboratory, the laboratory should consider… (Indicate all that apply.)
	A. evidence in published literature on the intended use of the new test.
	B. test methodology needed to perform the new test.
	C. whether laboratory personnel are capable of performing the test and communicating test results to the laboratory’s clients.
	D. the needs and demands of the new test based on a market analysis.
	10. Information on the clinical validity of a test to diagnose or predict risk for a health condition is often affected by… (Indicate all that apply.)
	A. clinical sensitivity.
	B. prevalence of the disease or health condition.
	C. clinical specificity.
	D. penetrance.
	E. current knowledge and testing technology.
	11. How often should control procedures be performed for molecular genetic testing for heritable diseases or conditions?
	A. Each time patient testing is performed.
	B. Once each day patient specimens are assayed.
	C. Once each week patient testing is performed.
	D. Once each month patient testing is performed.
	12. When a laboratory uses a purified DNA sample extracted from a cell line containing a rare mutation as a positive control in patient testing, which of the following is considered appropriate for monitoring the DNA extraction step of the testing process? (Indicate all that apply.)
	A. Testing patient samples for a housekeeping gene to determine specimen quality and integrity each time patient testing is performed.
	B. Testing patient samples for a spiked-in control sequence to assess the presence of inhibitors each time patient testing is performed.
	C. Testing patient samples for a housekeeping gene to determine specimen quality and integrity once each day patient testing is performed.
	D. Testing patient samples for a spiked-in control sequence to assess the presence of inhibitors once each day patient testing is performed.
Confidence in describing knowledge gained	After reading this document, I am confident I can describe …
	• the recommended quality practices for each of the three phases of the molecular genetic testing process (SA/A/N/D/SD3).
	• the qualifications, responsibilities, and competency of laboratory personnel (SA/A/N/D/SD).
	• issues to consider when planning or introducing new molecular genetic testing (SA/A/N/D/SD).
	• procedures for ensuring confidentiality of patient information in molecular genetic testing (SA/A/N/D/SD).
Perceived value	These recommendations will improve the quality of my practice (SA/A/N/D/SD).
Commitment to applying the knowledge gained	I plan to use these recommendations as the basis for… (Indicate all that apply.)
	A. health education materials.
	B. insurance reimbursement policies.
	C. local practice guidelines.
	D. public policy.
	E. other.
Satisfaction with the learning experience	• The learning outcomes (objectives) were relevant to the goals of this document (SA/A/N/D/SD).
	• he content was appropriate given the stated objectives of the document (SA/A/N/D/SD).
	• The content experts demonstrated expertise in the subject matter (SA/A/N/D/SD).
	• Overall, the quality of the document was excellent (SA/A/N/D/SD).
	• The MMWR format was conducive to learning this content (SA/A/N/D/SD).
	• The instructional strategies used in this document (text and appendices) helped me learn the material (SA/A/N/D/SD).
	• Overall, the length of the journal report was (Much too long/A little too long/Just right/A little too short/Much too short)
	• Do you feel this course was commercially biased? (Yes/No).
Factors that influenced CE participation	• The availability of continuing education credit influenced my decision to read this document (SA/A/N/D/SD).
	• How did you learn about the continuing education activity? (Indicate all that apply.)
	A. Internet.
	B. Advertisement (e.g., fact sheet, MMWR cover, newsletter, or journal).
	C. Coworker/supervisor.
	D. Conference presentation.
	E. MMWR subscription.
	F. Other.

¹See Ref. 7: Centers for Disease Control and Prevention. Good laboratory practices for molecular genetic testing for heritable diseases and conditions. MMWR Recomm Rep 2009; 58(RR-6):1–37.

²Correct answer options are shown in boldface.

³SA: Strongly agree; A: Agree; N: Undecided; D: Disagree; SD: Strongly disagree.

Collection of Participants’ Responses

The CE activity could be completed online via the CDC TCEO system website from June 12, 2009 through April 11, 2012 or by mail using the response form included in print copies of the document from June 12, 2009 through September 30, 2010. Responses received by mail were subsequently entered into the CDC TCEO system by CDC staff. Data collection on CDC-provided CE activities through the TCEO system has received continued clearance by the Office of Management and Budget (OMB) as required by the Paperwork Reduction Act (OMB clearance #0920–0017). For the purpose of this study, individual identifiers were removed so that no linkage could be made to any individual participant. This study was determined to be a non-research evaluation by CDC.

Statistical Analyses

Participants were stratified according to the categories they selected to best describe their professional activities. Responses were analyzed for all participants combined and for each participant category. Only those participants who characterized themselves into a single participant category were included in the respective professional group. Any individual group that represented less than 1% of all participants was combined with the “Other” category. Participants who did not select any professional category or selected more than one category were included in all participants (N=3,265) but were excluded from the analyses for any specific group. Data analyses were performed with Excel 2010 (Microsoft, Redmond, WA) and the Statistical Analysis System^® (SAS^®) Version 9.3 programs (SAS Institute, Cary, NC).

For the knowledge-check questions, a correctly answered question was defined as selecting all the correct and only the correct answer choice(s) provided for the question. The correct responses were summed into Poisson variables representing the number of correct answers given by each participant. Descriptive statistics were computed as counts and percentages for the correct answers to the knowledge check questions and for the categorical responses to the ‘yes/no’-type or the Likert-scale evaluation questions. Confidence intervals were computed using binomial distributions, generally as 95%, 2-tailed binomial confidence intervals if not otherwise noted. The bar charts in the figures display the confidence intervals. Statistical comparisons and tests of significance included Chi-square tests, Likelihood ratio, and Pearson correlations as appropriate and indicated for the result analyses.

RESULTS

Participants’ Professional Activities

Of a total of 3,265 individuals who participated in the CE activity, 3,139 (96%) characterized themselves in one of the provided participant categories, including 1,535 nurses, 572 laboratory professionals, 451 physicians, 133 health educators, 71 public health professionals, 47 office staff, 7 payers of laboratory services, and 323 who selected the “Other” category. Due to the small number of the payers of laboratory services, we combined the last 2 categories as “Other professionals” for the purpose of this study (Table 2). The remaining 4% of participants, which included 8 individuals who did not select any participant category and 118 who selected 2 or more categories, could not be included in the analyses for the specific professional groups but were still accounted for in the analyses for all participants (N=3,265).

Table 2.

Professional Activities of CE Participants

	Professional Activities	No. of Participants	% Total Participants
Participants who selected a single category	Nurses	1535	47%
	Laboratory professionals	572	18%
	Physicians	451	14%
	Health educators	133	4%
	Public health professionals	71	2%
	Office staff	47	1%
	Other professionals1	330	10%
Participants who selected more than 1 category	2 categories	99	3%
	3 and more categories	19	0.6%
Participants who did not specify their professional activities	Not specified	8	0.2%
Total		3265	100%

¹“Other professionals” include the participants who selected “Other” (N=323) and those who selected “Payer of laboratory services” (N=7).

²Actual total regardless the rounding of some participant categories’ percentages”

Responses to the Knowledge-check Questions

Participants’ ability to use the knowledge and information provided in the CE activity to correctly respond to the 12 knowledge-check questions is presented in Figure 1. The proportion of participants correctly answering all 12 knowledge-check questions varied significantly among the different professional groups, ranging from 4% for the office staff to 42% for the public health professionals (p<0.0001 by Chi-square analysis). Similarly, participants who correctly answered 9 or more knowledge-check questions (corresponding to a correct response rate of 75% or above) also varied significantly (19–65%, p<0.0001 by Chi-square analysis). Overall, approximately 32% of all participants provided correct responses to all 12, 21% to between 9 and 11, and 43% to only 0–8 of the knowledge-check questions (Knowledge-check results were not calculated for the 4% participants who did not select any participant category or selected 2 or more categories).

Figure 1.

CE Participants’ Responses to Knowledge-check Questions: Distribution of Results by Professional Groups

Confidence in Describing the Knowledge Gained

Participants expressed varying degrees of confidence in describing the 4 major areas of the good laboratory practice recommendations upon completion of the CE activity (Figure 2). Among all participants who responded to these questions, 83% agreed or strongly agreed that they were confident in describing the recommended practices for each of the three phases of the molecular genetic testing process. The proportions selecting the “strongly agree” or “agree” responses were 86% for the qualifications, responsibilities, and competency of laboratory personnel; 81% for the issues to consider when planning to introduce new molecular genetic tests; and 87% for ensuring confidentiality of patient information in molecular genetic testing (Figure 2). For the first 2 areas of the recommended practices, the differences among professional groups were generally not statistically significant except for the office staff participants, which reported significantly lower levels of confidence in these areas (72% and 72%; p=0.05 and respectively across all professional groups but p=0.14 without office staff). No significant differences were observed among the professional groups for planning to introduce new molecular genetic tests or ensuring confidentiality of patient information (p=0.16 and 0.33 respectively) (Figure 2).

Figure 2.

Confidence Reported by All Participants in Describing Major Areas of the Recommended Good Laboratory Practices

Perceived Value of the CE Activity and Commitment to Applying the Knowledge Gained

Of all participants, 75% agreed or strongly agreed that the good laboratory practice recommendations will improve the quality of their practice (Figure 3). Among the different professional groups, the combined proportions of the “agree” or “strongly agree” responses were 78% in nurses, 76% in laboratory professionals, 77% in health educators, 73% in office staff, and 55% in public health professionals. The inter-group differences were statistically significant (p<0.0001 by Chi-square analysis). In addition, public health professionals presented the highest and significantly different (Chi-square p=0.0006) percentage indicating they were undecided whether the recommendations would improve their practice (38%), as compared with 18–25% in all other groups (Figure 3).

Figure 3.

Perceived Value of the CE Activity: Participants’ Responses to the Evaluation Statement “These Recommendations Will Improve the Quality of My Practice”

Health education materials represented the major intended application of the recommendations by all participants (49%), followed by local practice guidelines (18%), public policy (5%) and insurance reimbursement policies (5%) (Table 3). However the responses from the different participant groups varied significantly for these intended uses (Chi-square p<0.0001 for health education materials, public policy, and insurance reimbursement policies; p=0.0007 for local practice guidelines). The proportion of participants selecting the “other” uses option also showed significant inter-group differences (Chi-square p<0.0001).

Table 3.

Participants’ Commitment to Applying the Knowledge from the CE Activity

I plan to use these recommendations as the basis for…	All participants	Nurses	Laboratory professionals	Physicians	Health educators	Public health professionals	Office staff	Other professionals
Health education materials	49%	53%	50%	40%	56%	24%	47%	37%
Local practice guidelines	18%	18%	20%	23%	18%	11%	21%	12%
Public policy	5%	3%	5%	9%	12%	10%	11%	2%
Insurance reimbursement policies	5%	5%	1%	6%	12%	7%	11%	1%
Other	37%	33%	38%	37%	20%	54%	21%	59%

Satisfaction with the CE Activity

Overall, participants expressed high levels of satisfaction with the CE activity by agreeing or strongly agreeing that the content was appropriate (92%), the content experts demonstrated excellent expertise (93%), the learning objectives were relevant (91%), the MMWR format was conducive to learning (91%), the instructional strategies were helpful (90%), and the overall quality of the guideline was excellent (90%) (Figure 4A). Similar response patterns were observed across the professional groups. No statistically significant differences were found among physicians, nurses, laboratory professionals, and public health professionals for any of the six satisfaction measures (p=0.27–0.83).

Figure 4.

Participants’ Satisfaction with the CE Activity

The length of the document was considered appropriate by 64% of all participants, including 67% of the laboratory professionals and 71% of the other professionals. However, 42% of the physician learners thought it was too long whereas 12% of the educator participants felt it was too short. While the majority of the participants did not consider this learning activity commercially biased, 11% of all participants and as high as 27% of office staff did think so despite the fact that no reference was made to any commercial entity throughout the document (data not shown). The CE activity did not include a space for the participants to provide any explanation for this response.

Factors Influencing CE Participation

Approximately 89% of all participants agreed or strongly agreed that the availability of CE credits influenced their decision to read the guideline document, with higher percentages observed in public health professionals (94%) and nurses (91%) than the other participant groups (82%−89%) (Figure 4B). In addition, participants identified the internet as the major source to be aware of this CE activity (44%), followed by coworker/supervisor (30%) and MMWR subscription (14%). Laboratory professionals and office staff were more likely to learn about this CE activity from coworkers or supervisors (53% and 40% respectively), whereas MMWR subscription represented a more frequent source of awareness in physicians (38%) and public health professionals (26%) (data not shown).

DISCUSSION

The overall results from this CE activity support the value of learning good laboratory practice recommendations for molecular genetic testing by individuals from diverse professional backgrounds, including laboratory and non-laboratory professionals. However, while the overall satisfaction and immediate learning outcomes were evident, the differences among the responses from the participants of different professional backgrounds regarding their learning experiences, possible ways of using the knowledge gained, and factors influencing their CE participation are worth noting. These observed differences are not meant to be generalizable to the respective health professions because the participants in this CE activity were self-selected and might also be subject to differences in awareness and access to the MMWR publication and the CE activity. Nevertheless, the different responses from the participants in this CE activity could reflect their different needs or expectations for CE participation, different perceptions in how specific areas of the recommendations might relate to their work settings or job responsibilities, different learning behavior, and different degrees of familiarity with molecular genetic testing practices. It is important to consider these needs in developing future learning activities for quality practices in genetic testing, which may need to entail separate CE activities for the different health professions.

An interesting observation in this study is the differences between the participants’ perceived self-efficacy with the good laboratory practice recommendations and their ability to correctly answer the knowledge-check questions using the information provided in the CE activity. This CE activity was provided during 2009–2012 in keeping with the standard CE practice at that time to make the answer keys to the knowledge-check questions available and to award CE credits to the participants with no minimum passing score as long as all questions were answered. The 12 open-book knowledge-check questions were intended to reinforce the participants’ learning and correct answers were expected for all of the 12 questions, because the participants were instructed to refer to the learning material as needed and also could use the answer keys as a self-check. However, only about 32% of all participants provided correct answers to all knowledge-check questions and 53% correctly answered 9 or more. In contrast, greater than 80% of all participants expressed confidence in describing the key areas of the recommended practices. Discrepancies between self-efficacy and applied skills in health professionals have been reported, suggesting while learners’ confidence in their capacity could encourage them to engage in activities requiring the relevant knowledge and/or skills, further learning opportunities would be necessary to allow knowledge enhancement, skill building, and incorporation of the learning experiences in practice.¹⁴ In our study, it is likely that some participants did not use the instructional guides provided because the format of the learning activity did not adequately motivate them to do so. These findings however, did suggest the needs for developing further educational materials and tools to improve understanding of the recommended molecular genetic testing practices by laboratory as well as non-laboratory health professionals. CDC therefore developed an online course in 2012 that includes simulated case scenarios, interactive learning, and actual measurement of knowledge gain to help learners enhance competencies in quality practices for molecular genetic testing.¹⁵

Another lesson learned from this study was that no pre-test was administered in this CE activity to ascertain the participants’ baseline knowledge of the good laboratory practices for molecular genetic testing, therefore it is not possible to determine the degrees of knowledge improvement for each individual participant after taking the CE activity. The differences among the participant groups in providing correct answers to the knowledge-check questions might also reflect their abilities or interest to use the learning tool provided in addition to understanding the learning material. These findings could be used to improve future competency-based learning and education efforts for the broader health professional community. As an example, a pre-test component has been included in the current online course to encourage participants to comprehend the instructional content and to provide better means for measuring their learning.¹⁵ In addition, efforts are underway to follow up with the participants to determine the extent of adoption and implementation of the recommendations into practice.

CONCLUSION

One of the key lessons learned from this study is that CE learners’ perceived self-efficacy may not reflect their actual ability to use or apply the knowledge gained. Therefore, continuous learning opportunities are often necessary to facilitate knowledge enhancement, skill building, and incorporation of the learning experiences into practice. It is also important to consider the different backgrounds, perception, and learning needs of diverse health professionals in developing effective shared learning activities. The inclusion of pre- and post-learning assessments will be crucial for improved CE activities as well as any competency-based professional development activity.¹⁶

Cooperation of multiple healthcare disciplines to achieve effective implementation of recommended practices is increasingly recognized as important in many healthcare settings.^17,18 Implementation of best practice guidelines and recommendations, especially those affecting the laboratory-user interfaces, requires the collaboration and cooperation of all professions involved. While this CE activity did not provide a platform for interactive learning among the participants, quality practice recommendations for processes in which laboratory professionals, healthcare providers and other stakeholders have shared responsibilities were imparted to the learners of diverse professional backgrounds. Therefore, insights gained from this study also could be useful for developing future inter-professional learning activities in genetic testing and other areas of laboratory medicine to promote the quality and effective use of laboratory services in health care. ^19,20

ABBREVIATIONS:

CE

Continuing education

CDC

Centers for Disease Control and Prevention

MMWR

Morbidity and Mortality Weekly Report

CLIA

Clinical Laboratory Improvement Amendments of 1988

CDC TCEO

CDC Training and Continuing Education Online