Effectiveness of an online curriculum for medical students on genetics, genetic testing and counseling

Abstract

Background

It is increasingly important that physicians have a thorough understanding of the basic science of human genetics and the ethical, legal and social implications (ELSI) associated with genetic testing and counseling.

Methods

The authors developed a series of web-based courses for medical students on these topics. The course modules are interactive, emphasize clinical case studies, and can easily be incorporated into existing medical school curricula.

Results

Results of a ‘real world’ effectiveness trial indicate that the courses have a statistically significant effect on knowledge, attitude, intended behavior and self-efficacy related to genetic testing (p<0.001; N varies between 163 and 596 for each course).

Conclusions

The results indicate that this curriculum is an effective tool for educating medical students on the ELSI associated with genetic testing and for promoting positive changes in students' confidence, counseling attitudes and behaviors.

The National Human Genome Research Institute (NHGRI) and the National Coalition for Health Professional Education in Genetics (NCHPEG) have expressed the need for the education of physicians about the ethical, legal and social implications (ELSI) related to genetic testing and counseling. To help address this need, we have developed and assessed an Internet-based curriculum to educate medical students about topics related to the ELSI associated with genetic testing and counseling. We developed content and interactive features for five core modules and evaluated their effect on students' knowledge, attitudes, self-efficacy and intended behavior. Statistically significant increases in knowledge and positive changes in attitude, self-efficacy and intended behavior were observed in all these studies. We believe that this web-based curriculum is a valuable supplement to typical medical school genetics coursework and an effective means of teaching students about ELSI associated with genetic testing and counseling. The modules are available online for review by the general public for free and for use by medical student groups at a nominal charge at www.GeneticSolutions.com.

Need for training of physicians and future physicians

With the sequencing and annotation of the human genome and the significant advances in biotechnology that have occurred in recent years, the field of medical genetics has undergone a dramatic increase in complexity. In particular, as we gain a better understanding of the genetic basis of complex traits and disease, the number of genetic tests and other molecular diagnostic tools available is rapidly growing.

The drastic increase in genetic testing implies that physicians will need to become more capable of providing care that includes genetic considerations, including ELSI. Physicians will need to understand how to obtain and evaluate family histories, when to utilize genetic testing, how to interpret testing results, and when and how to refer patients to genetic counselors or medical geneticists. However, since 1999 there has been evidence that many physicians and health care providers are not sufficiently trained to make decisions about genetic tests, interpret test results or counsel patients regarding results and implications (1–7). For example, in a survey of physicians' attitudes regarding genetic testing for cancer risk, only 29% reported that they felt qualified to provide genetic counseling to their patients (8). A survey of obstetricians and gynecologists regarding practices of genetic counseling and screening for Down syndrome identified deficiencies in knowledge and dissatisfaction with their training in these topic areas (9).

In addition, previous research has shown that physicians do not refer patients to genetic counseling services as frequently as is appropriate. Physicians have indicated that access to genetics consultation, particularly in rural and remote areas, can be difficult to obtain (10). When referrals are made, physicians do not fully describe to their patients what to expect during genetic counseling and, consequently, patients may be poorly prepared for genetic counseling sessions (11, 12). Other studies have shown that the lack of appropriate referrals to genetic counseling services may be partially due to a lack of understanding of the role that genetic counselors play in patient care (13, 14).

Combined, these findings indicate that there is an urgent need for increased education and greater availability of informational resources on medical genetics and genetic counseling for health care providers. While there are a variety of online genetics continuing education courses and resources available, many of these programs target primary care practitioners and do not specifically focus on ELSI concerns (for review, see Guttmacher et al. (2)). Since genetics issues are relevant to most medical specialties, medical school may be the ideal time to educate physicians-in-training about genetic testing, counseling and related ELSI.

Supplemental Internet-based curricular materials

Due to the stringent curriculum requirements of medical schools and the need for medical students to complete many courses, adding courses to the medical school curriculum can be difficult (15). Therefore, a more pragmatic approach involves using supplemental course modules that can be incorporated into existing curricula and may have the potential to educate more medical students. As medical school class sizes increase, the need to reach more students without increasing the strain on existing facilities and faculty becomes clear. Internet-based curricular supplements offer a solution. The use of supplemental curricula saves time, provides flexibility to the instructor, and is consistent and reproducible across classes. Advantages to students include self-direction, interactivity, and flexibility in scheduling the learning experience. Additionally, the use of computer-aided instruction allows the curriculum modules to be conveniently utilized by a large population of medical students across the country.

Program development

Web-based curriculum

We developed a curriculum plan following genetics core competency guidelines from the NCHPEG, the American Society of Human Genetics (ASHG), the American Academy of Family Physicians (AAFP) and the Association of American Medical Colleges (AAMC). We worked with a professional instructional designer to develop the curriculum; genetic counselors, genetics counselors-in-training and allied health professionals wrote the module content. Multiple experts in the field of medical genetics and genetic counseling reviewed the modules during development, and we performed an additional round of revisions based on their comments. We also incorporated feedback from medical school faculty and administrators.

The online modules were created with in-house Java-based software that we previously used to develop online continuing medical education (CME) courses for health professionals. A dynamic XML interface allowed for rapid formatting of content to web format, while an easily updated database allowed for data collection, including tracking learner progress and success and data reporting.

Educational modules

The curriculum includes five online modules that focus on the skills and attitudes required for successful incorporation of medical genetics into clinical practice. Our educational approach is based on many theoretical models, including those of Adult Learning Theory – Andragogy (16), Social Learning Theory (17) and Cognitive Theory (18). Research in the field of adult education suggests that problem-solving strategies, as opposed to subject-centered learning, are more effective (16). Additionally, Dr. Francis Collins, former director of the NHGRI (19), has enthusiastically supported education that includes patient examples and case studies. Thus, the program modules are specifically designed to help the learner develop knowledge, skills, confidence and self-efficacy, awareness of available resources, and changes in counseling attitudes with regard to genetics issues.

Genetics content is geared to be accessible to medical students in their first year of study. Topics covered include pediatric genetics, newborn screening, prenatal genetic testing, genetic testing for breast and ovarian cancer risk, and the genetics and inheritance of complex disorders. A sixth module on research ethics was completed but judged to be not relevant to medical school curriculum by the faculty we surveyed; thus it is not included in the final program. In general, each module takes an hour to complete. The format is flexible, and learners are able to stop any module and complete sections at a later time. Students can pursue the material at their own speed and repeat materials, if desired.

Relevant genetics knowledge, clinical scenarios and case studies form the core of these modules to create learning experiences that represent real life situations and emphasize the impact that genetic diagnoses have on families. Although the modules are not intended to teach the basic science of genetics, we did include a high volume of background material on disease genetics in order for students to develop a context in which to understand the ELSI issues of various genetic tests. There are also two non-core modules available on the website that provide an overview of basic human genetics (A Basic Human Genetics Primer, Parts I and II) to be used as a refresher course, if needed.

Case studies

Case studies form the core of the modules and are used to make the learning experience more engaging and representative of real world experiences. Details of each case are presented in sections as the student progresses through the module. Video clips of a patient interview (described below) are incorporated in a way that correlates with the lesson content. Students have the option of viewing the videos or reading the interview transcripts as they proceed through each module. Examples of case study subjects include the mother of a child diagnosed with the metabolic disorder phenylketonuria, a mother and daughter undergoing genetic testing to evaluate their predisposition for breast cancer, an adult male with Fabry disease who participated in a clinical trial, and the mother of a 12-year-old girl diagnosed with autism.

Video vignettes

The video clips feature actual genetics counseling patients who underwent genetic testing or who have a genetic condition present in their family. During program development, we collaborated with the University of Alabama–Birmingham (UAB) to recruit adult volunteers over the age of 18 who were either current or former patients of the genetics clinic and their family members. Interested volunteers were screened by Clinical Tool Inc. (CTI) staff and selected based on the compatibility of their clinical case with the learning objectives of the curriculum plan. Genetics professionals from UAB and CTI performed the patient interviews. The UAB staff and faculty facilitated institutional review board (IRB) approval, informed consent, technical staff support and equipment for videotaping.

Interactive features

The modules contain several additional informative and interactive features, including graphical illustrations, short quizzes, hypertext for uncommon words that give a ‘pop-up’ definition or a link to a glossary, and links to websites with additional information and resources (Figs 1 and 2). For quizzes integrated into the modules, learners answer multiple-choice, matching and fill-in-the-blank questions and receive immediate corrective feedback (Fig. 1). Key points, case reviews and informational resources that physicians may recommend to patients are listed along the right side of the screen throughout the lessons.

Fig. 1.

Representative screenshot from the ‘Prenatal Genetic Testing’ course module. This screen shows examples of several features of the instructional courses (arrows). Definitions for important terms are provided as ‘pop-up’ windows when the learner moves the mouse over the term. Interactive quizzes are included and provide feedback for both correct and incorrect answers. Additional resources and information are provided in lists with links along the side of each screen.

Fig. 2.

Screenshot from the ‘Pediatrics: a family study of a child with multiple congenital anomalies’ course module. This screenshot shows a medical history and pedigree work up, which would take place during an appointment with a genetic counselor.

Integrated assessments

Learning assessments are also built into the web-based experience. Each module begins and concludes with a knowledge test consisting of 7–10 multiple-choice questions. After working through the online module, the student repeats the same knowledge test to assess change. Feedback on test answers is provided only after course completion, with links to the appropriate section of the course for additional review. In addition, at the end of each module, learners complete retrospective instruments that evaluate changes in attitude, self-efficacy and intended behavior with regard to ELSI genetics issues. Also included is a survey to determine learners’ overall opinions of the module, how well it met its learning objectives, and their general level of satisfaction with the learning experience. The project content experts reviewed all assessment forms and questions to ensure content accuracy and content validity.

Usability testing

To ensure that the website and module design were intuitive and understandable, we conducted three iterative rounds of usability testing during program development. The modules were evaluated for ease of use and navigation, audience interest in material, relevance and appropriateness of the content for medical students, and overall participant satisfaction.

A total of 42 medical students from various institutions participated in these studies. Traditional usability testing methods, such as ‘think aloud’ and task completion, were included (see Rubin (20) for a full description). In addition to these qualitative measures, participants completed a Likert-style survey rating satisfaction and ease of use. Usability testing included both the module content and the integrated assessments (e.g., the knowledge test and satisfaction survey). After completing each course module, learners were also surveyed regarding their opinions on how well it achieved its learning objectives and were asked to rate statements regarding general learner satisfaction. We made iterative changes to the modules as usability testing proceeded to address participant feedback. Overall, satisfaction with the modules’ design and content was high.

Evaluation methods

Overview

When the modules were complete, we conducted a ‘real world’ effectiveness trial with medical students. Each module was evaluated as a stand-alone learning experience (N varied between 163 and 596 for each course). Participants completed knowledge, attitude, self-efficacy, intended behavior and satisfaction assessments after each module.

Assessment development

We developed the study assessments and pre-tested for face and construct validity using two methodologies. First, consultants reviewed the instruments for clarity and relevance and assessed if the identified correct answer was actually correct and unambiguous. Secondly, medical students involved in usability testing reviewed the materials. The students completed the instruments as part of the usability experience and then provided feedback to indicate if the items were clear and relevant. Questions were revised as necessary, based on the feedback from consultants and students.

Knowledge assessments

Knowledge gain was measured using a 7–10 question pre/post-test taken by each learner. The knowledge items were multiple choice questions intended to assess different levels of cognitive learning, adapting the concepts of Bloom's taxonomy of educational objectives (21). Particular emphasis was placed on the levels of application, analysis and synthesis. Knowledge test items were based specifically on the educational objectives defined for each module; each educational objective had at least one related question. For example, two of the seven educational objectives for the module on Prenatal Genetic Testing were:

After this course you will be able to:

• Identify indications for population-based reproductive risk screening, prenatal diagnosis and genetic counseling referral (content-focused question, on Bloom's ‘knowledge’ level)

• Show respect for patient values, beliefs and autonomy through the use of non-coercive, value-neutral counseling when discussing prenatal testing and reproductive options (application-focused question according to Bloom's taxonomy)

An example knowledge test item related to the educational objectives above is as follows:

Which of the following women should be offered diagnostic testing in lieu of or in addition to first or second trimester screening?

1. A 31-year-old woman who has a sister with sickle cell disease.

2. A 19-year-old woman known to carry a balanced chromosome translocation.

3. A 27-year-old woman whose first child was born prematurely at 28 weeks.

Feedback at the conclusion of the educational experience will tell the learner that #2 is correct:

Diagnostic testing in addition to or in lieu of first or second trimester screening is recommended for any woman who: is known to carry a balanced translocation, over age 35 with a single fetus, over age 32 and pregnant with twins, have a history of a pregnancy with a detected single gene disorder, have a history of abnormal ultrasound results, or is known to carry a balanced translocation.

Attitude, self-efficacy, and intended behavior assessments

Knowledge change alone will not impact the behavior of the future physicians, so each module also included a simple measure of attitude, self-efficacy and intended behavior items. These constructs were measured using a retrospective pre/post-design with a Likert-style instrument. Qualitative feedback from usability testing indicated that students had a low tolerance for items not associated with what they perceived to be the educational goal (i.e., obtaining knowledge and skills), and thus we limited these assessments to one item per area.

For each of the constructs, the learner was asked to rate a statement on a five-point Likert scale (1 = ‘Strongly Disagree’, 5 = ‘Strongly Agree’). All rating scale items were presented with an ‘N/A’ option.

The attitude items ask the learner to rate his/her role as a health care professional in dealing with issues addressed in the modules. An example of an attitude item includes:

It is part of my role as a healthcare provider to recognize that the decision to pursue or decline genetic testing is often based on an individual's values and beliefs and, therefore, should be a personal decision that is free of coercion by healthcare professionals, family members, or peers.

The self-efficacy items were tied to learning objectives for each module, thus providing an additional check on the success of the modules. The intended behavior items rated whether the student plans to incorporate the ELSI principles learned from the modules into clinical practice.

As mentioned, we utilized a retrospective design to assess changes in attitude, self-efficacy and intended behavior for students who completed the course modules. A retrospective assessment occured after the experience, but asked the learners to rate an item on their level of agreement before and after participating in the course. This method has been suggested to be more valuable when adult learners are not in a position to judge accurately their abilities prior to learning more about a topic (22). Utilizing a retrospective format allowed the students to learn about the topic in question before being asked to respond to questions related to the topic. In addition, it required only one administration after the experience rather than two (before and after) and simplified the experience for students (22).

Learning objectives and satisfaction items post-course

Lastly, learners evaluated satisfaction with the learning experience. Satisfaction was rated on a five-point Likert scale (1 = ‘Strongly Disagree’, 5 = ‘Strongly Agree’) using a standard assessment tool that we had developed and validated previously. Learners also rated how well they felt each learning objective was met, with the end points defined as 1 = ‘Completely Failed to Meet Objective’ and 5 = ‘Completely Met Objective’.

Results

Between 1 December 2004 and 1 May 1 2007, 897 students at eight different medical schools completed between one and five modules per student as part of their regular medical school experience, resulting in a ‘real world’ effectiveness trial. The total number of module completions by all students was 1,750. Medical school faculty used the online modules that they felt were best suited to their curriculum goals. Some students were required to complete specific modules, while others had the option to use up to five of the modules as ‘extra credit’. Data from the knowledge assessments and retrospective measures were collected for all participants. Only data for learners who completed the pre- and post-test assessments were included in the data analysis. All students were aware that the modules were grant-funded by the NHGRI and were being evaluated based on the student's use and feedback.

Knowledge

We analyzed each knowledge assessment item using a two-tailed t-test with each student serving as his/her own control. A two-tailed t-test was chosen since there was the possibility that the student's score could decline on the post-test (if they found the material confusing, for example). Additionally, each module was assessed individually, in order to assess variation among modules. We used 70% as the passing score for each individual post-test.

Using the two-tailed t-test, all five modules met the stringent statistical cutoff of p<0.001, indicating a statistically significant difference in the pre- and post-test knowledge scores. Each module's average post-test score met or exceeded 70%. Average knowledge scores for all the courses combined were 53.83% correct pre-test and 70.4% correct post-test.

There was variation among the pre/post-test scores for the different modules, indicating that some had a more positive effect on knowledge gain than others. The largest score improvements were observed with the Newborn Screening module, for which learners averaged 43.3% correct on the pre-test and 70% correct on the post-test (26.7% difference) and the Prenatal Genetic Testing module, for which learners averaged 54% correct pre-test and 79.1% post-test (25.1% improvement). The smallest gains were seen in Presymptomatic Testing: Genetic Testing for Breast and Ovarian Cancer Risk, where the average pre-test score was 63.3% correct, and the average of the post-test scores was 72.8%.

For each individual module, we calculated Cohen's d, a statistic used to determine the actual size of the effect. While a t-test is affected by sample size, Cohen's d is not affected by sample size, since it is an estimate of the population as whole. This allows one to compare among modules by withdrawing the potential impact of the sample size. This is particularly useful since some sample sizes were quite large (e.g., the 628 students who took the Complex Inheritance module), while others were smaller (the 184 students who used Newborn Screening). In each case, we used the pooled standard deviation of pre/post-knowledge scores when calculating Cohen's d. For two of the modules, the effect size was medium to high (0.75–0.8); for the remaining three modules, the effect size was large (0.9 to above 1.0) (see Table 1).

Table 1.

Average scores on the knowledge pre/post-test, by module, total modules = 1,750

Course name	N	Average pre-test (% correct)	Average post-test (% correct)	Statistically significant (y/n?)
Pediatrics	372	59.7 (st dev = 15%)	70.7 (st dev = 16%)	Yes, p<0.001
				Cohen's d = 0.71
				Med-high effect size
Complex inheritance: genetics of common complex disorders	628	60.9 (st dev = 16%)	74.7 (st dev = 15%)	Yes, p<0.001
				Cohen's d = 0.9
				Large effect size
Presymptomatic testing: genetic testing for breast and ovarian cancer risk	189	63.3 (st dev = 13.8%)	72.8 (st dev = 11.7%)	Yes, p<0.001
				Cohen's d = 0.76
				Med-large effect size
Newborn screening	184	43.6 (st dev = 19.2%)	70 (st dev = 17.6%)	Yes, p<0.001
				Cohen's d>1.0
				Large effect size
Prenatal genetic testing	377	54 (st dev = 17.8%)	79.2 (st dev = 16.6%)	Yes, p<0.001
				Cohen's d>1.0
				Large effect size

Student year of training appears to have had an impact on the size of the knowledge increase, as students in their first year of medical school tended to score lower on the pre-tests than students in their third and fourth years. For instance, the Complex Inheritance: Genetics of Common Complex Disorders module was taken by students in their first, second and third years of training. As expected, first-year medical students scored the lowest of the three groups on the pre-test. Data analysis by year of training showed the greatest increase in knowledge for students in their first year of medical school.

Attitude, self-efficacy, and intended behavior

Each attitude, self-efficacy, and intended behavior item showed a statistically significant increase. A two-tailed t-test was used to assess change in the measures, with each student serving as his/her own control. We used a cutoff p-value of 0.05 to determine significance, but the majority of items passed the stringent cutoff of p<0.001 (Table 2).

Table 2.

Summary of attitude, self-efficacy and intended behavior assessment results for each module

Course module	Statement	N	Average score pre-test (st dev)	Average score post-test (st dev)	p Value
Pediatrics: a family study of a child with multiple congenital anomalies	Attitude – It is part of my role as a health care provider to use effective and respectful language when obtaining family history and discussing genetic conditions and birth defects with patients	351	4.39 (0.83)	4.53 (0.75)	<0.001
	Self-efficacy – I am confident in my ability as a health care provider to identify at-risk family members when there is a family history of a chromosome abnormality	345	3.69 (1.01)	4.12 (0.79)	<0.001
	Intended behavior – I intend to identify at-risk members when there is a family history of a chromosome abnormality	344	4.09 (0.93)	4.42 (0.77)	<0.001
Complex inheritance: genetics of common complex disorders	Attitude – It is part of my role as a health care professional to assess the benefits and limitations of susceptibility gene tests for common conditions	586	3.96 (0.93)	4.27 (0.82)	<0.001
	Self-efficacy – I was/am confident I can determine appropriate recurrence risks for complex conditions	596	3.29 (1.14)	3.74 (1.00)	<0.001
	Intended behavior – I intend to communicate the current (and often limited) knowledge base about the inheritance of complex conditions and the options for dealing with those risks to patients	588	3.97 (0.90)	4.28 (0.81)	<0.001
Presymptomatic testing: genetic testing for breast and ovarian cancer risk	Attitude – It is part of my role as a health care provider to provide adequate informed consent, including a discussion of the ethical, legal and psychosocial implications, prior to initiating genetic testing	178	4.37 (0.83)	4.6 (0.68)	<0.001
	Self-efficacy – I was/am confident in my ability to recognize applications of the four main biomedical ethical principles (autonomy, beneficence, non-maleficence and justice) in medical genetics	179	4.05 (0.91)	4.39 (0.66)	<0.001
	Intended behavior – I did/intend to seek assistance from specialists when my knowledge, time or personal values limit my ability to manage genetic cases	175	4.42 (0.76)	4.63 (0.65)	<0.001
Newborn screening	Attitude – It is part of my role as a future health care provider to obtain fundamental knowledge about genetic conditions, screening and long-term management	164	3.90 (1.05)	4.34 (0.87)	<0.001
	Self-efficacy – I was/am confident in my ability to describe criteria used to determine which medical conditions should be tested in the newborn screening panel	163	3.23 (1.2)	4.07 (0.87)	<0.001
	Intended behavior – I did/intend to detect treatable conditions in newborns	159	3.86 (1.09)	4.23 (0.96)	<0.001
Prenatal genetic testing	Attitude – It is part of my role as a health care provider to provide adequate informed consent, including a discussion of the ethical, legal and psychosocial implications, prior to initiating genetic testing	351	4.32 (0.85)	4.52 (0.77)	<0.001
	Self-efficacy – I was/am confident in my ability to identify potential common ethical and/or legal conflicts in managing the genetic testing process for patients and strategies to avoid such conflicts	354	3.79 (0.98)	4.23 (0.79)	<0.001
	Intended behavior – I did/intend to provide and document adequate informed consent prior to ordering genetic testing on my patients	343	4.28 (0.89)	4.5 (0.81)	<0.001

Note: Based on a Likert scale with 1 = Strongly Disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly Agree.

Learning objectives and satisfaction items

Learners rated satisfaction on a five-point Likert scale (1 = ‘Strongly Disagree’, 5 = ‘Strongly Agree’). Learners appear to be quite satisfied with the learning experience from these modules; average satisfaction scores for the five modules were high (≥ 4). Learners rated the module learning objectives by rating how well the module had achieved each on a five-point Likert scale (1 = ‘Completely Failed to Meet Objective’ and 5 = ‘Completely Met Objective’). Average scores on objective-based items ranged from 4.08 to 4.26 for all five modules.

Qualitative feedback

In addition to the quantitative evaluations, each module included an open-ended survey question at the end to collect qualitative feedback. Comments from students reinforced the findings of the other assessments. Samples of comments regarding the modules include: ‘These courses added some more pertinent details to the information I received in class’. ‘It is always useful to have thought through some of these dilemmas before encountering them’. ‘It's nice to understand, with clinical examples, how the education will affect our practice in the future’. ‘Medical schools integrate genetics in many different ways. While simply memorizing facts or statistics is not useful, the discussions about patient counseling are much more practical. Plus, not all schools do an adequate job of preparing students to counsel patients in this manner’.

Discussion

Curriculum strengths

Overall, our assessments of the modules indicate that they succeeded in our goals of helping the learner develop knowledge and skills, changes in attitudes and increased awareness of available resources in ELSI issues related to genetics testing and counseling. The satisfaction results show that the greatest strengths of the curriculum were its ease of use and the subject matter's relevance and importance for medical students. The students indicated that the case studies and patient interviews integrated into the course were particularly useful, a finding consistent with previous studies that demonstrated the effectiveness of using real life scenarios for teaching clinical topics to medical students (23). In addition, our results indicated that students at different stages of medical school training can benefit from the courses, so the lessons may be used as a supplement for either the pre-clinical or clinical curricula. These results should be of great interest to medical schools, since the courses can be quickly, easily and inexpensively incorporated into existing curriculum and provide training in an area that is often underrepresented.

Not all students improved equally, suggesting that previous background has a significant effect on the value of the material. As noted, first-year students in particular had the most improvement from pre-test to post-test, presumably due to their lack of baseline knowledge about the topics. Additionally, there was variation in improvement in measures among modules as well, indicating that some modules (specifically Prenatal Genetic Testing and Newborn Screening) were of more use than others. Primarily, this appears to reflect that student knowledge of these areas was low entering the experience (an average pre-test score of 54 and 43% correct). Average post-test knowledge scores for all modules ranged from 70 to 79% correct (see Table 2). This narrow range suggests that there may well be a ceiling effect for learning related to the topic, or alternatively, related to the content of the items on the pre/post-test. We initially anticipated that our educational materials would reflect a ‘medium’ effect on knowledge, however, the data indicate that the online training modules had a medium-high or even high effect. Since effect size (when calculated with Cohen's d) is felt to be an indication of the population as a whole, these strong effect sizes should be seen when the modules are used with any group of medical students.

Impact of online experiences on knowledge and attitude

Several other Internet-based medical school curricula have been shown to be effective in changing knowledge and attitudes of students (24–26). Our results provide additional support for the use of Internet-based learning tools to teach medical students. Advantages of this format include easy accessibility, flexibility and individualized, independent learning that can be targeted to address specific weak areas. It also allows for the use of a combination of text, graphics, audio and video, which can be helpful in explaining complex problems and linking these concepts to real world situations. Another advantage of computer-based learning is that it may benefit specific groups of students who don't gain as much from traditional teaching methods (27, 28). For example, in a study of Irish medical students who participated in a computer-aided learning program, students with lower class ranks utilized the computer learning system more frequently, and usage positively correlated with improvements in class rank (28). Finally, online course content can easily be adapted and updated, which is especially advantageous for material in the quickly evolving field of genetics.

This study lends support to the growing literature that online educational interventions have the ability to impact student knowledge, as well as reported attitudes. While medical students are known to be ‘expert learners’, it cannot be assumed that any specific educational material presented to them is of value. The significance of the work reported here is not so much that medical students can increase their knowledge per se, but that when developed carefully, online educational materials have the ability to perform successfully and enhance the educational experience for medical students. Curriculum developers must work with content experts to review materials to ensure content accuracy, face validity and clarity of presentation. In addition to a pre/post-knowledge test, items measuring attitude can provide an additional check on the ‘take home message’ of the content to verify that educational goals are being delivered effectively.

Study limitations

The limitation of the research methods must be acknowledged, the most obvious being the quasi-experimental design, with students assigned due to their participation in a specific medical school class. This created an effectiveness trial, rather than an efficacy trial. The effectiveness trial included students from different schools whose motivations for using the online modules were not comparable – some were required to participate by their faculty, while others chose to do so for ‘extra credit’. Nonetheless, the data reported do reflect the actual use of the materials by the target audience in a truly ‘real world’ sense, allowing the results to be less amenable to analysis but more reflective of the results faculty members could realistically expect if they used the materials at their institutions.

There may have been some effect due to exposure to the same questions in the pre- and post-tests, but any learning effect due to exposure was mitigated by the lack of feedback on the pre-test – students were not told if they got the pre-test question ‘right’. Further, the questions were designed to reflect specifically the educational objectives; thus, the only thing students were likely to have ‘learned’ by taking the pre-test was to focus their attention on the module content related to the learning objectives. While this was a valuable asset from an instructional design perspective, it did impact the value of the questions as purely research instruments. Since this intervention was designed primarily for actual use, the limitation on research value was outweighed by the ability of the pre-test as a pedagogical tool.

As recommended by the theoretical model, we have given the learner control over the learning experience, and he or she may (rightfully) choose to focus on some specific areas and ignore others. In addition, some aspects of learning are not suited to measurement without direct interaction with the student (e.g., flexibility or ‘bedside manner’). Nonetheless, by including a variety of outcomes, including measures of effect on attitude, self-efficacy, intended behavior, and knowledge, we have attempted to provide a reasonably full evaluation that allowed us to assess whether or not the modules met our stated learning objectives.

Exposure to the pre-test questions is a limitation from an experimental design perspective, with the possibility that pre-exposure to the question influenced the student's better performance on the post-test using the same questions. We attempted to mitigate this effect through the lack of feedback on the student's performance on the pre-test. From an educational perspective, the exposure to the pre-test questions could be considered a strength, since each question was directly linked to an educational objective, the pre-test can serve as a stimulus to learning for the student, as well as an assessment.

Additionally, attitude, self-efficacy and intended behavior constructs were measured with only one item per module since, during usability testing, medical students indicated that the target audience would only tolerate one item per module for each of these constructs. The target audience was more tolerant of items related to knowledge, since they saw this as the ‘reason’ for the learning experience. The small number of total items limited the reliability of the results for these areas, which is frustrating given that these areas are both of great concern and notoriously difficult to measure with reliability. These constructs are perhaps the most significant as well, given the focus on teaching the students not only basic genetics knowledge, but also the appropriate application of that knowledge in future practice. The use of a pre/post-retrospective test is controversial, since both measurements take place after the experience (unlike the traditional pre/post-test). However, in situations where there is good reason to believe that response shift could occur (in this case, the learner would be unable to accurately self-assess due to insufficient knowledge), the use of the retrospective test is well-established (29).

Conclusion

Using online materials to supplement the genetics curriculum

The results of our module evaluations were positive; however, we do not yet know the long-term effects of these modules on medical student knowledge and behavior. It is a challenge to determine the long-term educational significance of any educational intervention. However, we are encouraged that previous studies have demonstrated the long-term effectiveness of online medical courses. Students that completed a distance course in medical informatics showed good knowledge retention 12 months after completing the course (30). The degree of retention correlated positively with the students' satisfaction with the curriculum and the average amount of time the students spend on the computer. To investigate the long-term effectiveness of our curriculum, future work will include additional knowledge testing to measure retention of the material over time. In addition, due to the encouraging results of our module assessments and the significant advantages of the online format, we are already in the process of preparing other web-based modules on topics targeted to the medical school audience. The medium-high and high effect sizes strongly support that these online learning experiences were educationally significant for the student audience.

Previous research has shown that physicians do not refer patients to genetic counseling services as frequently as is appropriate. Further, when referrals are made, a proper description of genetic counseling is often not conveyed; consequently, patients may be poorly prepared for genetic counseling sessions (11, 12). Additionally, other studies have shown that the lack of appropriate referrals to genetic counseling services may be partially due to a lack of understanding of the role that genetic counselors play in the care of patient (13, 14). By utilizing these modules in medical school curriculum, providers will gain a better understanding of the issues that are addressed by genetic counselors. Consequently, they will be better prepared to make and appropriately discuss genetic counseling referrals.

There is a demonstrated need for increased education and training of health care providers in topics related to genetics, genetic testing and counseling. By initiating this education early in their training, we hope these courses will result in long-term behavioral effects and continued efforts by these health care providers to pursue self-directed learning in this complex and rapidly changing field. We hope that future student learners of the materials will be better prepared to handle patients’ questions regarding genetics issues, have a better understanding of the importance of genetics-related ELSI, and be more comfortable and knowledgeable about the appropriate use of genetic testing for the benefit of their patients.

Advantages of online educational materials for medical schools

There are advantages to online training, such as we have described here, as a supplement to traditional educational interventions. Once effect is that established, online modules allow schools that currently do not have structured training in given topics to easily incorporate the material into their programs. Schools that already provide some training could similarly benefit from a standardized, easily deployed educational opportunity.

Medical school enrollment was at an all-time high of 18,000 students in the 2008 entering class and is slated to continue to increase (31), increasing the demands on faculty. While 113 medical schools report an increase in enrollment, less than 50% have hired new faculty. Fifty-eight percent are changing teaching and curriculum methods (32), acknowledging the role that technology now plays in medical school curricula; current medical school students are of the digital age and are quick to embrace electronic resources when studying. We propose that supplemental online training is an appropriate and useful method for meeting the challenge of increasing numbers of students, with minimal burden to resources such as current faculty or classroom/laboratory space.

The improvement in knowledge and the movement toward appropriate attitudes coupled with the positive qualitative feedback from learners indicates that this intervention specifically has educational value and is enjoyed by the students. The combination creates a powerful argument for the use of this and similar online case-focused materials, particularly in topic areas that are underrepresented in the current curricula of medical schools.

Conflict of interest and funding

This work was funded by grant #R25-HGO2266 from the National Human Genome Research Institute (NHGRI) to T. Bradley Tanner, MD at Clinical Tools, Inc.