Ensuring best practice in genomics education: A scoping review of genomics education needs assessments and evaluations

Summary

A health workforce capable of implementing genomic medicine requires effective genomics education. Genomics education interventions developed for health professions over the last two decades, and their impact, are variably described in the literature. To inform an evaluation framework for genomics education, we undertook an exploratory scoping review of published needs assessments for, and/or evaluations of, genomics education interventions for health professionals from 2000 to 2023. We retrieved and screened 4,659 records across the two searches with 363 being selected for full-text review and consideration by an interdisciplinary working group. 104 articles were selected for inclusion in the review—60 needs assessments, 52 genomics education evaluations, and eight describing both. Included articles spanned all years and described education interventions in over 30 countries. Target audiences included medical specialists, nurses/midwives, and/or allied health professionals. Evaluation questions, outcomes, and measures were extracted, categorized, and tabulated to iteratively compare measures across stages of genomics education evaluation: planning (pre-implementation), development and delivery (implementation), and impact (immediate, intermediate, or long-term outcomes). They are presented here along with descriptions of study designs. We document the wide variability in evaluation approaches and terminology used to define measures and note that few articles considered downstream (long-term) outcomes of genomics education interventions. Alongside the evaluation framework for genomics education, results from this scoping review form part of a toolkit to help educators to undertake rigorous genomics evaluation that is fit for purpose and can contribute to the growing evidence base of the contribution of genomics education in implementation strategies for genomic medicine.

A scoping review of needs assessments for, and/or evaluations of, genomics education interventions for health professionals showed variability in evaluation approaches and terminology and identified few articles considering long-term outcomes of interventions. The reviewed evaluation questions, outcomes, and measures form part of a toolkit to help educators undertake rigorous evaluation.

Introduction

Effective genetics and genomics education for health professionals is an essential component of genomic medicine implementation.¹ Genomic testing is complex and involves many health professionals, including those not trained in genetics,² many of whom report being unprepared for genomic medicine.³ Increasingly, programs have been developed to address the educational needs of non-genetics health professionals—medical specialists, general practitioners/family physicians, nurses, midwives, and allied health professionals.⁴ These programs aim to enable different health professionals to integrate genomic information into practice,⁵ identifying patients who could benefit from genomic testing, referring for or ordering genomic tests, and interpreting and acting on test results.⁶

However, these programs have been inconsistently described and evaluated to assess impact.⁴ Educators need tools that support them to rigorously evaluate their programs so that others can use the information and insights to enable better education and health outcomes.

Context and conceptual framework

Program funders require evidence of program effectiveness and impact.⁷ However, a landscape analysis of genomics education produced in Australia showed most program convenors were genetics/genomics experts with no qualifications or experience in education or evaluation, few used needs assessments to inform education program design or content, and few programs were evaluated.⁸

To address these issues, Australian Genomics, a national translational genomics program, convened an international group of experts in program evaluation, genetics/genomics education, and implementation science.⁹ The expert group identified the need for tools to support these genetics/genomics expert educators to develop, evaluate, and report genomics education interventions. The group contributed to design of a program logic model⁹ and reporting standards¹⁰ for genomics education and its evaluation. The RISE2 Genomics standards include 12 items for reporting evaluation of genomics education interventions, including evaluation aims, approaches, foci, design, recruitment, outcome measures, analysis, funding, results, and impact.

The expert group also recommended additional guidance and support be developed for educators, which would be particularly helpful for those unfamiliar with evaluation practice. Specifically, they recommended an evaluation framework and associated evaluation questions and measures. The framework (reported separately)¹¹ describes the different stages of evaluation across an education program—from planning (pre-implementation) through delivery (implementation) and impact (immediate, intermediate, and long-term outcomes)—as well as different stakeholders involved at different points and external influences on delivery and impact. Here, we present the findings of a scoping review of needs assessments for, and evaluations of, health professional genetics/genomics education. We aimed to identify how the evaluation of needs and outcomes of genomics education for health professionals are captured and described in the published literature.

Our intent was to explore and summarize the breadth of approaches and measures used to assess educational need and evaluate genomics education interventions and to identify and clarify common concepts, outcomes, and/or measures.¹²

Material and methods

This scoping review was conducted according to the JBI manual for Evidence Synthesis (2020 version)¹² and is reported according to the Preferred Reporting Items for Systematic Review and Meta Analysis Protocols Extension for Scoping Reviews (PRISMA-ScR) checklist.¹³ There is no a priori protocol for this review.

Eligibility criteria

To address the research aim of collating examples of evaluating genetics/genomics education interventions across of a range of health professions, countries, types of education, and stages of evaluation, eligible articles had to meet the following criteria: (1) journal articles or reviews/meta-analyses; (2) published in English between January 1, 2000 and June 1, 2023; (3) targeted a non-genetic specialist health workforce audience with education intervention(s); and (4) described evaluation outcomes related to a specific genetics or genomics education intervention.

Information sources

Articles were identified by searching MEDLINE and PubMed using search terms outlined below. The most recent search was conducted June 16, 2023.

In addition to the search described below, purposive sampling was also employed.¹⁴ We leveraged a previous literature review addressing the role of education in preparing medical specialists, conducted by our group.¹⁵ The authors (experts in genetics/genomics education and evaluation and/or program evaluation) recommended inclusion of additional articles, as did international experts convened to develop the RISE2 Genomics reporting standards.¹⁰ We also performed forward and backward citations of review articles identified.

Search

The search terms (see Note S1) were divided into components—three for needs assessments and two for genomics education evaluations—with the aim to capture the key inclusion criteria while keeping the scope sufficiently broad for subsequent manual review. For the needs assessment, these comprised genetics/genomics [genet^∗ OR genom^∗] AND education intervention [educat^∗ OR train^∗ OR learn^∗ OR instruct^∗ OR “professional development” OR CPD OR “continuing medical education” OR CME OR workshop] AND terminology broadly reflecting needs, developed by reviewing terms present in preliminary review [need^∗ OR prepar^∗ OR ready OR readiness OR know^∗ OR attitude^∗ OR skill^∗]. For the evaluation of an education intervention, search terms comprised genetics/genomics [genet^∗ OR genom^∗] AND education intervention [educat^∗ OR train^∗ OR learn^∗ OR instruct^∗ OR “professional development” OR CPD OR “continuing medical education” OR CME OR workshop]. We did not use search terms to filter by target audience because the nomenclature of the clinical workforce varies. We intentionally included review articles to examine key words to inform further searches and also to perform ancestry searches. Because this was an exploratory scoping review, we also did not include any exclusion criteria in our search terms (AND NOT …).

Selection of sources of evidence

Search results were imported into Covidence, a web-based management tool, to manage the screening and selection process. Articles were screened to only include those that focused on the “clinical" workforce, defined as medical, nursing/midwifery, pharmacy, dentistry, or other allied health (e.g., occupational health, nutrition, physiotherapy, speech therapy, social work, public health, community health work). Commentaries/editorial and gray literature were excluded. Other exclusion criteria included articles that described needs assessments or education interventions that did not focus on genetics or genomics; were targeted to predominantly non-clinical audiences or to professions who are not patient facing (e.g., laboratory professionals, pathologists); or focused solely on genetic specialists (e.g., genetic counselors, clinical geneticists). Education interventions focused on students at the pre-clinical stage (i.e., university degrees) were excluded because their focus is primarily knowledge gain rather than professional practice. Articles were also excluded from the evaluation scoping review if the authors did not provide evaluation data (e.g., described a planned or ongoing study without providing actual data) or if education was included in a larger program but the contribution or evaluation of the education component was not described in detail. Broader inclusion criteria were applied to articles on needs assessments, recognizing that the broader aim of these articles is not only to provide examples of relevant measures but also to inform design of education interventions.

The literature search was comprehensive and integrative,¹⁶ rather than systematic, to collate examples of evaluating genetics/genomics education interventions across of a range of health professions, countries, types of education, and stages of evaluation. At monthly review meetings throughout screening, the Project Working Group identified additional relevant articles that had not met the search criteria (e.g., used different title/key words) but were deemed relevant to one or more components of the evaluation framework.¹⁴

Data charting process

Article screening and review was performed using Covidence software. Retrieved articles were divided between M.J., A.N., and B.T., with each article reviewed by at least two authors. Articles that met inclusion criteria were downloaded for further review. Extracted data included author(s), publication year, title, abstract, journal, volume, issue, pages, location(s), intervention mode/format, profession(s) or population investigated, study design, data collection methods, topic of interest related to genomics, and evaluation stage (pre-implementation, implementation, and/or immediate, intermediate, or long-term outcomes). Study designs were categorized as cross-sectional (single or multiple time points), longitudinal (multiple time points with the same sample), and/or randomized controlled trial. Data collection methods were categorized as qualitative (interviews, focus groups, etc.), quantitative (surveys, clinical audits), consensus, or mixed. Surveys with open-text questions and audits of documents that required review of text were considered mixed methods.

Matrices were created in Excel to manage extracted data and independent author and working group review of data against eligibility criteria.

Critical appraisal of individual sources of evidence

The articles were reviewed for descriptions of needs assessments and/or evaluation of genetics/genomics education interventions. As this was an exploratory scoping review rather than systematic review, we aimed to map the breadth and extent of the literature rather than include all articles that met inclusion criteria.¹² Due to the high number of articles reporting evaluation of genetics/genomics education interventions targeted to physicians, articles that met inclusion criteria but did not contribute additional data (e.g., were not novel in profession, country, study design, and/or evaluation measures) were excluded. Related articles from the same group reporting evaluation of an intervention across multiple stages of the framework were prioritized. Articles that investigated intermediate (change in professional practice or appropriate use of genomic medicine) and/or long-term health outcomes were also prioritized. Articles that had detailed descriptions of measures or data collection tools in supplemental materials were favored for inclusion to maximize access to original source materials for readers. Review articles were critiqued to identify additional articles that met inclusion criteria that had not been identified in the searches; the review articles were then excluded from results synthesis.

Data reduction was used to summarize metrics and evaluation measures from the primary sources into evidence tables.¹⁶ Depending on the type(s) of evaluation reported, each article was defined against one or more components of the evaluation framework.¹¹ Data were then extracted and standardized by coding for setting, study design, data collection methods, and research group. Data were co-coded by M.J., B.T., and A.N. and compared to reach consensus.

Synthesis of results

Data labels were used to summarize and categorize articles according to stages of the evaluation framework, i.e., articles that reported measures for needs assessments, delivery of education, then immediate, intermediate, and/or long-term evaluation of genomics education programs. The framework defines immediate outcomes as meeting program learning objectives, intermediate outcomes as change in individual or workplace genomic competency and appropriate use of genomic medicine, and long-term outcomes as improved health.¹¹ Some articles were assigned to multiple stages. Summary tables were developed to describe the focus of each stage, and all articles were reviewed to extract measures into the relevant tables. This process was iterative: as tables were populated, the descriptions for measures became more detailed and so were regularly reviewed, grouped, and/or reworded by M.J., A.N., and/or B.T. Final table groupings and the selection of articles reported in the body of the results were agreed on by M.J., A.N., and/or B.T., with input from all authors.

Results

Characteristics of included articles

The scoping review identified 4,659 articles (435 needs assessments and 4,175 evaluations of education interventions from database searches; 25 needs and 49 education evaluations from purposive sampling). Of these, 363 met the eligibility criteria and underwent full-text review (Figure 1). After excluding articles that targeted other sectors, were not linked to a specific intervention, or did not contribute additional data, 104 articles remained. Sixty articles (57.7%) reported needs assessments for genomics education of health professionals. Fifty-two articles (50.0%) described evaluation of a genomics education intervention. Eight articles (7.7%) included both needs assessments and education evaluations.

Figure 1.

Results of scoping literature review to audit evaluation questions and measures in genomics education showing the flow of article identification and selection

(A) Search terms provided in Material and methods.

(B) Purposive sampling¹⁴ via previous literature review conducted by our group,¹⁵ recommendations from the Project Working Group, previous interactions with the workshop experts,⁹ and Reporting Item Standards in Education and Evaluation in Genomics Expert Group.¹⁰.

Key characteristics of the 104 articles are summarized in Table 1, including publication year, target audience (sample), location of the education program, and stage of evaluation reported; more detail on the education program and/or evaluation study design for each article is provided in Table S1. There was an increase in articles published in recent years; articles described education interventions targeted primarily to those with, or in training for, medical qualifications. Articles were included from more than 30 countries, with the most common location being the USA (39.4%), followed by Australasia (19.2%) and Europe (17.3%). Of the genomics education interventions, 35 (33.7%) included evaluation measures of implementation, 49 (47.1%) of immediate outcomes, 26 (25.0%) of intermediate outcomes, and 4 (3.8%) of long-term outcomes. Evaluation of external influences was reported in 8 (7.7%) articles. Forty-seven articles (45.0%) reported multiple stages of evaluation (Table S1).

Table 1.

Characteristics of the 104 included articles by publication year, location, sample/target audience for the education program, and stage of the evaluation framework reported

Characteristics of included articles	Number of papers
Year of publication	–
2000–2004	7
2005–2009	17
2010–2014	17
2015–2019	22
2020–2023	41
Location of education programa	–
USA	43
Australasia	24
Europe	22
UK	8
East and NE Asia	7
South and SE Asia	3
Western Asia	8
Africa	3
Canada	3
Multiple	3
Target audiencea	–
Medical specialists/physiciansb	38
Nurses and midwives	26
Family physicians/general practitionersb	18
Primary care practitionersb	17
Allied health practitioners	17
Medical trainees in clinical trainingb	7
Genetic health professionals	4
Non-health professionals	3
Multiple	5
Evaluation stagea	–
Pre-implementation	60
Implementation	35
Immediate outcomes	49
Intermediate outcomes	26
Long-term outcomes	4
External influences	8

^aArticles in these categories total more than 104 as some articles described education interventions provided in multiple locations, and/or to multiple audiences, and/or evaluated at multiple stages.

^bThe terms used to denote health professionals with medical qualifications or in training (i.e., “doctors”) varied by location.

Table S1 includes descriptions of the 52 education interventions and their evaluation, noting that the articles were selected to ensure that a range of programs, goals, and settings were represented. Modes of education delivery included online (e-learning and resource libraries), in-person, or blends of both. Education formats ranged from intensive workplace rotations and counseling sessions with real or standardized patients to online short courses and/or tools. The education interventions were targeted to physicians/medical specialists (including trainees); nurses and midwives; genetic counselors; allied health; dentists; pharmacists; community health workers; and health system managers. The genomics applications included upskilling health professionals to improve delivery of existing genetics/genomics services; supporting genomic testing for rare disease; enabling precision medicine (including pharmacogenomics); mainstreaming genomic medicine; or facilitating prenatal and population screening.

The literature reviewed included reports of many different evaluation study designs, including cross-sectional, longitudinal, and randomized controlled trials. Methods were quantitative, qualitative, or mixed, including audits (e.g., use of online resources, clinical practice, test referrals) and consensus methodologies to develop competencies and learning objectives. Data were obtained through objective and subjective sources. Objective sources included audits of documents, health service data, website data analytics, observing meetings, site visits, and visual artifacts, whereas subjective data sources included interviews, focus groups, and journaling. Surveys could include subjective and/or objective measures. Audits were conducted internally (by study investigators) or externally (by relevant specialists). Evaluating whether genomic testing was ordered appropriately was assessed against professional standards by study investigators or by independent specialists.

Measures across stages of an evaluation framework for genomics education

To assist educators and evaluators, here, we summarize potential evaluation questions, measures, and/or metrics identified in these articles. We illustrate the breadth of approaches evident at each stage of evaluation¹¹: planning (pre-implementation, Box 1); development and delivery (implementation, Box 2); and impact (immediate, intermediate, and long-term outcomes, Boxes 3, 4, and 5) as well as external influences (Box 5). Referenced versions of these boxes are provided in Table S2.

Box 1. Example concepts, questions, and measures for evaluating the planning (pre-implementation) of genomics education interventions through needs assessments. A referenced version is provided in Table S2.

Questions include how do I know there is a need? In which profession or region? In which areas of competence (knowledge, skills, and/or attitude) or confidence?

Relevant experience

• •Clinical practice characteristics relevant to genetics, including patient types, numbers, etc.
• •Experience with testing, referring to genetics services; ordering or interpreting tests
• •Genetic counseling aspects
• •Direct-to-consumer testing, e.g., patients bringing results for discussion
• •Adding/retrieving information from electronic medical records
• •Time spent on tasks related to testing
• •Barriers or enablers to implementing genomic testing in clinical practice

Knowledge

• •Sources of information about genetics/genomics, including professional competencies or guidelines
• •Fundamental concepts of genetics/genomics
• •Common or specific genetic conditions
• •Prenatal screening or preimplantation genetic diagnosis (PGD)
• •Pharmacogenomics
• •Different tests
• •How to access a test, including awareness of genetics services and test availability/funding
• •Genetic consultations and counseling: risk assessment; indications for genomic testing or referral criteria
• •Ethical, legal, and (psycho)social implications of testing

Skills

• •Elicit genetic information as part of a medical/family history and construct a pedigree
• •Risk assessment
• •Identify patients suspected to have a genetic condition who may benefit from testing
• •Develop a testing strategy, including evaluating the clinical usefulness of a test
• •Interpret test results
• •Apply test results to patient management

Attitudes

• •Perceived relevance/clinical utility of testing or perceived acceptability to the target population
• •Delivery of genomic testing: role of different clinicians, preferred service delivery model, perceived barriers
• •Ethical, legal, (psycho)social, and/or economic implications, including regulatory or insurance issues

Confidence, comfort, or self-efficacy

• •Elicit a family history
• •Explain concepts to patients, including those with low health literacy
• •Facilitate informed consent
• •Refer or order the appropriate test
• •Interpret test results and apply to patient management
• •Provide results and follow-up genetic counseling
• •Feel prepared to practice genomic medicine

Educational experiences and preferences

• •Previous genetics/genomics education experience, including educating others
• •General or individual desire for genetics/genomics education or training
• •Educational topic areas of interest/need
• •Preferred education format or provider

Box 2. Example concepts, questions, and measures for evaluating the development and delivery (implementation) of genomics education interventions. A referenced version is provided in Table S2.

Implementation

Questions include did the intervention function as intended?

Reach

• •In-person: attendance numbers
• •Online: frequency or total visits over a given period of time; revisitation; time spent; completion rates
• •Participant characteristics, e.g., profession(s), career stage, etc.

Engagement

• •Awareness of the intervention
• •Motivation to attend/complete
• •Reasons for lower or non-attendance/completion

Program improvement

Questions include how can the program be improved?

Target audience needs

• •Clear educational aim, goal, or learning objectives. If so, relevance to current training level and/or alignment of content with aim/goal/learning objectives
• •Meeting participants’ educational needs, either local/cultural or individual
• •Need for further educational opportunities of this kind

Educational strategy

• •Pedagogical approach, e.g., problem-based learning, workplace learning
• •Mode of delivery, e.g., in-person versus online or lectures versus facilitated small-group case discussions
• •Setting, e.g., single or mixed disciplines

Design

• •Learning design and format (in-person or online)
• •Realism and similarity to routine practice for simulated environments
• •Structure
• •Duration, including adequate time for reading/studying during the intervention
• •Assessment, e.g., informal versus formal

Content

• •Quality
• •Clarity, complexity, or level of detail
• •Relevance, appropriateness, or helpfulness
• •Variety or novelty, including desired content not currently included
• •Currency

Overall

• •Participant experience
• •Satisfaction
• •Value or usefulness to professional practice
• •Cost-effectiveness or willingness to pay
• •Whether would and/or did recommend to others
• •Whether the intervention should be repeated

Box 3. Example outcomes, evaluation questions, and measures for evaluating the immediate outcomes of genomics education interventions. A referenced version is provided in Table S2.

Questions include were learning objectives or participants’ educational needs met? Were there changes to genomic knowledge, skills, attitudes, or confidence as a result of the intervention (compared with a control group and/or retained over time)?

Knowledge

• •Sources of information about genetics/genomics, including any professional competencies
• •Fundamental concepts of genetics/genomics
• •Genetic conditions, including in context of pregnancy, adult-onset, or surveillance/management
• •Different tests, including possibilities and limitations
• •Applications of testing, such as pharmacogenomics and polygenic risk scores
• •Multidisciplinary delivery of testing, including awareness of role of genetic counselors and others
• •How to access a test, including awareness of genetics services and test availability/funding
• •Genetic consultations and counseling: importance of a medical/family history, risk assessment, indications for testing or referring, explaining genetic/genomic concepts, patient consent
• •Ethical, legal, and (psycho)social implications of testing, including consumer health literacy; regulatory or insurance requirements or misconceptions

Skills

• •Obtain up-to-date and correct knowledge through appropriate resources
• •Clinical skills relating to specific genetic conditions, including in pregnancy, pediatrics, or adult conditions
• •Length of consultation
• •Elicit genetic information as part of a medical/family history
• •Assess risk and manage high-risk populations
• •Identify patients suspected of having a genetic condition who may benefit from genetic counseling or a test
• •Make differential diagnoses
• •Develop a testing strategy, including evaluating clinical utility
• •Make/facilitate referrals to genetics services or order tests
• •Provide clinical or genetic counseling insight to cases presented in a multidisciplinary team
• •Interpret test results and apply to patient management
• •Genetic counseling: communicate risk; discuss testing or screening options; discuss ethical, legal, and (psycho)social implications; facilitate informed consent; protect patient confidentiality; disclose test results, including prevention/surveillance strategies
• •Provide patient-centered communication

Attitudes and beliefs

Also includes awareness of others’ attitudes

• •Wanting to learn genetics/genomics
• •Perceived clinical utility of testing and feasibility or impact of incorporating testing in practice
• •Delivery of testing, including multidisciplinary delivery, roles of different professions, importance of family history, adherence to clinical recommendations, importance of person-centered counseling, ethics and values
• •Wanting to become proficient in applying genomic testing to patient care
• •Importance of ethical, legal, and (psycho)social implications of testing, including consumer health literacy, regulatory, guideline or insurance issues, and equitable reference data
• •Intent to change practice based on the intervention, including continuing to use the intervention, teaching practice, or creating opportunities for improvement

Confidence

Change in (self-)confidence, comfort, perceived preparedness, professional self-efficacy, or beliefs about capabilities.

Box 4. Example outcomes, evaluation questions, and measures for evaluating the intermediate outcomes of genomics education interventions. A referenced version is provided in Table S2.

Building individual genomic competence

(Measurable activities that reveal benefit to the individual only.)

Questions include did participants undertake any new education activities as a result of the intervention? Or (if applicable) were the genomics education resources used?

Activities that reflect change in genetic/genomic competency to a different extent than before the intervention.

• •Information seeking and sharing: more genomics education, reading journal articles, advice from colleagues
• •Use of online resources for patient management
• •Use of education resources: circumstances; frequency; specific sections/modules/resources; barriers to use

Appropriate individual use of genomic medicine

Questions include were there changes to professional practice by applying new knowledge and/or skills (compared with a control group and/or retained over time)?

• •Use of intervention resource/s to provide care for patients in practice
• •Change in practice, compared with a control group, and/or retained over time
• •More attention on genetics/genomics in differential diagnoses
• •Frequency of managing/treating patients with genetic conditions
• •Genetic consultations and counseling (frequency, content, appropriateness, length): taking/using family history, discussing relevant topics, disclosing results, paying attention to patient perspective
• •Engagement in research
• •Appropriate referral for genomic tests, including use of genetics services for information, consulting regarding a patient, referring or facilitating referrals (timing, frequency, appropriateness, equity)
• •Appropriate ordering of genomic tests (“the right patient is offered the right test at the right time”)
• •Proportion of patients offered the correct screening test at the appropriate time interval
• •Appropriate application of test results in patient care, including conveying information to patients

Building genomic competence and fostering appropriate use in others

(Measurable activities by the individual that reveal benefit to others, including interactions with the wider “professional/healthcare community” or public.)

Questions include did participants develop and/or deliver any genomics education for others? Were there changes to others’ genomics practice as a result of the intervention?

• •Change in practice that impacted others’ genetic/genomic competency
• •Impact on professional networks (new and/or strengthened)
• •Professional development for colleagues, including being a “genomic champion”; disseminating information or resources; being a resource for health professionals or patients; teaching; updating medical school/professional organization or other curricula; or contributing to knowledge base, e.g., peer-reviewed publications
• •Public education
• •Fostering multidisciplinary network between specialty and genetics
• •Helping to coordinate genomic testing
• •Organization able to support future internal training/continuing professional development (CPD)

Box 5. Example outcomes, evaluation questions, and measures for evaluating the long-term outcomes of genomics education interventions and the external influences on outcomes of genomics education interventions. A referenced version is provided in Table S2.

Improved health outcomes

(Measurable activities that reveal health benefit to patients or their families.)

Questions include were there changes in patient behavior or health outcomes after the education intervention?

Change in the activities of patients or their families following changes in practice of health professionals who completed an educational intervention, compared with a control group

• •Health professional self-reported patient visits to genetics services for follow up
• •Patient self-reported: knowledge about condition, worry, risk perception, risk screening behavior, lifestyle changes, consulting health providers, discussing family health history with family members
• •Audit of appropriateness of patient self-reported screening

External influences

Questions include what other factors may have influenced the outcomes (compared with a control group)?

Immediate

• •Knowledge of condition
• •Confidence in performing risk assessment

Intermediate

• •Self-reported perceptions of work environment, including organizational attitudes to incorporating genetic/genomic testing, working across disciplines, and barriers and facilitators
• •Self-reported barriers and facilitators to personal practice, including perceived acceptability, benefits and role, social influences, implementing family health history taking and workload

Long-term

• •Demographics of patients (age, gender, risk category)

Planning (pre-implementation)

Evaluation questions and measures to determine genomics educational needs can inform planning and are provided in Box 1. Potential evaluation questions at this stage may assess the role of genetics/genomics in a particular discipline to determine a need for genomics education and inform educational goals that align with the intended service delivery model for genomics. Education providers may evaluate individual preparedness or self-efficacy to use genomics in practice against required core knowledge, skills, and attitudes. For example, Rasouly et al. used previously validated questions to assess US nephrologists’ awareness, knowledge, and recognition of genomic terminology.¹⁷ In contrast, Van der Giessen et al. used novel multiple-choice questions to assess health literacy knowledge of breast surgeons and specialized nurses to support effective genomics support and communication in the Netherlands.¹⁸

Education providers may also evaluate the target audience’s relevant prior knowledge of genetics/genomics, relevant clinical skills such as taking a family history, clinical risk management, or discussing limitations and/or ethical or psychosocial aspects of genomic medicine. For example, Peter et al. used validated competency-informed case-based questions to assess the genomics education needs, confidence, and procedural knowledge of US audiologists and speech and language pathologists.¹⁹ Similarly, Barr and colleagues sought the views of UK nurses and midwives in primary care on competencies in genomics but asked what they already knew, did not need to know, or needed to learn.²⁰

Needs assessment methodology can include literature reviews or other desktop research/online searches, qualitative or quantitative approaches, and consensus methods (e.g., expert workshops or Delphi process and/or mixed methods).

During planning, a range of genomic stakeholder perspectives can be considered beyond the target audience. These may include those involved in delivering genetics services^21,22 or the genomics education intervention,²³ patients,¹⁸ or representative consumer groups,²³ education/evaluation experts,^23,24 and/or funding agencies.²³ For example, Houwink and colleagues conducted a Delphi process with general practitioners (GPs, family physicians), GP educators, clinical geneticists, a genetic counselor, and patient advocacy group representatives to prioritize topics for GP genetics education.²⁵

Implementation

Evaluation at the implementation (development and delivery stages) of the program is more operational and can inform program improvement or help interpret evaluation findings. Potential questions and measures to evaluate the implementation of a genomics education intervention are provided in Box 2. Evaluators may assess whether the delivery of the intervention was as intended through metrics on attendance at an event or use of a resource. Data on the characteristics of evaluation participants (such as profession, specialty, career stage, location, etc.) could be used to identify potential barriers or enablers to engagement with the intervention and intended outcomes (“external influences” in the framework). For example, when evaluating an education program that included decision-support resources for inherited breast cancer risk, while Wilson et al. did not find evidence of a difference between their intervention and control groups of Scottish GPs, implementation data showed that many GPs in the intervention arm were not aware of, nor had accessed, the resources, which limited the interpretation of this finding.²⁶

Educators may also examine whether content aligned with target audience needs and re-evaluate strategies if these did not align. For example, Nguyen and colleagues evaluated a US-based pediatric subspecialty rotation in medical genetics for quality, effectiveness, and appropriateness for the target audience.²⁷ Similarly, when evaluating a pilot genetic counseling program for Ethiopian nurses, Quinonez et al. found that “Western” teaching materials were not suitable for local medical and cultural practices.²⁸

During implementation, participants may be given the opportunity to suggest improvements to the intervention and provide feedback on value or applicability to their professional role. Implementation evaluation was used to refine the pilot Medicine’s Future: Genomics for Practicing Doctors course for community-based physicians.²⁹ Reed et al. adjusted key instructional messages in their program for American physicians, simplified content on some topics, included more opportunities to apply and practice new skills, and provided targeted facilitator training.²⁹ Bokkers and colleagues also developed a short multiple-choice survey for health professionals who did not complete the education intervention to understand their motivations and feedback.³⁰

Immediate outcomes

Evaluating immediate outcomes of genomics education focuses on the impact of the education on an individual participant, such as changes in knowledge, skills, attitude or confidence, and intention to apply in practice. We found that immediate outcomes were defined in different ways across the literature, using variable terminology and measures (Box 3). Immediate outcome measures may be derived from professional core competencies, such as those published by the National Coalition for Health Professional Education in Genetics (https://www.jax.org/education-and-learning/clinical-and-continuing-education/ccep-non-cancer-resources/core-competencies-for-health-care-professionals), the American Nurses Association,³¹ or European Society of Human Genetics.³²

“Knowledge” spans awareness, understanding of concepts, or procedural knowledge that could be applied in the clinic. For example, Calzone and colleagues found that nurses’ awareness of genomics in clinical practice increased in a longitudinal study of a genomics education program run through North American hospitals,³³ while Murakami and colleagues investigated perceived change in genetic knowledge after an introductory-level, case-based workshop for nurses in Japan.³⁴ Others assessed knowledge pre- and post-education: Maxwell et al., who used items from validated genetic and health literacy assessments to evaluate genetic knowledge gains in active-duty Air Force primary care practitioners,³⁵ and Lee et al. assessed the knowledge of Korean medical and surgical oncologists.³⁶ Genomic procedural knowledge or “skills”³⁷ could be self-reported by participants, assessed through case-based scenarios, or observed during role-plays and educational activities. Swank and colleagues assessed change in nurses’ self-reported ability to identify genetic risk in candidate egg donors to evaluate a self-directed learning module for American nurses working in reproductive health centers,³⁸ while Maher et al. used case-based questions and a mock genomic test report to assess physicians’ understanding of, and confidence with, genomics.³⁹

In a longitudinal evaluation of an introductory genetics workshop for Canadian primary care physicians, Carroll et al. examined change in attitudes.⁴⁰ They asked participants to reflect on the importance of maintaining current knowledge about genetic tests, clinical utility of tests, making time to incorporate testing into practice, and the difficulty of explaining the concept of risk to patients.

Confidence was often assessed as a health professional’s self-belief in their capability to undertake a specific task. For example, Yu and colleagues assessed primary care physicians’ confidence in managing patients with genetic-related issues,⁴¹ while Lee et al. focused on specific areas such as confidence to convey information about non-BRCA genes in genetic counseling or determining when confirmatory testing is appropriate.⁴²

Intermediate outcomes

Intermediate outcomes are not defined by time but rather if and how new knowledge, skills, and insights are applied in practice at any time following education. Potential evaluation measures for intermediate outcomes are also provided in Box 4, characterized as genomic competency leading to appropriate use of genomic medicine, at either the individual or group level. Measures of genomic competence may be either self-reported or via audit. For example, to evaluate whether their education program reduced disparities in referral patterns for breast cancer screening, Van der Giessen et al. audited referral patterns of Dutch oncologists and nurses before and after their health literacy education program.⁴³

As part of a mixed methods, longitudinal evaluation of the Gen-Equip project resources (www.primarycaregenetics.org), Jackson et al. gathered data on use, re-use, and sharing of online resources as well as change in practice for both the individual and the professional community 2–3 months after accessing the resources.⁴⁴ Huq et al. evaluated how frequently junior hospital doctors shared new knowledge with colleagues after completing a 3-month rotation in genomic medicine in addition to measuring change in individual genomic practice.⁴⁵

Intermediate outcomes may be self-reported or evaluated by other stakeholders. After a seminar, Pestka et al. asked Singaporean nurses to describe a patient scenario involving genomics in their practice and reflect on their capability.⁴⁶ To assess change in practice and perceived value of a cancer genetics training course for nurses, Gaff and colleagues sent surveys to the participants’ supervisors 6 months post-training.⁴⁷

Long-term outcomes

The underpinning evaluation framework defines long-term outcomes in terms of improved health,¹¹ be it through direct patient participation in care of those receiving the genomics education intervention or indirectly through participation in screening programs (for example). As the framework articulates, long-term outcomes do not necessarily relate to the time point at which evaluation is performed.

There is limited literature reporting long-term outcomes of genetics/genomics education interventions in terms of improved health. Some authors note a need for, or intention to, measure long-term outcomes but either did not report them,^18,33 were unable to achieve them,⁹ or the data were unable to demonstrate that the education intervention contributed to the outcomes measured.^48,49

Measures from the few articles that did evaluate improved health as a result of a genetics/genomics education intervention are summarized in Box 5. Carroll and colleagues reported patient benefits from an education intervention on cancer risk.⁵⁰ In addition to collecting data on self-reported patient cancer worry 5 and 10 years after a point-of-care education intervention for family physicians, the evaluation also monitored actual patient risk screening behavior by auditing the proportion of patients who had appropriate screening at the appropriate time interval. This study bridged the gap between intermediate outcomes (appropriate use of genomic medicine) and long-term outcomes (defined in Caroll et al.’s study in terms of patient behavior change as a precursor to improved health, i.e., the proportion of patients who actually performed the recommended screening). In another study, Chen and colleagues evaluated the long-term health benefits of a training program for community health workers for both patients and their families by measuring clinician-reported patient compliance with clinical/lifestyle recommendations and whether patients had discussed family health history with family members.⁵¹

Stakeholders

Stakeholders were defined in different ways across the articles reviewed. Broad stakeholder groups were consulted during planning (pre-implementation), including health system influencers, genetic specialists, representatives of the target audience for the genomics education intervention, and/or patient/family advocates and consumers. For implementation and immediate outcomes, the participants of the education interventions were the primary stakeholders. However, if the intervention was evaluated for intermediate or long-term outcomes, stakeholders included supervisors; colleagues who received advice, referrals, or education, or became involved in professional networks; as well as patients and their families.

External influences

While external factors cannot be controlled, they can be recognized in evaluations, as shown in Box 5. Measures include assessments of work and social environments to identify barriers and facilitators to change in confidence (immediate), clinical/teaching practice (intermediate), or long-term outcomes. For example, the predictors of patient screening behavior identified by Carroll and colleagues— patient gender and cancer risk level—are outside the control of the education intervention but nonetheless affected the impact of their colorectal risk assessment education program for family physicians.⁵⁰ When evaluating a national genetic testing training program in oncology, Australian health professionals identified the lack of infrastructure, funding, local policy, and human resources to support the service as barriers to intermediate and long-term outcomes.⁵²

Definitions and terminology

Throughout the literature, similar concepts and terms were used interchangeably. For example, knowledge was measured as awareness, perceived knowledge, understanding, or skills/procedural knowledge. Skills were measured as ability, competence, capability, or application. Confidence, comfort, preparedness, self-efficacy, and beliefs about competence or capability were also measured. Measures of attitudes and beliefs included desire, importance, perception, feasibility, preference, intent, willingness, perceived relevance, and interest in learning more.

Use of genomics included a broad range of behaviors. At an individual level, this included test referral or ordering; provision of patient care; alignment with guidelines; practice change; and engagement with research. “Appropriateness” or “correctness” of use was variously defined as individual alignment with the education intervention or relevant patient characteristics (e.g., family history). It was either assessed by study investigators using existing professional guidelines or by independent geneticists.

Discussion

This exploratory scoping review synthesizes data from more than 100 articles over two decades to support genomics stakeholders and educators who are considering how to evaluate an education program or resource. The articles included were selected to include studies from different geographical areas, with different approaches and foci. Audiences for the genomics education programs and resources evaluated ranged from non-genetics-trained medical specialists, who may be able to order genomic tests, to allied health professionals who are integral to the support and care of people undergoing genomic testing.

We categorized evaluations and measures using an evaluation framework adapted for genomics education.¹¹ This process revealed that few genomics education programs have reported long-term health-related outcomes, with the vast majority reporting formative/implementation data or immediate outcomes of changes in knowledge, skills, attitudes, and confidence. Search and analysis processes also identified inconsistencies in education and evaluation terminology that are potentially confusing and may introduce additional barriers to effective education and evaluation in the future.

Less than 5% of reviewed articles reported data on long-term outcomes. This likely reflects the challenges of evaluating the outcomes. Some articles we retrieved reported long-term outcome measures, such as psychological or behavioral impact on patients or health care utilization,^48,53 but were excluded from our review as the authors could not, or did not, attribute long-term outcomes to the education intervention.

Our detailed examination of the literature also confirmed the need for a consistent language to describe concepts that inform metrics: different terms are used to measure similar concepts, and these are rarely defined in articles. We summarize these varying uses here to encourage researchers to purposefully identify appropriate outcomes and measures that reflect the program goal⁵⁴ and align with underpinning theories. Inconsistent use of terms also made it challenging to review articles in this study and extract evaluation measures, highlighting the need for the use of standards for reporting of genomics education interventions and their evaluation.¹⁰

A wealth of genetics/genomics education was evident in the evaluations reviewed. The articles spanned dozens of topics and educational approaches. Genetics/genomics education is typically characterized as formal programs or resources across the emerging (university/college) or current workforces (continuing professional development, continuous medical education). We included articles that defined experiential or workplace learning as an education intervention,^{27,33,45,55,56,57} as it is highly relevant to medical education⁵⁸ and valued by those learning genomics.⁵⁹ It is worth noting that experiential learning is variably described in the literature.

Strengths and limitations

We purposively searched the literature to review articles representing a breadth of countries, study designs, genomics applications, and stages of evaluation. The articles that met the criteria may therefore not be representative of all genomics education literature and instead over-represent countries with more developed national implementation strategies for genomic medicine and/or funding for evaluation of genomics education programs. Our review is practice focused, which is an area of education perceived as being inadequate globally.^1,60 While the accompanying evaluation framework can be applied to evaluate medical school genetics/genomics education,¹¹ we centered this review on continuing professional education. Genetics/genomics content is minimal in medical school curricula⁶⁰ and sometimes spread across multiple subjects or years of study as part of problem-based learning, so it is difficult to evaluate separately. This may warrant a separate review for those interested in evaluation of genomics in university medical education.

This exploratory scoping review does not aim to capture all potential evaluation approaches, measures, or settings relevant to genetics/genomics education. We did not aim to conduct a systematic review or comment on the quality of the evaluation studies. Our review only included peer-reviewed articles in English and may therefore have missed novel approaches reported in other languages or the gray literature. As the review included only articles presenting the results of evaluations published from 2000 to 2023, it will not capture emergent practice documented in published protocols where the investigators have not yet reported their findings. This review is intended to prompt genomics educators to navigate inconsistencies in the literature, prompt those relatively inexperienced in evaluation to consider their approach, and provide a starting point for more detailed development of an evaluation plan. We encourage educators and evaluators to conduct specific searches for recent literature relevant to their context as they develop their evaluation study. We have included insights and recommendations on search terms in Note S1 for those wishing to undertake their own literature analysis.

We only included articles that described education interventions separate to usual professional practice to highlight a clear contribution of the education to outcomes measured. However, interventions that are not sustainably championed or embedded in professional practice are more vulnerable to being decommissioned.⁶¹ An education intervention may be highly effective, but the magnitude of its impact in health care will be limited if delivery is restricted by time-limited research funding.

Conclusion

We have previously advocated for an evidence base to enable comparison of approaches to genomics education across different contexts.¹⁰ To this end, we worked with experts to design a suite of tools^9,10,11 that aim to make theory- and evidence-informed education and evaluation practice more accessible to individual educators/evaluators.

The scoping review we present here is an important part of this toolkit. We hope it will assist genomics educators to introduce or strengthen evaluation of their programs, identifying designs and measures best suited to determining the real-world impact of their program with the resources available to them. Lowering barriers and enabling evaluation of education programs should support more to contribute meaningfully to a growing evidence base. This would also assist emerging genomics implementation initiatives to establish effective education programs that positively impact genomic practice. They are also best placed to address the “wicked problem” of evaluating the contribution of health professionals’ education to improved health outcomes.

Data and code availability

The published article and supplemental materials include all datasets generated or analyzed during this study.