Beyond Multiple Choice: Clinical Simulation as a Rigorous and Inclusive Method for Assessing Genetic Counseling Competencies

Abstract

Educational use of clinical simulation is a way for students to immerse themselves within a realistic yet safe and structured environment as they practice clinical skills. It is widely used in healthcare training and evaluation, and there are best practices for design, implementation, debriefing, and assessment. An increasing number of genetic counseling graduate programs use simulation in various ways, ranging from role-plays to working with professional simulated/standardized patient (SP) actors. At this time, there is very little consistency across programs, research on the approaches, and standards by which simulation is incorporated into training. Simulation is an understudied but promising approach for genetic counselor (GC) education and assessment. After graduation, GCs demonstrate their competence as entry level providers through American Board of Genetic Counseling (ABGC) multiple choice examination (MCE), along with their participatory clinical encounters from graduate training. Data from genetic counseling and other professions highlight the limitations and biases of MCEs, suggesting they not only fail to accurately capture competency, but also that they disadvantage underrepresented individuals from entering the field. In addition, MCEs are limited as a tool for assessing nuanced counseling and communication skills, as compared to more quantitative scientific knowledge. We propose that innovative, evidence-based approaches such as simulation have the potential to not only enhance learning, but also to allow GCs to better demonstrate competency during training and in relation to the board examination. Collaborative approaches, research, and funding are needed to further explore the viability of routinely incorporating simulation into GC training and assessment.

Introduction

Genetic counseling graduate training programs facilitate the development of students’ skills to ensure they enter the field as competent providers, defined by the practice-based competencies (PBCs) from the Accreditation Council for Genetic Counseling (ACGC). To become board-certified, and in some states licensed, individuals must graduate from an ACGC accredited program, including completion of a minimum number of participatory cases (clinical encounters). After graduation, genetic counselors (GCs) must achieve a passing score on the 200-item multiple choice American Board of Genetic Counseling (ABGC) certification examination. The current exam has limitations, including its indirect measurement of clinical competence and concerns around racial bias (Myers, 2021). We argue that both GC training and formal assessment can be enhanced through research and development of simulation-based education and assessment.

In many fields, national multiple-choice examinations (MCEs) are administered with the goal of ensuring patients receive care only from competent providers; yet standardized tests come with an array of limitations, including racial and gender bias, financial and logistical barriers, and socioeconomic bias due to disparities in access to standardized test-taking preparation (Archer, 2016; Devito, 2022; Dewhurst, 2007; Rubright, 2019; Sharpless, 2019). Further, the relationship between exam performance and patient outcomes is unclear (Archer, 2016; Kaplonyi, 2017). Although not yet well studied, the pattern within genetic counseling appears to be similar. Recent data suggest that race, gender, performance on the GRE, and undergraduate GPA are associated with examination scores, indicating that the exam is not exclusively measuring genetic counseling competence and could, in fact, be discriminating against underrepresented individuals in the field. This finding is in direct conflict with the important priority of increasing the number of GCs with diverse backgrounds (McGinniss, 2022; Myers, 2021). While providing more resources, fostering belonging and cultural humility, lowering hostility, and related strategies employed by GC programs should remain a priority, changing the certification process itself has the potential to improve equity and encourage educational practices that support practice-based competence.

We propose that it is time for our field to explore a movement away from the exclusive MCE format in favor of incorporating alternative methods of assessing and demonstrating competence in new graduates. While the assessment of knowledge is necessary, it is not sufficient and should be combined with other approaches that allow critical thinking and counseling skills to be applied in varied ways and assessed by people with culturally diverse perspectives (Kaslow, 2009; Nelson, 2007). A shift in our certification approach would also be in line with the ABGC Continuing Competence Committee’s exploration of alternatives to current recertification practices and would lend itself to a more inclusive definition of competence (Epstein & Hundert, 2002). Examples of alternative approaches include but are not limited to script concordance testing (Stange, 2022), portfolios, situational judgment tests, written essay/short answers, and simulated patient encounters. In this paper, we explore simulation as an innovative means of demonstrating competence specifically in the counseling aspects of genetic counseling and propose it as an evidence-based approach that warrants further study and consideration.

Simulation: background and approaches

Healthcare simulation is “a technique that allows persons to experience a representation of a real event for the purpose of practice, learning, evaluation, or to gain understanding of systems or human actions” (Lioce, 2020). It is usually paired with debriefing, a reflective conversation, to maximize efficacy by promoting learning for the future. There are various guidelines and standards of best practice for the development and use of simulation and debriefing (Lewis, 2017), some of which are broad and others which pertain to specific fields of practice. Simulation is used widely in the training and maintenance of certification for a variety of healthcare professions such as physicians/residents (Motola, 2013), nurses (MacLean, 2017), social workers (Asakura, 2021; Logie, 2013), occupational and physical therapists (Bennett, 2017; Mori, 2015), audiologists (Alanzi, 2017), and psychologists/psychiatrists (Kühne, 2018). The Objective Structured Clinical Examinations (OSCEs) are used in various fields to evaluate trainees’ specific skills at multiple stations in which trained actors portray patients for brief encounters (Khan, 2013). Additionally, simulated encounters can serve as a clinical rotation placement, such as in nursing where simulated encounters in one prospective randomized-control study replaced up to 50% of clinical encounters without negatively impacting learning outcomes (Hayden, 2014).

In training environments, simulation can be used as both a learning and evaluation tool – within the evaluation category, there can be formative (evaluating how a student is doing during a course or rotation) and summative assessment (evaluating how much a student can apply acquired knowledge and skills). Formative assessments are designed to be lower stakes with a focus on growth and skill building. They can foster an environment which encourages students to take risks in the development of their counseling skills. Reflections of, peer, faculty, and self is often the basis for evaluation rather than a scored grade. Summative assessments (for example, OSCEs in which a student is graded on their performance) are higher stakes with trainees demonstrating specific skills to be evaluated, often with standardized scoring (Blamoun, 2021). There is evidence that a simulation program incorporating both formative assessment (activities to build learning) and summative assessment (activities to assess learning) maximizes the student experience and education. An important benefit of well-structured simulation activities is the facilitated debriefing following the activity. The reflection and discussion that occurs during debriefing is key for effective student learning and advancement of clinical and non-clinical skills (Palaganas, 2016; Sawyer, 2016).

In GC training, simulation beyond peer and faculty role-plays is an understudied but promising approach to developing clinical skills. Simulation is also increasingly being adopted into genetic counseling master’s programs, possibly aided by the ACGC decision in 2019 to allow 10 simulated encounters meeting their criteria to count as participatory cases towards graduation. In a recent survey, approximately 68% (17/25 programs responding to the survey – or 40% for 17/43 total programs) reported working with SPs (Kessler, 2021). Forty-eight percent of programs responding (or 28% of all programs) work with SPs in a manner consistent with ACGC criteria for SP engagement (ACGC Standards of Accreditation, 2019). Another study involved three medical genetics resident physicians and five GC students conveying an abnormal amniocentesis result in a simulated prenatal encounter utilizing standardized patients. Participants felt it was a useful and more realistic experience than a role-play, suggesting high acceptability and face validity with learners (Holt, 2013). Simulation has also been a mechanism for practicing the delivery of difficult news in genetic counseling, with one study finding approximately 50% of 28 genetic counseling programs use OSCEs in that context (Andoni, 2018). Despite widespread involvement of SPs, there is no central or defined evaluative criteria that is used across training programs. Moreover, only a small minority of programs are currently working with SPs as part of a summative evaluation of clinical skills (Kessler, 2021).

While simulated encounters allow learners to demonstrate clinical judgment skills, there are limitations such as assessor biases and validity/reliability of assessment tools (Oermann, 2016). Some strategies to mitigate these limitations include having more than one session and/or more than one assessor per session, using standardized scoring (Brannick, 2011), and possibly having the SP actors themselves rate the sessions (Gude, 2015). No assessment measure or certification tool is bias-free, and there is room for error when humans administer them. However, given that standardized MCEs are rooted in biases and systemic racism, there is a need to examine alternatives. Well-designed and evidence-based alternate certification approaches would theoretically allow new GCs to demonstrate competence using a range of counseling approaches, evaluated by multiple people on several occasions, thus creating a more inclusive and equitable assessment.

Assessing counseling skills

Our counseling and communication skills in genetics/genomics set us apart from other non-specialist providers and educational tools such as chatbots (Abacan, 2019; Frievogal, 2019; Riconda, 2018). Possible futures where reimbursement is tied to quality of care or where triage models help identify individuals most in need of individual care from a GC will rely on the honing and highlighting of our counseling strengths for complex cases (Rashkin, 2019). Thus, it is also critical that evidence-based assessments of psychological counseling and communication skills are developed, with simulation as a key component to fill the gap. Incorporating standardized simulations for summative evaluation in GC programs could allow for assessment consistency in competency-based student evaluation and potentially be a means of demonstrating competence to obtain certification.

Simulation scenarios can be designed to assess some of the essential counseling skills that multiple choice questions are not able to capture (INACSL, 2021a). Such a design would also better allow for the assessment of competence (INACSL, 2021b) which is typically defined as skills-related, vs. quality or knowledge-based measurements. Presenting a short list of discrete answer choices on an exam in response to a question about what one should do first for a tearful client presents a limited view of a highly nuanced situation and may not reflect what one would do with an actual patient. A simulated counseling encounter could allow a candidate to determine and demonstrate a range of approaches (e.g., silence, empathy, exploring underlying feelings) that could be observed by multiple assessors and captured by an assessment tool designed around minimal competency rather than a single best answer. Examples of some potential formats that could be investigated further are simulations of full sessions, parts of sessions focused on certain skills, and responses to simulated client prompts (Lowe, 2022a; Massey & Roter, 2016; Setzer & Roter, 2020). GC programs, in turn, would need to ensure that their new graduates are competent in basic counseling skills assessed either in preparation for graduation or as part of the certification examination.

Scientific topics (e.g., genetics knowledge, risk assessment skills) are often easier to assess via a range of established methods, while programs and practitioners may struggle with defining and ensuring minimal counseling competence in part because the assessments typically reflect underlying cultural values and beliefs. Programs should use tools that promote and evaluate counseling competencies, and which do not suppress individual styles or viewpoints. Although more work needs to be done to investigate best pedagogical approaches for counseling skills in genetic counseling, studies have shown beneficial effects in the areas of tailoring communication through video-supported feedback (Pieterse, 2006), motivational interviewing (Winchester, 2022), appropriate psychological counseling and literacy levels training (Riddle, 2021), an evidence-based psychotherapeutically oriented counseling approach (Austin, 2020), and patient-centered communication skills (Lowe, 2022b). The communication and counseling related skills taught in studies such as those may be the most ideal targets for the development of simulation-based assessments of entry-level GC competence. Other related healthcare fields, such as psychotherapy, have taken similar approaches. The Competency Assessment Toolkit for Professional Psychology was created both to meet a need in the education of trainees as well as licensing boards desiring to assess competence acquisition (Kaslow 2009). One study proposed a competency research agenda for therapists after completing a scoping review, due to the dearth of reliable, standardized methods for competency assessment, and a future potential for non-specialist providers to be delivering therapy services (Ottman, 2020). A commonly used measure for ensuring baseline competency of non-specialist mental health providers is the Enhancing Assessment of Common Therapeutic Factors (ENACT) tool (Bond, 2022; Kohrt, 2015). Providers are assessed on their use of basic and advanced counseling skills in 15 categories of performance while working with SPs.

Call to action

To expand the use of simulation in genetic counseling, more work must be done to increase the quality and evidence base (Lewis, 2017; Shepherd & Burton, 2019; Zierhut, 2022), especially if it is to be incorporated into the certification process. GC programs could collaborate in sharing cases and resources, building on the cases that were shared when the COVID-19 pandemic began. It could also be a goal for all GC students to have access to high quality simulation-based learning experiences that are standardized or shared across programs, and to collect data on the process and effects of those experiences. Funding and support would be needed for costs such as simulation software, space, and staff; hiring, training and compensating actors; and research and quality improvement efforts to establish best practices. Part of the development would include the creation and evaluation of assessment measures and training assessors to be able to mitigate cultural and racial biases inherent in any assessment.

Simulation has the potential to not only bolster learning in graduate programs but also to better capture the nuanced and diverse skills of GCs. Working to include simulation as a component of the certification process is emerging as an important area of exploration. In fact, the ABGC has not only noted a need to reduce the number of MCE style questions but also that they are considering alternative exam methodologies (ABGC DEIJ Certification Assessment Report, 2021), suggesting an openness to concepts such as simulation-based assessment. A next step for consideration is to create a formal task force for investigating alternative exam methods such as simulation, which would include members from ACGC, the National Society of Genetic Counselors (NSGC), Genetic Counselor Educators Association (GCEA; formerly known as the Association of Genetic Counseling Program Directors), Minority Genetics Professionals Network (MGPN), and individuals with expertise in healthcare education simulation and assessment. As graduate programs, researchers, and the genetics professional societies envision the future and what it means to practice as a GC, hopefully future collaboration and research efforts will continue building the evidence base for use of simulation in genetic counseling training and assessment.