Abstract
Genetic testing is a valuable tool to guide care of pancreatic cancer patients, yet personal and family uncertainty about the benefits of genetic testing (i.e., decisional conflict) may lead to low adoption. Enabling patients to learn more about genetic testing before their scheduled appointments may help to address this decisional conflict problem. We completed a feasibility assessment of a chatbot to provide genetic education (GEd) with 60 pancreatic cancer patients and using the chatbot to deliver surveys to assess: (a) opinions about the GEd, and (b) decisional conflict about genetic testing. Findings demonstrate intervention and study feasibility with about 80% of participants engaging with the GEd chatbot, 71% of which completed at least one survey. Overall, participants appear to have favorable opinions of the chatbot-delivered education and thought it was helpful to decide about genetic testing. Furthermore, patients who chose to get genetic testing spent more time interacting with the chatbot. Findings will be used to improve chatbot design and to facilitate a well-powered future trial.
INTRODUCTION
Pancreatic cancer (PC) is highly lethal. The American Cancer Society’s estimates for pancreatic cancer in the United States for 2020 are: ~57,600 people (30,400 men and 27,200 women) will be diagnosed with PC. About 47,050 people (24,640 men and 22,410 women) will die of PC. It is predicted to become the second leading cause of cancer related deaths by 20301. Inherited genetic factors play an important role in PC risk. Studies have demonstrated that 4.6-9.7% of PC harbor pathogenic variants in a predisposition gene. As such, in October 2018, guidelines from the National Comprehensive Cancer Network (NCCN) suggest that patients diagnosed with PC should undergo multigene genetic testing2. When there are only a few genetic counselors available to give services for the growing volume of testing, automated communication methods are starting to become more prevalent3-6. Automated, pre-programmed, and receptive conversational bots or chatbots are employed to simulate human interactions. Many bots assess user input using natural language processing and reply accordingly using spoken or written human language3. The collection of patient data, the provision of genetic information, the delivery of findings, and the facilitation of cascade testing of high-risk patients in clinical settings are some of the procedures where chatbots have been deployed7-11. Growing evidence suggests that many patients prefer web-based, interactive, 24-hour accessible, updatable, and/or simpler options for genetic education and counseling12,14,15. More than 80% of US adults own a smartphone with high (>71%) ownership in persons with <30K$ income16. In this work, we developed and applied a chatbot with PC patients, as one population with potential to benefit from enhanced education around genetic testing.
To study the potential use of chatbots in genetic education (GEd), we have designed a chatbot experience using the virtual Genetic Information Assistant (Gia®) powered by InvitaeTM and are piloting its delivery to new PC patients. While there are other chatbot development platforms that could be tailored for this application7,17, we choose to use Gia because it is HIPAA compliant and has been used previously for genetic test education4. In a study to evaluate the effectiveness of chatbots more broadly on patient medication adherence rates, researchers found that if the agent was attentive, knowledgeable, smart/interactive, convincing, and supportive then they were more effective at simulating a doctor-patient relationship and led to higher rates of overall user satisfaction13. Similarly, several studies have shown that the addition of an interactive tool enhances knowledge acquisition in self-identified minority patients with low health literacy18,19. While some have studied the influence of providing broader access to genetic education on genetic test (GT) adoption4 few have studied decisional conflict as a potential mechanism influencing adoption. We hypothesize that genetic education will help reduce decisional conflict around genetic testing in PC patients. We further hypothesize that there will be higher GT adoption among individuals with lower decisional conflict about genetic testing. As an initial step, we sought to better understand opinions influencing the acceptability of GEd use prior to genetic testing, and how to best integrate GEd and GTs in routine care for PC patients. GEd integration in the cancer treatment education continuum may help to ensure future study scalability.
METHODS
Intervention Description
The Health Insurance Portability and Accountability Act compliant Invitae platform was used to create the conversation. We customized Gia for tailored GEd delivery and to administer post-intervention survey questions related to opinions about user experience with the GEd material and genetic testing decisional conflict for our research. GEd content for Gia was developed by experts in genetic counseling and pancreatic cancer care (JB and AK). It covers the goal for genetic testing, the types of GTs available, and the potential outcomes. The scripted pretest GEd chat first asked about personal health history and then covered the following major genetics education areas: (a) basic genetics, (b) genetic test being offered, (c) types of the genetics results, and (d) possible relevance of genetic results to family members.
When the patients first accessed the chat window, Gia introduced the chat’s objective, reaction buttons, and menu options. Gia delivered the pre-scripted material in the form of real-time messages, using text bubbles with three periods (ellipses) to simulate an instant chat exchange (Figure 1). For pretest genetics education, all the patients were provided with the same content. Gia was customized to show options for answers on the bottom of the screen at specified moments during the dialogue, and the patient’s response influenced the subsequent content delivered without the need to enter free-text responses. Additional material included a wide variety of subjects, such as requesting basic explanations of cancer and genetics, the many outcomes of genetic testing, and the risk of genetic mutation for different cancer types (e.g., breast, pancreatic, and colon). Depending on what options were selected, patients might view more or less information on a topic through this method. At the end of the conversation, 10 pre-visit survey questions were presented to the patient to assess opinions about Gia and the GEd content.
Figure 1.
Screenshots of the Genetics Information Assistant (Gia®), Genetics Eduction (GEd) content, and a pre-visit survey question. A. First interaction with an enrolled patient; B. Introduction screen to Gia; C. Gia chat with real-time messaging; D. Gia delivered GEd chat with answer options; and E. Pre-visit survey question with submit button.
Study flow and Sample
Figure 2 shows study design, including the sequence and timing of study procedures (distinguishing research procedures from those that are part of routine care). All the patients older than 18 years of age with known or suspected PC diagnosis were eligible to participate in this study. Patients with prior germline multi-gene genetic testing for PC within the past 3 years were excluded. Following the oral consent process, willing participants provided their preferred contact information for Gia access. Consented patients who did not respond to any of the pre- or post-visit survey were considered ‘full dropouts’, and those who completed either or both the surveys were considered ‘partial dropout’. For the data analysis we selected the ‘partial dropout’ cohort of patients.
Figure 2.
Study Flow. Research procedures are in white diamonds and routine care steps are in green boxes. SKCCC= [Sidney Kimmel Comprehensive Cancer Center], Gia= [Genetic Information Assistant]. Red dot indicates the end of a process.
Data collection
The data were collected through pre- and post-visit surveys delivered to the participant via Gia. The demographic data was collected by the coordinators. We also collected data about the time spent interacting with Gia. We assessed feasibility to use Gia for GEd, opinions about GEd, and the decision to get a GT. Feasibility was characterized based on whether consented patients engaged with Gia. Opinions about Gia and GEd were collected using pre-visit survey questions. The post-visit survey questions were used to assess self-reported choice to receive genetic testing (primary outcome) and the decision to get a GT. Study engagement was assessed based on the time it took the participant to complete each chat (pre-visit and post-visit) experience. This time was determined by comparing the time when each Gia chat was started to the time when each survey was completed. For some participants, this total duration lasted multiple days, which may represent users who stepped away from the chat and returned at a later time. To limit the effect of these long periods of usage, when calculating the average chat interaction time for each survey, we excluded data of chats with time of interaction longer than 15 minutes11.
To design our pre- and post-visit survey instruments, we drew from existing surveys and measurement scales. The pre-visit survey instrument included 10 survey questions to assess the GEd content and usability of Gia. These questions were adapted from the Post-Study Usability Questionnaire (PSSUQ)14 with a Likert-based response options ranging from 1 (“strongly disagree”) to 5 (“strongly agree”), with 3 representing “neither agree nor disagree”. The post-visit survey instrument had 4 questions about their status of genetic testing. Based on their response, the patient was then directed to the next set of 6 questions which were adapted from the satisfaction with healthcare decision scale15. The 6 questions were used to assess the decision to get a GT after GEd via Gia. The responses for these questions used 5 scale Likert- based response options ranging from 1 (“strongly disagree”) to 5 (“strongly agree”), The responses were converted into two response categories : “agree” (including “strongly agree” and “somewhat agree”), and “disagree/neither” (including “neither agree nor disagree”, “somewhat disagree”, and “strongly disagree”) to facilitate the analysis and better interpret the frequencies. This study was approved by the Institutional Review Board of Johns Hopkins University (protocol code IRB0027915).
Data analyses
We performed descriptive statistical analysis of the cohort stratified by patients’ self-reported genetic testing status after using Gia: “yes”, “not yet decided”, and “missing”. To assess whether there was a relationship between opinions about GEd via Gia (pre-visit survey) and the ability to decide on genetic testing (post-visit survey), we used Spearman’s Correlation coefficients. The threshold for statistical significance was set at p=0.05. The questions were divided into a set of positive and negative questions for the pre-visit survey. All the even numbered questions were considered positive (i.e., directed towards Gia being considered useful), and all the odd numbered questions were negative (i.e., directed away from Gia being considered useful). Spearman’s correlation was also used to assess if a relationship exists between questions within each survey. All statistical analyses were performed using Python 3.8.8.
RESULTS
Characteristics of Study Participants According to Genetic Testing Status
Sixty patients were enrolled into the study and sent the Gia link for GEd (Figure 3). Of those who enrolled, 80% (n=48) opened the GEd link and 71% (n=43) completed at least one of the pre- or post-visit surveys. About 26% (n=16) completed both the surveys. Among patients that completed at least one survey (n=43), characteristics stratified by choice of genetic testing are summarized in Table 1. Patients who reported getting genetic testing were majorly female (n=16, 100%), few had a family history of pancreatic cancer (n=2, 13%), and few indicated that their provider offered the testing (n=9, 57%). Patients who did not decide to receive a GT were all females (n=4, 100%), few had a family history of pancreatic cancer (n=1, 25%), and in all the cases, the provider offered the testing (n=4, 100%). Patients who did not answer the question about genetic testing status were majority female (n=22, 96%), with one patient reporting a family history of pancreatic cancer (n=1, 4%). Patients who report getting a GT were younger when compared to patients whose genetic testing status was undecided (average age of 69 and 79 years, respectively).
Figure 3.
Flow diagram for cohort selection
Table 1.
Characteristic of study population (n=43)
|
Study Engagement According to Genetic Testing Status
Among patients who completed at least one survey (n=43), the analysis of time spent completing the study, stratified by choice of GT, are summarized in Table 2. Findings indicate that patients who reported getting a GT spent more time (11 minutes) interacting with Gia, compared to those who were not sure if they wanted to have a GT (9 minutes). All participants spent approximately the same amount of time on the post- visit survey.
Survey Responses According to Genetic Testing Status
Results from completing at least one survey, stratified by genetic testing status are summarized in Table 2. The average score for GEd content and usability of Gia is highest among patients who reported getting a GT (4.50). The average score for the ability to make an informed decision about genetic testing after using Gia is also slightly higher for patients that indicated getting a GT (3.47), when compared to people who were unsure if they want a GT (3.25).
Table 2.
Characteristic of study population who completed at least one survey (n=43)
|
Opinions about Chatbot for Genetics Education
Among patients who completed both surveys (n=16), the frequency distribution of opinions about Gia for GEd is summarized in Table 3. Most patients indicated that they were satisfied with the usability of Gia and the GEd content. To inform survey instrument revisions, analyses of pre-test survey items indicated that question number 9 & 10 were highly correlated with question 5 (>0.9, p<0.001), indicating potential to remove 9 & 10 in the final survey instrument.
Table 3.
Frequencies of satisfaction with opinion for chatbot (Gia) for GEd for 16 patients who completed both surveys.
|
Self-reported Decision to get Genetic Testing
For the patients who completed both surveys (n=16), the survey scores showed that the ability to make an informed decision with GEd via Gia is neutral. Responses showed mixed agreement that GEd via Gia was helpful to make an informed decision about GT (Table 4). Half of the patients agreed that Gia helped them to realize a decision needed to be made (Question 1). Most of the patients agreed that Gia helped them the make a better decision about GT and helped them to think about what advantages and disadvantages were most important (Questions 2 and 3). Most patients showed disagreement or were neutral about Gia helping them to organize their thoughts about a decision to GT (Question 4), come up with questions to ask their doctor about GT (Question 5), or prepare for their doctor’s appointment (Question 6). To guide survey instrument revisions, we found that questions 5 and 6 were highly correlated with each other (>0.8, p<0.001) and we may consider removing one in the final survey.
Table 4.
Frequencies of satisfaction with decision to get GT with GEd for 16 patients who completed both surveys.
|
Opinions about GEd and Decision to Genetic Test
The correlation between the two surveys for the cohort of patients who completed both the surveys (n=16) using a scatter plot showed that there was a positive relationship between the two scales (Figure 4). One outlier was removed to see the correlation clearly. The relationship signified that the people who had more positive opinions about Gia and GEd (pre-visit survey responses), also had felt more able to decide about getting genetic testing with GEd (post-visit survey responses).
Figure 4.
Correlation between opinion of chatbot (pre-visit survey) and the decision to get the GT with GEd (post-visit survey).
The relationships between pre-test survey items to assess opinions about GEd via Gia and post-visit survey items on the ability to decide on genetic testing are summarized in Figure 5 and Figure 6. Pre-visit items showed low correlation with post-visit items (-0.54 to 0.51), indicating that the survey instruments measure distinct concepts.
Figure 5.
Correlation between positive questions. Pre-visit survey questions on opinion of Gia for GEd: 1. Gia was easy to use, 2. The various functions in Gia were well integrated, 3. I would like to use a tool like Gia again, 4. I think most people would learn to use Gia quickly, 5. I felt confident using Gia). Post-visit survey questions on decision to get a genetic test: 6. Gia helped me realize that a decision had to be made, 7. Gia prepared me to make a better decision, 8. Gia helped me to think about which advantages and disadvantages are most important, 9. Gia helped me to organize my thoughts about the decision, 10. Gia helped me to come up with
Figure 6.
Correlation between negative questions. Pre-visit survey questions on opinion of Gia for GEd: 1. It would be better to have a technical person’s help to use Gia, 2. Gia had too many inconsistencies, 3. Gia was too complex, 4. Gia was awkward to use, 5. I needed to learn a lot of things before I could get going with Gia. Post-visit survey questions on decision to get a genetic test: 6. Gia helped me realize that a decision had to be made, 7. Gia prepared me to make a better decision, 8. Gia helped me to think about which advantages and disadvantages are most important, 9. Gia helped me to organize my thoughts about the decision, 10. Gia helped me to come up with questions to ask my doctor, 11. Gia prepared me for my appointment
DISCUSSION
Our aim was to create and test the feasibility of an intervention for pre-visit GEd that provided baseline information on genetics and testing to preface in-person genetic counseling. This study provides insight into the components of Gia and GEd (interface features and topics presented) that may be most important to pancreatic cancer patients and lays the foundation for future work to study its impact on decisions to get genetic testing in this patient population. Conversational agents like Gia are becoming more important in precision medicine, with potential to replace some types of education that have more traditionally been led by people18. The majority of the enrolled pancreatic cancer patients chose to engage with the GEd provided by Gia. This finding adds to earlier research on the acceptability and usefulness of conversational agents in a variety of genetic counseling scenarios4,7,19-22.
Gia was customized for this study to deliver tailored GEd and to collect participant responses related to our outcomes of interest. By embedding survey questions after our designed GEd materials, we collected user opinions and assessed usability of Gia’s pre-visit GEd in a novel way. We provide evidence that Gia can be adopted as a multi-use tool to inform patients about genetic considerations for cancer as well as support researchers to conduct surveys specific to their clinical interventions. To assess the impact of education obtained via Gia on the choice of genetic testing, we stratified the patients based on their choice of GT post-GEd. A few patients who reportd getting a GT had a family history of pancreatic cancer (Table 1), which might impact their inclination towards getting tested. These patients also spent more time interacting with Gia (Table 2), which signifies that these patients may be motivated to learn more about GT and genetics, and that interacting with Gia may be an acceptable strategy to educate patients about genetic testing and related topics. Our findings are also in alignment with previous studies showing that greater use of visuals to educate people about genetics and testing may result in educational initiatives that are more successful and discoveries that are more transferable to other contexts4,18.
To assess opinions about Gia and GEd and the ability to make an informed decision about genetic testing after using Gia, the pre-visit and post-visit survey responses were analyzed for patients who completed both surveys. Results from the pre-visit survey to assess patients’ opinion about Gia for GEd showed high satisfaction among those indicating they chose genetic testing, when compared to those that were not sure if they want to have a genetic test (Table 2). Our findings also highlighted that most patients felt that Gia was an easy and feasible tool to learn about GT (Table 3). Furthermore, the post-visit survey analysis showed mixed agreement that GEd via Gia was helpful to make an informed decision about GT (Table 4). To address areas of low or moderate agreement, potential areas to improve GEd content to better help patients with genetic testing decisions include: clarifying that GT should be offered to them as a standard care practice (Question 1), providing a summary of advantages and disadvantages to help with organizing their thoughts about a decision to GT (Questions 3 & 4), and providing talking points to discuss with their doctor (Questions 5 & 6).
To identify potential revisions for our survey instruments, we performed correlation analyses for the two surveys to identify highly correlated questions. Removing those question may help to minimize study dropout due to survey fatigue. In addition, the correlation between pre-visit and post-visit survey items show that the instruments are measuring distinct concepts (Figure 5 and Figure 6).
When assessing the relationship between pre-visit survey scores and post-visit survey scores, we observed a positive relationship (Figure 4). This finding indicates that the people who had more positive opinions about Gia and GEd, also felt more able to decide about getting genetic testing with GEd (i.e., less decisional conflict). The major shortcoming of our feasibility study is the small sample size and lack of gender diversity. In the future, we aim to implement the study with a more diverse and larger cohort that includes patients with other cancer types. We also seek to better understand patients that choose not to use the chatbot when it is offered. The primary goal of our future work will investigate decisional conflict as a mechanism that moderates the process by which using Gia for GEd influences the choice to get genetic testing. In a broader deployment of Gia for GEd, we seek to build evidence on its use to increase adoption of genetic testing recommendations in the care of pancreatic cancer patients and of patients with other explored cancer types. We hypothesize that its use in conjunction with physician care will assist in reducing decisional conflict about getting genetic testing and lower the burden on providers to educate patients about GT.
CONCLUSION
Overall, this analysis quantified opinions about the usability and acceptability of Gia for GEd, captured how it was used by patients participating in this study, and captured areas to improve GEd to help patients decide about genetic testing. Our findings reveal high feasibility to use Gia for GEd, and we hope to deploy it as an intervention in a larger study, including considering other cancer types.
ACKNOWLEDGEMENT
The authors would like to thank our study participants, the Sidney Kimmel Comprehensive Cancer Center (SKCCC) Pancreas Cancer team at Johns Hopkins Medicine, and the team at Invitae. Special thanks to Emilie Simmons, MS, Sarah Savage, MS, and members of the Gia technical team at Invitae for their assistance in the development, launch, and support of the Pancreatic GEd Gia chat. This work was funded in part by the Sol Goldman Pancreatic Cancer Research Center.
Figures & Table
References
- 1.Rahib L, Wehner MR, Matrisian LM, Nead KT. Estimated Projection of US Cancer Incidence and Death to 2040. JAMA Netw Open. 2021;4(4):e214708. doi: 10.1001/jamanetworkopen.2021.4708. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Walker EJ, Carnevale J, Pedley C, et al. Referral frequency, attrition rate, and outcomes of germline testing in patients with pancreatic adenocarcinoma. Fam Cancer. 2019;18(2):241–251. doi: 10.1007/s10689-018-0106-2. [DOI] [PubMed] [Google Scholar]
- 3.de Cock C, Milne-Ives M, van Velthoven MH, Alturkistani A, Lam C, Meinert E. Effectiveness of Conversational Agents (Virtual Assistants) in Health Care: Protocol for a Systematic Review. JMIR Res Protoc. 2020;9(3):e16934. doi: 10.2196/16934. Published 2020 Mar 9. doi:10.2196/16934. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Chavez-Yenter D, Kimball KE, Kohlmann W, et al. Patient Interactions With an Automated Conversational Agent Delivering Pretest Genetics Education: Descriptive Study. J Med Internet Res. 2021;23(11):e29447. doi: 10.2196/29447. Published 2021 Nov 18. doi:10.2196/29447. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Wang Y, Golesworthy B, Cuggia A, et al. Oncology clinic-based germline genetic testing for exocrine pancreatic cancer enables timely return of results and unveils low uptake of cascade testing. J Med Genet. 2022;59(8):793–800. doi: 10.1136/jmedgenet-2021-108054. doi:10.1136/jmedgenet-2021-108054. [DOI] [PubMed] [Google Scholar]
- 6.Ramsey ML, Tomlinson J, Pearlman R, et al. Mainstreaming germline genetic testing for patients with pancreatic cancer increases uptake [published online ahead of print, 2022 Jun 17] Fam Cancer. 2022:1–7. doi: 10.1007/s10689-022-00300-5. doi:10.1007/s10689-022-00300-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Schmidlen T, Schwartz M, DiLoreto K, Kirchner HL, Sturm AC. Patient assessment of chatbots for the scalable delivery of genetic counseling. J Genet Couns. 2019 Dec 24;28(6):1166–1177. doi: 10.1002/jgc4.1169. [DOI] [PubMed] [Google Scholar]
- 8.Biesecker B. Genetic counseling and the central tenets of practice. Cold Spring Harb Perspect Med. 2020 Mar 02;10(3):a038968. doi: 10.1101/cshperspect.a038968. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Flannery DB. Challenges and opportunities for effective delivery of clinical genetic services in the U.S. healthcare system. Curr Opin Pediatr. 2018;30(6):740–745. doi: 10.1097/MOP.0000000000000693. [DOI] [PubMed] [Google Scholar]
- 10.Gordon E, Babu D, Laney D. The future is now: technology’s impact on the practice of genetic counseling. Am J Med Genet C Semin Med Genet. 2018 Mar;178(1):15–23. doi: 10.1002/ajmg.c.31599. [DOI] [PubMed] [Google Scholar]
- 11.Rashkin MD, Bowes J, Dunaway K, Dhaliwal J, Loomis E, Riffle S, et al. Genetic counseling, 2030: an on-demand service tailored to the needs of a price conscious, genetically literate, and busy world. J Genet Couns. 2019 Apr 09;28(2):456–465. doi: 10.1002/jgc4.1123. [DOI] [PubMed] [Google Scholar]
- 12.Katapodi MC, Jung M, Schafenacker AM, et al. Development of a Web-based Family Intervention for BRCA Carriers and Their Biological Relatives: Acceptability, Feasibility, and Usability Study. JMIR Cancer. 2018;4(1):e7. doi: 10.2196/cancer.9210. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Nazareth S, Hayward L, Simmons E, et al. Hereditary Cancer Risk Using a Genetic Chatbot Before Routine Care Visits. Obstet Gynecol. 2021;138(6):860–870. doi: 10.1097/AOG.0000000000004596. doi:10.1097/AOG.0000000000004596. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction. 1995 Jan 1;7(1):57–78. [Google Scholar]
- 15.Bennett C, Graham ID, Kristjansson E, Kearing SA, Clay KF, O’Connor AM. Validation of a preparation for decision making scale. Patient education and counseling. 2010 Jan 1;78(1):130–3. doi: 10.1016/j.pec.2009.05.012. [DOI] [PubMed] [Google Scholar]
- 16.Nilsson MP, et al. BRCA search: written pre-test information and BRCA1/2 germline mutation testing in unselected patients with newly diagnosed breast cancer. Breast Cancer Res Treat. 2018;168(1):117–126. doi: 10.1007/s10549-017-4584-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Ireland D, Bradford D, Szepe E, et al. Introducing Edna: A trainee chatbot designed to support communication about additional (secondary) genomic findings. Patient Educ Couns. 2021;104(4):739–749. doi: 10.1016/j.pec.2020.11.007. doi:10.1016/j.pec.2020.11.007. [DOI] [PubMed] [Google Scholar]
- 18.Mobile Fact Sheet. 2022, November 16. Pew Research Center: Internet, Science & Tech. https://www.pewresearch.org/internet/fact-sheet/mobile/
- 19.Wang C, et al. Acceptability and feasibility of a virtual counselor (VICKY) to collect family health histories. Genetics in Medicine. 2015;17(10):822–830. doi: 10.1038/gim.2014.198. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Yee LM, et al. A randomized trial of a prenatal genetic testing interactive computerized information aid. Prenat Diagn. 2014 Jun;34(6):552–7. doi: 10.1002/pd.4347. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Holmes-Rovner M, Kroll J, Schmitt N, et al. Patient satisfaction with health care decisions: The satisfaction with decisions scale. Med Decis Making. 1996;16:56–64. doi: 10.1177/0272989X9601600114. [DOI] [PubMed] [Google Scholar]
- 22.DeMarco TA, Peshkin BN, Mars BD, Tercyak KP. Patient satisfaction with cancer genetic counseling: a psychometric analysis of the Genetic Counseling Satisfaction Scale. J Genet Couns. 2004;13(4):293–304. doi: 10.1023/b:jogc.0000035523.96133.bc. [DOI] [PMC free article] [PubMed] [Google Scholar]