User-centered design of multi-gene sequencing panel reports for clinicians

Abstract

The objective of this study was to develop a high-fidelity prototype for delivering multi-gene sequencing panel (GS) reports to clinicians that simulates the user experience of a final application. The delivery and use of GS reports can occur within complex and high-paced healthcare environments. We employ a user-centered software design approach in a focus group setting in order to facilitate gathering rich contextual information from a diverse group of stakeholders potentially impacted by the delivery of GS reports relevant to two precision medicine programs at the University of Maryland Medical Center. Responses from focus group sessions were transcribed, coded and analyzed by two team members. Notification mechanisms and information resources preferred by participants from our first phase of focus groups were incorporated into scenarios and the design of a software prototype for delivering GS reports. The goal of our second phase of focus group, to gain input on the prototype software design, was accomplished through conducting task walkthroughs with GS reporting scenarios. Preferences for notification, content and consultation from genetics specialists appeared to depend upon familiarity with scenarios for ordering and delivering GS reports. Despite familiarity with some aspects of the scenarios we proposed, many of our participants agreed that they would likely seek consultation from a genetics specialist after viewing the test reports. In addition, participants offered design and content recommendations. Findings illustrated a need to support customized notification approaches, user-specific information, and access to genetics specialists with GS reports. These design principles can be incorporated into software applications that deliver GS reports. Our user-centered approach to conduct this assessment and the specific input we received from clinicians may also be relevant to others working on similar projects.

1. Introduction

Customizing healthcare based on each person's unique genetic makeup could enable an era of precision medicine that would improve prevention, diagnosis and treatment for many types of health conditions. Routine precision medicine is rapidly approaching due to increased use of whole genome, whole exome, and other types of multi-gene next generation sequencing panels, hereafter referred to as genomic sequencing (GS). Results from GS will be used more often as cost goes down and evidence of clinical utility increases [1,2]. With the overwhelming amounts of data that can be generated from GS, the laborious task of manually prioritizing clinically significant results often falls to clinical and laboratory geneticists [3]. With the anticipated increase in the use of GS, manual review of genetic data by clinicians, however, is not scalable. We believe that developing computerized tools to help non-genetics experts make sense of GS results will greatly increase the likelihood of achieving the vision of successful use of these data.

The healthcare environment in which such computerized tools might be deployed can be complex and high-paced. Thus, there is a need to use a design methodology that aims to support the current way of working. The primary objective of this study was to develop a high-fidelity prototype for delivering GS reports to clinicians that simulates the user experience of a final application. Target stakeholders were clinicians involved in two University of Maryland Program for Personalized and Genomic Medicine (PPGM) initiatives that are exemplary use cases for precision medicine programs more broadly. These initiatives are the Translational Pharmacogenetics Program (TPP) and the Personalized Diabetes Medicine Program (PDMP). The TPP project aims to use a patient's CYP2C19 genotype results to tailor antiplatelet therapy after a cardiac stent has been placed [4]. The PDMP is designed to implement, disseminate and evaluate an approach to identifying and genomically diagnosing highly penetrant genetic forms of diabetes. The goal for PDMP is to enable personalized treatment for better and potentially less invasive glucose control, prognosis, and assessment of familial risk for patients [5,6].

We employed a user-centered software design approach in a focus group setting in order to facilitate gathering rich contextual information with a diverse group of stakeholders potentially impacted by the delivery of GS reports relevant to the TPP and PDMP projects. We completed two studies: The goal for the first study (phase 1) was to understand current genetic and laboratory testing processes for documenting results, notifying clinicians of those results, and viewing the results. Findings from the first study were used to propose the design of software, which we called the Genomic Medicine Assistant (GMA), for delivering GS reports to clinicians. In particular, we proposed a design that would potentially mitigate issues with the reporting and notification processes we identified in phase 1. The goal for the second study (phase 2) was to gather feedback on a proposed design of the GMA software including preferences for interactions with, content contained in and the perceived usability of the design.

2. Background

Much of the research surrounding the delivery of individual GS results to date has been regarding the ethical, legal and social implications for communicating findings to patients [7–9]. Indeed, GS technology is not mature and appropriate in all clinical scenarios and there is the potential for cascade effects (i.e., a chain of events initiated by an unexpected result leading to unnecessary additional testing or treatments) [10–13]. We are now seeing guidance for clinicians ordering, interpreting, and communicating GS with their patients [13,14], as well as clinical laboratories supporting the ordering of GS and delivery of reports that include results that are both clinically actionable and directly relevant to the patients' indication. While there are some studies exploring the potential for GS to replace traditional tests in terms of sensitivity, specificity and completeness [15], there are few studies to date investigating technologies for delivering GS reports under this new paradigm. One group has developed a web-based platform to automatically generate a clinical report, based on pre-defined templates, from raw assay results or specified diplotypes [16]. Their framework has potential to help provide consistent and reproducible reporting while also saving time by calculating diplo-types and assembling report content. Our work adds to this emerging literature by taking a user-centered design approach to explore a range of scenarios for software generating reports from GS for use by clinicians.

Related to our goal to investigate preferences for content contained in the proposed design of software are studies of the information needs of clinicians when interpreting GS. These studies serve as the groundwork to inform the design of software tools [17,18]. While most of these studies have focused on genetic counselors' information needs, some findings may be generalized to other clinicians. Genetic counselors have expertise when reviewing GS reports; therefore, information that they deem necessary should be emphasized when non-genetic expert clinicians are reviewing GS reports. For example, a study surveying genetic counselors showed that there is a perceived lack of information on the classification system for variants of unknown significance (VUS) included in laboratory reports, highlighting a need for improved GS reports to provide transparency [18]. That study also found that genetic counselors wanted more information, such as patient information, in reports to help contextualize VUS results [18]. We believe this need for more information about the patient within laboratory reports can be generalized to other types of clinicians in order to make patient-specific recommendations. This belief is supported by findings of others indicating that clinicians desire information available to discuss with the patient [19]. Another study of interviews with genetic counselors identified specific information needs for risk communication [17]. Information needs described in that study include clinical patient characteristics, social and cognitive patient characteristics, and patient motivation and goals for the genetic counseling session. Those needs may also generalize to non-genetics experts communicating GS results to patients more broadly, and thus were considered with findings from phase 1 studies in our proposed software design.

While there exist tools and resources for clinical laboratory professionals interpreting and reporting genetic test results, the design and purpose of these tools varies. One web-based application, SeqReporter, has been developed for clinical molecular laboratory use for next-generation sequencing data analysis [20]. That tool is an automated web-based application for GS result classification. While it was designed to optimize laboratory reporting, it did not support the anticipated information needs of non-genetics expert clinical end users. While the published information provides a starting point, our work gathers additional end-user input on preferences for interactions with, content contained in and the perceived usability of the software design by non-genetics clinician experts.

2.1. Methodological Background

In order to gather input on the proposed design of software, we used a task-centered system design (TCSD) protocol in a focus group setting. Other studies have used similar task-centered approaches to investigate the usability of genetic data interpretation software. For example, Shyr et al. administered surveys, conducted interviews and performed cognitive task analyses to assess the usability of a clinical exome analysis software [21]. In another study a think-aloud, graded-task protocol was used to evaluate the GeneInsight Suite. That study highlighted a need to provide the most current genetic information [22]. The graded task protocol facilitated identifying design improvements that could be easily made and determining larger issues with how the interface was interpreted by end-users. While our goals to gather input on the design of software were similar, in our modified TCSD protocol, we did not have participants grade each task they completed. Rather, by conducting our study in a focus group setting, we allowed for open discussion about proposed software design features in order to gather feedback on specific clinical contexts in which those features would be most useful.

In proposing software design characteristics, we explored the inclusion of features that might be supported by existing clinical system vendors. These included alert messages, and an HL7 standard, the “Infobutton Standard” [23,24]. The use of infobuttons in electronic health records (EHRs) is an important method for delivering the most current healthcare information to clinicians at the point-of-care. The technology lends itself to use with GS reports, as treatment guidelines and recommendations associated with genetic variation are ever-improving [25]. Linking to those resources from infobuttons could provide a mechanism to communicate genomic data and clinical decision support [26]. One study piloted a pharmacogenomic-based prescribing alert to examine the clinical impact of embedding infobuttons within an EHR alert [27]. That study offered improvements to make the infobutton content more useful and increase clinician trust in the alert. Findings from those studies reinforced our decision to include alert messages and infobuttons in the proposed design of software capable of providing clinicians with important, current information with GS results.

3. Materials and Methods

3.1. Recruitment Approach

Our two-phase study was conducted within a focus group setting. We sent emails to individuals influenced by the two precision medicine projects at the University of Maryland School of Medicine. These included 14 TPP/PDMP research team members (clinical research coordinators, genetic counselors, laboratory professionals), 17 PDMP clinicians (endocrinology fellows, nurse practitioners, diabetes educators), and 34 TPP clinicians (cardiology fellows, nurse practitioners, clinical pharmacists). The email was sent with an information sheet describing the goals of the project. Two weeks later, a second email was sent to the same pool of potential participants asking for their participation again. A third email was sent out to interested participants to schedule a focus group. Interested participants were contacted a fourth and final time to confirm the date and time of the focus group. A $100 gift card was given to participants. Participants in phase 2 were recruited in the same fashion and from the same pool of PDMP and TPP clinicians as in phase 1. Those who participated in phase 1 were allowed to participate in phase 2. This study was reviewed and judged by the University of Maryland Institutional Review board (IRB) as non-human subjects research (HP-00061346). At the start of each focus group, participants were given a participant information sheet. This sheet captured the participant's involvement in TPP and PDMP projects, years of experience in their profession, gender, and their choice to be contacted, or not, for future focus groups.

3.2. Phase 1 Data Collection and Analysis

In phase 1, three focus groups were conducted in order to examine existing and anticipated genetic testing processes among two precision medicine projects (TPP and PDMP). Our focus group protocol was organized into three topics about genetic test results: laboratory documentation of genetic test results, making genetic test results available to view in the EHR, and notifying the clinician about genetic test results. For each topic, open-ended questions (e.g., “what is your overall opinion of how genetic test results should be documented by the testing lab?”) elicited general information followed by more specific questions and probes (e.g., “what kinds of information do you think should or shouldn't be in the report?” “when would you want to access information you just mentioned?”). A moderator and note taker were present at all focus groups, each of which lasted 90–120 min. Focus group discussions regarding PDMP were modified to cover laboratory test processes more generally given that data collection occurred prior to the start of genetic testing for that project. Sessions were audio recorded, transcribed, and analyzed with the aid of Dedoose, a qualitative research analysis software [28].

Two coders (EC, CO) mapped statements made by study participants to terminology common to user experience and design, and to organizational and business process modeling heuristics. Our assessment of user experience in phase 1 was focused on usability (effectiveness, efficiency and satisfaction) and design (missing functionality, problems and proposed solutions, and desired features). Similar to another study of the usability of clinical systems [29], we also captured ‘positive’ and ‘negative’ quotes. Positive quotes allowed us to identify features that satisfied the needs of clinicians and negative quotes allowed us to identify possible features to improve the use clinical systems for genetic testing processes. Mappings of statements to usability codes facilitated determining perceived issues with the use of current clinical systems for genetic testing processes and proposed solutions acceptable by a range of stakeholders to address those issues. Mappings of statements to organizational and business process modeling heuristics facilitated determining existing and anticipated genetic testing processes for documenting, viewing, and notifying individuals about genetic test results (Appendix B).

We completed two analyses of data collected in phase 1. First, we analyzed the data to understand genetic testing processes of TPP specifically. Findings from that analysis is described elsewhere [30] and is summarized in the following section. Second, we analyzed phase 1 data in order to define scenarios and GMA design requirements. Methods and results from that analysis are described in this manuscript.

3.2.1. Summary of Methods and Findings from Previous Work Analyzing Translational Pharmacogenetics Program Genetic Testing Processes

The coding team analyzed data from two focus groups (TPP/PDMP research team members and TPP-related clinicians) in order to document TPP genetic testing processes at the time, and to propose more streamlined processes enabled by technology. With information gained from focus groups, we documented TPP genetic testing processes using a Business Process Model and Notation (BPMN) model [30]. We also performed member checking to verify the accuracy of our model. In summary, findings indicated a reliance on TPP research team members to complete tasks that were labor-intensive and potentially error-prone. Findings also indicated insufficient support and resources to assist clinicians in treatment decisions with the lab report alone. At the time, important information for making treatment decisions were provided by the TPP research team in the form of a letter, separate from the lab report itself. Specific recommendations to enhance TPP genetic testing processes included reducing the reliance on the TPP research team by establishing a Laboratory Information Management System (LIMS) for genetic test results that is capable of communicating with the EHR, and the implementation of clinical decision support (CDS). CDS implementation could also provide a mechanism to improve access to information for clinicians to make treatment decisions.

3.2.2. Assessing Desired Characteristics for Precision Medicine Scenarios and Genomic Medicine Assistant Design

We drew from data collected for two focus groups (TPP-related clinicians and PDMP-related clinicians) to provide evidence for scenario and software design choices. Scenarios were designed to be relevant to TPP and PDMP projects and validated with project advisory committee members. Committee members consisted of experts in clinical genomics, clinical vocabulary standards and clinical decision support implementation in commercial EHRs (Appendix A). After scenario validation, we designed project-specific “storybook packets” that included two scenarios, each of which had associated tasks for accessing GS reports, mockups of those reports and information resources viewed using GMA. Those notification methods found to be acceptable by phase 1 study participants were described in both storybook packets. One team member (EC) proposed initial GMA design mockups informed by participant recommendations from phase 1 focus groups, the literature describing laboratory report design principles [31], and model laboratory reports. The packets underwent multiple rounds revision based upon feedback from the project team prior to finalizing.

3.3. Phase 2 Data Collection and Analysis

In phase 2, a modified task-centered system design (TCSD) protocol was applied in a focus group setting in order to gather feedback from clinicians on a proposed design of the GMA software. Traditional TCSD involves four main phases: (a) identification of tasks, (b) user-centered requirements analysis, (c) design through use of scenarios or storybooks, and (d) evaluation of design using walkthrough analyses [32]. In this study we leverage our understanding of preferences for genetic testing processes from phase 1 analyses in order to complete TCSD phases a–c. TPP and PDMP clinicians were each asked to complete task walkthroughs involving two clinical scenarios where GS was performed (see Appendix A for “Scenarios relevant to the Translational Pharmacogenomics Project” and for “Scenarios relevant to the Personalized Diabetes Medicine Program”).

3.3.1. Administering Storybook Packets for Task Walkthroughs With clinicians

In phase 2, storybook packets were administered to study participants within a focus group setting. A moderator and note taker were present at both focus groups, each of which lasted 90– 120 min. Focus groups were split into two parts, with no discussion during the first part of the study. Participants were first asked to spend 15–20 min reviewing and writing down comments or questions about each task/screen presented in the storybook packet. For the second part, the moderator led a discussion of each task as a group. The moderator facilitated discussion through the use of probes (e.g., “Is this alert message informative?”) corresponding to each task (e.g., “Clinician views alert message”). Sessions were audio recorded, transcribed, and analyzed with the aid of Dedoose, qualitative research analysis software [28].

3.3.2. Analysis of Clinician Perceptions of Tasks and Screen Mockups

Two coders (EC, MB) mapped statements made by study participants to terminology common to user experience and design, processes for viewing and using the GMA, characteristics of the GMA design (see Appendix C. Genomic Medicine Assistant Demo Multi-Gene Panel Laboratory Reports), and roles for different stakeholders (e.g., patient, lab professional, ordering physician, genetic counselor). Our assessment of user experience was broader in phase 1 than in phase 2 with the addition of codes for content and usefulness added to our codebook. Codes for usability and design remained. Codes for organizational and business processes from phase 1 were replaced by a single code to capture processes for viewing and using the GMA. Mappings of statements to codes facilitated identifying preferences for and issues with proposed design features and areas for which there was agreement or disagreement among focus group participants.

3.3.3. Analysis to Characterize Study Participant Views Using System Usability Scale Survey

In order to characterize the views of our study participants, we used the System Usability Scale (SUS) survey [33]. The SUS has become a standard of usability measurement [34]. While other usability evaluation surveys exist [35,36], the SUS is among the most broadly used and has a number of advantages including its ease of use with only 10 items rated on a 5-point scale; its comprehensibility given a resultant score ranging from 0 to 100; and it is “technology agnostic” [33,37]. Participants are asked to score each item on a scale of one to five that range from Strongly Disagree to Strongly Agree. Scores were interpreted and analyzed using Sauro's method (30). For each odd numbered item, one was subtracted from the user response. For each even numbered item, the user response was subtracted from five. Each user's responses were added up and multiplied by 2.5 to convert the range of values to 0–100. A score of 68 is an average usability score. A score above 68 would be considered above average, a score below 68 is considered below average. Scores were then converted to their percentile rank [33].

4. Results

4.1. Study Population

Phase 1 focus groups (three totals) were conducted between December 2014 and February 2015. Phase 2 focus groups with clinicians (two totals) were conducted in May 2015 at UMMC. Each focus group had a total of four participants (Table 1). With the exception of one focus group conducted during phase 1 with TPP/PDMP research team members, all focus groups included 1–2 nurse practitioners and 1–3 physicians. 6 clinicians (nurse practitioners and physicians) participated in both phases 1 and 2.

Table 1. Demographic Information collected about focus group participants: Gender, specialty, years of experience.

Characteristics	N (%)
Gender
Female	13 (65)
Male	7 (35)
Study
Study 1	12 (60)
Study 2	8 (40)
Profession
Clinical Pharmacist	1 (5)
Physician (Fellow)	13 (65)
Nurse practitioner	2 (10)
TPP/PDMP research team (clinical research coordinators)	3 (15)
TPP/PDMP research team (laboratory professionals)	1 (5)
Clinical specialty
Cardiology	8 (40)
Endocrinology	8 (40)
NA	4 (20)
Years of experience (mean, range)	6.7, 1–35

4.2. Phase 1 Results: Validated Scenarios and Proposed Software Design

The phase 1 analysis uncovered several desired characteristics for scenarios and for the design of software to support processes for documenting, viewing, and being notified of genetic and laboratory test results.

4.2.1. Feedback From Clinicians Informing Genomic Medicine Assistant design

Preferences for documenting and viewing genetic test results informed the design of the GMA software. Specific clinical preferences for report documentation included embedding recommendations for the course of treatment based on the patient's results. This design choice was based primarily upon discussions occurring in the TPP focus group session. The goal for this would be to reduce the number of steps required for clinicians to review recommendations for using test results. At the time, recommendations were provided to clinicians primarily within a letter sent by the study team.

TPP and PDMP clinicians had similar preferences for viewing test results. Given that multiple variants in a single gene or multiple genes may be presented in a single report, participants expressed a desire for data to be organized by degree of urgency and clinical significance. We proposed addressing this issue by proposing the use of a multiple tabs to separate essential and nonessential information in the proposed software design. We also incorporated a specific recommendation of the TPP clinicians to use to allow for quick review of the report (e.g., use of the color red to indicate high urgency).

Decisions regarding the organization of content were also informed by model laboratory reports and recommendations from the literature [31]. Many features are generalizable across scenarios and are described in Table 3. Potentially generalizable content and content sections are also illustrated in Fig. 1. Given differences in the availability of information resources, the inclusion or exclusion of infobuttons varied slightly between scenarios.

Table 3. Proposed genomic medicine assistant software design features.

Deign feature	Description
Screens
Inbox for test reports	This screen contains sortable patient demographic information and the test report status for quick selection of reports
Results tab	This screen contains patient demographic information, test indication, clinically relevant variant results, results interpretation, and recommendations
Test Information tab	This screen contains information about the test method and limitations
Infobuttons
Gene (all scenarios)	This button opens a new window that displays a gene resource (e.g., OMIM [47], a resource that provides information about human genes and genetic phenotypes, including supporting references to confirm genephenotype relationships)
Variant information (all scenarios)	This button opens a new window that displays a gene variant resource (e.g., ClinVar [48], a resource provides information about the variant identified and the classification of the variant)
Clinical significance (TPP scenario 2 and PDMP scenarios) [NOTE: this button is not shown in Fig. 1]	This button opens a new window that displays a resource to assist with assessing clinical significance resource (e.g., GeneReviews [49], a resource provides information on genetic diseases, focusing on clinically relevant and medically actionable information on the diagnosis, management, and genetic counseling of patients and families with specific inherited conditions)
Evidence (all scenarios)	This button opens a new window that lists citations or presents a resource that synthesizes evidence to support interpretations and recommendations included in the laboratory report
Patient resource (all scenarios)	This button opens a new window that displays a patient resource (e.g., Genetics Home Reference [50], a resource that contains patient-friendly information about the gene tested and disease-associated information)

Fig. 1.

Results tab features. This example shows a pharmacogenomic sequencing panel report. Note: this image was modified to illustrate the generalizable content sections and features including tabs and infobuttons embedded in the demo. For screenshots of the Genomic Medicine Assistant demo see Appendix C.

Focus group participants also reported need for sufficient information to help non-specialist clinicians explain test results to their patients. TPP clinicians expressed that the information in current reports was insufficient and that links to external information resources would be beneficial in their decision-making. PDMP clinicians liked the idea of providing ready access to electronic resources with GS reports, with a preference for familiar and trusted resources such as UpToDate^® (www.uptodate.com). They made specific suggestions for access to general information about monogenic diabetes and the ability to print patient-targeted information. These preferences were incorporated in our choice of information resources, organization of content, and incorporating support for print and email capabilities into a proposed software design. Other features incorporated into the GMA design included proposed access to web-based resources using infobuttons and an inbox of all reports available within the GMA.

4.2.2. Validated Scenarios and Proposed Tasks for Reviewing The genomic Medicine Assistant Design

Notification preferences informed the design of our scenarios. Study participants indicated a need for improved mechanisms of being notified of GS and laboratory results. Epic^® Systems (Verona, WI) alert and reminder capabilities in particular were discussed during focus group sessions. Both TPP and PDMP recommended considering alert messages in the EHR as a preferred mechanism to notify and deliver GS reports. One suggestion brought up in the TPP focus group specifically was to consider Epic InBasket messaging. EHR alert and reminder mechanisms (Epic Best Practice Alert [BPA] and InBasket messages) were therefore incorporated into scenarios as two proposed mechanisms for notifying clinicians of GS reports. The BPA provides an interruptive form of clinical decision support notification, while the InBasket provides a more passive form of notification where new messages are indicated on a sidebar as the number of new messages that have been received.

Scenarios are summarized in Table 2. Notification occurred when a clinician opened the patient chart and methods of notification included Epic BPA and InBasket messages. Both contained summary information about GS results and a link to view the report in the GMA. In order to assess both types of notification approaches, the first scenario presented to cardiology and endocrinology clinicians used BPAs, while the second scenario contained an InBasket message.

Table 2. Summary of scenarios.

Lab report in the GMA	When opening a new encounter, the healthcare provider is notified…
TPP Scenario 1: BPA notification of a pharmacogenomic sequencing panel report (CYP2C19)	…that their patient is a carrier of a genetic variant that causes a reduction in the ability to metabolize clopidogrel. The message contains a link to view the lab report in the GMA
TPP Scenario 2: InBasket notification of a pharmacogenomic sequencing panel report (SCN5A)	…that their patient is a carrier of a pathogenic variant in the SCN5A gene relevant to Brugada Syndrome (an incidental finding). The message contains a link to view the lab report in the GMA
PDMP Scenario 1: BPA notification of a monogenic diabetes sequencing panel report (HNF1A)	…that their patient has a result consistent with a diagnosis of a genetic form of diabetes (MODY3) caused by a mutation in HNF1A. The message contains a link to view the lab report in the GMA
PDMP Scenario 2: InBasket message notification of a monogenic diabetes sequencing panel report (HNF1A)	…that their patient has a result consistent with a diagnosis of a genetic form of diabetes (MODY3) caused by a mutation in HNF1A. The message contains a link to view the lab report in the GMA

Tasks outlined for the scenarios involved a user viewing an EHR alert or reminder message, clicking on a link within that message to access the GMA software, then viewing the GMA inbox containing the status of all ordered GS tests and reports. The user opens the pre-selected GS report and is brought to a lab report Results Tab that contains patient demographic data, the result, clinical significance, and recommendations. The Results Tab page also contains infobuttons that link to resources such as OMIM^® [47], GeneReviews^® [49], ClinVar [48], supporting literature, and a patient resource. The user then clicks the Test Information tab, which contains test limitations and other information. See Fig. 1 for an example of the Results Tab of the report.

4.3. Phase 2 Results: Clinician Perceptions of Tasks and Screen Mockups

Two main themes came out of the phase 2 focus group analysis. First, there were differing opinions about proposed notification methods. Second, there were user-specific needs for information and consultation (Table 4).

Table 4. Phase 2 focus group excerpts that support major themes.

Theme	Excerpts
Differing opinions about notification methods	“I'm personally not a fan, that [the alert] needs to be closed. […] It stops what you're doing. It will do nothing but aggravate providers” –Cardiology Fellow, scenario 1
	“ At the end of the day, […] you're going through a ton of stuff and you just see words and words and words, and word. So you have that danger [of alert fatigue] – I'm just going to breeze through that” – Endocrinology Fellow, scenario 3
	“ I wouldn't want a box to pop up and stop what you're doing every time, but if the person is on Plavix and has any test that says they're a poor metabolizer, they should at least be noted on the side – like a beacon, something that's alerting you to it without actually stopping what you're doing.” –Cardiology clinician, scenario 1
	“I would probably do both [methods of notification]. I think the problem is this is – this is potentially important information…for the physician” – Cardiology clinician, scenario 2
More information needed to support understanding	“At least having some background on that [the content for clopidogrel poor metabolizers] to get what it means [to “effectively metabolize”] …. That [the common variants table] actually might be a little bit vague for a provider who's not familiar with the test.…It's a small change, but it may make a difference for a lot of people that don't understand the test well…. It adds a degree of uncertainty”–Cardiology clinician, scenario 1
	“[…] what if in the variants table where it says ‘clinical significance’, that column, if you say ‘strong’ – [where] it says ‘pathogenic variant in gene associated with MODY 3.’ What if you [also] put a [classification of pathogenicity] or something, just so you know how much evidence backs it up.” –Endocrinology clinician, scenario 3
Differences in user's decisions to refer to specialist	“ […] I would explain and say I don't have all the information. What I could tell you is this… but I think we're going to a genetic – for a genetic consultation is probably in the best interest.” – Cardiology clinician, scenario 2
	“I think the reason to get a genetic person involved is not so much just for the patient but for family members as well, because they're going to have a lot of other questions, regarding ‘what about my children’… and I think that's going to be something that the genetics people would have to sort out for you.”– Cardiology clinician, scenario 2
Information needed within the GMA may differ based on the user	“I still think the PCPs wouldn't be comfortable to [make a clinical decision] and that's not surprising.” – Endocrinology clinician, scenario 3
	“I mean, based on [the resources provided] I would make a decision” – Cardiology clinician, scenario 1

For both of the EHR alert and reminder notification methods we explored, participants expressed concerns about the frequency of being notified of GS reports (Table 4). For BPAs specifically, the potential for a high frequency of pop-up BPA alerts was a concern. Participants agreed that the way the GMA Genetic Test Report tab was structured to provide optional access to information resources along with concise information summaries could help to overcome issues with alert fatigue (or desensitization to frequent alerts) [38]. Which methods of notification were preferred also depended on the scenario. For example, TPP clinician participants preferred both methods of notification for the pharmacogenomic sequencing panel report SCN5A result scenario, where as they preferred only one method of notification for the CYP2C19 result scenario. Study participants also noted that the preferred method of notification might depend on familiarity with the scenario for using GS results.

User-specific needs for information and consultation became evident from comments about clinicians' familiarity with scenarios, comfort with using GS results and the GMA to make clinical decisions, and preferences for referral processes (Table 4). Participants explained that confidence with using the GMA to make clinical decisions and their preferred referral processes may differ for a non-specialized primary care provider (PCP) who may be unfamiliar with the GMA and/or GS results. Many participants agreed that they would seek consultation from a genetics or pharmacogenetics specialist after viewing the GS report.

4.4. Phase 2 Results: Design Revision Recommendations and Perceptions of Usability

Study participants offered recommendations to improve the navigation of the software to be more intuitive and the content more easily and quickly understood (Table 5). These included specific content improvements such as more clearly defining certain phenotypes. One important recommendation was to add a resource to view evidence summaries regarding the clinical utility of the data such as supporting clinical trials. Participants also perceived the GMA design to be usable with scores ranging from 72.5 to 95 out of 100 on the SUS Survey. These scores representing the perceptions of our study participants, indicated an above average usability score of the GMA.

Table 5. Phase 2 focus group design recommendations.

Timing and navigation recommendations Alert and reminder messages: – Improve the visibility of the alert/reminder message link that would bring the user to the GMA software – Provide with the alert/reminder message, an option to refer the patient to a genetics professional Throughout the software: – Use mouse-overs to display definitions and spell-out acronyms Inbox: – Provide support to archive reports, rather than to delete reports – Provide support to indicate that a report has been reviewed – Use color to indicate the status of a report (e.g., indicate that a final report is available to view with the color green) – Include a “Result” column that shows a short description of the result Results tab: – Make links to resources imbedded in the text more visible (e.g., UpToDate® [51] in the Monogenic Diabetes Sequencing Panel - Results Page (HNF1A), see Appendix C, Fig. C.5) – Format the “Evidence” button bibliography to include the abstract of references

Content recommendations Alert and reminder messages: – Include with generic drug names, the commercial names in parentheses – Consider the order of treatment alternatives (e.g., Prasurgrel was recommended to be listed first in the TPP focus group) – Consider including the same recommendation on the “Recommendations” section of the lab report as in the alert and reminder messages TPP example: Add contraindications for prescribing alternative medications included in the “Recommendations” section to the alert message PDMP example: Introduce link to UpToDate® [51] in the “Interpretation Summary” section earlier, possibly in the alert message Results tab – Include with the “Evidence” button bibliography, citations from clinical studies Pharmacogenomic Sequencing Panel Report (CYP2C19): – Bring table of common variants to Results tab [NOTE: this was incorporated into the demo report, see Appendix C, Fig. C.1] – Define interpretation categorizes (e.g., It was recommended to include quantitative measures of decreased metabolism for poor metabolizers) Monogenic Diabetes Sequencing Panel (HNF1A): – “Supporting literature” button should be called “Evidence” or “References” to minimize confusion about the type of information provided. [NOTE: this was incorporated into the demo reports shown in Appendix C]

5. Discussion

We have proposed a design for the Genomic Medicine Assistant (GMA) software for generating a report from GS data and provide resources to orient clinicians when viewing and interpreting results. In phase 1 of our study, focus group participants offered insights into the complex reporting process of genetic test results within two precision medicine programs (TPP and PDMP). Key areas of improvement were identified and considered when defining scenarios and design features for the GMA. In phase 2 we assessed the GMA soft-ware design by conducting focus groups and task walkthroughs with clinical end-users (physicians and nurse practitioners). Major findings we identified were the existence of (a) differing opinions regarding preferred methods of notification and (b) user-specific needs for information and consultation.

Familiarity with a technology can assist with its integration into everyday routines. To bolster clinician familiarity with information provided in the GMA, clinician training and educational programs may be effective. For example, one study piloted a “Genome Report” containing whole genome sequencing results with non-geneticist physicians. They found that users viewed the report initially as daunting, but noted that as they became more familiar with the report, they would be able to better manage subsequent reports [39]. In our study, while familiarity with genetic test results differed between clinicians involved in our two precision medicine programs, there were common preferences for information. For example, regardless of previous experience and education about GS described in our scenarios, many participants agreed that they would likely seek consultation from a genetics or pharmacogenetics specialist after viewing the test report.

In addition, participants made several recommendations to improve the proposed design of the GMA that can serve as design principles for other pursuing similar efforts. Focus group participants reported a desire for information to support more in depth understanding of GS results. Addressing this desire would need to be considered with care given findings from others that using too much bioinformatics or genetics jargon causes confusion [21]. In addition, while actual navigation of the GMA software interface could not be tested during our task walkthroughs, participants still proposed several areas to improve (Table 5).

Toward improving the likelihood of successful integration with busy healthcare environments, we employed user-centered design approaches to understand genetic testing workflow processes and design preferences in order to understand points where technology could help support those processes (see “Summary of Findings from Analyzing Translational Pharmacogenetics Program Genetic Testing Processes” and [30]) and to provide evidence for scenario and software design choices. We found that the documentation of GS results by multiple individuals, at different points in the testing process, and into independent locations can potentially lead to the incorrect communication of a result. We therefore proposed mechanisms for clinical decision support notification in our scenarios and proposed design features for documenting and viewing GS reports with potential to mitigate some of those issues. Overall the proposed scenarios (Table 2) and GMA design features (Table 3) were well accepted by our study participants. Preferences for notification by EHR alerts or reminders, however, may depend in part on familiarity with GS results and urgency of the scenarios. For example, for GS results that cardiology participants' were less familiar with, they preferred both BPA and InBasket notifications. With CYP2C19 test results, for which they were more familiar, the cardiology participants' preferred InBasket notifications.

5.1. Limitations and Future Directions

We employed a user-centered design approach with the aim of designing software that fits within current ways of working (i.e., GS reporting processes that are similar to current laboratory test delivery processes). There are some processes for GS reporting, however, that are distinct from other types of laboratory tests such as the need to support mechanisms for continually updating and improving genetic information. Our proposed design feature to embed within GS reports links to external references and to continually updated genomic knowledge bases such as ClinVar [40] would partially supports such mechanisms. The scenarios we present in this work, however, were for the delivery of results at one point in time, and do not explore potential changes to interpretations over time. One technology exploring mechanisms to provide updated interpretations of GS results is GeneInsight, whose processes have been shown to greatly improve the probability that users receive updated variant information [41]. Complementing that work would be research to better understand approaches to notify clinicians of updates to GS findings and interpretations. There is potential to draw from approaches in human factors engineering to analyze complex work systems [42,43]. Approaches in contextual inquiry in particular may be used in order to design technology to support current work systems, or to design new ways of working that are well supported by technology [44].

Another area for further exploration is scenarios for the return of incidental findings unrelated to the indication for testing. This work explored a scenario related to TPP project for the return of a potential incidental finding upon ordering a pharmacogenomic sequencing panel test for another purpose. Of note is that the scenario was highly exploratory and not based on current standards of care. While others are exploring similar scenarios for returning GS results with healthy individuals [9], current practice recommendations do not support GS for that purpose [45].

Some challenges brought up in phase 1 focus groups could not be addressed directly in the design of the GMA. For example, a clear need identified by participants was to improve the communication between laboratory systems and the EHR. Focus group participants also reported that some processes for notifying clinicians of genetic test results do not notify the correct members of the care team of an available test result in the TPP project [30]. In both instances, solutions for these needs were beyond the scope of this project. These are however broader implementation challenges that will be addressed in the respective TPP and PDMP projects.

While our prototype can provide a starting point for exploring scenarios for the reporting of multi-gene sequencing test results, further research is needed to determine applicability to other scenarios and healthcare organizations. Future studies would benefit from gathering feedback on multiple design concepts in order identify the best solution for different environments. In addition, usability studies where individuals are able to interact directly with a prototype system may help users' understanding of the GMA application.

Furthermore, mechanisms to improve the communication between laboratories and clinicians will be important. As we found in our assessment of TPP genetic testing processes, laboratory processes for preparing the report are vital for the delivery of results. The communication between clinicians and laboratories becomes increasingly complex when GS is involved. Multiple factors now play a part in this communication and should be considered when building such a connection. These include informed consent, quality of testing, regulations of the testing laboratory, and most importantly, interpretation of results [46]. While the GMA provides a framework for delivering reports to clinicians, laboratory and genetics professionals will continue to guide the development of content for inclusion of GS reports and the identification of appropriate timing for delivering reports to clinicians.

In addition to these findings, study participants perceived the GMA design to be usable and offered numerous design and content recommendations to further improve its look and feel (Table 5). Participant preferences for information resources and organization of content from the first phase of the study were incorporated in the prototype software design presented during the second phase. Above average perceptions of usability were therefore unsurprising. Overall design principles resulting from our study can be incorporated into software applications that support the delivery of GS reports.

6. Conclusion

We employed a user-centered software design approach in a focus group setting in order to facilitate gathering rich contextual information from a diverse group of stakeholders potentially impacted by the delivery of multi-gene sequencing panel (GS) reports relevant to two precision medicine programs at the University of Maryland Medical Center. Our evaluation of the genetic testing processes facilitated identifying preferred mechanisms for clinicians to be notified of results and preferred information resources to support interpreting results. Notification by EHR alerts and reminders were explored within scenarios for ordering and delivering GS reports. Preferences for notification, information and consultation from genetics specialists appeared to depend upon previous experience and education about the GS described in scenarios. Despite familiarity with some aspects of the scenarios, many of our participants agreed that they would likely seek consultation from a genetics specialist after viewing GS reports. Findings illustrate a need to support customized notification approaches, user-specific information, and access to genetics specialists with GS reports. Design principles from this work can be incorporated into software applications that deliver GS reports. Our user-centered approach to conduct this assessment and the specific input we received from clinicians may also be relevant to others working on similar projects.