The design of decisions: Matching clinical decision support recommendations to Nielsen’s design heuristics

Abstract

Objective

While general design heuristics exist for graphic user interfaces, it remains a challenge to facilitate the implementation of these heuristics for the design of clinical decision support. Our goals were to map a set of recommendations for clinical decision support design found in current literature to Jakob Nielsen’s traditional usability heuristics and to suggest usability areas that need more investigation.

Materials and methods

Using a modified nominal group process, the research team discussed, classified, and mapped recommendations, organized as interface, information, and interaction, to design heuristics. A previous narrative review identified 42 recommendations from the literature to define the design and functional characteristics that impact the performance of CDS in terms of provider preference, process of care, and patient outcomes.

Main findings

We matched 20 out of 42 recommendations to heuristics. The mapping reveals gaps in both heuristics and recommendations, identifying a set of Nielsen’s heuristics that are underrepresented in the literature and subsets of recommendations important to design not covered in Nielsen’s heuristics. We attributed this, in part, to the evolution of technology since the inception of Nielsen’s heuristics. The team created a new interaction heuristic: Integration into real-time workflow to consider the needs of the end-user in the clinical space.

Discussion

Clinical decision support has enabled clinicians to better address arising information needs; however there remains a lack of evidence-based guidelines in terms of functional and design requirements.

Conclusion

Results from this review suggest that interaction design principles were not fully satisfied by the current literature of clinical decision support.

1 Introduction

Clinical Decision Support (CDS) is defined as “providing clinicians with clinical knowledge and patient-related information, intelligently filtered, and presented at appropriate times to enhance patient care” [1]. CDS systems are widely used in clinical care, but are often ineffective due to poor usability [2]. Human factors, usability, and human computer interface principles are critical to the success of CDS. By evaluating CDS from a human factors perspective, frustrations and error-prone conditions can be minimized to create tools that are designed with the user in mind. While general design heuristics exist for graphic user interfaces [3,4], it remains a challenge to facilitate the implementation of these traditional interface interaction heuristics for design, and literature lacks studies that evaluate CDS as interactive computer systems. Guidelines for systems design, such as rules specifically focusing on designing graphical user interfaces, require better design and functional characteristics related to specific rules with a focus on clinical usability. Evaluating the design of these interactive computer systems from a human factors engineering perspective can optimize interactions between clinicians and the system, and eventually improve both patient and clinician outcomes.

Jakob Nielsen’s ten general principles for human computer interaction design are recognized not only in the world of design [4], but by healthcare researchers as the gold standard of heuristic evaluation [5–7]. In systems engineering, heuristics are defined as systematically designed procedures that do not guarantee an optimal solution, but provide near-optimal solutions [8]. In the context of computer systems design, heuristics represent broad rules of thumb to achieve optimal design. A heuristic evaluation is a usability inspection method for computer software that helps to uncover CDS design and interface deficiencies and to identify usability problems in human computer interface design. This technique typically requires three or more expert usability evaluators to independently apply a set of usability heuristics to a product, identify violations of the heuristics, and assess the severity of each violation [9].

By identifying successful components of current CDS and mapping them to Nielsen’s set of interaction design heuristics, this study provides insight into CDS evaluation and identifies design characteristics that lack guidance in the literature. The aims for this project were two-fold: i) to map a set of recommendations for CDS design found in current literature to Nielsen’s usability heuristics and ii) to suggest usability areas that need more investigation based on design heuristics that are not adequately covered in the matching process. We include a brief discussion of matched recommendations for each heuristic. In better understanding CDS design recommendations and their adherence to usability heuristics, we can provide guidance to improve interaction design.

2 Material and methods

2.1 Literature search and recommendations pulled

A brief narrative review was conducted from January 1, 2000 to December 31, 2016, and used the literature database PubMed and the peer-reviewed journal The Journal of Human Factors and Ergonomics Society, using the following MeSH terms: “clinical decision support”, “decision support systems”, “clinical alerts”, “alert systems”, AND “characteristics”, “features”, “human factors”, “usability heuristics”, “usability factors”, and “recommendation” [9]. Only one database was used for brevity as well as a snapshot of CDS literature. Articles were included if they were in English, electronic health record (EHR) based, provided specific characteristics in terms of CDS design, and defined success factors used to evaluate CDS performance.

Of the 14 articles identified, 42 recommendations were pulled from the full-text articles, to define the design and functional characteristics that impact the performance of CDS in terms of provider preference, process of care, and patient outcomes [9]. Recommendations “had to represent consensus, defined as recommendations identified in a systematic review and/or a design recommendation made by more than one paper (referenced within the article), demonstrating success factors” [9]. This review found that no reviews have been completed that discussed or recommended design features of CDS with heuristic evaluation. Recommendations from this narrative review were organized into three “I” categories: Interface, Information, and Interaction, further divided by four “P” categories (presentation, placement and positioning, and provision of multiple presentation layers); three “C” categories (clean and concise, content guidance, and consistency); and five “F” categories (fast, fit, feedback, forgiveness, and flexible design) extracted from 14 papers [9]. Included studies assessed CDS success and impact by studying patient/clinician outcomes in clinician workflow [10–23].

2.2 Consensus and evaluation

Members of the research team conducted the initial inclusion of articles, with the final selection of articles discussed in a group setting [9]. A modified nominal group process was conducted with the full-text articles to discuss, classify, and map recommendations to Nielsen’s heuristics.

The research team applied usability knowledge to drive the heuristic evaluation. A codebook was developed based on iterative discussions to assist in matching with the standardized recommendation methods. Recommendations were matched to Nielsen’s heuristics by members of a multi-disciplinary team without prior discussion [24]. Following this independent process, the team discussed each recommendation’s relevance to the applicable design heuristic and a consensus was reached on every recommendation-heuristic match.

2.3 Heuristics

This evaluation utilized Jakob Nielsen’s ten general principles for interaction design (Table 1).

Table 1.

Nielsen’s ten general principles for interaction design.

Principles	Definition
1 Visibility of system status	The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.
2 Match between system and the real world	The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions,making information appear in a natural and logical order.
3 User control and freedom	Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.
4 Consistency and standards	Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions
5 Error prevention	Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action.
6 Recognition rather than recall	Minimize the user's memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
7 Flexibility and efficiency of use	Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.
8 Aesthetic and minimalist design	Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
9 Help users recognize, diagnose, and recover from errors	Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
10 Help and documentation	Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.

2.4 Quantitative analysis of associations

Our mapping process refers to mapping a finite number of items in one set (n1 = 10 design heuristics) to a finite number of items in another set (n2 = 42) design recommendations extracted from the studies included in the narrative review (e.g., recommendations regarding the front size/spacing/color.) In this context, we are utilizing a generalized and simplified version of a point-to-set mapping framework [25], where the items in one set are assigned to items in the others set using an n-to-n mapping. In other words, since there are n1 = 10 design heuristics and n2=recommendations, n-to-n mapping indicates that any one item in set one (a design heuristic) can be matched to 0, 1,…n2 recommendations. Any one recommendation in set 2 can be matched to 0, 1,…,10 design heuristics. Recommendations that matched zero design heuristics were considered as revealing a potential underrepresented area specific to CDS and need for further expansion of Nielsen’s heuristics. The relationship between recommendations and matched design heuristics were quantified with the goal of: i) analyzing the relevance for each recommendation with regards to CDS design by quantifying the number of matches for each recommendation where higher number of matches represent stronger relevance, and ii) analyzing the relevance of the design heuristics with regards to CDS design by quantifying the total number of recommendations matched to each design heuristic. In addition, association rules methods were applied to find patterns of association between recommendations and design heuristics. Association rules explored combinations of recommendation-heuristic matches and searched for groups of matches that are commonly found in a set (e.g., Interface, Information, and Interaction sets) with the goal of identifying rules, such as “if a given combination of design heuristics in set A are matched, then so are they matched in set B with probability > p”.

3 Results

Through this consensus process, we were able to match 20 out of 42 recommendations to design heuristics (Table 2). Among the 22 recommendations that were not matched, five were in the Interface category, five in Information, and 12 in Interaction.

Table 2.

Recommendations taken from CDS literature that has been matched to Nielsen’s design heuristics.

	Design Heuristica	1	2	3	4	5	6	7	8	9	10
Interface	Presentation				1				1
	Make it Simple								✔
	Use appropriate font sizes
	Use meaningful colors
	Ensure acceptable contrast
	Keep presentation consistent				✔
	Deploy space filling techniques
	Make icons bold or bigger in size
	Total Matches				1				1
	Placement and Positioning	1	2		2
	Display information in prominent positions	✔
	Allow for reading left to right		✔		✔
	Localize information		✔		✔
	Total Matches	1	2		2
	Provision of Multiple Presentation Layers								1
	Avoid using only text								✔
	Total Matches								1
Information	Clean and Concise				1				1
	Standardize terminology				✔
	Use concise and effective language								✔
	Total Matches				1				1
	Content Guidance					1	1	1
	Provide a recommendation
	Justify recommendations
	Suggest alternative recommendations
	Provide additional resources
	Evidence-based recommendations as default					✔	✔	✔
	Keep recommendations up to date
	Total Matches					1	1	1
	Consistency				2
	Recommendations from same place				✔
	Same display for healthcare team members				✔
	Total Matches				2
Interaction	Fast	1
	Provide timely feedback	✔
	Reduce the amount of time required for use
	Total Matches	1
	Fit					3	3
	Minimize cognitive load (e.g. mouse clicks)					✔	✔
	Minimize cognitive load (e.g. manual input)					✔	✔
	Reduce screens to facilitate navigation
	Automatically pull data from the EHR					✔	✔
	Navigate to appropriate locations
	Initiate interventions (interactivity)
	Provide a route to get provide specific info
	Adapt behavior according to clinician actions
	Incorporate functions supporting dialogue
	Total Matches					3	3
	Feedback	1				1
	Provide decision support automatically
	Automate alerting	✔				✔
	Request documentation for override
	Total Matches	1				1
	Forgiveness			2
	Allow the user to modify orders			✔
	Integrate a reset button			✔
	Total Matches			2
	Flexible Design							1			1
	Involve the patient
	Utilize adaptive design and feedback
	Provide indication of data availability
	Incorporate functions to support the team
	Give access upon request to extended info							✔			✔
	Total Matches							1			1
	Total Table Matches	3	2	2	6	5	4	2	3	0	1

^aDesign heuristics are written as numbers which correspond to the below list.

3.1 Matched heuristics

3.1.1 Visibility of system status

Three recommendations were found to match the heuristic visibility of system status: Interface recommendation to display information in prominent positions to ensure content is seen [14,22,26,27], interaction recommendations to provide timely feedback [10,14,18,19,23,28–35], and to automate alerting and develop automating prompting of users by the CDS [12,13,17,21,36].

3.1.2 Match between system and the real world

Two recommendations were found to match the heuristic match between system and the real world: the interface recommendations to allow for conventional mapping by reading left to right [37–39] and to localize information, grouping pieces of information together to facilitate on-screen searches [22,26,27,40].

3.1.3 User control and freedom

Two recommendations were found to match the heuristic user control and freedom: the interaction recommendations to allow the user to modify orders [23] and to integrate a reset button [17,20].

3.1.4 Consistency and standards

Six recommendations were found to match the heuristic consistency and standards: three interface recommendations to promote consistent terminology [22,27,40–42], to allow for reading left to right [37–39], and to localize information [22] and three information recommendation, to standardize terminology [22,26,27,40,42], to ensure recommendations come from the same place [19], and to have the same display of basic CDS information for the case at hand for all professionals of the healthcare team [19].

3.1.5 Error prevention

Five recommendations were found to match the heuristic error prevention: one information recommendation, to make evidence-based recommendations the default [12,14,32], and four interaction recommendations to minimize cognitive load (using both mouse clicks and free text as well as reducing the amount of manual input) [11,12,22], to automatically pull data from the electronic health record (EHR) [11–13,19,31,33,37,43–47], and to automate alerting [12,21].

3.1.6 Recognition rather than recall

Four recommendations were found to match the heuristic recognition rather than recall: one information recommendation, to make evidence-based recommendations the default [12,14,32], and three interaction recommendations, to minimize cognitive load [11,12,22] (using both mouse clicks and free text as well as reducing the amount of manual input) and to automatically pull data from the HER [11–13,19,31,33,37,43–47].

3.1.7 Flexibility and efficiency of use

Two recommendations were found to match the heuristic flexibility and efficiency of use: the information recommendation to make evidence-based recommendations the default [12,14,32], and to give access upon request to extended information (justification of the rule, attached scientific documentation, etc.) that should be structured depending on the user profile [16,19,22,37,40,48–50].

3.1.8 Aesthetic and minimalist design

Three recommendations were found to match the heuristic aesthetic and minimalist design: the interface recommendations to make the design simple [11,16,18,22,32] and to avoid using only text [11,20,22], and the information recommendation to use concise and effective language [22,26,27,40,42].

3.1.9 Help users recognize, diagnose, and recover from errors

No recommendations were found to match the heuristic help users recognize, diagnose, and recover from errors.

3.1.10 Help and documentation

One recommendation was found to match the heuristic help and documentation: the interaction recommendation to give access upon request to extended information (justification of the rule, attached scientific documentation, etc) that should be structured depending on the user profile [16,19,22,37,40,48–50].

3.2 Associations

Following the iterative matching process of 42 recommendations to ten design heuristics, we first quantified the number of recommendations matched to each heuristic to compare the heuristics’ applicability. Fig. 1 illustrates the ten design heuristics (“DH”) categorized by applicability where size and color of the circles is determined by the number of matched recommendations (i.e., larger, darker circles refer to more recommendations matching the given design heuristic). Design heuristic 4 (consistency and standards) was matched to the most number of recommendations (six recommendations). Design heuristic 9 is not included because it matched no recommendations. Overall, this visualization highlights the discrepancy between the applicability and practicality of different design heuristics for CDS. While some heuristics are more relevant for clinical interface design, others may not add significant value to the clinical setting.

Fig. 1.

Recommendations from CDS literature quantified and matched to each heuristic. **Heuristic 9: Help users recognize, diagnose, and recover from errors, was not matched to any recommendation.

Next, we evaluated the individual recommendations grouped by type (where types refer to the categories Interface, Interaction, and Information) with regards to number of design heuristics they are mapped to. Fig. 2 quantifies how frequently the recommendations in each category matched to any design heuristics. In Fig. 2, DH stands for design heuristic. “No Rec to DH match” represents the set of recommendations in each type that were mapped to no design heuristics.

Fig. 2.

CDS literature recommendations stratified, quantified, and matched to each heuristic.

Fig. 2 highlights the heterogeneity between the three categories of recommendations derived from 14 papers. The Interaction category contained the highest number of total recommendations (n = 21), although the majority of the recommendations in this group were matched to zero design heuristics (n = 13 out of 21). The Information category had the lowest number of total recommendations (n = 9), and the majority of recommendations in this group were matched to either zero recommendations (n = 4 out of 9) or exactly one design heuristic (n = 4 out of 9). In addition, the Information category was the only group including a recommendation that matched to three or more design heuristics. In the Interface category, the majority of the recommendations matched to zero or one design heuristics (n = 5 and 4 out of 11, respectively). Due to a limited number of matches in each category and the large number of recommendations matching zero design heuristics, the association rule algorithms were not able to identify statistically significant patterns.

4 Discussion

In this follow-up study to a narrative review outlining the current state of CDS design, we successfully mapped a set of recommendations for CDS design to Jakob Nielsen’s ten usability heuristics. The mapping reveals gaps in both heuristics and recommendations, identifying a set of Nielsen’s heuristics that are underrepresented in the literature and subsets of recommendations important to CDS design not covered in Nielsen’s heuristics.

By only matching 20 of 42 recommendations to heuristics, we were able to identify areas of research not represented in Nielsen’s list. The low match rate can be attributed to both the high specificity and age of Nielsen’s definitions. For example, the heuristic “flexibility and efficiency of use” is narrowly defined as “Flexibility and efficiency of use: accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.” While the expansion to more general definitions of flexibility and efficiency would have increased our match rate, we were unwilling to compromise the protocol of the study by changing the definitions of well-established heuristics. Similarly, while respected and heavily utilized in the field of CDS design, Nielsen’s heuristics were published over two decades ago, and the capabilities and context of technology has evolved immensely in those years. We identify the need for additional focus on the design interaction between the user and the system’s interface, the arguably most evolved aspect of technological advancements.

To address the consequences of these two limitations, we contemplated additional heuristics that could be considered when designing CDSs based on the unmatched recommendations. This research identified existing recommendations that do not match any of the ten heuristics, which led to the suggested development of domain-specific recommendations. For example, the team created a new interaction heuristic: Integration into real-time workflow to consider the needs of the end-user in the clinical space. CDS offers powerful technology to aid clinicians in delivering high quality care to patients and reduce costs; however in order for CDS to reach its full potential, the information must be rapidly accessible and understood from within the clinicians’ workflow. This new heuristic supplements Nielsen’s heuristics to form a complete set of guidelines for effectively designing CDSs. Future work should consider additional design heuristics to the list that address the evolving technology of CDS.

In addition to unmatched recommendations, we also identified under-researched heuristics in the field of CDS. In particular, the heuristic “help & documentation” was matched to zero recommendations. This heuristic’s definition is extremely limiting (and, arguably, less relevant) for CDSs. While some CDSs do offer documentation, it is more frequently related to topics beyond instructions for use or troubleshooting. For example, MDCalc, a library of interactive score calculators, offers documentation regarding the source of the data used to develop the scoring system, the appropriate settings and populations to use it in, and its limitations [51]. While the heuristic is helpful in designing manual-like documentation, it is too narrow to guide the development of documentation referring to clinical data or predictive model development aspects of the tool that may be of interest to clinicians. Other heuristics with low match rates (two recommendations each for design heuristics 2, 3, and 7) would more likely benefit from additional research than the modification of the heuristic definition.

Previous research has identified similar challenges with regards to traditional interaction design and interface heuristics when applying heuristics to the evaluation of modern day healthcare technologies [14]. The evolution of EHR, medical devices, and mobile interfaces raises new usability challenges. Traditional heuristics have been modified for application to medical devices [52], radiotherapy systems [53], and health information systems [54]. The application of additional methodologies like ergonomic criteria [55], heuristic walkthroughs [56], and new evaluation checklists [57] has supplemented heuristic evaluation of human-computer interfaces. These methodologies are similar to a heuristic evaluation in that they are easy, fast, cheap, and suitable for every life-cycle software phase while serving as a compilation of recommended best practices. One specific evaluation rearranged traditional heuristics adding the following new heuristic: skills defined as “prepare workarounds for frequent users”; pleasurable and respectful interaction with the user; and privacy [57]. Furthermore, this evaluation expanded up on the traditional heuristic definitions by including subheuristics that, while kept unchanged from their corresponding references, provided increased flexibility. For example, the heuristic Help and Documentation include subheuristics like: “navigation: is information easy to find?” and “is there context-sensitive help?”

There is growing recognition that CDS, when well-designed and implemented, holds great potential to improve healthcare quality, increase efficiency, and reduce healthcare costs [14,32,58]. Optimizing the design of CDS and health information technology in general is of increased importance as providers struggle with high cognitive workloads and challenging workflow [59,60]. Much work has been done to identify features critical to success, evaluating clinician performance, workflow, and building frameworks for implementation [14,32,58,61] and yet, there is little consensus on how CDS should be generated and displayed to the user to optimize response [62]. Successful adoption of CDS requires careful consideration of not only the knowledge driving the alert system but also requires application of human factors principles in alert development and design [62].

5 Conclusion

CDS has enabled clinicians to better address arising information needs with evidence-based guidelines to deliver the best available care; however there remains a lack of evidence-based guidelines in terms of functional and design requirements of CDS. Results from this review suggest that interaction design principles were not fully satisfied by the current literature of CDS. Considering design and usability heuristics in the development and evaluation of CDS is a proactive approach to provide unique insight to address failing CDS and will allow developers and researchers to identify areas for improvement to ensure a concise and practical improved implementation.

Summary Table.

• CDS systems are widely used in clinical care, but are often ineffective due to poor usability.

• General design heuristics exist for graphic user interfaces; it remains a challenge to facilitate the implementation of these heuristics for the design of clinical decision support.

• The mapping reveals gaps in both heuristics and recommendations, identifying a set of Nielsen’s heuristics that are underrepresented in the literature and subsets of recommendations important to CDS design not covered in Nielsen’s heuristics

• This research created a new interaction heuristic: Integration into real-time workflow to consider the needs of the end-user in the clinical space due, in part, to the gaps in Nielsen’s heuristics.

Highlights.

• A previous narrative review identified 42 recommendations of design and functional characteristics for clinical decision support; this paper classified and mapped these recommendations to Nielsen’s heuristics.

• We identified a set of Nielsen’s heuristics that are underrepresented in the literature important to clinical decision support decision.

• We created a new interaction heuristic that was integration into real-time workflow to consider the needs of the end-user in the clinical space.