Best practices for data visualization: creating and evaluating a report for an evidence-based fall prevention program

Abstract

This case report applied principles from the data visualization (DV) literature and feedback from nurses to develop an effective report to display adherence with an evidence-based fall prevention program. We tested the usability of the original and revised reports using a Health Information Technology Usability Evaluation Scale (Health-ITUES) customized for this project. Items were rated on a 5-point Likert scale, strongly disagree (1) to strongly agree (5). The literature emphasized that the ideal display maximizes the information communicated, minimizes the cognitive efforts involved with interpretation, and selects the correct type of display (eg, bar versus line graph). Semi-structured nurse interviews emphasized the value of simplified reports and meaningful data. The mean (standard deviation [SD]) Health-ITUES score for the original report was 3.86 (0.19) and increased to 4.29 (0.11) in the revised report (Mann Whitney U Test, z = −12.25, P < 0.001). Lessons learned from this study can inform report development for clinicians in implementation science.

INTRODUCTION

In today’s healthcare environment, data are continuously being collected at the patient, clinician, and organizational levels.^1,2 Data visualization (DV) techniques are competing with the rate of data collection.^3,4 Data pertaining to patient risk status^5,6 and outcome measures⁷ are constantly presented to clinicians. Previous research has focused on the use of technologies and dashboards⁸ to identify patterns in electronic health record (EHR) data,⁹ but there is a lack of research on best practices for DV of clinician adherence with quality improvement initiatives.

DV is defined by Stephen Few as “the graphical display of abstract information for 2 purposes: sense making and communication.”¹⁰ Effective data displays are crafted based on the message the creators intend to communicate and consideration of the best means to display variables.^11,12 The issue with poor quality reports is that they can lead clinicians to overlook patterns in performance and miss opportunities for improvement.¹³ Reports created by leveraging the innate capacities for pattern recognition among other best practices for data display could be valuable to clinicians and prompt positive practice change.

Previous research shows that tailoring reports to the end users’ knowledge and skills can reduce the cognitive burden associated with report comprehension;^14,15 this is often a barrier to the implementation of evidence-based practices.^15–17 Furthermore, a systematic literature review by Wu et al concluded that future work was needed to create DV frameworks to apply broadly and validate in healthcare.⁷

To address this knowledge gap, this study used a literature review to consolidate best practices from cognitive science and computer science. Lessons learned were applied to data displays for clinician end users and validated by them. DV principles that emerged in the literature were applied to refine reports that were originally created to display nurse adherence with an evidence-based fall prevention program, Fall TIPS (Tailoring Interventions for Patient Safety). Fall TIPS demonstrated a 25% reduction in falls in a randomized controlled trial¹⁸ and has over a decade of evidence to support its use in acute care settings.^19–24 Fall TIPS protocol adherence is measured using the Fall TIPS Audit Tool (FTAT); audit results are the basis for the Fall TIPS Monthly Reports (FTMR). The FTMR focuses on process measures instead of outcome measures because the outcome measures (falls/fall-related injuries) are rare events and often require additional time to collect, process, and share with staff. Furthermore, the need for FTMR improvement was emphasized by nurse feedback at practice committee meetings. The Institute for Healthcare Improvement’s Framework for Spread emphasizes the importance of continuous monitoring and feedback related to the implementation of new initiatives,²⁵ thus, identifying optimal strategies for depicting adherence is in line with improving the quality of healthcare.

OBJECTIVE

The objective of this study was: 1) To identify best practices for DV through a systematic literature review, 2) To apply these principles and collect semi-structured feedback from nurses to iteratively refine the FTMR, and 3) To evaluate FTMR usability. By harnessing best practice standards established in the literature as well as qualitative and quantitative feedback, we sought to establish guiding principles for creating reports for clinician use in implementation science projects and clinical practice. The methods used were tailored for this process.

MATERIALS AND METHODS

This study is a part of a 3-year project to evaluate the generalizability and spread of an evidence-based fall prevention toolkit: Fall TIPS.²⁶ The protocol was approved by the Partners HealthCare human subjects’ committee. Participants included nurses working on general medical and surgical units at a large academic medical center located in the northeastern USA.

Literature review

With the assistance of a medical librarian, we searched the literature published between 1940 and 2019 to identify best practices for communication of quantitative data via visual display and principles for effective DV for clinicians (see Supplementary MaterialFigure S1 for a list of databases, search terms, inclusion/exclusion criteria, and PRISMA diagram).²⁷ Literature review results directed FTMR improvement.

Usability testing

The original FTMR (Figure 1) was created by the research team and used for 6 months in 1 large academic medical center. Qualitative and quantitative feedback from nurses was collected from April–July 2018 (original report) and August–September 2018 (revised report) and applied to improve the FTMR. The Research Computing Core (RCC) at our hospital modified the reports.

Figure 1.

Original Fall TIPS Monthly Report (FTMR). This figure shows the original FTMR, prior to implementing any modifications. It was developed by the research team using Microsoft Powerpoint.

Qualitative data collection

Two authors adapted Few’s 6 requirements for effective data display^11,28,29 into questions specific to the FTMR and used them to guide semi-structured nurse interviews in individual or group settings based on availability (Table 1). Participation was voluntary; nurses were not required to have a DV background to participate. Basic content analysis methods³⁰ were used to interpret qualitative feedback using a 2-person consensus approach for identifying and organizing themes.

Table 1.

Few et al. Requirements for data displays adapted to a Questionnaire to assess fall TIPS monthly reports

Number	Question
1	Does the display clearly indicate how the values relate to 1 another?
	How do the 3 audit questions relate?
	How does your unit/service compare to others?
	How does your unit compare to the aggregate data?
2	Does it make it easy to compare the quantities?
	How easy is it to interpret the bar graph (# of audits)?
3	Are the ranked order values easily recognizable?
	Is it easy to see which unit is doing the best?
	Is it easy to see which units are meeting the 5 audits/month target?
4	Is it clear how the information display should be used?
	Are the takeaways clear?
	Is it clear how this can be used to provide targeted feedback?

Nurses from medical, surgical, or combined medical surgical units were interviewed; some nurses were Fall TIPS champions (FTCs), nurse leaders, and staff nurses. The FTCs completed the Fall TIPS audits for their respective units and disseminated the among staff. Feedback was collected from independent groups of nurses in 2 phases: once based on the original FTMR and once based on the revised FTMR. Interviews were semi-structured and not transcribed.

Abbreviations: FTC, Fall TIPS champion; FTMR, Fall TIPS Monthly Reports; TIPS, Tailoring Interventions for Patient Safety.

Quantitative data collection

Nurses completed a Health Information Technology Usability Evaluation Scale (Health-ITUES) survey. The 20-item Health-ITUES is a validated tool designed to be customizable for use outside its original context.³¹ Each item is based on task-specific concepts using the Technology Acceptance Model³²: “Using [system] is useful in [task].” The Health-ITUES addresses 4 usability factors: quality of work life, perceived usefulness, perceived ease of use, and user control.³² Participants responded on a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree); a Mann-Whitney U test was used for analysis using SPSS.³³ Two researchers modified 15 items to ask about FTMR use and eliminated 5 that were irrelevant if redundant or out of scope (see Supplementary MaterialTable S2). Since the Health-ITUES was adapted for Fall TIPS, Cronbach’s alpha was calculated to retest reliability.

RESULTS

Literature review

After removing duplicates and assessing titles and abstracts for eligibility, 956 articles were retained. Since 791 articles did not meet established inclusion criteria, 165 were retained for full text review. Two researchers identified 54 publications as relevant to the study (references^{8,10–12,14,15,28,29,34–79}; Supplementary Material Figure). Lessons learned from the literature aligned with the qualitative themes (Table 2).

Table 2.

Changes to the Fall TIPS monthly report were rooted in the qualitative themes and lessons learned from the literature

Change to Fall TIPS monthly report	Theme	Subtheme	Principle from literature
Rotated the bars on the graph from vertical to horizontal and integrated “goal” into bar graph (denoted by red line as opposed to a separate box on the report)	Simplified Reports	Ease of Comprehension	The best data displays take advantage of one’s ability to visually process what is seen with limited thinking power, thereby reducing the cognitive burden involved in data interpretation.10,12,15,41,51,59,76 Color should be used conservatively.8,46,64,75
2. Ordered units based on those that submitted the highest number to lowest number of audits	Simplified Reports	Ease of Comprehension	Making the message stand out is vital to creating an effective visual display, which is easily perceived and remembered.15,41,51,62
3. Wrote out the full wording of the Fall TIPS Audit Tool questions	Simplified Reports	Ease of Comprehension	In designing all components of a visual display, it is best practice to simplify while providing sufficient context to orient the viewer through legends, titles, and axis labels.60,73,77,80
4. Changed the term “aggregate” to “average”	Simplified Report	Optimizing Visualization	A data display is effective when it accurately communicates as much information to the audience as directly as possible.10,34,35,40,57
5. Incorporated “aggregate results” into the “Patient Engagement Audit Results” table	Simplified Reports;	Optimizing Visualization;	Choosing the right visual display to represent data is vital in the clear communication of a message to the target audience.35,38,72
6. Added the “number of audits” to the “Patient Engagement Audits” table	Simplified Reports; Meaningful Data	Optimizing Visualization; Leveraging Numeracy	Repetition is an important component of visual literacy42,70 and interactions between different types of display reduce cognitive effort and foster easy interpretation.42,45
	Meaningful Data	Leveraging Numeracy	Data tables are particularly useful when trying to convey precise numbers or compare specific values.10,28,29,44,66,71 There is variability in numeracy and graph literacy in clinicians, but, in today’s healthcare environment, clinicians are more often exposed to numbers over graphs.14
7. Refined the criteria for calculating the top unit	Meaningful Data	Ensuring Accuracy of Metrics	Clinicians are concerned with their performance with respect to goals, which need to be clearly communicated to them.16,17,68
8. Clarified target adherence	Meaningful Data	Goal Clarification
9. Eliminated top champion metric	Meaningful Data	Goal Clarification	Experts emphasize the importance of a high data–ink ratio or ensuring that the ink on a report represents meaningful data.75,81

Usability testing results

Qualitative results

Staff interviews (original FTMR n = 79, revised FTMR n = 72, total n = 151) lasted 5–15 minutes depending on group size. Some nurses were interviewed in pre-existing practice committee meetings (original = 40, revised = 32), while others were interviewed individually or in groups of 2–3 nurses (original = 39, revised = 40). Interviews were conducted until saturation was reached. Two themes emerged emphasizing that clinicians prefer simplified reports (theme 1) that depict meaningful data (theme 2). Subthemes highlighted the importance of easy comprehension and optimizing visualization (theme 1) and the accuracy of the data, goal clarification, and numeracy (theme 2). When evaluating the revised FTMR (Figure 2), suggestions emerged related to optimizing visualization (minor changes related to wording and formatting). This implies that concerns related to FTMR were addressed by the changes made to the original report.

Figure 2.

Revised Fall TIPS Monthly Report (FTMR). Revisions of the FTMR included: 1) rotating the bars on the graph from vertical to horizontal, 2) ordering the units based on those that submitted the highest number to lowest number of audits, 3) writing out the full wording of the Fall TIPS Audit Tool questions, 4) changing “aggregate” to “average,” 5) incorporating “aggregate results” to the “Patient Engagement Audit Results” table, 6) adding “number of audits” to the “Patient Engagement Audit Results” table, 7) refining the criteria for calculating the top unit, and 8) clearly communicating Fall TIPS Audit Tool target adherence.

Quantitative results

A total of 151 nurses from medical and surgical units completed the Health-ITUES survey to evaluate the FTMR (original n = 79, revised n = 72). A reliability analysis using Cronbach’s alpha showed that the survey adapted for Fall TIPS use is reliable (original = 0.96, revised = 0.94). The mean (SD) score based on the survey was 3.86 (0.19) for the original FTMR and increased to 4.29 (0.11) for the revised FTMR (Mann-Whitney U, z = −12.25, P < 0.001). The mean score for all 15 items increased from the original to revised FTMRs (Figure 3; see Supplementary MaterialTable S2 for item-level statistics).

Figure 3.

Health-ITUES Scores by Factor for Original versus Revised Fall TIPS Monthly Reports. Improvements in Health-ITUES scores were observed across all 4 usability factors: quality of work life, perceived usefulness, perceived ease of use, and user control.

DISCUSSION

The changes made to improve the original FTMR emerged in the nurse interviews and were further supported by the literature. These modifications improved visualization but did not change the main metrics communicated in the reports (number of audits and FTAT adherence). DV principles that exist in cognitive science and computer science also apply to clinicians, but the main clinician-specific point that emerged was a strong preference for goal-related metrics. Nurses value reports where information can be found quickly and the visualization techniques facilitate easy comparison between quantities. The changes made to the FTMR to promote the ease of comprehension adhere to Kosslyn’s Psychological Principles of Effective Graphs^38,80 and reduce cognitive workload. Simple changes such as writing out the full questions, avoiding jargon/symbols, integrating information with graphs, and limiting the use of gridlines to increase comprehension are also important.^38,42,54,72 Comprehension improves when the data representation helps viewers cluster information^8,57,70 and recognize patterns. Improvements made to the Patient Engagement Audits table in the FTMR are in line with the idea that our visual processing capabilities allow us to better retain information that stand out from the norm. FTMR evaluation findings also highlight the benefit of communicating important metrics in 2 areas of reports. Reports including both numerical displays and graphical displays are beneficial as they cater to the DV needs of clinicians with varying levels of comfort with numbers. Reports that leverage viewers’ innate ability for pattern recognition and encourage the eye to connect different components of the report are most effective.^12,42,79

Nurses had positive perceptions of FTMR usefulness. Nurses were aware of how to use these reports in discussions with their staff to identify opportunities for improvement. However, the lack of a routine method for dissemination was a barrier to use. Even if validated reports exist, their impact is limited unless the report is widely viewed and used for continuous quality improvement.³⁵ A system for dissemination is a crucial facilitator for report usefulness. To facilitate dissemination, we collaborated with RCC to automate FTMR generation.

Following best practices for DV reduces cognitive workload.^14,15 Given that modifications were made based on best practices identified in the literature and nurses communicated positive perceptions of usefulness, a reduction in cognitive workload is expected. Best practices include the focus on facilitating the ease of comprehension, clarifying goals, ensuring metrics accuracy, leveraging clinicians’ numeracy and graphical literacy, and optimizing visualization.

The clinician-specific best practices for DV identified in this work can be leveraged to create effective reports that communicate feedback on process measures to clinicians, thus potentially improving the quality of care.

Limitations

Limitations of this study include that the FTMR was evaluated with nurses at 1 academic medical center and participants had varying levels of Fall TIPS exposure. Interviews were not transcribed. Given the preexisting system for FTMR dissemination, it was not feasible to have a control group for the qualitative portion of this study. The cognitive workload associated with the FTMR was not directly measured in this study.

Future work

Recommendations for future iterations of the FTMR include displaying temporal trends. Nurses suggested adding a “days since last fall” metric to emphasize the correlation between Fall TIPS adherence and falls. Next steps also include using the questionnaire based on Few’s requirements and the Health-ITUES of Yen et al to evaluate FTMR perceptions and usability at other hospitals.

CONCLUSION

Through a literature review and qualitative and quantitative evaluation of the FTMR, best practices for DV were identified. The novelty of this work is that best practices were validated by clinicians by refining the FTMR. Reports for clinicians need to be quick and easy to comprehend to be used effectively as a tool to disseminate feedback. These reports must contain accurate data and clarify goals. To maximize the benefit of the reports, a systematic approach to dissemination must be in place. If a report meets these criteria, it is more likely to reduce the attentive effort associated with understanding the report and procure positive perceptions of usefulness by clinicians. The lessons learned can be applied to developing reports for continuous monitoring and feedback regarding implementation progress of other programs for clinicians.

FUNDING

This work was supported by the Agency for Healthcare Research and Quality (AHRQ) Grant Number 1R18HS025128-01. The content is solely the authors’ responsibility and does not represent official AHRQ views.

AUTHOR CONTRIBUTIONS

SK, ZB, and PCD designed the study, interpreted the data, and drafted the manuscript. SK, ZB, MC, and MM acquired data. RN created the automated reports. All authors revised and approved the final manuscript.