Abstract
Clinical dashboards that display targets compared to performance metrics are increasingly used by healthcare organizations in their quality improvement efforts. However, few studies have evaluated the extent to which healthcare professionals can readily understand and interpret these data. This study explored associations between measures of graph literacy and numeracy in home care nurses from two agencies (N=195) with comprehension of quality targets presented in a graphical dashboard format. Data were collected using an online survey. Results from linear regression models indicated that nurses’ levels of graph literacy and numeracy were positively associated with comprehension of quality targets. Nurses with low levels of both graph literacy and numeracy tended to have the lowest target comprehension scores compared to those who had high levels of both graph literacy and numeracy. Nurses with low graph literacy and high numeracy also had significantly lower scores for comprehension of quality targets compared to those with high graph literacy and numeracy. These findings suggest that developers of clinical dashboards that incorporate quality target information need to evaluate users’ ability to understand the information displayed in graphs and tables before they release the product for general use in healthcare settings.
Introduction
Clinical dashboards are a form of Health Information Technology (HIT) that use data visualization techniques to provide feedback to healthcare professionals on their performance compared to quality metrics (1). Dashboards are increasingly used by healthcare providers to enhance clinical productivity and track performance on quality improvement efforts. Most clinical dashboards aim to provide timely information to care providers in a succinct visual format. One feature that is often incorporated is the presentation of performance relative to quality targets or benchmarks. In the context of home healthcare services, quality targets might present clinical performance against outcome metrics like hospital readmission and emergent wound infections, which clinicians can use to benchmark their current performance against organization goals (2).
Despite the widespread use of dashboards in clinical settings, researchers have yet to explore the extent to which healthcare professionals can understand and interpret quality targets presented in a dashboard format. Two factors that may influence comprehension of quality targets include numeracy and graph literacy. Numeracy is defined as the ability to understand basic probability and math concepts (2). Graph literacy represents the ability to comprehend information displayed in graph formats (3). Few studies have explored numeracy and graph literacy among health care professionals in general or nursing specifically (3, 4). Those studies that have explored the impact of graph literacy with either the general population or patients have highlighted that individuals with high graph literacy have greater comprehension of information presented in a graphical format, and individuals with low graph literacy have greater understanding if the same information is presented in a number formant (5-7). Studies exploring graph literacy and numeracy in physicians and nurses suggest that the type of graph used to display data interacts with graph literacy to impact data comprehension (3, 4). The present study builds upon earlier research by examining how numeracy and graph literacy skills are related to comprehension of quality targets presented in a graphical format among nurses sampled from two large regional home care agencies. We hypothesized that home care nurses with lower levels of graph literacy and numeracy would be less likely to comprehend graphical data presented to them about clinical quality targets.
Method
The study employed data collected via an on-line survey. Eligibility criteria included: 1) Registered Nurses (RN) who were currently employed at a Certified Home Health Agency (CHHA), and 2) visiting patients once or more per week. Nurses were recruited from two large, not-for-profit home health agencies located in the North East Region of the United States. All potential participants were sent an email invitation describing the purpose of the study and containing a link to the survey website. Reminders to complete the survey were sent to all non-responders two, three and four weeks after the initial email. Participants first completed screening questions to ensure their eligibility to participate in the study, and if eligible they were directed to a demographic questionnaire followed by both the numeracy scale and graph literacy scale. The survey then randomized participants into one of four groups; each group received information in different graph formats (the results of this experimental study have been reported elsewhere). Finally all participants completed the questions related to comprehension of quality target information (the results reported in this paper). On completion of the survey nurses could provide their contact details (separate from survey responses) to receive a $20 gift voucher in recognition of their time. All study procedures were approved by the institutional review boards at Columbia University Medical Center and the Visiting Nurse Service of New York.
Study Measures
Numeracy
Numeracy was measured using the expanded numeracy scale developed by Lipkus et al (8). This is an 11-item scale that evaluates numerical skills such as familiarity with probability, the ability to convert percentages into proportions and risk magnitude. It is widely used as a method for assessing objective numerical competence, with high reliability (Cronbach a ranging from 0.7 – 0.75). The median numeracy score was used to categorize respondents with lower (<8) and higher numeracy levels (≥8).
Graph Literacy
Graph literacy was measured using a scale developed by Galesic and Garcio-Retamero (9) specifically for the health domain, to measure both basic and advanced graph reading skills and comprehension across different types of graphs (9). The scale consists of 13 items and measures 3 levels of graph comprehension: i) the ability to read the data; ii) the ability to read between the data (to find relationships in the data); and iii) the ability to read beyond the data (e.g., to be able to predict a future trend from a line chart). The scale was originally tested in two samples from Germany and the US, with high internal consistency (a = 0.74 and 0.79 respectively), and has been used previously to evaluate the graph literacy of nurses (9). The median graph literacy score was used to categorize respondents with lower (<9) and higher graph literacy levels (≥9).
Comprehension of Quality Targets
Comprehension of quality targets was assessed with four questions that asked nurses to interpret data presented in a table and graph relative to a quality target. The four questions assessing comprehension of quality targets were chosen based on their importance for home health nursing practice. Questions were developed to assess comprehension of quality targets related to the completion of a home health plan of care, visiting high risk home health patients within 24 hours of admission, assessment of hospitalization risk, and tracking the signs and symptoms of patients with chronic illness. An example of one quality target graph and corresponding comprehension question is shown in Figure 1. Binary variables were created to code correct responses and then summed to create a single measure of quality target comprehension that ranged from 0 (no correct responses) to 4 (all correct responses).
Figure 1.
Example of Graph Displaying Quality Target and Corresponding Question Measuring Comprehension.
Demographic Characteristics
Nurse responders were asked about their demographic characteristics at the start of the survey. These characteristics included sex, age, race/ethnicity, education and nurse training level, number of years in a nursing role, number of years working at the agency from which they were sampled, and full-time or part-time status.
Data Analysis
Descriptive statistics were used to examine the demographic characteristics of the sample population, and chi-square or t-test were used to compare these characteristics between the two agencies. Linear regression models were used to estimate bivariate relationships of quality target comprehension with 1) graph literacy, 2) numeracy, and 3) a four-category interactive variable specifying combinations of graph literacy and numeracy. All data were analyzed using R (version 3.1.3).
Results
The survey was conducted between February 16th and May 26th 2016. Email invitations were sent to 1,052 nurses across the two agencies. 322 individuals accessed the survey (30.6 % response rate); of these 125 failed to complete all of the survey questions. Of the 125 respondents who did not complete the entire survey, 23 participants were disqualified because they did not meet all study inclusion criteria and 102 participants gave partial responses before dropping out of the survey. Of those with partial responses, 44 completed screening and/or demographic questions, 8 dropped out before completing the graph literacy scale, 38 dropped out before completing the numeracy scale, and 12 dropped out before completing the data comprehension questions. Overall we received a total of 195 completed responses (129 from Agency 1 and 66 from Agency 2). Reliability of the numeracy scale for our sample was a= 0.68 and for the graph literacy scale a=0.76. Similar to previous studies (9), the numeracy and graph literacy scales were strongly correlated (r=0.56; p<0.001), but with enough variance to indicate that they were measuring different concepts.
The characteristics of nurses who participated in the study are displayed in Table 1. Most nurses were female (89.7%), with an average age of 49 years (SD=11.0). The majority of nurses held a Bachelor’s (52.3%) or Post-Graduate degree (18.5%). Further, most nurses had a Bachelor’s (52.8%) or Post-Graduate nursing degree (11.8%). Nurse respondents differed in their average tenure at their respective agencies, with nurses surveyed at Agency 1 being employed for significantly (<0.001) longer on average (M=12.0; SD=7.7) compared to those surveyed at Agency 2 (M=5.5;SD=6.9). Further nurses surveyed at Agency 1 tended to have higher levels of education compared to those at Agency 2 (<0.001).
Table 1.
Characteristics of Study Participants
| ALL N=195 | AGENCY 1 N = 129 | AGENCY 2 N = 66 | p-value | |
|---|---|---|---|---|
| Female Sex (%) | 175 (89.7) | 115 (89.2) | 60 (90.9) | 0.70 |
| Mean Age in Years (SD) | 49.0 (11.0) | 48.3 (11.0) | 50.5 (11.0) | 0.19 |
| Race/Ethnicity (%) | ||||
| American Indian/Alaska native | 1 (0.5) | 0 | 1 (1.5) | |
| Asian | 28 (14.4) | 23 (17.8) | 5 (7.6) | |
| African American/Black | 46 (23.6) | 33 (25.6) | 13 (19.7) | |
| Native Hawaiian/Other Pacific Islander | 2 (1.0) | 2 (1.6) | 0 (0.0) | |
| Native Hawaiian/Other Pacific Islander | 2 (1.0) | 2 (1.6) | 0 (0.0) | |
| White | 97 (49.7) | 54 (41.8) | 43 (65.2) | |
| More than one race | 11 (5.6) | 9 (7) | 2 (3.0) | |
| Other | 10 (5.1) | 8 (6.2) | 2 (3.0) | |
| Education | <0.001 | |||
| Associate/Diploma Degree (%) | 57 (29.2) | 27 (20.9) | 30 (45.5) | |
| Bachelors Degree (%) | 102 (52.3) | 70 (54.3) | 32 (48.5) | |
| Post Graduate Degree (%) | 36 (18.5) | 32 (24.8) | 4 (11.1) | |
| Nursing Training Level | <0.001 | |||
| Pre Bachelors (%) | 69 (35.4) | 34 (26.4) | 35 (53.0) | |
| Bachelors (%) | 103 (52.8) | 75 (58.1) | 28 (42.4) | |
| Post Graduate (%) | 23 (11.8) | 20 (15.5) | 3 (4.6) | |
| Mean Years as a Nurse (SD) | 19.7 (11.4) | 20.2 (11.4) | 18.7 (11.4) | 0.39 |
| Agency Tenure (Years) Mean (SD) | 9.8 (8.1) | 12.0 (7.7) | 5.5 (6.9) | <0.001 |
| Staff Nurse (vs. Per diem) (%) | 150 (76.9) | 101 (78.3) | 49 (74.2) | 0.53 |
| Mean Total Numeracy Score (SD) | 8.4 (2.0) | 8.4 (2.0) | 8.5 (2.0) | 0.86 |
| Mean Graph LiteracyScore (SD) | 9.7 (2.4) | 9.5 (2.5) | 9.9 (2.2) | 0.34 |
| Mean Target Comprehension Score (SD) | 3.4 (0.8) | 3.4 (0.7) | 3.4 (0.9) | 0.46 |
There was no difference in mean numeracy or graph literacy between nurses at the two agencies; the mean numeracy score for the total sample was 8.4 (SD 2.0) and graph literacy was 9.7 (SD 2.4). The average quality target comprehension score was 3.4 (SD=0.8) out of 4 possible points; this score also did not differ between the two agencies.
In bivariate and multivariate analyses mean graph literacy did not differ by sex, age, race (white vs. non-white), education level, nurse training level, whether the nurse was staff or per-diem, agency, experience (years as a nurse), or agency tenure. Mean numeracy was slightly and inversely associated with years as a nurse (each year of additional nursing experience was associated with a 0.03 lower numeracy score (p=0.014) in bivariate analyses. After adjusting for all demographic characteristics in multivariate analyses, years as a nurse was no longer a significant predictor of numeracy level (p=0.08).
We categorized nurses as having either high (score ≥ 8; N=147, 75%) or low (score < 8; N=48, 25%) numeracy and high (score > 9; N=146, 75%) or low graph literacy (score < 9; N=49, 25%). Nurses were then further categorized into four groups based on their level of numeracy and graph literacy: 1) low numeracy and low graph literacy (n=24; 12%), 2) high numeracy and low graph literacy (n=24; 12%), 3) low numeracy and high graph literacy (n=25; 13%) and 4) high numeracy and high graph literacy (n=122; 63%).
Table 2 presents the results of three linear regression models examining associations between graph literacy and numeracy with comprehension of quality targets presented in a graphical format. Findings from Models 1 and 2 suggest that both high graph literacy and high numeracy, respectively, are positively and significantly associated with level of quality target comprehension. Model 3 explores the relationship of a categorical variable representing the interaction between graph literacy and numeracy with level of quality target comprehension. Results from this model suggest that, compared with nurses who scored as having both low graph literacy and low numeracy, greater comprehension of quality targets is associated with the combination of low graph literacy and high numeracy, high graph literacy and low numeracy, and high graph literacy and high numeracy.
Table 2.
Linear Regression of Quality Target Comprehension on Graph Literacy and Numeracy Scores (N=195)
| Variable | Model 1 Est (SE) | Model 2 Est (SE) | Model 3 Est (SE) |
|---|---|---|---|
| High Graph Literacy | 0.68 (0.13)*** | ||
| High Numeracy | 0.40 (0.13)** | ||
| Graph Literacy and Numeracy Group | |||
| High Graph Literacy and High Numeracy | 1.00(0.17)*** | ||
| High Graph Literacy and Low Numeracy | 1.06 (0.22)*** | ||
| Low Graph Literacy and High Numeracy | 0.67 (0.22)** | ||
| Low Graph Literacy and Low Numeracy | -- | ||
| Model Intercept | 2.88 (0.11) | 3.08 (0.11) | 3.54 (0.68) |
| R2 | 0.13 | 0.05 | 0.17 |
NOTE: *p<0.05; **p<0.01; ***p<0.001
Conclusion
This study identifies the importance of numeracy and graph literacy levels among home care nurses for their comprehension of quality targets presented in graphical displays—a format that is increasingly used by healthcare organizations to share performance feedback with staff. We found variation in both numeracy and graph literacy in our sample population. Our sample of nurses had higher average numeracy and graph literacy skills than that identified in the wider US population (9). Compared to previous studies of nurses, respondents in our study scored higher on measures of numeracy and graph literacy (10). The findings also highlight the importance of considering numeracy and graph literacy when evaluating data presentation formats. Our results suggest that individuals with low numeracy and low graph literacy are less likely to comprehend quality targets compared with those who have high numeracy and high graph literacy. These findings may help to clarify prior research on the impact of clinical dashboards on patient outcomes, by raising the possibility that comprehension of quality targets is an additional factor that may be related to clinical practice (11). Individuals who have low numeracy and graph literacy may have difficulty effectively comprehending quality targets and other data presented in dashboards, potentially influencing their ability to make decisions related to patient care.
The findings of our study have important implications for the design of clinical dashboards that include quality targets. In general, there is an assumption that the presentation of data in a visualized format will improve data comprehension, through reduction in cognitive workload associated with processing that information (12, 13). However, our findings would suggest that consideration also needs to be given to individual variation in clinicians’ ability to understand that information. Although individuals with high graph literacy and numeracy may be able to comprehend quality targets presented in graphical formats, those with low graph literacy or numeracy may not. One unexpected finding was the lack of any significant relationships between demographic characteristics and graph literacy or numeracy level.
Strengths of our study include a large home care nurse sample recruited from two health care agencies, with variation in age, experience, and race/ethnicity, the use of a validated objective measure of numeracy rather than subjective measures used in prior research (4, 10), and a validated measure of graph literacy. Study limitations included that the sample was self-selected by choosing to respond to the survey invite. This sample excludes individuals who started but did not complete the survey. Analysis of partial responses to the survey revealed that those who did not complete the full survey tended to have significantly lower scores on the graph literacy scale, suggesting that they may have been more challenged by the questionnaire and decided not to continue with the survey.
As healthcare organizations expand use of complex data presentations for key performance indicators, the ability of end-users to comprehend that data will represent an important linchpin in efforts to improve healthcare quality. Further research is needed to explore numeracy and graph literacy in nursing populations across health care settings and to investigate the influence of these factors on clinical practice and patient outcomes. In addition, future research should investigate how best to provide support to clinicians to enable them to comprehend dashboards that include data visualizations such as quality targets. Researchers may also seek to examine whether providers in
References
- 1.Daley K, Richardson J, James I, Chambers A, Corbett D. Clinical dashboard: use in older adult mental health wards. The Psychiatrist Online. 2013;37(3):85–8. [Google Scholar]
- 2.Russell D, Rosenfeld P, Ames S, Rosati RJ. Using technology to enhance the quality of home health care: Three case studies of health information technology initiatives at the Visiting Nurse Service of New York. Journal for Healthcare Quality. 2010;32(5):22–9. doi: 10.1111/j.1945-1474.2010.00098.x. [DOI] [PubMed] [Google Scholar]
- 3.Elting LS, Martin CG, Cantor SB, Rubenstein EB. Influence of data display formats on physician investigators’ decisions to stop clinical trials: prospective trial with repeated measures. Bmj. 1999;318(7197):1527–31. doi: 10.1136/bmj.318.7197.1527. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Febretti A, Sousa VEC, Lopez KD, Yao Y, Johnson A, Keenan GM, et al. One Size Doesn’t Fit All: The Efficiency of Graphical, Numerical and Textual Clinical Decision Support for Nurses. 2014 [Google Scholar]
- 5.Gaissmaier W, Wegwarth O, Skopec D, Muller AS, Broschinski S, Politi MC. Numbers can be worth a thousand pictures: individual differences in understanding graphical and numerical representations of health-related information. Health Psychol. 2012;31(3):286–96. doi: 10.1037/a0024850. [DOI] [PubMed] [Google Scholar]
- 6.Okan Y, Galesic M, Garcia-Retamero R. How People with Low and High Graph Literacy Process Health Graphs: Evidence from Eye-tracking. Journal of Behavioral Decision Making. 2016;29(2-3):271–94. [Google Scholar]
- 7.Okan Y, Garcia-Retamero R, Cokely ET, Maldonado A. Individual Differences in Graph Literacy: Overcoming Denominator Neglect in Risk Comprehension. Journal of Behavioral Decision Making. 2012;25(4):390–401. [Google Scholar]
- 8.Lipkus IM, Samsa G, Rimer BK. General Performance on a Numeracy Scale among Highly Educated Samples. Medical Decision Making. 2001;21(1):37–44. doi: 10.1177/0272989X0102100105. [DOI] [PubMed] [Google Scholar]
- 9.Galesic M, Garcia-Retamero R. Graph literacy: a cross-cultural comparison. Med Decis Making. 2011;31(3):444–57. doi: 10.1177/0272989X10373805. [DOI] [PubMed] [Google Scholar]
- 10.Lopez KDPMPH, Wilkie DJPRNF, Yao YP, Sousa VPMSNRN, Febretti AMS, Stifter JPMSRN, et al. Nurses’ Numeracy and Graphical Literacy: Informing Studies of Clinical Decision Support Interfaces. Journal of Nursing Care Quality April/June. 2016;31(2):124–30. doi: 10.1097/NCQ.0000000000000149. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Dowding D, Randell R, Gardner P, Fitzpatrick G, Dykes P, Favela J, et al. Dashboards for improving patient care: Review of the literature. International Journal of Medical Informatics. 2015;84(2):87–100. doi: 10.1016/j.ijmedinf.2014.10.001. [DOI] [PubMed] [Google Scholar]
- 12.Van Der Meulen M, Logie RH, Freer Y, Sykes C, McIntosh N, Hunter J. When a graph is poorer than 100 words: A comparison of computerised natural language generation, human generated descriptions and graphical displays in neonatal intensive care. Applied Cognitive Psychology. 2010;24(1):77–89. [Google Scholar]
- 13.Hutchinson J, Alba JW, Eisenstein EM. Heuristics and biases in data-based decision making: Effects of experience, training, and graphical data displays. Journal of Marketing Research. 2010;47(4):627–42. [Google Scholar]