To the Editor
Electronic health record (EHR)-based alerts are often used to notify practitioners of abnormal test results, but follow-up failures (missed results) continue to occur in outpatient settings.1 In the Department of Veterans Affairs (VA), abnormal test result alerts are generated automatically for pre-specified abnormal laboratory values or manually by the interpreting radiologist when an unexpected finding is noted.2–4 Factors such as workflow, user behaviors, and organizational characteristics likely affect EHR-based test result follow-up.1,5 Thus, we examined the “sociotechnical” predictors of missed test results in the setting of EHR-based alerts.
Methods
Between June 2010 and November 2010, we conducted a cross-sectional survey of VA primary care practitioners (PCPs); trainees and subspecialists were excluded. The survey content was informed by an 8-dimension sociotechnical model6 representing multiple complex facets of EHR-based test result notification. Survey items assessed PCPs’ perceptions of technological factors (e.g., EHR notification software, its ease of use, content of alerts PCPs received, EHR user-interface) and social factors (e.g., workflow, people, and organizational policies and procedures) related to alert follow-up. We assessed potential information overload by asking if practitioners received more alerts than they could effectively manage or received too many alerts to focus on the most important ones.
After pilot testing, we administered the 105-item survey using a web-based survey administration service. To increase response rates, invitation e-mails and reminders were followed by telephone attempts to reach non-respondents. In accordance with VA policies, we did not use monetary incentives for participation.
We defined primary outcomes related to missed test results based on respondents’ answers to two items: Outcome 1 (potential for missed results) based on “The alert notification system in CPRS (Computerized Patient Record System) as currently implemented makes it possible for practitioners to miss test results”, and Outcome 2 (personal history of missed results) based on “In the past year, I missed abnormal lab or imaging test results that led to delayed patient care.”
We examined correlation coefficients to determine significant relationships between sociotechnical variables and each primary outcome. Variables significantly related to the outcomes in bivariate analyses were then included in multivariable regression analyses. For ease of interpretation, we subsequently recoded the outcome responses into dichotomous categories and conducted logistic regression analyses. Responses of “agree” or “strongly agree” were coded affirmative. Independent variables in the logistic regression models were those that were significant for both outcomes in the initial linear models. Stepwise selection was used to identify predictors significantly related to the outcome (p≤0.05).
Results
Of 5001 PCPs invited, 2590 (51.8%) responded. Median number of alerts PCPs reported receiving each day was 63; 86.9% perceived the quantity of alerts they received to be excessive and 69.6% reported receiving more alerts than they could effectively manage (marker of information overload).
Over half (55.6%) reported that the EHR notification system as currently implemented made it possible for practitioners to miss test results. Almost a third (29.8%) reported having personally missed results that led to care delays.
Table 1 indicates the significant predictors in logistic regression. Perceived ease of EHR use was related to a lower likelihood of both the perception of potentially missing results (OR=0.52,95%CI 0.32–0.86) and to reporting missed results that led to care delays (OR=0.64,95%CI 0.43–0.96). Greater concern over electronic hand-offs (i.e., routing alerts to the EHR of a surrogate covering-practitioner) was also related to potential for (OR=2.00,95%CI 1.38–2.89) and personal history of (OR =1.86,95%CI 1.28–2.69) missed test results. PCPs who reported receiving more alerts than is manageable (information overload) were more likely to report having missed results that led to delayed patient care (OR=2.20,95%CI 1.37–3.52). Notably, the number of alerts that respondents reported they received per day was unrelated to either outcome.
Table 1.
Statistically Significant Predictors of Missed Test Results
| Outcome 1: Perception of Potential for Missed Test Results a, b | Outcome 2: Personal History of Missed Test Results a, b | |||||
|---|---|---|---|---|---|---|
| Variable | Odds Ratio | 95% CI | Odds Ratio | 95% CI | ||
| Lower | Upper | Lower | Upper | |||
| Practitioner Characteristics | ||||||
| Native language not English | .63 | .42 | .95 | .54 | .35 | .81 |
| Sociotechnical Characteristics | ||||||
| Finds the alert notification system in CPRS (Computerized Patient Record System) easy to use | .53 | .32 | .86 | .64 | .43 | .96 |
| Number of alert notifications exceeds what they can effectively manage c | -- | -- | -- | 2.20 | 1.37 | 3.52 |
| Receives too many alert notifications to easily focus on the most important ones c | 3.05 | 1.84 | 5.07 | -- | -- | -- |
| Uses additional paper-based methods to help manage test results | 1.72 | 1.23 | 2.42 | -- | -- | -- |
| Consistently notifies patients of abnormal test results | -- | -- | -- | .45 | .25 | .79 |
| Thinks alert notifications related to surrogates create new safety concerns | 1.99 | 1.38 | 2.89 | 1.86 | 1.28 | 2.69 |
| Follows up on all alert notifications received | -- | -- | -- | .42 | .27 | .65 |
Responses of “neither agree nor disagree” were not included in the analyses.
Covariates considered in the stepwise regression but that did not meet criteria for inclusion in the final logistic regression: age, gender, race, job classification, years at VA, has used an electronic medical record system other than CPRS, number of alerts/day, follows up on all high priority alert notifications received.
Marker of Information Overload.
-- Not statistically significant.
Comment
Our data suggest that PCPs using comprehensive EHRs are vulnerable to information overload, which might lead them to miss important information. While the alert quantity data was self-reported, it was strikingly similar to what we found using objective methods querying the EHR of a single VA facility (mean 56.4/day per PCP).7
Our study also suggests an association between usability and patient safety in the context of missed results. Because the EHR and its user-interface is the same across all VA facilities, these perceptions were likely affected by other sociotechnical contextual factors that affect EHR-based test result follow-up. Current efforts to measure and improve usability should encompass a broad, “real-world” context that includes broader sociotechnical facets of the work-environment.6,8 Moreover, an isolated reduction in alert numbers without attention to the broader PCP experience related to information overload might be insufficient to improve outcomes.9
Because this was a cross-sectional survey, we cannot determine causation. Nevertheless, our findings suggest that missed results in EHRs might be related to information overload from alert notifications, electronic hand-offs in care, and practitioner perceptions of poor EHR usability. Interventions to improve safety of test result follow-up in EHRs must address these factors.
Acknowledgments
The research reported here was supported by the Department of Veterans Affairs National Center of Patient Safety, an NIH K23 career development award (K23CA125585) to Dr. Singh and in part by the Houston VA Health Services Research & Development Center of Excellence (HFP90-020). Dr. Sittig is supported in part by a SHARP contract from the Office of the National Coordinator for Health Information Technology (ONC #10510592). The funders did not have any role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. Dr. Singh had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. We would like to thank additional members of our project team: Donna Espadas, BS; Daniel Murphy, MD, MBA; Michael Smith, PhD, and Archana Laxmisan, MD, MA (all affiliated with the Houston VA Health Services Research & Development Center of Excellence) for their valuable contributions to the survey. None of them received additional compensation for this work.
Footnotes
The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.
There are no conflicts of interest for any authors.
Reference List
- 1.Callen JL, Westbrook JI, Georgiou A, Li J. Failure to Follow-Up Test Results for Ambulatory Patients: A Systematic Review. J Gen Intern Med. 2011 Dec 20; doi: 10.1007/s11606-011-1949-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Singh H, Thomas EJ, Mani S, et al. Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential? Arch Intern Med. 2009 Sep 28;169(17):1578–86. doi: 10.1001/archinternmed.2009.263. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Singh H, Thomas EJ, Sittig DF, et al. Notification of abnormal lab test results in an electronic medical record: do any safety concerns remain? Am J Med. 2010 Mar;123(3):238–44. doi: 10.1016/j.amjmed.2009.07.027. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Singh H, Vij MS. Eight recommendations for policies for communicating abnormal test results. Jt Comm J Qual Patient Saf. 2010 May;36(5):226–32. doi: 10.1016/s1553-7250(10)36037-5. [DOI] [PubMed] [Google Scholar]
- 5.Hysong SJ, Sawhney MK, Wilson L, et al. Understanding the management of electronic test result notifications in the outpatient setting. BMC Med Inform Decis Mak. 2011;11:22. doi: 10.1186/1472-6947-11-22. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care. 2010 Oct 1;19(Suppl 3):i68–i74. doi: 10.1136/qshc.2010.042085. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Murphy DR, Reis B, Sittig DF, Singh H. Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis. Am J Med. 2012 Feb;125(2):209e1–7. doi: 10.1016/j.amjmed.2011.07.029. [DOI] [PubMed] [Google Scholar]
- 8.Roth EM, Eggelston RG. Forging New Evaluation Paradigms: Beyond Statistical Generalization. In: Patterson E, Miller J, editors. Macrocognition Metrics and Scenarios: design and evaluation for real-world teams. Burlington, VT: Ashgate Publishing; 2010. pp. 203–19. [Google Scholar]
- 9.Singh H, Wilson L, Reis B, Sawhney MK, Espadas D, Sittig DF. Ten Strategies to Improve Management of Abnormal Test Result Alerts in the Electronic Health Record. Journal of Patient Safety. 2010;6(2) doi: 10.1097/PTS.0b013e3181ddf652. [DOI] [PMC free article] [PubMed] [Google Scholar]