ABSTRACT
BACKGROUND
Failure to follow up microbiology results pending at the time of hospital discharge can delay diagnosis and treatment of important infections, harm patients, and increase the risk of litigation. Current systems to track pending tests are often inadequate.
OBJECTIVE
To design, implement, and evaluate an automated system to improve follow-up of microbiology results that return after hospitalized patients are discharged.
DESIGN
Cluster randomized controlled trial.
SUBJECTS
Inpatient and outpatient physicians caring for adult patients hospitalized at a large academic hospital from February 2009 to June 2010 with positive and untreated or undertreated blood, urine, sputum, or cerebral spinal fluid cultures returning post-discharge.
INTERVENTION
An automated e-mail-based system alerting inpatient and outpatient physicians to positive post-discharge culture results not adequately treated with an antibiotic at the time of discharge.
MAIN MEASURES
Our primary outcome was documented follow-up of results within 3 days. Secondary outcomes included physician awareness and assessment of result urgency, impact on clinical assessments and plans, and preferred alerting scenarios.
KEY RESULTS
We evaluated the follow-up of 157 post-discharge microbiology results from patients of 121 physicians. We found documented follow-up in 27/97 (28%) results in the intervention group and 8/60 (13%) in the control group [aOR 3.2, (95% CI 1.3-8.4); p = 0.01]. Of all inpatient physician respondents, 32/82 (39%) were previously aware of the results, 45/77 (58%) felt the results changed their assessments and plans, 43/77 (56%) felt the results required urgent action, and 67/70 (96%) preferred alerts for current or broader scenarios.
CONCLUSION
Our alerting system improved the proportion of important post-discharge microbiology results with documented follow-up, though the proportion remained low. The alerts were well received and may be expanded in the future.
KEY WORDS: reminder systems, pending test results, transitions of care, test result management, delays in diagnosis
INTRODUCTION
Microbiology results are frequently pending at the time patients are discharged from the hospital.1,2 Many of these results may require a change in antimicrobial therapy.3,4 Failure to follow up on these results in a timely fashion can delay the diagnosis and treatment of infections, leading to patient harm and risk of litigation.5,6 The proper communication of test results that are pending at hospital discharge is becoming more crucial as hospitalized patients are increasingly cared for by hospital-based clinicians rather than their primary care providers (PCPs).7,8 However, this communication process is often hindered by inpatient clinician unawareness of pending test results1, and failure to provide the PCPs with discharge summaries9,10 or list pending tests in the discharge summaries.2,11
Clinicians are often dissatisfied with their systems to manage test results,12,13 and health information technology has the potential to assist clinicians in this important aspect of patient care.14–16 One study found that 38/533 (7.1%) positive post-discharge non-urine microbiology results that were reviewed led to initiation, prolongation or change of antimicrobial therapy in the outpatient setting.4 Comprehensive electronic result management systems have been successfully implemented in ambulatory settings;16,17 however, gaps in test result management remain.18 Initial attempts to improve inpatient test result management have not shown significant impact.12,19 These results have illustrated that the workflow of hospital-based clinicians and the requirement to ensure follow-up of results after discharge from the hospital may necessitate different result management solutions for inpatient providers than those implemented in the ambulatory setting.
The most effective decision support interventions anticipate the users’ needs, fit into their workflow and provide timely information.20 With these goals in mind, our objective was to design, implement, and test an electronic system that alerted inpatient physicians to microbiology results that returned post-discharge and were likely to require a change in antibiotic therapy.
METHODS
Design Overview
We conducted a prospective, cluster randomized controlled trial of electronic alerts for post-discharge microbiology results from February 2009 through June 2010 with clustering at the level of the inpatient physician. We randomly assigned inpatient physicians to receive electronic alerts for potentially untreated infections identified through blood, urine, sputum, and cerebral spinal fluid culture results that returned after their patients were discharged from our hospital. The Partners Institutional Review Board approved the study protocol and granted waivers of informed consent for both the physicians and patients.
Setting and Participants
We conducted our study at a 777-bed academic hospital in Boston, Massachusetts. During the study period, 67,449 hospitalized patients were discharged from 12 medical and 19 surgical or procedural services. The hospital had a well-established computerized order entry system, an electronic system to record and access lists of medications prescribed at hospital discharge, and paper inpatient clinical notes. The outpatient practices affiliated with the hospital used an internally developed electronic health record (EMR) with access to test results obtained during hospitalization.
Inpatient physicians were included in our study if they were listed as the responsible attending providers at the time of discharge for patients of age 18 years or older with clinically important post-discharge blood, urine, sputum, or CSF culture results that were not adequately treated with an antibiotic at the time of discharge. Our criteria for eligible results are summarized in Table 1. These criteria have been evaluated at our institution and were found to have a positive predictive value of >50% for potential need for change in antimicrobial therapy.3 Patients were excluded if their code status was comfort measures only at the time of discharge.
Table 1.
Criteria for Clinically Important Cultures
| Culture type | Criteria |
|---|---|
| All | No antibiotic prescribed at time of discharge to which organism found to be susceptible |
| Blood | 50% likelihood of being true positive24 |
| Urine | >100,000 colony forming units and growth of ≤2 organisms |
| Sputum | Growth with a discharge diagnosis of pneumonia* |
| CSF | Any growth |
*Diagnosis based on ICD-9 codes
Randomization and Interventions
We employed a pseudorandomization technique to assign all inpatient physicians to control or intervention groups based on the odd or even status of their physician identification numbers. Our goal was to create an alert that aggregated patient-specific data and provided physicians with enough information to properly triage post-discharge microbiology results. We developed an e-mail-based alert that included patient demographic information, the inpatient and outpatient physicians responsible for the patient, hospital admission and discharge dates, culture result, discharge medication list, and patient allergy information. A sample alert is shown in Figure 1.
Figure 1.
Sample alert for post-discharge microbiology result.
An overview of the study is shown in Figure 2. Both groups received the standard care in our institution regarding notification of post-discharge microbiology results. For post-discharge blood and CSF cultures, the microbiology laboratory attempted to contact the ordering or attending physician by phone. Sputum and urine cultures did not trigger phone calls from the microbiology laboratory. Reports for microbiology results were available in the EMR but were not sent directly to inpatient or outpatient physicians.
Figure 2.
Overview of study design.
Our intervention supplemented the standard care in our hospital. Each day, an automated algorithm evaluated our hospital discharge database, electronic discharge medication lists, and all culture results from our microbiology laboratory to identify post-discharge microbiology results that met our eligibility criteria as described in Table 1. In the intervention group, inpatient physicians were sent the alert via e-mail immediately. Our internal e-mail system was protected by password and encryption protocols, ensuring security of patient information. In both the intervention and control groups, the same alert was sent 3 days after the results returned, along with a link to an online survey (www.surveymonkey.com). As post-discharge results often return during a transition from inpatient to outpatient settings, at the time we sent the inpatient physicians alerts, we also sent alerts to the patients’ outpatient physicians if they were within our hospital system. The outpatient physicians were those that the patients identified as their primary care providers at the time of registration in the hospital. Neither recipient could see whether the other had received an alert.
We surveyed the physicians on day 3 after each eligible result. We assessed whether they were aware of the culture result. On a 5-point Likert scale from “strongly disagree” to “strongly agree,” we asked whether physicians agreed that the result changed the assessment or plan for the patient and that the result required urgent action. We also assessed the conditions under which they would like to receive the alerts in the future with the following options: “all cultures finalized after discharge (growth or no growth);” “only cultures with growth;” “only cultures with growth that do not appear to be treated by an antibiotic on the discharge medication list;” “I would prefer no alerts for post-discharge cultures.”
Outcomes and Follow-up
Our primary outcome was the presence in the patient’s outpatient electronic chart of documentation of follow-up action within 3 days of the post-discharge microbiology result. We defined a response as at least one of the following in the patient’s outpatient chart: a note describing follow-up with the patient, and a note with acknowledgement of the result or a new antibiotic prescription. Inpatient providers had access to the outpatient chart and documented post-discharge care in that system. In addition, all discharge orders and discharge summaries were available in the outpatient EMR. Two members of our study staff reviewed the inpatient and outpatient charts in all cases, and any disagreements were resolved through discussion. Reviewers were blinded to whether charts were from intervention or control cases. Upon review, cases for 40 inpatient physicians (24/97 intervention physicians and 16/64 control physicians) were excluded for not meeting the inclusion criteria. In these cases, the results were either acknowledged in inpatient notes or treated with antibiotics prior to the patient’s discharge from the hospital, or the patient received comfort measures only at the time of discharge. After exclusions, there were 73 intervention physicians and 48 control physicians. Secondary outcomes included inpatient physician survey results on day 3, including awareness of the results, whether the result changed the assessment and plan for the patient, the urgency of the result and preferred conditions under which alerts would be sent.
Sample Size
We calculated our required sample size taking into account our clustered design. We assumed that the proportion of cases with evidence of follow-up would be 25% in the control group and 50% in the intervention group. In addition, we assumed an intraclass correlation coefficient (ICC) of 0.15 based on published ICCs of 0.05-0.15 for process variables in primary care21 and assumed an average cluster size of 3. Under these assumptions, with α = 0.05 and a power of 0.8, we required a sample size of 86 patients in each group. This estimate did not acknowledge the categorization of results into urine and other types of cultures. Based on a prior retrospective study of post-discharge microbiology results at our institution, we anticipated that our 16-month study period would provide an adequate sample size.
Statistical Analysis
Our analysis of our primary outcome was performed on an intention-to-treat basis. We used the generalized estimating equation approach to account for clustering within inpatient physicians. We chose patient age, hospital service and culture type as covariates for our model a priori. Patient age was included as we hypothesized that the physicians’ follow-up procedures may be affected by perceived risk of potential clinical decline with older patients. We included hospital service to account for variation in practice patterns in responding to microbiology results between different specialties. We also included patient sex as a predictor to adjust for the difference in the proportion of female patients in the control and intervention groups. For our analysis, each hospital service was placed into oncology, medical, or surgical/procedural categories. The culture type was included to account for differences in perceived urgency among the types of results. We anticipated that urine cultures may be considered less urgent than the other culture types, and we dichotomized this variable into urine and non-urine cultures.
For our secondary outcomes and when comparing our intervention and control groups, we used Student’s t-test for normally distributed continuous variables. We used a Wilcoxon signed rank test for non-normally distributed continuous variables and analyzed categorical variables using Fisher’s exact test. We calculated proportions of responses for each item in our physician survey. For items “the result changed the assessment or plan for the patient” and “the result required urgent action,” we considered responses of “agree” or “strongly agree” to indicate agreement with the statement. All analyses were performed using SAS version 9.2 (SAS Institute, Inc., Cary, NC).
Role of the Funding Sources
This study was funded through grants from CRICO/Risk Management Foundation and the National Library of Medicine. Neither organization played any role in the study design, data collection, or the analysis or interpretation of the results.
RESULTS
From February 2009 through June 2010, we evaluated 229 post-discharge microbiology results for hospitalized patients cared for by 161 inpatient physicians. We excluded 72 results for patients from 40 physicians as they did not meet our inclusion criteria. We then analyzed 157 results for patients from 121 physicians (Fig. 3). The physicians in the intervention and control groups had a similar distribution among medicine, surgery, and oncology specialties. Urine cultures comprised 85% of the results in the intervention group and 93% in the control group (p = 0.13). The proportion of female patients was 62% in the intervention group and 80% in the control group (p = 0.02). The mean age in years, median length of stay and race were similar between the two groups (Table 2). We found evidence in the EMR of follow-up within 3 days for a significantly higher proportion of results in the intervention group than in the control group [28% vs. 13%; adjusted OR 3.2 (95% CI 1.3-8.4); p = 0.01] (Table 3). Additional analyses included the Elixhauser comorbidity score22, discharge disposition and whether the outpatient provider received an alert. These additional variables were not significant predictors of our outcome and did not affect our results. They were not included in our final model.
Figure 3.
Study flow diagram.
Table 2.
Baseline Characteristics of Intervention and Control Groups
| Intervention group | Control group | p-value | |
|---|---|---|---|
| Physician factors at baseline | |||
| No. of physicians | 73 | 48 | |
| Specialty, n (%) | 0.33 | ||
| Medicine | 35 (48) | 19 (40) | |
| Surgery | 32 (44) | 21 (44) | |
| Oncology | 6 (8) | 8 (17) | |
| Patient factors at baseline | |||
| No. of patients | 97 | 60 | |
| Mean age, years (SD) | 63.8 (17.3) | 65.4 (19.9) | 0.60 |
| Median length of stay, days (IQR) | 3 (2–6) | 3 (2–7) | 0.81 |
| Culture type, n (%) | 0.13 | ||
| Urine | 82 (85) | 56 (93) | |
| Non-urine | 15 (15) | 4 (7) | |
| Female, n (%) | 60 (62) | 48 (80) | 0.02 |
| Race, n (%) | 0.85 | ||
| White | 74 (76) | 47 (78) | |
| Non-white or unknown | 23 (24) | 13 (22) |
Table 3.
Primary Outcome: Documented Follow-Up of Microbiology Results Within 3 Days
| Variable | Intervention group n (%) | Control group n (%) | Adjusted odds ratio (95% CI)* | p-value* |
|---|---|---|---|---|
| No. of physicians | 73 | 48 | ||
| No. of patients | 97 | 60 | ||
| Documented follow-up | 27/97 (28) | 8/60 (13) | 3.2 (1.3–8.4) | 0.01 |
*Adjusted for patient age and sex, hospital service at time of discharge, culture type, and clustering of patients within physician
Physician Survey
We received survey responses from the inpatient physicians for 83/157 (53%) of the post-discharge results and 19/61 (31%) for outpatient physicians. Of the inpatient respondents, 21/48 (44%) of intervention physicians indicated awareness of the results prior to receiving the survey compared with 11/34 (32%) of control physicians (p = 0.36). For the combined group of intervention and control physicians, 28/75 (37%) of respondents were aware that the test was ordered, 45/77 (58%) indicated that the test result changed their assessment and/or plan for the patient, and 43/77 (56%) indicated that the result required urgent action. The data for the outpatient physician respondents are also presented and show a low proportion of respondents indicating awareness of the results. There were no significant differences between intervention and control groups (Table 4).
Table 4.
Secondary Outcome: Physician Awareness and Assessment of Post-Discharge Microbiology Results
| Survey question | Intervention group n (%) | Control group n (%) | p-value |
|---|---|---|---|
| Inpatient physicians | |||
| Aware of result prior to survey | 21/48 (44) | 11/34 (32) | 0.36 |
| Result changed your assessment and/or plan for the patient | 24/46 (52) | 21/31 (68) | 0.24 |
| Result required urgent action | 27/46 (59) | 16/31 (52) | 0.64 |
| Aware that the test was ordered | 13/45 (29) | 15/30 (50) | 0.09 |
| Outpatient physicians | |||
| Aware of result prior to survey | 5/13 (38) | 1/6 (17) | 0.60 |
| Result changed your assessment and/or plan for the patient | 6/12 (50) | 5/5 (100) | 0.10 |
| Result required urgent action | 4/12 (33) | 4/5 (80) | 0.13 |
| Aware that the test was ordered | 8/12 (67) | 4/5 (80) | 1.00 |
When asked the conditions in which they would prefer to receive the post-discharge microbiology alerts, the 70 responding inpatient physicians (intervention and control groups combined) were fairly equally divided among “All cultures finalized after discharge (growth or no growth),” “Only cultures with growth,” and “Only cultures with growth that do not appear to be treated by an antibiotic on the discharge medication list.” The eight respondents who answered this question more than once responded similarly each time. Again, the data for the outpatient respondents are also shown and indicate that at least half the respondents preferred expanding the scenarios in which post-discharge microbiology alerts are generated. No significant differences were detected between responses for the intervention and control groups (Table 5).
Table 5.
Secondary Outcome: Physician Preferred Alerting Conditions for Post-Discharge Microbiology Results
| Survey question | Intervention group n (%) | Control group n (%) | p-value |
|---|---|---|---|
| Inpatient physicians | n = 40 | n = 30 | |
| All cultures finalized after discharge (growth or no growth) | 9 (23) | 11 (37) | 0.24 |
| Only cultures with growth | 16 (40) | 8 (27) | |
| Only cultures with growth that do not appear to be treated by an antibiotic on the discharge medication list | 12 (30) | 11 (37) | |
| No alerts should be generated | 3 (8) | 0 (0) | |
| Outpatient physicians | n = 12 | n = 4 | |
| All cultures finalized after discharge (growth or no growth) | 3 (25) | 2 (50) | 0.62 |
| Only cultures with growth | 4 (33) | 0 (0) | |
| Only cultures with growth that do not appear to be treated by an antibiotic on the discharge medication list | 5 (42) | 2 (50) | |
| No alerts should be generated | 0 (0) | 0 (0) |
DISCUSSION
We performed a prospective, cluster randomized controlled trial of an automated e-mail-based system to alert inpatient physicians to microbiology results that returned post-discharge and potentially required a change in antibiotic therapy. We found that the system significantly increased the proportion of cases with documented follow-up action in response to the results. However, even after our intervention, less than one third of results had documented follow-up in the electronic chart. In addition, our survey indicated that many of the responding physicians were unaware of these post-discharge results, felt the results changed their assessments and plans for the patients, and valued the alerting system. Our survey results indicated that a sizable proportion of respondents would have preferred a less restrictive alerting algorithm with alerts sent for cultures with growth, regardless of whether an adequate antibiotic had been prescribed at the time of discharge. Given that finding, it may be reasonable to simplify the alerting algorithm to remove the comparisons between the discharge antibiotic list and the antibiotic susceptibilities in the microbiology result.
Our findings extend the work done in prior studies that evaluated post-discharge test results. Roy et al. highlighted the magnitude of the problem of pending test results for patients discharged from the hospital.1 Wilson et al. described a notification system targeting infectious disease specialists with post-discharge non-urine microbiology results, but did not evaluate the impact of the system with a concurrent control group.4 Dalal et al. found that hospital-based clinicians recognized the importance of proper test result management, but failed to adopt an electronic test result management system because of workflow limitations and excessive clinically irrelevant results.12 To improve adoption, we strove to maximize the likelihood that our alerts represented true untreated infections. In addition, we attempted to aggregate useful information in the alert to enhance the efficiency of result follow-up. A similar methodology could be applied to other tests with relatively long delays between test ordering and the return of results that would make them at high risk for follow-up errors. Two important examples include pathology results and tests sent to outside laboratories.
The transition of care between inpatient and outpatient settings is complicated, and the responsibility of ownership of pending test results is often unclear. We targeted inpatient attending physicians for three reasons: (1) the inpatient teams ordered the microbiology tests and best understood their clinical contexts; (2) attending physicians were ultimately responsible for the care delivered by their teams; (3) each hospitalized patient was assigned an inpatient attending physician, but may not have had an outpatient physician. In addition, we targeted physicians rather than staff because we considered the interpretation of microbiology results and determination of whether a change in therapy was warranted to be a physician-level responsibility.
This study had some limitations. First, our system was developed at an institution with internally developed hospital order-entry and discharge medication list systems, as well as a custom-built ambulatory EMR, potentially limiting the generalizability of the results, although many vendor-based solutions support similar functionality.23 Our results provide evidence that in this setting an automated e-mail alerting system can improve follow-up of pending tests and may represent a worthwhile intermediate step in settings without a comprehensive EMR. A second limitation was the low response rate for our survey of physicians, especially in the outpatient setting. Our results may overestimate the positive response to our alerting system because of a response bias. However, the effect on our primary outcome is likely small as an additional analysis did not detect a significant difference in the proportion of test results with documented follow-up between survey respondents and non-respondents. Third, our intervention and control groups were imbalanced in size. With the exception of patient gender, the composition of the two groups was similar. In addition, by adjusting for patient gender in our analysis, we accounted for this difference. Fourth, our outcome measure was a process of care rather than patient outcomes. Finally, we acknowledge that our study examined the documentation of follow-up of results and appropriate actions may be performed but not documented in the chart, leading to ascertainment bias. However, we believe that the intervention and control groups should have been equally impacted by this effect.
In summary, our study showed that our automated electronic alerting system for post-discharge microbiology results improved the proportion of results with documented follow-up, though this proportion remained low. Physicians felt the alerts were valuable, and many wanted alerts for more microbiology results. In the future, a similar methodology could be applied to other tests pending at the time of discharge.
Acknowledgements
This study was supported by grants from CRICO/RMF, Cambridge, MA, and the National Library of Medicine (2T15 LM 007092-17). The funding agencies played no role in the conduct of the study, collection, management, analysis, and interpretation of the data, or the preparation, review, or approval of the manuscript. Dr. El-Kareh had full access to all the data in the study and takes responsibility for the integrity and the accuracy of the data analysis.
Conflict of Interest
None disclosed.
Footnotes
This study was funded by grants from CRICO/RMF and the National Library of Medicine (2T15 LM 07092-17), and was presented at the 34th Annual Meeting of the Society of General Internal Medicine on May 5, 2011 in Phoenix, AZ.
References
- 1.Roy CL, Poon EG, Karson AS, et al. Patient safety concerns arising from test results that return after hospital discharge. Ann Intern Med. 2005;143:121–128. doi: 10.7326/0003-4819-143-2-200507190-00011. [DOI] [PubMed] [Google Scholar]
- 2.Were MC, Li X, Kesterson J, et al. Adequacy of hospital discharge summaries in documenting tests with pending results and outpatient follow-up providers. J Gen Intern Med. 2009;24:1002–1006. doi: 10.1007/s11606-009-1057-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.El-Kareh R, Roy CL, Brodsky G, Perencevich M, Poon EG. Incidence and predictors of microbiology results returning post-discharge and requiring follow-up. J Hosp Med 2011. [DOI] [PMC free article] [PubMed]
- 4.Wilson JW, Marshall WF, Estes LL. Detecting delayed microbiology results after hospital discharge: improving patient safety through an automated medical informatics tool. Mayo Clin Proc. 2011;86:1181–1185. doi: 10.4065/mcp.2011.0415. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Gandhi TK, Kachalia A, Thomas EJ, et al. Missed and delayed diagnoses in the ambulatory setting: a study of closed malpractice claims. Ann Intern Med. 2006;145:488–496. doi: 10.7326/0003-4819-145-7-200610030-00006. [DOI] [PubMed] [Google Scholar]
- 6.Kachalia A, Gandhi TK, Puopolo AL, et al. Missed and delayed diagnoses in the emergency department: a study of closed malpractice claims from 4 liability insurers. Ann Emerg Med. 2007;49:196–205. doi: 10.1016/j.annemergmed.2006.06.035. [DOI] [PubMed] [Google Scholar]
- 7.Kuo YF, Sharma G, Freeman JL, Goodwin JS. Growth in the care of older patients by hospitalists in the United States. N Engl J Med. 2009;360:1102–1112. doi: 10.1056/NEJMsa0802381. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Wachter RM, Goldman L. The hospitalist movement 5 years later. JAMA. 2002;287:487–494. doi: 10.1001/jama.287.4.487. [DOI] [PubMed] [Google Scholar]
- 9.Kripalani S, LeFevre F, Phillips CO, Williams MV, Basaviah P, Baker DW. Deficits in communication and information transfer between hospital-based and primary care physicians: implications for patient safety and continuity of care. JAMA. 2007;297:831–841. doi: 10.1001/jama.297.8.831. [DOI] [PubMed] [Google Scholar]
- 10.Walraven C, Seth R, Laupacis A. Dissemination of discharge summaries. Not reaching follow-up physicians. Can Fam Physician. 2002;48:737–742. [PMC free article] [PubMed] [Google Scholar]
- 11.Walz SE, Smith M, Cox E, Sattin J, Kind AJ. Pending laboratory tests and the hospital discharge summary in patients discharged to sub-acute care. J Gen Intern Med 2010. [DOI] [PMC free article] [PubMed]
- 12.Dalal AK, Poon EG, Karson AS, Gandhi TK, Roy CL. Lessons learned from implementation of a computerized application for pending tests at hospital discharge. J Hosp Med. 2011;6:16–21. doi: 10.1002/jhm.794. [DOI] [PubMed] [Google Scholar]
- 13.Poon EG, Gandhi TK, Sequist TD, Murff HJ, Karson AS, Bates DW. “I wish I had seen this test result earlier!”: Dissatisfaction with test result management systems in primary care. Arch Intern Med. 2004;164:2223–2228. doi: 10.1001/archinte.164.20.2223. [DOI] [PubMed] [Google Scholar]
- 14.Singh H, Wilson L, Reis B, Sawhney MK, Espadas D, Sittig DF. Ten strategies to improve management of abnormal test result alerts in the electronic health record. J Patient Saf. 2010;6:121–123. doi: 10.1097/PTS.0b013e3181ddf652. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Singh H, Wilson L, Petersen LA, et al. Improving follow-up of abnormal cancer screens using electronic health records: trust but verify test result communication. BMC Med Inform Decis Mak. 2009;9:49. doi: 10.1186/1472-6947-9-49. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Poon EG, Wang SJ, Gandhi TK, Bates DW, Kuperman GJ. Design and implementation of a comprehensive outpatient Results Manager. J Biomed Inform. 2003;36:80–91. doi: 10.1016/S1532-0464(03)00061-3. [DOI] [PubMed] [Google Scholar]
- 17.Ferris TG, Johnson SA, Co JP, et al. Electronic results management in pediatric ambulatory care: qualitative assessment. Pediatrics. 2009;123(Suppl 2):S85–91. doi: 10.1542/peds.2008-1755G. [DOI] [PubMed] [Google Scholar]
- 18.Wahls TL, Cram PM. The frequency of missed test results and associated treatment delays in a highly computerized health system. BMC Fam Pract. 2007;8:32. doi: 10.1186/1471-2296-8-32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Graumlich JF, Novotny NL, Stephen Nace G, et al. Patient readmissions, emergency visits, and adverse events after software-assisted discharge from hospital: cluster randomized trial. J Hosp Med. 2009;4:E11–9. doi: 10.1002/jhm.469. [DOI] [PubMed] [Google Scholar]
- 20.Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10:523–530. doi: 10.1197/jamia.M1370. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Campbell MK, Mollison J, Steen N, Grimshaw JM, Eccles M. Analysis of cluster randomized trials in primary care: a practical approach. Fam Pract. 2000;17:192–196. doi: 10.1093/fampra/17.2.192. [DOI] [PubMed] [Google Scholar]
- 22.Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care. 1998;36:8–27. doi: 10.1097/00005650-199801000-00004. [DOI] [PubMed] [Google Scholar]
- 23.Jha AK, DesRoches CM, Campbell EG, et al. Use of electronic health records in US hospitals. N Engl J Med. 2009;360:1628–1638. doi: 10.1056/NEJMsa0900592. [DOI] [PubMed] [Google Scholar]
- 24.Wang SJ, Kuperman GJ, Ohno-Machado L, Onderdonk A, Sandige H, Bates DW. Using electronic data to predict the probability of true bacteremia from positive blood cultures. Proc AMIA Symp 2000; 893–7. [PMC free article] [PubMed]