Abstract
Background:
Follow-up of abnormal outpatient laboratory test results is a major patient safety concern. Electronic medical records can potentially address this concern through automated notification. We examined whether automated notifications of abnormal laboratory results (alerts) in an integrated electronic medical record resulted in timely follow-up actions.
Methods:
We studied four alerts: hemoglobin A1c (HbA1c) ≥15%, positive hepatitis C antibody (HCV), prostate specific antigen (PSA) ≥15 ng/mL, and thyroid stimulating hormone (TSH) ≥ 15 mIU/L. An alert tracking system determined whether the alert was acknowledged (i.e. provider clicked on and opened the message) within two weeks of transmission; acknowledged alerts were considered read. Within 30 days of result transmission, record review and provider contact determined follow-up actions (e.g. patient contact, treatment etc.). Multivariable logistic regression models analyzed predictors for lack of timely follow-up.
Results:
Between May 2008 and December 2008, 78,158 tests (HbA1c, HCV, TSH and PSA) were performed, of which 1163 (1.48%) were transmitted as alerts; 10.2% of these (119/1163) were unacknowledged. Timely follow-up was lacking in 79 (6.8%) and was statistically not different for acknowledged and unacknowledged alerts (6.4% vs. 10.1%; p =.13). Two-hundred two alerts (17.4% of 1163) arose from unnecessarily ordered (redundant) tests. Alerts for a new versus known diagnosis were more likely to lack timely follow-up (OR: 7.35; 95% CI: 4.16-12.97) whereas alerts related to redundant tests were less likely to lack timely follow-up (OR: 0.24; 95% CI: 0.07-0.84).
Conclusions:
Safety concerns related to timely patient follow-up remain despite automated notification of non-life threatening abnormal laboratory results in the outpatient setting.
Keywords: diagnostic errors, abnormal diagnostic test results, electronic medical records, patient follow-up, patient safety, health information technology, communication, primary care
BACKGROUND
Outpatient care is often busy and fragmented, and therefore follow-up of abnormal laboratory test results is prone to error. 1-10 Moreover, many laboratory test results in the outpatient setting may not be immediately life threatening and hence not verbally reported to the ordering provider. Therefore, missed laboratory results or delayed recognition of results leads to a significant potential for outpatient diagnostic errors, adverse events and liability claims.4;9;11-17
Most paper-based methods of communication between laboratories and ordering physicians are especially vulnerable to failures. Automated systems notifying providers about abnormal test results in integrated electronic medical record systems offer a potential solution.18 These systems usually communicate through “alerts” (computerized notifications of abnormal clinical information) transmitted directly to the provider's desktop, facilitating a rapid review of patient information.15 The integrated electronic medical record used at all Veterans Affairs (VA) facilities [the Computerized Patient Record System (CPRS)] uses the View Alert system for automated notification of abnormal laboratory test results.
Abnormal result follow-up, however, will occur only if electronic communication of test results (either through alerts or direct access of test result) is reviewed and acted upon by providers. In previous work, we found abnormal diagnostic imaging alerts may not always be reviewed by ordering practitioners, and practitioners who review them may not always act upon the transmitted results in a timely manner.19 We thus hypothesized that a similar phenomenon would exist for abnormal laboratory results. In this study we evaluated follow-up actions on abnormal diagnostic laboratory tests transmitted as high-priority automated notifications to ordering providers in an integrated electronic medical record. We also determined predictors of lack of timely follow-up of these tests.
METHODS
The study was conducted in a large multispecialty ambulatory clinic of the Michael E. DeBakey Veterans Affairs Medical Center (MEDVAMC) and its five satellite clinics located in Southeast Texas from May 2008 to December 2008. To reliably assess follow-up actions on outpatient test alerts, we only focused on abnormal tests that generated a “high-priority” mandatory automated notification to a specified ordering provider without a concomitant verbal notification. Hence we excluded abnormal tests which:
1) would be potentially life-threatening and hence meet our institution's criteria for verbal notification or would result in immediate hospitalization at certain values (e.g., high potassium values) or
2) were not ordered through computerized provider order entry and hence ordering provider was not consistently specified or
3) did not generate mandatory alerts i.e,. the provider had an option to turn off the notification for a specified level of abnormality.
Four tests met our inclusion criteria: hemoglobin A1c (HbA1c) at a level ≥15%, positive hepatitis C antibody (HCV), prostate specific antigen (PSA) at a level ≥15 ng/mL, and thyroid stimulating hormone (TSH) at a level ≥ 15mIU/L.
Data Collection
While some site specific configurations exist, many automated notification processes in CPRS are similar within the VA system. All VA health care providers receive important clinical information in a “View Alert” window of the electronic medical record screen and life-threatening findings are communicated by telephone. As their only “inbox” for all types of CPRS notifications, providers are very dependent on the View Alert system. Providers see all of their alerts when they first log on and again when they switch between patient records. New alerts remain active in the window for two weeks unless acknowledged, after which they disappear. Providers are expected to click on and open the alert to review the report (considered acknowledged), after which the alert disappears from the window. The provider also may become aware of an abnormal result without clicking on the alert if they were reviewing the medical record for other purposes, hence directly accessing the result. Primary care providers assign surrogate covering providers to receive their alerts when they are out of office.
On a daily basis we queried an Alert Tracking File of the electronic medical record to identify outpatient alerts transmitted two weeks earlier. We tracked acknowledgement status of alerts and extracted additional information such as patient identifiers, names of providers to whom the alert was sent, the date and type of laboratory study. Within a week of data query, a reviewer blinded to acknowledgement status evaluated the medical record using a standardized pre-tested data collection instrument to determine any follow-up actions such as ordering a follow-up test or referral, prescribing or changing treatment, contacting the patient, or patient refusal of additional work-up. Based upon our pilot work, we unexpectedly found a fair number of abnormal tests that were clinically determined to be unnecessary. For instance, we found some tests to be redundant based on the frequency of being ordered i.e., ordered too soon after a previous test or, in cases of hepatitis C, the ELISA was repeated unnecessarily despite a previous positive value. We used objective pre-defined explicit criteria to determine redundancy of tests (criteria specified in Table 3).20
Table 3.
Criteria to determine alerts related to redundant laboratory tests and their respective frequencies (total n=202)
| n | (%) | |
|---|---|---|
| HCV AB (Elisa) (n=131) * | ||
| Known HCV ELISA positive and ongoing follow-up in Hepatitis C clinic ^ | 32 | (24.4) |
| Known HCV ELISA positive and confirmed disease by either PCR+, viral load, or treatment genotype ^ | 104 | (79.4) |
| Known HCV ELISA positive (but no viral load, PCR, or genotype) ^ | 29 | (22.1) |
| PSA (n=60) * | ||
| Documentation of patient refusal for PSA work-up already present before PSA test was ordered | 37 | (61.7) |
| Similar level of PSA that already had appropriate action taken within previous 4-weeks ^ | 21 | (35.0) |
| Patient not a candidate for screening PSA testing | 10 | (16.7) |
| TSH (n=9) | ||
| Test repeated within 4-weeks of last adjustment ^ | 9 | (100.0) |
| Hemoglobin A1C (n=2) | ||
| Test repeated within 3-months of last adjustment ^ | 2 | (100.0) |
Percentages add up to more than 100 because more than one criteria for redundancy may have been met in certain cases
Tests deemed redundant based on repetition
In cases of no documented follow-up, a second study investigator evaluated the medical record to confirm the findings and contacted the provider by telephone to obtain any additional evidence of undocumented follow-up or a decision to not pursue follow-up. If we could not contact any of the patient's providers or if the provider offered no additional information about follow-up, we considered the alert to lack timely follow-up. This determination was made approximately 30 days after alert transmission. In certain cases, we determined calling the provider would be unnecessary or make no impact on outcome either because the alert offered no new information over what was previously documented, the patient was already receiving appropriate care for the condition, or the patient had died. These were considered as timely follow-up; however, categorized as “no-impact”. Not all redundant tests were automatically considered to be of no impact. For example, a redundant positive HCV would still need follow-up if there was none documented for either the new or the previously positive test. Hence, follow-up actions on a redundant test did not make it any less redundant.
Data analysis
In addition to descriptive data, we identified two groups of alerts corresponding to the two outcome variables in our study: 1) alerts lacking electronic acknowledgment versus acknowledged alerts; 2) alerts lacking timely follow-up versus those receiving timely follow-up. We compared the distribution (as proportions) of several independent variables within each group including: ordering provider specialty (primary care, medicine subspecialties, surgery etc.), ordering provider type (physician, trainee, and allied health professionals), redundant tests and alerts signifying a new versus known diagnosis. To assess significance, chi-square test was used for categorical variables and Fisher's exact test was used when chi-square assumptions were not met. Hierarchical multivariable logistic regression models accounting for clustering of laboratory tests by providers were used to identify factors associated with the outcome variables, lack of acknowledgment and lack of timely follow-up. Covariates with p values <0.2 in univariate testing were tested as predictor variables. For multivariable testing, we combined specialties from univariate analysis into three groups: primary care, specialty care and mental health care. The models were fit using maximum likelihood estimation and odds ratios (OR) and 95% confidence intervals (CI) were calculated. We also described the frequencies and types of redundant test alerts.
RESULTS
Between May 2008 and December 2008, 27,092 HbA1c, 22,837 PSA, 6271 HCV and 21,958 TSH tests were performed. A total of 1163 (1.49%) results were electronically transmitted as mandatory high-priority alerts (including 29 HbA1c ≥15%, 448 PSA ≥15 ng/mL, 433 positive HCV, and 253 TSH ≥15 mIU/L.). Acknowledged alerts constituted 89.8% of the total high-priority alerts (1163). Figure 1 illustrates the outcomes of these 1163 alerts. No evidence of documented follow-up action was found in 307 (26.4%) of the alerts, however 213 (18.3% of 1163) cases were categorized as no-impact alerts. In the remaining 94 cases, we called providers to determine if for some reason follow-up had occurred but was not documented or if they had additional knowledge that would affect outcome including patient or provider decision not to pursue follow-up. Of these, 79 alerts (6.8% of total) were determined to lack timely follow-up. Two-hundred two alerts (17.4% of 1163) were considered redundant of which 159 were considered redundant based on repetition (recent or previous test).
Fig 1.
Study Flowchart and Outcomes
Lack of Acknowledgement
Table 1 shows the distribution of the four laboratory tests between the acknowledged and unacknowledged groups. We also show a comparison of several independent variables between the two groups using univariate testing. Ordering provider type and specialty were significantly different across the two groups. Trainees were less likely to acknowledge alerts compared to attending/staff physicians and allied health providers (physician assistants and nurse practitioners) whereas specialty services (including mental health) were less likely to acknowledge alerts compared to primary care providers (p<.0001 for both). In a nested logistic regression model (data not shown in Table 1), the following variables were significantly associated with lack of acknowledgment of the alert, compared to attending physicians: allied health care providers as ordering providers (OR, 4.32; 95% CI: 1.21-15.52) and trainees as ordering providers (OR, 8.39, 95% CI: 2.97-23.68).
Table 1.
Comparison of types of abnormal labs, providers, diagnosis characteristics and test characteristics for acknowledged and unacknowledged alerts
| Unacknowledged n= 119 |
Acknowledged n=1044 |
p-value | |||
|---|---|---|---|---|---|
| n | (%) | n | (%) | ||
| Types of Abnormal Labs Reported | |||||
| HCV AB (Elisa) | 53 | (12.2) | 380 | (87.8) | |
| PSA | 53 | (11.8) | 395 | (88.2) | |
| TSH | 10 | (4.0) | 243 | (96.0) | |
| Hemoglobin A1C | 3 | (10.3) | 26 | (89.7) | |
| Independent Variables | |||||
| Ordering Provider Characteristics | |||||
| Attending physician | 69 | (9.5) | 656 | (90.5) | |
| Allied Health Professionals | 27 | (7.3) | 345 | (92.7) | |
| Trainees (interns, residents, fellows) | 18 | (31.6) | 39 | (68.4) | |
| Other | 5 | (55.6) | 4 | (44.4) | <.0001 |
| Ordering Provider Specialty | |||||
| Primary care | 43 | (5.7) | 706 | (94.3) | |
| Specialty surgery | 5 | (15.6) | 27 | (84.4) | |
| Medicine subspecialties | 31 | (14.8) | 178 | (85.2) | |
| Other non-specified specialties | 8 | (17.0) | 39 | (83.0) | |
| Mental health | 32 | (25.4) | 94 | (74.6) | <.0001 |
| Diagnosis Characteristics | |||||
| Alerts signified newly diagnosed condition | 19 | (7.8) | 226 | (92.2) | |
| Diagnosis already known | 60 | (9.3) | 583 | (90.7) | .46 |
| Considered Redundant | 0 | (0.0) | 36 | (100.0) | .04 |
| No Impact * | 35 | (16.4) | 178 | (83.6) | <.001 |
| Test not ordered by the PCP ^ | 50 | (11.4) | 388 | (88.6) | .3 |
No impact on outcome either because the diagnosis was not new, the patient was already receiving appropriate care for the condition, or had died.
Primary care provider
Lack of Timely Follow-up
Table 2 shows the distribution of the four laboratory tests between the timely follow-up versus lack of timely follow-up groups. We show results of a univariate analysis comparing the several independent variables between the 79 alerts determined to lack timely follow-up versus 1084 that received timely follow-up: Statistically, there was no significant difference in rates of lack of timely follow-up between acknowledged and unacknowledged laboratory alerts (6.4% vs. 10.1%; p=.13). There was no significant difference in provider type but specialty differences were significant. Redundant tests were more likely to receive follow-up (p<.01). Alerts for conditions signifying new diagnoses were more likely to lack timely follow-up than alerts for pre-existing conditions (p<.0001).
Table 2.
Comparison of types of abnormal labs, providers, diagnosis characteristics and test characteristics for alerts with and without timely follow-up at 30days
| Lack of Timely Follow-Up n=79 |
Timely Follow-Up* n=1084 |
p-value | |||
|---|---|---|---|---|---|
| n | (%) | n | (%) | ||
| Types of Abnormal Labs Reported | |||||
| HCV AB (ELISA) | 57 | (13.2) | 376 | (86.8) | |
| PSA | 4 | (0.9) | 444 | (99.1) | |
| TSH | 16 | (6.3) | 237 | (93.7) | |
| Hemoglobin A1C | 2 | (6.9) | 27 | (93.1) | |
| Independent Variables | |||||
| Ordering Provider Characteristics | |||||
| Attending physician | 45 | (6.2) | 680 | (93.8) | |
| Physician Assistants & Nurse Practitioners | 28 | (7.5) | 344 | (92.5) | |
| Trainees (interns, residents, fellows) | 5 | (8.8) | 52 | (91.2) | |
| Other | 1 | (11.1) | 8 | (88.9) | .67 |
| Ordering Provider Specialty | |||||
| Primary care | 50 | (6.7) | 699 | (93.3) | |
| Specialty surgery | 1 | (3.1) | 31 | (96.9) | |
| Medicine subspecialties | 5 | (2.4) | 204 | (97.6) | |
| Other non-specified specialties | 4 | (8.8) | 43 | (91.5) | |
| Mental health | 19 | (15.1) | 107 | (84.9) | <.001 |
| Diagnosis Characteristics | |||||
| Alerts signified newly diagnosed condition | 57 | (23.3) | 188 | (76.7) | |
| Diagnosis already known | 21 | (3.3) | 622 | (96.7) | <.0001 |
| Alert Status | |||||
| Acknowledged | 67 | (6.4) | 977 | (93.6) | |
| Unacknowledged | 12 | (10.1) | 107 | (89.9) | .13 |
| Test not ordered by the PCP | 34 | (7.7) | 404 | (92.2) | .30 |
| Redundant Test | 4 | (5.1) | 198 | (18.3) | |
| Appropriate Test | 75 | (94.9) | 886 | (81.7) | <.01 |
All tests with no impact considered timely follow-up
In a nested logistic regression model the following were significantly associated with lack of timely follow-up (data not shown in Table 2): redundant tests (OR: 0.24; 95% CI: 0.074-0.84) with appropriately ordered tests as referent; alerts for conditions newly diagnosed i.e., diagnosis was not made until the lab was done (OR: 7.35; 95% CI: 4.16-12.97) with previously known diagnosis as referent and mental health as a specialty (OR: 2.82; 95% CI: 1.06-7.54) with primary care as referent.
Alerts Related to Redundant Tests
In Table 3 we list criteria to determine if an alert was related to a redundant laboratory tests and their respective frequencies (total n=202). Because the VA's electronic medical record has a reminder system that prompts the ordering provider about recent orders of the same test, we further evaluated the subcategory of 159 alerts deemed redundant based on repetition. Overall, we found 28 cases where the provider ordered the test too soon after the last PSA, TSH or HbA1c and in 131 cases a second HCV ELISA test was ordered despite the presence of a previous positive test. In our institution the reminder is set to prompt the ordering provider about repetition when the new test is being ordered within 60 days from last TSH or PSA order or within 90 days from last HCV or HbA1c order. However, in only 11 (7%) of these 159 cases would the provider have received a computerized reminder prompting them of potential redundancy based on repetition.
DISCUSSION
We tested whether certain abnormal outpatient laboratory tests were followed-up in a timely manner in a multi-specialty clinic that used an integrated electronic medical record for automated notification. We found 6.8% of alerts lacked follow-up at 30 days, suggesting that follow-up of abnormal outpatient test results is not fail-safe even when providers are alerted about abnormal results through the electronic medical record. Of concern was the finding that there was lack of timely follow-up even when providers acknowledged notifications through the electronic medical record, which was comparable to when they did not acknowledge them. These findings are similar to our previous findings of follow-up on abnormal imaging alerts in the same electronic medical record suggesting that this phenomenon may exist for all alerts of abnormal diagnostic test results. Unexpectedly, we found 17% of abnormal test alerts were related to tests that we deemed redundant based on pre-determined criteria. Alerts related to these tests were less likely to lack timely follow-up.
Our findings have several significant implications for electronic medical record use in the future. One, it cannot be assumed that automated notification of abnormal lab results within an electronic medical record and the resultant acknowledgement will translate into timely actions to address these alerts. Two, notifications of abnormal redundant tests appear to be a distracting influence on providers who are missing essential alerts for newly diagnosed conditions. Three, high-reliability tracking systems to monitor potential patient harm and outcomes are needed, which also should account for follow-up actions by providers. Currently, the only way to track follow-up actions on abnormal alerts is through medical record review, a time-consuming and expensive procedure. However, individual-level tracking of follow-up actions taken in response to abnormal test result notifications could be designed within the electronic medical record. Thus, when providers process an alert, they could be provided order sets of appropriate follow-up actions in a separate “pop-up” window (such as having a nurse call the patient, setting up a return appointment, ordering a consultation or follow-up test, or an option indicating no further action is required such as when a patient is already in hospice care). These actions could be tracked through the electronic medical record and a reporting process could be created for clinic administrators to review and identify patients who may have truly “slipped through the cracks” without performing extensive record reviews. For instance, in cases of inaction on an abnormal lab at 2 weeks, the ordering provider or their surrogates could be informed.
We previously determined rate of lack of timely follow-up for abnormal imaging alerts in the same system and setting and found comparable results. Future work needs to confirm the extent to which these findings exist in other electronic medical record systems. Because there could be many potential reasons why busy providers in the front lines of health care delivery miss abnormal test results, a multidisciplinary approach is needed to address test result follow-up in future. For instance, an approach involving human computer interaction and informatics3;21 that accounts for issues related to usability, organizational characteristics, technology, work-flow and provider factors could be useful to explain why providers are unable to follow-up results despite reading them and hence improve safety in this area. 22;23
Computerized reminders have been shown to reduce redundant tests in the inpatient setting.24 However, our computerized reminder system would have prompted the ordering provider only 7% of the time. Notably, in 131 cases where a positive Hepatitis C test previously existed in the electronic medical record, the reminder logic was not set to prompt providers that they were ordering a repeat, redundant test. Current computerized reminder systems could be better designed to reduce test redundancy. For instance, these reminders are designed to prompt providers based only on the date of test order and not the date of the result of the last test. If the system had been configured as giving off a prompt based on both these dates, all 28 providers who unnecessarily ordered a PSA, Hba1c or a TSH would have received a prompt. Future work is essential to better document whether the use of information technology can reduce the enormous costs associated with redundant tests especially in the outpatient setting. 25;26
Our study had several limitations. Because of the study population (e.g. predominantly male veterans) and the unique VA setting, our findings may not be generalizable outside the VA. However, with increasing emphasis on electronic medical records the potential relevance of these findings is significant. We also lack comparable data from non-electronic medical record based systems and cannot comment on the effectiveness of automated notification compared to these systems. Due to the lack of similar tracking and documentation capabilities, such an evaluation study would be very difficult to carry out. Conversely, many factors including a large sample size, multiple clinics, large number of providers (over 500 from different specialties), rigorous methods to determine follow-up, explicit criteria for determination of redundant tests, various types of abnormal lab alerts and an advanced integrated electronic medical record used in VA facilities throughout the US, all add several unique strengths to our study.
In conclusion, current systems of mandatory automated notification of abnormal lab results do not guarantee timely follow-up on the abnormality in the outpatient setting. Additionally, provider acknowledgment of receipt of the test result also does not automatically result in timely follow-up. Multidisciplinary interventions involving human-computer interaction 3;21 and high-reliability tracking systems to monitor test result notification outcomes, such as follow-up actions by providers on these tests, are needed to alleviate these safety concerns.
Acknowledgments
Funding Source
The study was supported by an NIH K23 career development award (K23CA125585) to Dr. Singh, the VA National Center of Patient Safety, Agency for Health Care Research and Quality and in part by the Houston VA HSR&D Center of Excellence (HFP90-020).
These sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.
Data
All authors had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.
Reference List
- 1.Edelman D. Outpatient diagnostic errors: unrecognized hyperglycemia. Eff Clin Pract. 2002;5(1):11–16. [PubMed] [Google Scholar]
- 2.Hickner J, Graham DG, Elder NC, Brandt E, Emsermann CB, Dovey S, et al. Testing process errors and their harms and consequences reported from family medicine practices: a study of the American Academy of Family Physicians National Research Network. Qual Saf Health Care. 2008;17(3):194–200. doi: 10.1136/qshc.2006.021915. [DOI] [PubMed] [Google Scholar]
- 3.Hickner JM, Fernald DH, Harris DM, Poon EG, Elder NC, Mold JW. Issues and initiatives in the testing process in primary care physician offices. Jt Comm J Qual Patient Saf. 2005;31(2):81–89. doi: 10.1016/s1553-7250(05)31012-9. [DOI] [PubMed] [Google Scholar]
- 4.Poon EG, Gandhi TK, Sequist TD, Murff HJ, Karson AS, Bates DW. “I wish I had seen this test result earlier!”: Dissatisfaction with test result management systems in primary care. Arch Intern Med. 2004;164(20):2223–2228. doi: 10.1001/archinte.164.20.2223. [DOI] [PubMed] [Google Scholar]
- 5.Schiff GD, Kim S, Krosnjar N, Wisniewski MF, Bult J, Fogelfeld L, et al. Missed Hypothyroidism Diagnosis Uncovered by Linking Laboratory and Pharmacy Data. Arch Intern Med. 2005;165(5):574–577. doi: 10.1001/archinte.165.5.574. [DOI] [PubMed] [Google Scholar]
- 6.Wahls TL, Haugen T, Cram P. Diagnostic errors associated with failure to follow up on abnormal results. Joint Commission Journal on Quality and Patient Safety/Joint Commission Resources. 2007 [Google Scholar]
- 7.Wahls T. Diagnostic errors and abnormal diagnostic tests lost to follow-up: a source of needless waste and delay to treatment. J Ambul Care Manage. 2007;30(4):338–343. doi: 10.1097/01.JAC.0000290402.89284.a9. [DOI] [PubMed] [Google Scholar]
- 8.Bates DW, Leape LL. Doing better with critical test results. Jt Comm J Qual Patient Saf. 2005;31(2):66–67. doi: 10.1016/s1553-7250(05)31010-5. [DOI] [PubMed] [Google Scholar]
- 9.Schiff GD. Introduction: Communicating critical test results. Jt Comm J Qual Patient Saf. 2005;31(2):63–65. doi: 10.1016/s1553-7250(05)31009-9. [DOI] [PubMed] [Google Scholar]
- 10.Nepple KG, Joudi FN, Hillis SL, Wahls TL. Prevalence of delayed clinician response to elevated prostate-specific antigen values. Mayo Clin Proc. 2008;83(4):439–448. doi: 10.4065/83.4.439. [DOI] [PubMed] [Google Scholar]
- 11.Gandhi TK. Fumbled handoffs: one dropped ball after another. Ann Intern Med. 2005;142(5):352–358. doi: 10.7326/0003-4819-142-5-200503010-00010. [DOI] [PubMed] [Google Scholar]
- 12.Poon EG, Haas JS, Louise PA, Gandhi TK, Burdick E, Bates DW, et al. Communication factors in the follow-up of abnormal mammograms. J Gen Intern Med. 2004;19(4):316–323. doi: 10.1111/j.1525-1497.2004.30357.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Gandhi TK, Kachalia A, Thomas EJ, Puopolo AL, Yoon C, Brennan TA, et al. Missed and delayed diagnoses in the ambulatory setting: A study of closed malpractice claims. Ann Intern Med. 2006;145(7):488–496. doi: 10.7326/0003-4819-145-7-200610030-00006. [DOI] [PubMed] [Google Scholar]
- 14.Graber ML, Franklin N, Gordon R. Diagnostic error in internal medicine. Arch Intern Med. 2005;165(13):1493–1499. doi: 10.1001/archinte.165.13.1493. [DOI] [PubMed] [Google Scholar]
- 15.Singh H, Arora HS, Vij MS, Rao R, Khan M, Petersen LA. Communication outcomes of critical imaging results in a computerized notification system. J Am Med Inform Assoc. 2007;14(4):459–466. doi: 10.1197/jamia.M2280. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Singh H, Naik A, Rao R, Petersen L. Reducing Diagnostic Errors Through Effective Communication: Harnessing the Power of Information Technology. Journal of General Internal Medicine. 2008;23(4):489–494. doi: 10.1007/s11606-007-0393-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Singh H, Sethi S, Raber M, Petersen LA. Errors in cancer diagnosis: current understanding and future directions. J Clin Oncol. 2007;25(31):5009–5018. doi: 10.1200/JCO.2007.13.2142. [DOI] [PubMed] [Google Scholar]
- 18.Poon EG, Wang SJ, Gandhi TK, Bates DW, Kuperman GJ. Design and implementation of a comprehensive outpatient Results Manager. J Biomed Inform. 2003;36(1-2):80–91. doi: 10.1016/s1532-0464(03)00061-3. [DOI] [PubMed] [Google Scholar]
- 19.Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? Archives of Internal Medicine. doi: 10.1001/archinternmed.2009.263. In Press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.van Walraven C, Naylor CD. Do we know what inappropriate laboratory utilization is? A systematic review of laboratory clinical audits. JAMA. 1998;280(6):550–558. doi: 10.1001/jama.280.6.550. [DOI] [PubMed] [Google Scholar]
- 21.Horsky J, Zhang J, Patel VL. To err is not entirely human: Complex technology and user cognition. Journal of Biomedical Informatics. 2005;38(4):264–266. doi: 10.1016/j.jbi.2005.05.002. [DOI] [PubMed] [Google Scholar]
- 22.Carayon P, Schoofs Hundt A, Karsh BT, Gurses AP, Alvarado CJ, Smith M, et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care. 2006;15(suppl_1):i50–i58. doi: 10.1136/qshc.2005.015842. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Kushniruk AW, Patel VL. Cognitive evaluation of decision making processes and assessment of information technology in medicine. Int J Med Inform. 1998;51(2-3):83–90. doi: 10.1016/s1386-5056(98)00106-3. [DOI] [PubMed] [Google Scholar]
- 24.Bates DW, Kuperman GJ, Rittenberg E, Teich JM, Fiskio J, Ma'luf N, et al. A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests. The American Journal of Medicine. 1999;106(2):144–150. doi: 10.1016/s0002-9343(98)00410-0. [DOI] [PubMed] [Google Scholar]
- 25.Wang SJ, Middleton B, Prosser LA, Bardon CG, Spurr CD, Carchidi PJ, et al. A cost-benefit analysis of electronic medical records in primary care. The American Journal of Medicine. 2003;114(5):397–403. doi: 10.1016/s0002-9343(03)00057-3. [DOI] [PubMed] [Google Scholar]
- 26.van Walraven C, Raymond M. Population-based Study of Repeat Laboratory Testing. Clin Chem. 2003;49(12):1997–2005. doi: 10.1373/clinchem.2003.021220. [DOI] [PubMed] [Google Scholar]