Abstract
Electronic decision support systems are an important tool for improving performance and improving quality of care. We investigated the relationship between physicians’ estimated resolution times for computerized clinical reminders and adherence rates in VA outpatient settings. We surveyed 10 expert physician users to assess the resolution times of four targeted CCRs for three cases: pessimistic (worst case), expected (average), and optimistic times (best case). ANOVA test shows that physicians’ adherence rates for the four CCRs differed significantly (p = 0.01). CCR adherence rate and resolution time were highly linearly correlated (R-square= 0.876 for the best case, R-square= 0.997 for the average case, and R-square= 0.670 for the worst case). This study suggested that future efforts in designing CCRs need to take resolution time into consideration during design, usability testing and implementation phases.
Introduction
The Veterans Health Administration (VHA) electronic health record (EHR) is one of the most widely implemented in the United States1. The VHA has developed the informatics infrastructure necessary to support an EHR, the Computerized Patient Record System (CPRS), which enables clinicians to review and analyze patient clinical data, order laboratory tests and medications, document care, review radiology and other data and support clinical decision-making. It has been widely implemented throughout the VHA’s healthcare system of acute care medical centers, outpatient care clinics and long-term care facilities.1
Computerized clinical reminders (CCRs) are a type of decision support system which utilize pre-existing electronic patient data and a set of Boolean CCR definitions to alert providers when an action is recommended. In VHA, CCRs are one of the major functions of CPRS with electronic decision support developed either locally or nationally.2 In order to resolve a CCR, a series of procedures need to be performed, including opening and reviewing the electronic health records, consulting with the patient regarding the dialog boxes and recommendations, ordering laboratory tests, following-up on the laboratory tests, revising orders for the patient’s medication, and documenting resolution of the reminder. Thus, the resolution time of a CCR indicates the quantitative summation of the time required to accomplish the aforementioned procedures associated with the CCR when applicable.
CCRs have received increasing attention as a tool to improve evidence-based practice by augmenting the limited memory resources of time-pressured providers during the point of care.3 However, a review of controlled clinical trials found that CCRs did not always improve physician’s performance.4 Numerous studies have identified factors that may explain poor adherence with CCRs. In particular, the effectiveness of CCRs is greatly reduced by human factors related barriers, such as problems with the user interface or workflow.5 Several investigators have identified additional factors through observations, interviews, and surveys including increased workload, time for documentation, inapplicability of CCRs to practice, limited staff training, staff engagement, different stakeholder perspectives, etc. 1,2,6–9 These factors may result in either low adherence or prolonged time required to use the CCRs.
Retrospective time estimation is often used to study cognitive ability, working memory, and subjective task complexity.10,11 It has also been used as an implicit indicator of software usability and user satisfaction in software engineering.12,13 However, very few studies have linked retrospective time estimation to users’ utilization or adherence bahavior.
In this study, we asked the physicians to estimate how long it took to complete a CCR in the best, average, and worst cases. These three cases represent the optimistic time, expected time, and pessimistic time required to complete a CCR, respectively. The best case of CCR resolution implies the situation when a reminder is inapplicable or inappropriate, and most procedures related to the test or screening can thus be skipped. Contrarily, the pessimistic time incurs when all the procedures associated with a CCR are applicable and being followed. Our goal was to investigate quantitatively the relationship between subjective estimating of the time needed to resolve CCRs and physicians’ adherence to the CCRs.
Methods
Collection of CCR Average Adherence Rates:
Four national guidelines were selected for study because local CCRs had been developed, implemented, and their adherence was expected in the VAMC primary care clinics. The four CCRs are Annual HgbA1c, Diabetic Foot check, Fecal Occult Blood Test (FOBT), and Positive Hepatitis C. Annual HgbA1c and Diabetic Foot CCRs are automatically triggered for all patients who have a diagnosis of diabetes or have a prescribed medicine for diabetes in their medical record. FOBT is a colorectal cancer screening CCR for patients ages 50 to 80, whereas the Positive Hepatitis C reminder is a follow-up reminder that alerts the patient’s provider after one or more risk factors for Hepatitis C is found for the patient in the medical record.
Physician order entry and structured data documenting their responses to the four CCRs have been continuously captured by the VA CPRS system. However, collecting individual physicians’ adherence data was costly and politically sensitive due to confidentiality related concerns. Therefore, we collected the average adherence rates of all physicians in the Roudebush VAMC primary care outpatient clinics. The average adherence rates were collected for each of the four CCRs biweekly, and weekly in some rare cases, between August 2003 and July 2006. The study was approved by the institutional review board and VA research review committee.
Subjective Evaluation of CCR Resolution Time:
A survey instrument was distributed to a panel of expert users during a physician panel meeting in October 2006*. Ten physicians participated in the survey, representing one-third (36 %) of the total 28 physicians practicing in the Roudebush VAMC primary care clinics.
The design of this survey is based on time estimation in “optimistic”, “expected”, and “pessimistic” times, which is a common practice in the Program Evaluation and Review Technique (PERT) for project management, to calculate deterministically the fixed time estimate of each activity.14,15 The survey asked the physicians to estimate the resolution time for each of the four CCRs in minutes under three scenarios: the best, the average, and the worst cases. The CCR resolution time in the best case, also called the “optimistic” time, is the minimum possible time required to resolve a CCR, assuming everything proceeds better than normally expected. Similarly, CCR resolution time in the worst case, also called the “pessimistic” time, is the maximum possible time required to resolve a CCR, assuming every procedure relevant to resolving a CCR is performed. The resolution time in the average case, i.e. the “expected” time, is the best estimation of the time required to resolve a CCR, assuming everything proceeds as normal.
Results
CCR Average Adherence Rate Summary:
Physicians’ adherence rates for the four CCRs were plotted over time for an interval screening from August 2003 to July 2006, as shown in Figure 1. Since physician’s adherence was updated biweekly for most cases, there are 93 sampling instances during the three year span of data collection. Note that there were missing data during the process of data collection for “Diabetic Foot” reminder, which results in a shorter series of sampling instances.
Figure 1.
Adherence rate vs. time for the four CCRs from August 2003 to July 2006.
Table 1 shows the means and the standard deviations of the CCR adherence rate, as well as the number of sampling instances. ANOVA test on SAS program revealed a main effect of CCR adherence rate (p = 0.01). In other words, at least one of the four adherence rates was different from one other rate.
Table 1.
The mean and standard deviation of CCR Adherence Rate (8/2003 – 7/2006)
| Sampling instances | Mean | Stand. Dev. | |
|---|---|---|---|
| Annual HgbA1c | 93 | 94% | 5% |
| Diabetic Foot | 86 | 81% | 10% |
| FOBT | 93 | 73% | 5% |
| Positive Hepatitis C | 93 | 87% | 8% |
Subjective Evaluation of CCR Resolution Time:
Ten primary care physicians completed the survey. One of the providers had CPRS experience for less than 3 months, whereas the other nine experts had used CCRs for more than one year. Their level of experience was typical of the population of physicians in the VAMC primary care clinics. The results from the survey are summarized in Figure 2, where the bars represent the average resolution time of the ten panel experts for the “optimistic”, “expected”, and “pessimistic” cases, respectively. The chart also shows the interval corresponding to one standard deviation above and below the mean resolution time for each reminder under each scenario.
Figure 2.
Average CCR resolution time of the panel experts for the “optimistic”, “expected”, and “pessimistic” cases
Linear Regression Model:
A linear regression model was applied to examine a linear relationship between CCR adherence rate and resolution time. As shown in Figure 3–5, a linear relationship was found between CCR adherence rate and resolution time for each case. In Figure 3, a best-fit line was drawn to demonstrate the linearity between the average adherence rate and the average resolution time under the “optimistic” case.
Figure 3.
Average CCR adherence rate v.s. average resolution time in the “Optimistic” case
Figure 5.
Average CCR adherence rate vs. average resolution time in the “Pessimistic” case
Similarly, Figures 4 and 5 were drawn to represent the “expected” and the “pessimistic” cases. The R-square values, representing the fraction of variability in the response (in this case, adherence rate) explained by the explanatory variable (resolution time), were 0.876, 0.997 and 0.670 for the “optimistic”, “expected”, and “pessimistic” cases respectively. In terms of correlation, the correlation coefficients (R) for the three cases are 0.936, 0.998, and 0.819 respectively. The high degree of correlation between the average adherence rate and the resolution time shows that physicians’ subjective time estimates can serve as an effective estimator to explain their adherence practices with CCRs.
Figure 4.
Average CCR adherence rate v.s. average resolution time in the “expected” case
Discussion
This study examined the correlation between the average CCR adherence rate and estimates of resolution time, and found these two variables to be significantly linearly related. Thus, when physicians perceive on average a longer period of resolution time for a specific CCR, it is more likely that this group of physicians will have a lower adherence rate with this CCR, compared to other CCRs.
Providers in the VHA face extremely high patient loads. Further, patient encounters are generally limited to 30 minutes at the Indianapolis VAMC. Providers are expected to complete the CCRs for each patient under these constraints, in addition to all of the elements that must be completed in the patient encounter. If CCRs take a relatively long time to resolve in relation to the other patient encounter activities, providers are more likely to shed or defer resolution of some or all CCRs.6 Thus, it is critical that provider perceptions of CCR completion time, as well as actual completion time, is reasonably short.
The subjective time estimation refers to the human ability to assess how much time has elapsed for an activity. Previous research in time experience studies suggested that if people can retrieve a greater number of events from memory, they perceive the duration of a time period as being longer.16–18 Thus, if a person estimates retrospectively a longer duration of an activity, there could be relatively more tasks involved for completing that activity. Under high time-constraint clinical settings, the activities involving more steps (such as resolving a more complex CCR) may tend to be unattended or unaccomplished.
One of the study’s strengths is that it examined CCR use by experienced users in the setting of an EHR where clinical decision support is widely used. One potential limitation is that the adherence rates for individual CCRs was based on the average measured adherence of the twenty-eight primary care clinic physicians, whereas the resolution time was calculated from the average of the ten panel experts. However, since the panel of experts represented 36% of the population and the time estimates and adherence were highly correlated, the sample likely represents the population of all providers.
The perceived time required to complete a CCR can be regarded as a cost component associated with complying with a clinical guideline. Similarly, the perceived benefit of CCRs can be one of the indicators of physicians’ motivation to resolve CCRs. Previous studies of the Theory of Reasoned Action indicated that people’s behavior is determined by their intention to perform the behavior, which is a combination of their attitude and motivation toward the behavior.19,20 The implication of this theory in our study suggests that physician’s adherence to CCRs can be a weighted function of perceived completion time and perceived benefit of CCRs.
Future studies should develop methods for objective time estimation of CCR resolution by observation as a comparison to this research. In addition, more national or regional clinical guidelines should be included in future studies to strengthen the current research conclusion.
Conclusions
Increasing evidence points to the importance of usability testing in the staged development and implementation of clinical decision support.21 Potential usability issues, such as those that contribute to excessive time needed to complete CCRs, can be addressed prior to implementation with often modest design changes, as well as with less monetary and time cost than changes post-implementation. Our study demonstrates that providers’ estimates of the time required to resolve reminders predicts their use and adherence. Future studies should further investigate the perceived benefit of CCRs and the relationship between physicians’ adherence rate and perceived time and benefit. Furthermore, the time required to complete a given clinical reminder should be carefully weighed against its expected benefit during the development and usability testing phases.
Acknowledgments
The authors would like to thank Christopher Suelzer, Mindy Flanagan, Alan Zillich and Shawn Hoke for providing helpful suggestions during the study and reviewing this paper for accuracy. This research was supported by the Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service (HSRD) Center grant #HFP 04-148. The views expressed in this article are those of the authors and do not necessarily represent the view of the Department of Veterans Affairs.
References
- 1.Doebbeling BN, Vaughn TE, McCoy KD, Glassman P. Informatics implementation in the Veterans Health Administration healthcare system to improve quality of care. Proc AMIA Symp. 2006:204–8. [PMC free article] [PubMed] [Google Scholar]
- 2.Fung CH, Woods JN, Asch SM, Glassman P, Doebbeling BN.Variation in implementation and use of computerized clinical reminders in an integrated healthcare system Am J Managed Care 20041011Part 2878–85. [PubMed] [Google Scholar]
- 3.Bates DW, Cohen M, Leape LL, Overhage JM, Shabot MM, Sheridan T. Reducing the frequency of errors in medicine using information technology. J Am Med Inf Assoc. 2001;8(4):301. doi: 10.1136/jamia.2001.0080299. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support system on physician performance and patient outcomes: a systematic review. JAMA. 1998;280(15):1339–46. doi: 10.1001/jama.280.15.1339. [DOI] [PubMed] [Google Scholar]
- 5.Krall MA, Sittig DF. Clinician's assessments of outpatient electronic medical record alert and reminder usability and usefulness requirements. Proc AMIA Symp. 2002:400–4. [PMC free article] [PubMed] [Google Scholar]
- 6.Saleem JJ, Patterson ES, Militello L, Render ML, Orshansky G, Asch SA. Exploring barriers and facilitators to the use of computerized clinical reminders. J Am Med Inform Assoc. 2005;12(4):438–47. doi: 10.1197/jamia.M1777. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Patterson ES, Doebbeling BN, Fung CH, Militello L, Anders S, Asch SM. Identifying barriers to the effective use of clinical reminders: Bootstrapping multiple methods. J Biomed Inform. 2005;38:189–199. doi: 10.1016/j.jbi.2004.11.015. [DOI] [PubMed] [Google Scholar]
- 8.Patterson ES, Nguyen AD, Halloran JP, Asch SM. Human factor barriers to the effective use of the HIV clinical reminders. J Am Med Inform Assoc. 2004;11:50–59. doi: 10.1197/jamia.M1364. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Lyons SS, Tripp-Reimer T, Sorofman BA, et al. Informatics technology for clinical guideline implementation: Perceptions of multiple stakeholders. J Am Med Inform Assoc. 2005;12(1):64–71. doi: 10.1197/jamia.M1495. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Fink A, Neubauer AC. Individual differences in time estimation related to cognitive ability, speed of information processing and working memory. Intelligence. 2005;33(1):5–26. [Google Scholar]
- 11.Zakay D, Block RA. Prospective and retrospective duration judgments: an executive-control perspective. Acta Neurobiologiae Experimentalis. 2004;64(3):319–328. doi: 10.55782/ane-2004-1516. [DOI] [PubMed] [Google Scholar]
- 12.Cutrell E, Czerwinski M, Horvitz E. Subjective duration assessment: an implicit probe for software usability. Proc IHM-HCI 2001; Lille, France. Sep, 2001. pp. 167–170. [Google Scholar]
- 13.Davis MM, Vollmann TE. A framework for relating waiting time and customer satisfaction in a service operation. J Services Marketing. 1990;4(1):61–9. [Google Scholar]
- 14.Clark CE. The PERT model for the distribution of an activity time. Operations Research. 1962;10(3):405–406. [Google Scholar]
- 15.Keefer DL, Verdini WA. Better estimation of PERT activity time parameters. Management Science. 1993;39(9):1086–1091. [Google Scholar]
- 16.Fraisse P. In: The psychology of time. Leith J, translator. New York: Harper & Row; 1963. [Google Scholar]
- 17.Ornstein RE. On the Experience of Time. Middlesex, England: Penguin; 1969. [Google Scholar]
- 18.Vroon PA. Effects of presented and processed information on duration experience. Acta Psychologica. 1970;34:115–121. [Google Scholar]
- 19.Fishbein M, Ajzen I. Belief, attitude, intention, and behavior: An introduction to theory and research Reading, Mass; Don Mills, Ontario: Addison-Wesley, Pub. Co; 1975. [Google Scholar]
- 20.Ajzen I, Fishbein M. Attitudinal and normative variables as predictors of specific behavior. Journal of Personality and Social Psychology. 1973;27(1):41–57. [Google Scholar]
- 21.Doebbeling BN, Chou AF, Tierney WM. Priorities and strategies for implementation of an integrated informatics and communications technology system for evidence-based practices. J. Gen. Intern. Med. 2006;21:S98–S105. doi: 10.1111/j.1525-1497.2006.00363.x. [DOI] [PMC free article] [PubMed] [Google Scholar]