Evaluation of the Effect of Decision Support on the Efficiency of Primary Care Providers in the Outpatient Practice

Abstract

Background: Clinical decision support (CDS) for primary care has been shown to improve delivery of preventive services. However, there is little evidence for efficiency of physicians due to CDS assistance. In this article, we report a pilot study for measuring the impact of CDS on the time spent by physicians for deciding on preventive services and chronic disease management. Methods: We randomly selected 30 patients from a primary care practice, and assigned them to 10 physicians. The physicians were requested to perform chart review to decide on preventive services and chronic disease management for the assigned patients. The patients assignment was done in a randomized crossover design, such that each patient received 2 sets of recommendations—one from a physician with CDS assistance and the other from a different physician without CDS assistance. We compared the physician recommendations made using CDS assistance, with the recommendations made without CDS assistance. Results: The physicians required an average of 1 minute 44 seconds, when they were they had access to the decision support system and 5 minutes when they were unassisted. Hence the CDS assistance resulted in an estimated saving of 3 minutes 16 seconds (65%) of the physicians’ time, which was statistically significant (P < .0001). There was no statistically significant difference in the number of recommendations. Conclusion: Our findings suggest that CDS assistance significantly reduced the time spent by physicians for deciding on preventive services and chronic disease management. The result needs to be confirmed by performing similar studies at other institutions.

Introduction

The quality of preventive care delivered in the United States is suboptimal.^1,2 Recent studies have shown that many patients do not get the recommended best preventive services³—mammography, pap, and colonoscopy screening rates are 72%, 83%, and 59%, respectively.⁴ Adult vaccination rates are also low—pneumococcal vaccination coverage among high-risk adults is 20% and tetanus, diphtheria, and pertussis (Tdap) coverage of adults is 13%.⁵ Several studies have identified provider,^6,7 patient,⁸ practice,^9,10 and environmental factors^11-14 that affect care quality. The need to improve the care quality has motivated federal investments in health information technology.¹⁵

Although there is strong evidence that clinical decision support (CDS) assistance improves preventive care delivery, there is little evidence of CDS impact on workload and efficiency of care providers.^16-22 Besides increasing service delivery, CDS interventions may reduce the time required for service delivery. The time savings in turn can enable the physician to focus on the patient’s presenting problem, and consequently improve care quality. But there is no previous report of the time savings from the CDS assistance. In this article, we report a pilot study aimed at measuring the impact of CDS assistance on the time spent by physicians for deciding on preventive services and chronic disease management.

Methods

We randomly selected 30 patients from a primary care practice, and invited 10 physicians across multiple sites of the practice to participate in this study. Each patient was randomly assigned in a cross over design to 2 of the physicians, such that one physician had CDS assistance and the other physician did not have CDS assistance (Figure 1). The physicians were requested to perform chart review to decide on preventive services and chronic disease management (Table 1) for the assigned patients. To ensure equal representation of the physicians, each physician had CDS assistance for only half the assigned patients. Consequently, each patient had two sets of recommendations—one from a physician using CDS assistance and the other from a different physician without CDS assistance. We compared the physician recommendations made using CDS assistance, with the recommendations made without CDS assistance. The methodological details are described further in this section. This research was approved by the institutional review board at Mayo Clinic, Rochester.

Figure 1.

Crossover design of the study. Circles indicate patients, rectangles indicate physicians and connecting lines indicate the assignments. Each patient received 2 sets of recommendations—one from a physician with clinical decision support (CDS) assistance and the other from a different physician without CDS assistance. Each physician provided recommendations for half the assigned patients using CDS assistance, and made the recommendations without CDS assistance for the other half of the patients.

Table 1.

Preventive Care and Chronic Disease Management Decisions Investigated in This Study.^a

Lifestyle factors
Tobacco cessation advice	Patient is using tobacco
Alcohol counseling	Patient reports problems with alcohol
Lifestyle counseling	BMI is >30 kg/m2
Blood tests
Fasting glucose	Hyperglycemic with no glucose test in 1 year
Glycated hemoglobin (HbA1c)	Diabetic with no HbA1C report
Lipid panel	Nondiabetic with no cholesterol, HDL, or triglycerides in the past 5 years or diabetic with no cholesterol, HDL, or triglycerides in past 1 year
LDL	High risk for cardiovascular disease and has a high recent LDL
Creatinine	No creatinine report in the last 1 year, and patient is diabetic or receives diuretics, ARB, or ACE inhibitors
Microalbumin	Diabetic patient with no microablumin result in past year
TSH	Patient on thyroid meds, with no TSH in past year
Issue examinations/screenings
Eye examination	Diabetic with no eye examinations in the past year
Urinalysis	On cyclophosphamide, no urinalysis reported in the past 6 months
Echocardiogram	Heart failure with no reported ejection fraction
Papanicolaou (Pap)	If abnormal Pap or high-risk condition was not followed by Pap test in past year, or no Pap report for 21- to 30-year-old females in past 2 years, or 30- to 65-year-old female with no Pap report in past 3 years
Mammogram	40- to 75-year-old female with no mammogram in past year
Colonoscopy	(i) Polyp not followed up in past 3 years, or colon cancer diagnosis not followed up in past year. (ii) Family history of colon cancer with no colonoscopy in past 5 years. (iii) 50- to 80-year-old with no colonoscopy in the past 10 years, occult blood in the past 12 months, flexible sigmoidoscopy, colon x-ray or colonography in the past 5 years
Abdominal aortic aneurysm	65- to 75-year-old male with past tobacco habit and no imaging of abdominal aorta
BMD	>65-year-old female with no BMD since age 50 years
PSA	50- to 70-year-old male with no PSA in past year
STI screening	16- to 25-year-old with no report of chlamydia/gonorrhea screening
Depression screening (PHQ9)	Last PHQ9 with high score more than 6 months ago or PHQ9 not reported in past year for patients diagnosed with depression
AICD consult	CHF with ejection fraction less than 35%, who have no AICD
Immunizations due
Influenza	No immunization in flu season
HPV	9- to 25-year-old female with (i) no previous HPV vaccination, or (ii) first vaccination more than 2 months ago, or (iii) second vaccination more than 4 months ago
Pneumococcal	20- to 65-year-old with high risk and no pneumococcal immunizations in the past 5 years. Greater than 65 years with no vaccine in the past 5 years
Herpes zoster	More than 60 years with no zoster vaccine in the past
Tdap	20- to 64-year-old with no Tdap ever
Tetanus	>60 years with no Tdap or tetanus dose in the past 10 years
Hepatitis B	Diabetic, aged 19 to 59 years and (i) no past immunization, (ii) 1 month after first immunization, (ii) 2 months after second immunization and 4 months after first immunization
Start medication
Beta blocker	Not asthmatic or depressed, is not currently on beta blockers, and has either (i) heart rate is greater than 60 beats per minute, with CHF and ejection fraction is less than 0.4 or (ii) diagnosis of CAD
Warfarin	Has atrial fibrillation, who is not on warfarin, has no bleeding disorder and has high CHADS2 score
Aspirin	Not on aspirin or warfarin, and has no bleeding disorder but has either of CAD, DM, cerebrovascular accident, or peripheral vascular disease
ACE inhibitors/ARBs	On ACE or ARB, has DM, hypertension, high microalbumin, CAD or CHF, and has no history of renal failure
Medication-related blood work
Digoxin level	On digoxin with no digoxin report in past year
Potassium level	On diuretics, ARBs or ACE inhibitors with no potassium report in past year
Sodium level	On diuretics, with no sodium report in past year
Vitamin B12	No vitamin B12 in past year with anemia, bypass surgery, Crohns, achlorhydria, alcohol dependency, or bacterial overgrowth
Dilantin level	On anticonvulsants and dilantin, with no dilantin report in past year
Carbamazepine level	On carbamazepine, and no carbamazepine report in past year
ALT	On cyclosporin and no ALT report in past 3 months
AST	On azathioprine, leflunomide, methotrexate or tocilizumab, and no AST report in past 8 weeks
CBC	On cyclophosphamide, no CBC in last 6 weeks. On azathioprine or leflunomide or methotrexate or tocilizumab, and no CBC in last 8 weeks. On cyclosporin, sulfasalazine, anti-IL1β or anti-TNF, or rituximab and no CBC result in the past 3 months
INR	On warfarin and no INR in past month

Abbreviations: ACE inhibitor, angiotensin-converting enzyme inhibitor; AICD, automatic implantable cardioverter defibrillator; ALT, alanine aminotransferase; ARB, angiotensin II receptor blocker; AST, aspartate transaminase; BMD, bone mineral density; CAD, coronary artery disease; CBC, complete blood count; CHADS₂, congestive heart failure, hypertension, age, diabetes mellitus, stroke or transient ischemic attack; CHF, chronic heart failure; DM, diabetes mellitus; HbA1c, glycated hemoglobin; HDL, high-density lipoprotein; HPV, human papillomavirus; IL1β, interleukin-1 beta; INR, international normalized ratio; LDL, low-density lipoprotein; Pap, Papanicolaou test; PHQ9, Patient Health Questionnaire 9; PSA, prostate-specific antigen; STI, sexually transmitted infection; Tdap, tetanus, diphtheria, pertussis; TNF, tumor necrosis factor; TSH, thyroid-stimulating hormone.

^aThe left column gives the recommendation, and the right column summarizes the applicable trigger conditions.

Study Population

We randomly selected 30 patients from the population of approximately 103 000 patients older than 18 years who receive primary care at the Department of Employee and Community Health at Mayo Clinic, Rochester. Ten physicians were invited to participate from different locations in the practice.

Data Collection

We prepared a paper checklist of preventive services including screening and chronic condition management that are covered by the institutional CDS system. The services were grouped as lifestyle factors, blood tests, screenings, immunizations, medications, and medication-related blood work, and listed with checkboxes on a single page. Participating physicians were invited to complete the checklist for indicating which preventive services are due, by performing chart review.

Each physician was requested to complete checklists for 6 randomly allotted patients (Figure 1). For 3 of the allotted 6 patients, the physicians were requested to consider the CDS reminders that were provided as a printout, for deciding on the screening recommendations. The CDS decision logic for the preventive services is summarized in Table 1.²³ For the other 3 patients, the physicians were requested to skip the CDS assistance. The physicians recorded the start and end times on each of the checklists. The 30 cases were randomly distributed among the 10 physicians, such that 2 checklists were completed for each patient—for one of the checklists the physician was assisted by the CDS system, and the other checklist was completed by a different physician who did not use assistance from the CDS system.

Analysis

We compared the checklists across the unassisted and assisted groups, for time required to complete the checklist and number of recommendations per checklist using paired t tests. The time required to complete the checklists was computed as the difference in the start and end times recorded by the clinicians.

Results

The physicians required an average of 1 minute 44 seconds, when they were they had access to the decision support system and 5 minutes when they were unassisted. Hence the CDS system resulted in an estimated saving of 3 minutes 16 seconds (65%) of the physicians’ time, which was statistically significant (P < .0001).

Assisted and unassisted physicians made an average of 3.4 and 3.9 recommendations per patient, respectively (Table 2 and Figure 2). This difference was not statistically significant (P = .36). The results of the paired t test for difference in time and number of recommendations between the groups were robust to nonparametric assumption. The average statistics correspond to a respective total of 102 assisted and 116 unassisted recommendations, of the total possible 1290 (=43 preventive services × 30 patients) recommendations in each group.

Table 2.

Results Summary.

	Mean (SD)	No. of Observations	P Value
Time (minutes) for completing checklist			<.0001
Unassisted group	5.00 (2.73)	30
Assisted group	1.73 (1.57)	30
Number of recommendations in			.36
Unassisted group	3.87 (3.20)	30
Assisted group	3.40 (2.76)	30

Figure 2.

Time required to complete the checklist and number of recommendations per patient, for the assisted versus unassisted physicians.

The patient population in this study had more females (57%), and tended to be middle aged, with an average age of 56.53 years (standard deviation of 19.7 years). The age ranged from 25 to 94 years. The number of problems listed in the electronic health record (EHR) ranged from 1 to 95, with an average of 22.9 (20.9), and number of laboratory results ranged from 0 to 2603 with average of 324.1 (597.1).

Discussion

We compared the time for decision making between 2 groups of providers—one group had CDS assistance and the other did not. CDS assistance was found to significantly reduce the time required by primary care providers for deciding on the preventive services and chronic disease management.

Our finding is in agreement with a report by Del Fiol et al²⁴ that providing context-specific information in the EHR saves 17% of the physician’s time for searching information needed to make clinical decisions. Several other studies have reported mixed effect of health information technology on time efficiency of physicians, but none of them have specifically reported on the time savings for deciding on preventive services and chronic disease management due to CDS.^25-27

Yarnall et al²⁸ have estimated the time for delivering preventive services, but they assume zero time for deciding on the recommendations. However, our study shows that physicians spend considerable time for deciding on the recommendations.

During patient consultation, the physician’s time is divided into addressing the current patient complaints and carrying out services for preventive care and for management of chronic conditions. Although physician can order the services and delegate the administration of the services to assistants, the lack of time to perform chart review for deciding on the services, can itself pose as an obstacle for delivery of the services.⁹ By reducing the time requirements for deciding the services, CDS may augment the delivery of the applicable service and improve the quality of care.

Time utilization of the clinician²⁹ for different tasks has been previously reported using methodologies like work sampling,³⁰ surveys,³¹ and continuous observation.^31,32 The methodology of continuous observation is considered a gold standard. Although observational studies can provide a detailed perspective of how the clinicians distribute their time, they are resource intensive. Our aim was to measure the time component specific for decision making, which is difficult to determine by independent observation. Hence, we resorted to the approach of self-timing.

The time savings from decision support may seem counter intuitive, since the CDS reminders represent additional information that the physicians need to consider for deciding on the recommendations. The physicians are indeed expected to confirm the validity of decision support before acting on it, which entails chart review. As a special case, CDS interventions that generate explanations or summarization can reduce the chart review effort, by identifying the reports that are relevant for the decision.³³ However, such explanations or summarizations were not supplied in this study. Hence, the likely explanation for the time savings is that the physicians acted on the CDS reminders without performing the verifications, possibly because of the trust developed in the CDS system.^6,34

Although there was a statistically significant difference in the time required for deciding on the recommendations, there was no significant difference in number of recommendations between the 2 groups. This suggests that CDS assistance may have greater benefits in terms of time savings as compared to improvements in service delivery.

The need of services for preventive care and chronic disease management is expected to vary substantially across patients, and the physician characteristics are known to influence the delivery of preventive care.^14,36 Hence, we used a crossover design, to ensure equal representation of the physician characteristics and patients in the study groups.

Limitations

As the physicians in the study were not previously familiar with the assigned patients, the study may not reflect the real-world setting of a longitudinal physician–patient relationship, wherein previous knowledge about the patient reduces the need for chart review. Also, the chart review was not synchronous with the patient visit, and thereby excludes patient participation in the decision making. For instance, some patient data resulting from care obtained outside the institution may not be recorded in the EHR system, but may be provided by the patient. In addition, patients may refuse particular services. These factors could cause an over estimation of the efficiency resulting from the use of CDS. Moreover, our study sample is modest in size and is from a single institution. Hence, the time savings measured in this study need to be confirmed by similar studies at other institutions.

Conclusion and Future Work

Clinical decision support interventions for primary care have been shown to improve screening rates, and projected to reduce costs. However, the evidence for efficiency of physicians due to decision support is sparse. In our pilot study, the CDS assistance was found to significantly reduce the time spent by physicians for deciding on preventive services and chronic disease management. Further research is required to confirm the time savings at other institutions.

Author Biographies

Kavishwar B. Wagholikar, MBBS, PhD, is a Research Associate at Mayo Clinic. His research interest is in the optimization of Clinical Decision Making, through computer base decision support, guideline refinement, natural language processing, and secondary use of electronic health records.

Ronald A. Hankey is a senior analyst/programmer at Mayo Clinic with an interest in clinical decision support systems. Mr. Hankey holds BS, MS and MBA degrees from various institutions.

Lindsay K. Decker is a lead analyst/programmer at Mayo Clinic and has worked on clinical decision support systems since 2007. She holds a B.S. degree in management of information systems and a M.S. degree in Project Management.

Stephen S. Cha graduated from University of Pittsburgh with a MS degree in applied statistics 1983 and with another MS degree in biostatistics in 1985. He has been working in cancer center statistics at Mayo Clinic from 1985-2000, and in biomedical statistics and informatics from 2000-2014.

Robert A. Greenes is a Professor of Biomedical Informatics at Arizona State University and Mayo Clinic. He has a long history of work in computer-based clinical decision support, knowledge representation, and standards-based approaches to interoperability. He is the author/editor of a major textbook in clinical decision support, now in its second edition.

Hongfang Liu, PhD, is an Associate Professor and Director of the Natural Language Processing (NLP) program at Mayo Clinic Rochester. She has over 16 years’ experience in biomedical informatics, with an extensive background in statistical and expert-based NLP. Her mission is to make NLP accessible, portable, and interoperable.

Rajeev Chaudhry, MD, MPH, is a consultant and Assistant Professor in the Department of Internal Medicine at Mayo Clinic Rochester, and Co-director of Knowledge Delivery Center of the Office of Knowledge and Information Management at Mayo Clinic. He has championed the implementation of clinical decision support (CDS) systems for primary care at Mayo Clinic, and has lead the development and deployment of the CDS system investigated in this study.