Abstract
Objective
To quantify compliance with guideline recommendations for secondary prevention in peripheral artery disease (PAD) using natural language processing (NLP) tools deployed to an electronic health record (EHR) and investigate provider opinions regarding clinical decision support (CDS) to promote improved implementation of these strategies.
Patients and Methods
Natural language processing was used for automated identification of moderate to severe PAD cases from narrative clinical notes of an EHR of patients seen in consultation from May 13, 2015, to July 27, 2015. Guideline-recommended strategies assessed within 6 months of PAD diagnosis included therapy with statins, antiplatelet agents, angiotensin-converting enzyme inhibitors or angiotensin receptor blockers, and smoking abstention. Subsequently, a provider survey was used to assess provider knowledge regarding PAD clinical practice guidelines, comfort in recommending secondary prevention strategies, and potential role for CDS.
Results
Among 73 moderate to severe PAD cases identified by NLP, only 12 (16%) were on 4 guideline-recommended strategies. A total of 207 of 760 (27%) providers responded to the survey; of these 141 (68%) were generalists and 66 (32%) were specialists. Although 183 providers (88%) managed patients with PAD, 51 (25%) indicated they were uncomfortable doing so; 138 providers (67%) favored the development of a CDS system tailored for their practice and 146 (71%) agreed that an automated EHR-derived mortality risk score calculator for patients with PAD would be helpful.
Conclusion
Natural language processing tools can identify cases from EHRs to support quality metric studies. Findings of this pilot study demonstrate gaps in application of guideline-recommended strategies for secondary risk prevention for patients with moderate to severe PAD. Providers strongly support the development of CDS systems tailored to assist them in providing evidence-based care to patients with PAD at the point of care.
Abbreviations and Acronyms: ABI, ankle brachial index; ACEI, angiotensin-converting enzyme inhibitor; AME, AskMayoExpert; ARB, angiotensin II receptor blocker; CDS, clinical decision support; EHR, electronic health record; NLP, natural language processing; PAD, peripheral arterial disease; PPV, positive predictive value
Peripheral artery disease (PAD) is diagnosed in 1 of 20 Americans older than 50 years1 and is an expanding global pandemic affecting more than 200 million individuals in developed and developing countries.2, 3 It increases the risk of mortality and often coexists with ischemic heart disease, the leading cause of death worldwide.4, 5 Accordingly, patients with PAD are at increased risk for myocardial infarction, angina, and stroke.5
Numerous studies have identified gaps in patient and provider knowledge of PAD, which likely contributes to the lack of adoption of evidence-based guideline recommendations in clinical practice.6, 7, 8 The 2016 practice guidelines of the American College of Cardiology and the American Heart Association recommend that optimal secondary prevention therapy for patients with PAD include antiplatelet agents, statins, angiotensin-converting enzyme inhibitors (ACEIs) or angiotensin II receptor blockers (ARBs), and smoking abstention.9 However, patients typically receive only 1 or 2 of the 4 recommended therapies.8, 10 Many reasons likely contribute to the low adherence rates to guidelines including lack of provider knowledge and low provider comfort levels in treating patients with PAD.7, 10
The emerging field of clinical informatics brings innovative approaches to investigate where discrepancies may exist between guideline recommendation and clinical care and may also provide solutions that mitigate gaps in practice. While some institutions have tried to implement information technology systems to address these issues, many have not been adopted or successful because of lack of provider input during creation, leading to dissatisfaction once implemented.11, 12 In the present study, we investigated the gap between optimal and actual treatment for patients with PAD and evaluated provider opinions regarding clinical decision support (CDS) for the implementation of secondary prevention strategies for patients with PAD.
Patients and Methods
Automated Identification of PAD Cases Using Natural Language Processing
A previously validated natural language processing (NLP) algorithm was used to identify PAD patient cases from narrative clinical notes from the electronic health record (EHR).13 The PAD-NLP algorithm used a knowledge-driven approach and consisted of 2 main components: text processing and patient classification. The text processing component was used to find PAD-related concepts in clinical notes using an open source clinical pipeline, MedTagger,14 which analyzed text and identified PAD-related concepts. The PAD-related concepts were then mapped to specify categories used for patient classification. The NLP algorithm identified 73 patients with symptomatic PAD seen from May 13, 2015, to July 27, 2015, in Employee and Community Health, a community primary care practice in Rochester, Minnesota, which encompasses the divisions of primary care internal medicine, family medicine, and community pediatric and adolescent medicine. Employee and Community Health includes a main practice site and 4 additional clinic sites and provides care to approximately 152,000 patients residing in and around Olmsted County, Minnesota.
The performance of the PAD-NLP algorithm to identify PAD cases has been previously validated.13 In this study, criterion standard manual abstraction criteria required a clinical diagnosis of symptomatic lower extremity PAD supported by ancillary diagnostic tests including ankle brachial index (ABI) value of 0.9 or less at rest or 1 minute after exercise,9 presence of poorly compressible arteries based on ABI value of 1.40 or more,9 prior limb revascularization or amputation due to ischemia, acute or critical limb ischemia, or evidence of flow limiting stenosis or occlusion of aortoiliac, femoropopliteal, or infrapopliteal arterial segments by computed tomography angiography, magnetic resonance angiography, or Duplex ultrasound.9 For the study reported herein, patients with asymptomatic PAD with borderline ABI were excluded. The NLP algorithm for automated extraction of clinical characteristics also required the presence of key words describing PAD symptoms to classify a patient as a PAD case. Examples of key words and their lexical variants included claudication, leg (calf or calve) pain (discomfort, cramp), and ischemic ulcer.13 The comprehensive list of key words and the rules applied in the NLP-PAD algorithm have been previously published.13 For the present study, the performance of the PAD-NLP algorithm was also confirmed by manual abstraction of the EHR for all 73 cases.
Quality Indicators
Data collection included detailed review of the EHR of each patient. Characteristics of interest included age, sex, diabetes, hypertension, and smoking status (current smoker, ex-smoker, or not smoker). Medications used within 6 months of PAD diagnosis were recorded and included antiplatelet agents (aspirin or clopidogrel), statins, and ACEIs or ARBs. Patients known to be statin intolerant were included and counted as not following the statin guidelines. For each medication, both generic name and dosage were summarized. When available, the contraindications (ie, reasons to not use a guideline-recommended therapy) and smoking cessation strategies were also noted. In the present study, the manual review of medical records was conducted by trained chart abstractors and verified by a board-certified cardiologist.
Patients were categorized according to the number of guideline-recommended strategies in use within a 6-month interval after PAD diagnosis.15 This ranged from a patient receiving none of the recommended strategies (n = 0) to patients receiving all 4 recommended strategies (n = 4). The guideline-recommended strategies were smoking abstention, use of antiplatelet agents (aspirin or clopidogrel), use of moderate or high-intensity statin, and use of ACEIs or ARBs.9
Therapy with statins, antiplatelet agents, and smoking abstention each have class I strength of recommendation (level of evidence A) for patients with symptomatic PAD. Antihypertensive therapy also has a class I recommendation (level of evidence A) for patients with PAD and hypertension. The only category of antihypertensive agents recommended by published guidelines is ACEIs or ARBs,9 which have a class IIa recommendation (level of evidence A). However, this category of antihypertensive agent has been evaluated by other studies as a quality metric for patients with PAD6, 15, 16 and therefore was also evaluated for the study herein.
Provider Survey
A 10-question provider survey instrument (see Supplemental Appendix, available online at http://mcpiqojournal.org/) was developed in collaboration with the Mayo Clinic Survey Research Center. Initial questions assessed provider demographic characteristics, including role, primary work area, years in practice, and the number of patients seen with PAD. Subsequent questions investigated how helpful providers considered the current version of Ask Mayo Expert (AME), a Mayo Clinic online knowledge-based care process model17 for secondary prevention in patients with PAD as well as provider comfort level in caring for patients with PAD. The final questions investigated provider opinions on the potential helpfulness of an advanced CDS system that would automatically display secondary prevention strategies for patients with PAD and an automated risk score for mortality for these patients at the point of care. The answers were in a check box format, allowing 1-answer responses. Four questions were posed on a 1 to 5 Likert response scale, with 1 corresponding to “low” and 5 corresponding to “high.”18, 19 For analysis, these responses were grouped using the following categories: 1 to 2, 3, and 4 to 5.
The Survey Research Center sent an invitation letter by e-mail with a link to a self-administered web-based survey for 760 providers including all staff physicians, residents, fellows, nurse practitioners, and physician assistants who provide medical care for patients with PAD at Mayo Clinic, Rochester, Minnesota. Of these, 398 were general practitioners (including family medicine, internal medicine, and primary care providers) and 362 were specialists (including cardiology, vascular medicine, and vascular surgery) who provide care to patients with PAD. Of this group, 207 providers responded to the survey. A total of 4 reminder emails were sent until the survey closed on May 10, 2017.
Statistical Analyses
Baseline characteristics are presented as percentages, mean ± SD, or median (25th-75th percentile), as appropriate based on distribution. To assess the NLP algorithm, positive predictive value (PPV) and sensitivity were used to compare with criterion standard abstraction results and calculated as PPV = true positives/(true positives + false positives) and sensitivity = true positives/(true positives + false negatives). Differences in baseline characteristics by number of guideline-recommended strategies were tested using the Pearson χ2 test for categorical variables or t test for continuous variables. P value less than or equal to .05 was considered significant. JMP version 10 was used to conduct statistical analyses.
Results
NLP
The NLP algorithm identified a cohort of 73 patients with PAD with average age of 73 ± 11 years; 74% (54) were men. There were 41 (56%) patients with diabetes, and 69 (95%) had systemic hypertension. When compared with criterion standard manual abstraction, the NLP algorithm had a PPV of 97% and sensitivity of 99% for automated PAD case identification.
Clinical Characteristics of Patients With PAD
Clinical characteristics of 73 patients who comprised the study group are summarized in Table 1. All 73 patients had clinically diagnosed symptomatic PAD and met criteria for secondary prevention strategies per published guidelines.9
Table 1.
Clinical Characteristics of Patients With Symptomatic PADa
| Clinical characteristic | No. of patients |
|---|---|
| Limb amputation | 10 |
| Status after limb revascularization surgery | 11 |
| Status after angioplasty or stenting (femoral or iliac arteries) | 15 |
| ABI <0.9 (at rest or with exerciseb) | 22 |
| Poorly compressible arteries | 13 |
| Moderate or severe stenosis of femoral or iliac artery (or arteries) by magnetic resonance angiography or computed tomography angiography | 2 |
ABI = ankle brachial index; PAD = peripheral artery disease.
There was 1 patient with normal ABI at rest, which decreased with exercise.
Use of Guideline-Recommended Strategies
The most common strategy implemented within 6 months of PAD diagnosis was aspirin therapy used by 63 patients (86%), 15 of whom were also taking clopidogrel. None of the patients took clopidogrel alone; all patients on clopidogrel were also on aspirin. There were 4 patients with a history of aspirin intolerance, but none of the patients with aspirin intolerance was taking clopidogrel in place of aspirin. Smoking abstention was the second most common strategy, implemented in 49 patients (67%). Statins were used by 47 (64%) patients, while ACEIs or ARBs were used by 35 (48%) (Figure 1). Among 24 (33%) current smokers with PAD, smoking abstention counseling or medication was used by 18 patients. In the overall group, only 3 (4%) patients had a history of statin intolerance. For patients without statin intolerance, only 4 (10%) received 3 to 4 guideline-recommended prevention strategies.
Figure 1.
Number of patients receiving each guideline-recommended strategy within 6 months of PAD diagnosis. ACEIs = angiotensin-converting enzyme inhibitors; ARBs = angiotensin II receptor blockers; PAD = peripheral artery disease.
Further analysis showed that only 12 (16%) patients were on all 4 guideline-recommended strategies. Nearly one-third were on 2 (33%) or 3 (36%) strategies within 6 months of PAD diagnosis (Figure 2). Three (4%) patients were on no recommended strategies (Figure 2). Age, sex, or hypertension did not influence the number of strategies used (Table 2). However, diabetic patients (n = 41) were more likely to be on 3 to 4 strategies (n = 26 [63%]) than 1 to 2 strategies (n = 14 [34%]; P=.05).
Figure 2.
Number of guideline-recommended strategies received by each patient within 6 months of PAD diagnosis. PAD = peripheral artery disease.
Table 2.
Risk Profile of Participants
| Variables | Overall (n=73) | 1-2 Strategies (n=32) | 3-4 Strategies (n=38) | P value |
|---|---|---|---|---|
| Age (>60 y), No. (%) | 64 (88) | 28 (88) | 33 (87) | .93 |
| Sex (female), No. (%) | 19 (26) | 7 (22) | 11 (29) | .5 |
| Diabetes, No. (%) | 41 (56) | 14 (44) | 26 (68) | .05 |
| Hypertension, No. (%) | 69 (95) | 30 (94) | 36 (95) | .86 |
Provider Survey
The survey had an overall response rate of 27% (207 of 760 providers). Among these 207 responders, 123 (59%) were staff physicians, 58 (28%) were nurse practitioners or physician assistants, and 26 (13%) residents or fellows (see Supplemental Figure, available online at http://mcpiqojournal.org/). Within the responder group, 141 (68%) were generalists and 66 (32%) were specialists (25% cardiology, 5% vascular medicine, and 2% vascular surgery) (Figure 3). Seventy-seven (37%) respondents were in practice from 0 to 5 years, 32 (15%) from 6 to 10 years, 20 (10%) from 11 to 15 years, 26 (13%) from 16 to 20 years, and 52 (25%) more than 20 years. A total of 183 (88%) respondents currently cared for patients with PAD, and 129 (62%) reported seeing an average of 1 to 5 patients with PAD per month.
Figure 3.
Primary work areas of provider respondents.
Among the 207 respondents, 102 (49%) indicated they never use AME for recommendations regarding PAD risk factor modification. Only 20 (10%) of the providers indicated they use AME for PAD risk modification. When examining the results of how providers viewed the current influence of AME in their practice for PAD risk modification, 93 (45%) of the providers felt that it had no influence. Despite the availability of the PAD-AME care process model, 51 respondents (25%) indicated they felt uncomfortable discussing guideline-recommended risk factor modification strategies for patients with PAD.
When providers were asked how beneficial they felt the potential future availability of care recommendations for patients with PAD in a CDS system would be to their practice, 138 (67%) rated it 4 or 5, indicating it would be very helpful. One hundred forty-eight (71%) providers also indicated they thought having an automated EHR-derived mortality risk score for patients with PAD would be very helpful. Only 34 respondents (16%) indicated that they did not believe it would benefit their practice and patients with PAD (Figure 4).
Figure 4.
Provider opinions regarding future CDS. CDS = clinical decision support; EHR = electronic health record.
Discussion
A major observation of this study was the low proportion of patients with symptomatic PAD treated with all 4 guideline-recommended secondary prevention strategies for PAD. A recent cross-sectional study of adults seeking ambulatory care that investigated the use of guideline-recommended therapies in patients with PAD also reported underuse of these 4 guideline-recommended strategies.6
Previous studies suggest that millions of US adults with PAD do not receive secondary prevention therapies.6, 11 Hence, our study confirms a substantial gap in translation from recommended guidelines to practice implementation and provides evidence that solutions to address this gap are needed.13 A potential approach is the use of CDS tools to provide an individualized approach at the point of care.
Previous studies have investigated the benefits of introducing CDS systems in an effort to increase individualized and guideline-based patient care.20, 21, 22, 23, 24 Although implementations of CDS systems have been successful in increasing guideline-recommended care for other diseases,20, 21, 22, 24, 25 we are not aware of previous systems specifically designed for patients with PAD. Hence, we evaluated the opinions of clinicians regarding the design of a future automated CDS tool geared toward risk assessment and recommendation of secondary prevention strategies for patients with PAD. In comparison to other survey studies, the provider response rate of 27% was greater than a rate of 9% in a survey by Chin-Quee and Yaremchuk,23 investigating medical resident circadian preferences, as well as a response rate of 22% reported by Einstein et al26 in another resident survey on discussions of code status with patients. In addition, the response rate of the study herein was greater than the 10% to 15% range of responses from physicians across specialties (primary care, obstetrics/gynecology, and cardiology) in a national survey of guideline-recommended strategies for cardiovascular disease prevention.27
In our study, only 10% of providers indicated that they currently use AME for PAD risk modification. Although AME provides knowledge-based care process models,17 71% of respondents perceived greater potential benefit of individualized risk score calculators and guideline-recommended strategies at the point of care. Importantly, care process models are not individualized for use at the point of care. In addition, 66% of respondents indicated that if a CDS system had automated secondary prevention recommendations Only 1 level 2 head under “Results” - delete head, close up space, and para indent “Although the...” PAD deployed at point of care it would be very useful for their practice. These high proportions of favorable responses suggest that providers strongly support well-designed CDS systems to assist them in providing evidence-based care to patients with PAD at the point of care.
Although the number of patients identified by the PAD-NLP algorithm was small, this study shows the feasibility of using an automated approach for identification of patients with PAD from an EHR for quality initiatives. This pilot study also confirmed a clear gap between recommended secondary prevention guidelines and implementation for patients with PAD, underscoring the need for innovative solutions to address this gap. Because the study population was from a large community practice and because the practitioners surveyed represent a broad range of practitioners, the results may be generalizable to many practices across the United States. Moreover, the study also demonstrates the feasibility of deploying electronic tools based on NLP to an EHR to support quality initiatives that may be relevant to any practice that uses an EHR in which narrative text is in the English language. Finally, the NLP tools used in this study are portable to any EHR because the MedTagger software14 used for extraction of clinical information from narrative notes is fully functional regardless of vendor and can be downloaded from the Internet at no cost by any potential user.28
The present study was conducted within a single institution at Mayo Clinic, Rochester, Minnesota. However, in a separate multicenter study of the Electronic Medical Records and Genomics network (a national network organized and funded by the National Human Genome Research Institute), the same NLP-PAD algorithm used in this study has been validated by 2 other large academic institutions to demonstrate algorithm portability to other practices and EHRs.
Conclusion
The findings of this pilot study confirm gaps in application of guideline-recommended strategies for secondary risk prevention for patients with PAD as ascertained by electronic tools that use NLP. Providers strongly support the development of CDS systems to assist them in providing evidence-based care to patients with PAD at the point of care. Systems that use NLP to automatically identify patients with PAD may enable improved evidence-based and individualized patient care.
Acknowledgments
We thank the Mayo Research Survey Center and Rebecca M. Olson for secretarial support.
Footnotes
Grant support: The study was supported by grant K01HL124045 from the National Heart, Lung, and Blood Institute of the National Institutes of Health; grants HG04599 and HG006379 from the NHGRI, the Electronic Medical Records and Genomics Network; the Rochester Epidemiology Project (grant number R01-AG034676; Principal Investigators: Walter A. Rocca, MD, MPH and Jennifer L. St Sauver, PhD); and National Institute of General Medical Sciences award R01GM102282 for the natural language processing framework. The content is solely the responsibility of the authors and does not necessarily represent official views of the National Institutes of Health.
Potential Competing Interests: The authors report no competing interests.
Supplemental Online Material
Supplemental material can be found online at http://mcpiqojournal.org/. Supplemental material attached to journal articles has not been edited, and the authors take responsibility for the accuracy of all data.
References
- 1.National Heart, Lung, and Blood Institute NHLBI website. Facts About Peripheral Arterial Disease (P.A.D.) https://catalog.nhlbi.nih.gov/sites/default/files/publicationfiles/06-5837.pdf Accessed February 28, 2018.
- 2.Hirsch A.T., Duval S. The global pandemic of peripheral artery disease. Lancet. 2013;382(9901):1312–1314. doi: 10.1016/S0140-6736(13)61576-7. [DOI] [PubMed] [Google Scholar]
- 3.Fowkes F.G., Aboyans V., Fowkes F.J., McDermott M.M., Sampson U.K., Criqui M.H. Peripheral artery disease: epidemiology and global perspectives. Nat Rev Cardiol. 2017;14(3):156–170. doi: 10.1038/nrcardio.2016.179. [DOI] [PubMed] [Google Scholar]
- 4.World Health Organization WHO website. The top 10 causes of death. http://www.who.int/mediacentre/factsheets/fs310/en/ Accessed February 28, 2018.
- 5.Smolderen K.G., van Zitteren M., Jones P.G., et al. Long-term prognostic risk in lower extremity peripheral arterial disease as a function of the number of peripheral arterial lesions. J Am Heart Assoc. 2015;4(10):e001823. doi: 10.1161/JAHA.115.001823. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Berger J.S., Ladapo J.A. Underuse of prevention and lifestyle counseling in patients with peripheral artery disease. J Am Coll Cardiol. 2017;69(18):2293–2300. doi: 10.1016/j.jacc.2017.02.064. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.El Morr C., AlHamzah M., Ng P., Purewal A., Al-Omran M. Knowledge of peripheral arterial disease: results of an intervention to measure and improve PAD knowledge in Toronto. Vascular. 2017;25(5):479–487. doi: 10.1177/1708538116689355. [DOI] [PubMed] [Google Scholar]
- 8.Kim L.K., Swaminathan R.V., Minutello R.M., et al. Trends in hospital treatments for peripheral arterial disease in the United States and association between payer status and quality of care/outcomes, 2007-2011. Catheter Cardiovasc Interv. 2015;86(5):864–872. doi: 10.1002/ccd.26065. [DOI] [PubMed] [Google Scholar]
- 9.Gerhard-Herman M.D., Gornik H.L., Barrett C., et al. 2016 AHA/ACC Guideline on the Management of Patients With Lower Extremity Peripheral Artery Disease: executive summary; a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines. J Am Coll Cardiol. 2017;69(11):1465–1508. doi: 10.1016/j.jacc.2016.11.008. [DOI] [PubMed] [Google Scholar]
- 10.Hirsch A.T., Criqui M.H., Treat-Jacobson D., et al. Peripheral arterial disease detection, awareness, and treatment in primary care. JAMA. 2001;286(11):1317–1324. doi: 10.1001/jama.286.11.1317. [DOI] [PubMed] [Google Scholar]
- 11.Kostopoulou O., Porat T., Corrigan D., Mahmoud S., Delaney B.C. Diagnostic accuracy of GPs when using an early-intervention decision support system: a high-fidelity simulation. Br J Gen Pract. 2017;67(656):e201–e208. doi: 10.3399/bjgp16X688417. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Shaha J.S., El-Othmani M.M., Saleh J.K., et al. The growing gap in electronic medical record satisfaction between clinicians and information technology professionals: issues of most concern and suggested remediations. J Bone Joint Surg Am. 2015;97(23):1979–1984. doi: 10.2106/JBJS.N.01118. [DOI] [PubMed] [Google Scholar]
- 13.Afzal N., Sohn S., Abram S., et al. Mining peripheral arterial disease cases from narrative clinical notes using natural language processing. J Vasc Surg. 2017;65(6):1753–1761. doi: 10.1016/j.jvs.2016.11.031. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Liu H., Bielinski S.J., Sohn S., et al. An information extraction framework for cohort identification using electronic health records. AMIA Jt Summits Transl Sci Proc. 2013;2013:149–153. [PMC free article] [PubMed] [Google Scholar]
- 15.Armstrong E.J., Chen D.C., Westin G.G., et al. Adherence to guideline-recommended therapy is associated with decreased major adverse cardiovascular events and major adverse limb events among patients with peripheral arterial disease. J Am Heart Assoc. 2014;3(2):e000697. doi: 10.1161/JAHA.113.000697. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Arruda-Olson A.M., Moussa Pacha H., Afzal N., et al. Burden of hospitalization in clinically diagnosed peripheral artery disease: a community-based study. Vasc Med. 2018;23(1):23–31. doi: 10.1177/1358863X17736152. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.North F., Fox S., Chaudhry R. Clinician time used for decision making: a best case workflow study using cardiovascular risk assessments and Ask Mayo Expert algorithmic care process models. BMC Med Inform Decis Mak. 2016;16:96. doi: 10.1186/s12911-016-0334-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Weber C.K., Miglioranza M.H., Moraes M.A., et al. The five-point Likert scale for dyspnea can properly assess the degree of pulmonary congestion and predict adverse events in heart failure outpatients. Clinics (Sao Paulo) 2014;69(5):341–346. doi: 10.6061/clinics/2014(05)08. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Porat T., Delaney B., Kostopoulou O. The impact of a diagnostic decision support system on the consultation: perceptions of GPs and patients. BMC Med Inform Decis Mak. 2017;17(1):79. doi: 10.1186/s12911-017-0477-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Garg A.X., Adhikari N.K., McDonald H., et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293(10):1223–1238. doi: 10.1001/jama.293.10.1223. [DOI] [PubMed] [Google Scholar]
- 21.Wells S., Furness S., Rafter N., et al. Integrated electronic decision support increases cardiovascular disease risk assessment four fold in routine primary care practice. Eur J Cardiovasc Prev Rehabil. 2008;15(2):173–178. doi: 10.1097/HJR.0b013e3282f13af4. [DOI] [PubMed] [Google Scholar]
- 22.Silveira P.C., Ip I.K., Sumption S., Raja A.S., Tajmir S., Khorasani R. Impact of a clinical decision support tool on adherence to the Ottawa Ankle Rules. Am J Emerg Med. 2016;34(3):412–418. doi: 10.1016/j.ajem.2015.11.028. [DOI] [PubMed] [Google Scholar]
- 23.Chin-Quee A.L., Yaremchuk K. Medical residents' circadian preferences across specialties. Laryngoscope. 2017;127(10):2236–2238. doi: 10.1002/lary.26449. [DOI] [PubMed] [Google Scholar]
- 24.Durieux P., Nizard R., Ravaud P., Mounier N., Lepage E. A clinical decision support system for prevention of venous thromboembolism: effect on physician behavior. JAMA. 2000;283(21):2816–2821. doi: 10.1001/jama.283.21.2816. [DOI] [PubMed] [Google Scholar]
- 25.Tajmir S., Raja A.S., Ip I.K., et al. Impact of clinical decision support on radiography for acute ankle injuries: a randomized trial. West J Emerg Med. 2017;18(3):487–495. doi: 10.5811/westjem.2017.1.33053. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Einstein D.J., Einstein K.L., Mathew P. Dying for advice: code status discussions between resident physicians and patients with advanced cancer–a national survey. J Palliat Med. 2015;18(6):535–541. doi: 10.1089/jpm.2014.0373. [DOI] [PubMed] [Google Scholar]
- 27.Mosca L., Linfante A.H., Benjamin E.J., et al. National study of physician awareness and adherence to cardiovascular disease prevention guidelines. Circulation. 2005;111(4):499–510. doi: 10.1161/01.CIR.0000154568.43333.82. [DOI] [PubMed] [Google Scholar]
- 28.Wang Y., Wang L., Rastegar-Mojarad M., et al. Clinical information extraction applications: a literature review. J Biomed Inform. 2018;77:34–49. doi: 10.1016/j.jbi.2017.11.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.