Abstract
Background
Through our work, we have demonstrated how clinical decision support (CDS) tools integrated into the electronic health record (EHR) assist providers in adopting evidence-based practices. This requires confronting technical challenges that result from relying on the EHR as the foundation for tool development; for example, the individual CDS tools need to be built independently for each different EHR.
Objective
The objective of our research was to build and implement an EHR-agnostic platform for integrating CDS tools, which would remove the technical constraints inherent in relying on the EHR as the foundation and enable a single set of CDS tools that can work with any EHR.
Methods
We developed EvidencePoint, a novel, cloud-based, EHR-agnostic CDS platform, and we will describe the development of EvidencePoint and the deployment of its initial CDS tools, which include EHR-integrated applications for clinical use cases such as prediction of hospitalization survival for patients with COVID-19, venous thromboembolism prophylaxis, and pulmonary embolism diagnosis.
Results
The results below highlight the adoption of the CDS tools, the International Medical Prevention Registry on Venous Thromboembolism-D-Dimer, the Wells’ criteria, and the Northwell COVID-19 Survival (NOCOS), following development, usability testing, and implementation. The International Medical Prevention Registry on Venous Thromboembolism-D-Dimer CDS was used in 5249 patients at the 2 clinical intervention sites. The intervention group tool adoption was 77.8% (4083/5249 possible uses). For the NOCOS tool, which was designed to assist with triaging patients with COVID-19 for hospital admission in the event of constrained hospital resources, the worst-case resourcing scenario never materialized and triaging was never required. As a result, the NOCOS tool was not frequently used, though the EvidencePoint platform’s flexibility and customizability enabled the tool to be developed and deployed rapidly under the emergency conditions of the pandemic. Adoption rates for the Wells’ criteria tool will be reported in a future publication.
Conclusions
The EvidencePoint system successfully demonstrated that a flexible, user-friendly platform for hosting CDS tools outside of a specific EHR is feasible. The forthcoming results of our outcomes analyses will demonstrate the adoption rate of EvidencePoint tools as well as the impact of behavioral economics “nudges” on the adoption rate. Due to the EHR-agnostic nature of EvidencePoint, the development process for additional forms of CDS will be simpler than traditional and cumbersome IT integration approaches and will benefit from the capabilities provided by the core system of EvidencePoint.
Keywords: clinical decision support system, cloud based, decision support, development, EHR, electronic health record, evidence-based medicine, health information technology, platform, user-centered design
Introduction
The practice of evidence-based medicine has well-established benefits, including improving patient satisfaction and health outcomes and reducing costs [1,2]. Through our work, we have demonstrated how clinical decision support (CDS) tools integrated into the electronic health record (EHR) assist providers in adopting evidence-based best practices and improving care [1,3,4]. However, although numerous studies on the benefits of EHR-integrated CDS tools have been published, widespread CDS adoption is limited [5-8].
In previous studies, we have demonstrated that effective clinical workflow integration is key to CDS adoption [3,9-14]. The more seamlessly a CDS tool is integrated into real-world EHR interfaces and procedures, the more likely it is to achieve high user-adoption rates. Achieving effective EHR integration for CDS tools, however, requires confronting technical challenges. CDS tools are typically built using an EHR’s proprietary development environment. As such, these tools are limited by EHR functionality, which is often rudimentary, frequently leading to the creation of tools that diverge from the ideal clinical implementation and disrupt clinical and digital workflows [15,16]. In addition, building a CDS tool “into” an EHR limits the ability to distribute the resulting CDS tool beyond the specific EHR it was developed to work with.
Our research had three implementation and dissemination goals: (1) to build the EvidencePoint platform and implement vendor-agnostic integrated CDS solutions for emergency room, inpatient, and ambulatory settings; (2) to perform comprehensive workflow analysis and usability testing with end-user clinicians to achieve optimal levels of tool usefulness, usability, and performance; and (3) to measure outcomes related to tool adoption rates and optimizations to clinical practice workflows.
Methods
Overview
To address CDS challenges, we developed EvidencePoint, a novel, cloud-based, EHR-agnostic CDS platform for the development and hosting of CDS tools. Here, we describe the development of EvidencePoint and the deployment of its initial CDS tools. The platform provides a library of CDS functionality for key clinical scenarios such as antibiotic prescribing, pulmonary embolism diagnosis, and pharmacological thromboprophylaxis. The platform includes an EHR trigger, clinical prediction rule (CPR), and order integration for each CDS tool. This modular functionality allows for the efficient expansion of additional tools. Clinical impact metrics are monitored through the system’s reporting dashboard.
Our team at Northwell’s Usability Lab was awarded an R18 grant through the Agency for Healthcare Research and Quality to begin the development of the EvidencePoint system in March 2019. This required collaboration among the usability lab members and several Northwell EHR and health information exchange (HIE) technical teams, along with an applications development team, a clinical documentation team, and a research innovations and informatics team. Over the course of the design and implementation process, considerations of scalability, portability, workflow, and user experience were paramount. All study activities were approved by the Northwell Health Institutional Review Board.
Workflow Analysis
An important input to the integration of our CDS tools into EvidencePoint was to determine where in the clinical workflow our tools would appear and under which clinical circumstances. As an example, here we describe our process for determining where and how the Wells’ criteria CPR to detect pulmonary embolism CDS tool would be integrated into the emergency department’s (ED) clinical workflow. This methodology was also applied to the integration of all other CDS tools.
Initial workflow analysis for the Wells’ criteria CDS tool began with a task force meeting among the research team and key ED stakeholders, including frontline providers, the chairperson, administrators, and nurses. The meeting included semistructured questions with responses written into notes taken by a research coordinator and research administrator. The types of questions asked were the need for CDS in the ED, the current workflow of patients suspected of pulmonary embolism, where the tool would be needed for effective implementation, and how team members would interact with the tool. The focus was on system interactions and information flow.
After the workflow analysis meeting, research team members shadowed 4 different ED providers (frontline attending physician, resident physician, physician assistant, and primary ED nurse) to observe clinical workflow in real time. Each study team member was paired with 1 of the 4 ED providers. Study team members asked semistructured questions during the shadowing sessions and took notes on clinician responses. The observations, which ranged from 30 minutes to 1 hour, took place over several days in May 2019 in a tertiary care center’s ED.
User Experience
The usability lab team’s previous work has demonstrated that a CDS tool’s usability is crucial to facilitating its adoption by providers [9-11,17,18]. Therefore, parallel to the development work on the underlying EvidencePoint software platform and workflow analysis, we made an effort to ensure the resulting CDS tools would be useful and usable for providers. Following meetings with key stakeholders, including attending physicians, residents, nurses, and physician assistants at each clinical site, the usability lab team started to map out user journeys and workflow analyses of the ED, inpatient hospital unit, and ambulatory clinical environments. Keeping in mind that the primary objective of the EvidencePoint system is to trigger relevant CPRs to the correct providers at the appropriate moments during the clinical decision-making workflow, the workflow analyses served as the preliminary road map for the ensuing usability testing process.
In addition to mapping user journeys and completing workflow analysis, the team also developed a series of interactive wireframe prototypes of potential user interfaces for each CDS tool that we were building. We performed initial rounds of rapid-cycle usability testing on these wireframes (Figure 1).
Figure 1.
International Medical Prevention Registry on Venous Thromboembolism-D-Dimer clinical decision support application. CCU: critical care unit; DOB: date of birth; DVT: deep vein thrombosis; HIE: health information exchange; ICU: intensive care unit; MRN: medical record number; NSUH: North Shore University Hospital.
Our usability testing of the EvidencePoint platform and the integrated CDS tools is ongoing and will continue throughout the development process, including an assessment following the launch of each individual CDS tool to allow for an assessment of critical system analytics such as trigger rates. Our analyses will enable us to fine-tune the platform and the integrated CDS tools to maximize their impact on clinical outcomes while minimizing clinical workflow disruption.
Technical Development of the EvidencePoint Platform
The EvidencePoint platform is divided into 4 components that make up a front-end user interface and a back-end data exchange (Figure 2). The system was developed using standard web technologies, including HTML, CSS, and JavaScript. The front-end interface consists of the clinician-facing EHR and the integrated EvidencePoint CDS tool. The front-end EHR is responsible for launching and running the CDS tool, which is a web-based application that can be displayed within a native EHR window to appear as if it is an integrated part of the EHR clinical or digital workflow. The EvidencePoint CDS tools are hosted on a web server and calculate a predictive score based on CPR. CPRs are validated tools that quantify the individual contributions that components of history, physical, and laboratory results make toward a diagnosis, prognosis, or treatment response [19]. CPRs are prepopulated with patient data in the back-end data exchange, generating an automated CDS assessment.
Figure 2.
EvidencePoint platform structure (left) and communication scheme (right). API: application programming interface; CDS: clinical decision support; EHR: electronic health record.
The back-end data exchange consists of the EvidencePoint application programming interface (API) and the back-end EHRs, namely the HIE. Given the desired CDS tool, the EvidencePoint API translates patient health information (eg, test results and codes) from the HIE to the relevant assessment questions for prepopulation. Then, the API sends the prepopulated prompts and assessment-scoring scheme back to the CDS tool, where the doctor fills out the remainder of the assessment, corrects for errors, and calculates a score. The EvidencePoint software system thus bridges front-end, back-end, preexisting, and entirely bespoke software to bring CDS assessments to clinician workflows. The system currently supports several CDS applications, as described in the Results section.
To address scalability, each of the CDS tools—1 tool for each CPR—is configured on the EvidencePoint server with a text file. The text file specifies each assessment question, how the score is calculated, and the clinical codes relevant to assessment prepopulation. This modular design allows for a near “plug and play” development process for CDS administrators. As long as EHRs are configured to launch the EvidencePoint CDS tool, the system can provide clinicians with the desired CPR at the point of care, prepopulated with relevant patient data. The modular text-file configuration puts future CDS tools well within reach.
To ensure portability to outside institutions and make widespread dissemination possible, the platform takes an EHR-agnostic approach. The CDS tools and EvidencePoint API run on separate servers from the Northwell EHRs and communicate through standard protocols. Thus, the platform can be configured to launch from any health system’s EHR based on that EHR’s particular data (Figure 3).
Figure 3.
Schematics of the flow of data between the health care system, electronic health record (EHR), and health information exchange (HIE). Doctors launch an EvidencePoint clinical decision support tool from an EHR front-end workflow. The request includes the desired clinical prediction rule (eg, the Wells’ criteria) and the patient’s visit-specific ID. The clinical decision support tool forwards the request to the EvidencePoint application programming interface, which retrieves patient data, prepopulates evaluation answers, and sets the clinical prediction rule calculation logic. After calculating the patient score, the clinical decision support tool returns the score to the EHR front-end workflow.
For example, from an end user perspective, the Wells’ criteria for pulmonary embolism CPR comprises several components [20-23], including “heart rate greater than 100” and “hemoptysis.” The EvidencePoint platform supports user-editable fields for each specific component of the CPR (eg, “heart rate greater than 100” and “hemoptysis”) and automatically calculates the associated point values for each (such as +1.5 for “heart rate greater than 100” and +1 for “hemoptysis”; Table 1). The user-editable field is pulled from the EHR and can be updated by the user if needed.
Table 1.
The Wells’ score clinical prediction rule criteria.
| Wells’ criteria (as seen by end user) | User-editable field | |
| Heart rate greater than 100 | input:radio,Heart rate greater than 100,,,well | |
| selected:Medical=1.5 | ||
| unselected:Medical=0 | ||
| Hemoptysis | input:radio,Hemoptysis,,,wells | |
| selected:Medical=1 | ||
| unselected:Medical=0 | ||
The text file also provides a mechanism to specify, on a component-by-component basis, the source for retrieving extant patient-specific data with which to prepopulate specific CPR components. The methods used to retrieve these data can include vendor-specific calls in addition to general data requests.
CDS Tool Deployment
Following the development of the EvidencePoint platform and workflow analysis, we developed our first 3 CDS tools for the platform. We describe these CDS tools below.
Northwell COVID-19 Survival Tool
One of the most important features of EvidencePoint is that it enables the rapid development, implementation, and dissemination of new CDS tools at the point of care, especially during times of urgent need. EvidencePoint’s debut came as the COVID-19 pandemic hit the New York City area, putting thousands of patients on ventilators, straining health care personnel, and stretching scarce medical supplies. Because the EvidencePoint platform enables the rapid development of CDS, Northwell Health’s research and software teams were able to respond quickly to the needs of the situation, developing and releasing the Northwell COVID-19 Survival (NOCOS) CPR [24], a tool that estimates a patient’s probability of survival during hospitalization. Following its success, the teams developed NOCOS 2.0, which integrated with the Allscripts Sunrise EHR to prepopulate patient information and return the NOCOS score back to the EHR for subsequent use. This was the first implementation of an EHR-integrated CDS tool running on the EvidencePoint platform (Figure 4A).
Figure 4.
The Northwell COVID-19 Survival (NOCOS), the International Medical Prevention Registry on Venous Thromboembolism-D-Dimer (IMPROVE-DD), and the Wells’ Score clinical decision support applications. (A) The NOCOS tool for predicting a patient with COVID-19’s probability of surviving hospitalization. This noninterruptive tool is meant for use in an emergency department (ED) setting. (B) The IMPROVE-DD tool for predicting a patient’s risk of developing a venous thromboembolism. This interruptive tool is meant for use in an inpatient setting. (C) The Wells’ Score tool for assisting with the diagnosis of pulmonary embolism (PE). This interruptive tool is meant for use in the ED. CCU: critical care unit; CTA: computed tomography angiography; DVT: Deep Vein Thrombosis; EHR: electronic health record; ICU: intensive care unit.
International Medical Prevention Registry on Venous Thromboembolism-D-Dimer Tool
Following the success of the NOCOS tool, our team released the International Medical Prevention Registry on Venous Thromboembolism-D-Dimer (IMPROVE-DD) tool (Figure 4B). This tool incorporates the IMPROVE-DD score for determining appropriate venous thromboembolism (VTE) prophylaxis. Hospital-acquired VTE is prevalent in the United States [25], with hospitalization as a significant risk factor in patients who are medically ill [26,27]. In addition, a medical patient’s risk of developing a VTE remains high up to 3 months post discharge [27]. The IMPROVE-DD scoring tool was developed based on the previous IMPROVE score [26], with the additional inclusion of the D-dimer blood test to better predict VTE risk [28] and was externally validated both in medical inpatients and inpatients with COVID-19 [27,29,30]. Therefore, due to the need for VTE risk assessment to determine appropriate prophylactic measures, especially during the pandemic, the IMPROVE-DD score was an excellent candidate for incorporation into the EvidencePoint platform. In comparison to NOCOS, IMPROVE-DD features had better integration with EHR workflows and more extensive usability testing. A total of 7 usability testing sessions were conducted for the IMPROVE-DD tool. The tool integrates seamlessly into 3 clinician workflows (ie, at admission history and physical, order entry, and discharge summary); prepopulates the assessment with relevant patient information; and returns a score that provides actionable patient recommendations that are tied to anticoagulant medication order entry to prevent VTE.
Wells’ Criteria Tool
From an implementation perspective, the EvidencePoint platform also allows us to add new functionality seamlessly to enhance our simpler previous CDS tools. Before EvidencePoint’s inception, we developed a version of the Wells’ criteria CDS tool [31]. A total of 7 usability testing sessions were completed for the Wells’ criteria tool. This tool was integrated into the EHR in a traditional format and has not been widely adopted by our intended end users (ED physicians and physician assistants) [31,32]. Including the Wells’ criteria CDS on the EvidencePoint platform (Figure 4C) will allow for rapid modifications of the tool that should enhance its usability, usefulness, and adoption. For example, the Wells’ criteria CDS will incorporate behavioral nudges aimed at encouraging clinicians to use the tool. The nudges will address areas that we have identified as barriers to the CDS tool’s use [32]. While we do not yet know which behavioral nudge will be most effective in terms of encouraging adoption of the Wells’ criteria CDS, our EvidencePoint platform will enable us to seamlessly switch among nudges so that we can study their impact.
Ethical Considerations
The study was approved by Northwell Health’s institutional review board (protocol numbers 19-0045 and 18-0714) as minimal-risk research using observational data. Collected data were a part of routine clinical practice, and the requirement for informed consent was waived. Data were collected from the enterprise EHR (Sunrise Clinical Manager, Allscripts) reporting database. Adoption and usage data are aggregated, with no identifiable information included.
Results
Shadowing sessions from the workflow analysis revealed several key feedback points. We addressed as many feedback points as possible in order to develop a tool that would be of ideal use to our end users. Key points noted during the meeting were the need to include an alert for the CDS in a manner that would not slow down the clinical workflow and the incorporation of the CDS early on during the clinical decision-making process. Additionally, usability testing provided valuable insight into aspects of the user experience that could be improved. As an example, the initial wireframe prototype for the IMPROVE-DD tool (Figure 1) presented users with the numerical point value of the CPR calculations the tool was performing. We learned from users that, rather than the absolute point value, it would be more beneficial to display the absolute risk percentage that users believe derives greater clinical meaning (Figure 4B).
The results below highlight the adoption of the CDS tools, the IMPROVE-DD, the Wells’ criteria, and the NOCOS, following development, usability testing, and implementation. The IMPROVE-DD CDS was used in 5249 patients at the 2 clinical intervention sites. The intervention group tool adoption was 77.8% (4083/5249 possible uses). For the NOCOS tool, which was designed to assist with triaging patients with COVID-19 for hospital admission in the event of constrained hospital resources, the worst-case resourcing scenario never materialized and triaging was never required. As a result, the NOCOS tool was not frequently used, though the EvidencePoint platform’s flexibility and customizability enabled the tool to be developed and deployed rapidly under the emergency conditions of the pandemic. Adoption rates for the Wells’ criteria tool will be reported in a future publication.
Discussion
Principal Findings
The EvidencePoint system successfully demonstrated that a flexible, user-friendly platform for hosting CDS tools outside of a specific EHR is feasible. The forthcoming results of our outcomes analyses will demonstrate the adoption rate of EvidencePoint tools as well as the impact of behavioral economics “nudges” on the adoption rate. Due to the EHR-agnostic nature of EvidencePoint, the development process for additional forms of CDS will be simpler than traditional and cumbersome IT integration approaches and will benefit from the capabilities provided by the core system of EvidencePoint.
With feasibility and proof-of-concept work completed, the IMPROVE-DD tool running on the EvidencePoint platform was released in production environments at 2 of Northwell’s largest tertiary hospitals in December 2020. In September 2021, the Wells’ criteria CDS tool was released in 2 tertiary hospitals and 1 community hospital. Once field-tested in these initial production environments, EvidencePoint will be capable of expanding to include additional forms of CDS (ie, dissemination and implementation of a child abuse CDS funded by Patient-Centered Outcomes Research Institute award No. DI-2017C1-6215) [33] and will be deployed at additional locations.
A recent study reviewed the design and implementation of CDS using 4 interoperability standards: Fast Healthcare Interoperability Resources (FHIR), Substitutable Medical Applications, Reusable Technologies (SMART), clinical quality language, and CDS Hooks [34]. A total of 44 studies were included in the review, of which 43 used FHIR, 22 used SMART, 2 used clinical quality language, and 8 used CDS Hooks. It was noted that evaluation of the technology was reported in a small number of studies, and many studies were in the design and piloting stages. Additional studies on CDS standards will be necessary to inform design decisions for further implementation and dissemination of the EvidencePoint platform moving forward.
Future work on EvidencePoint will include wider-scale dissemination, enabling the platform to be used beyond the Northwell environment. We will enhance EvidencePoint to fully support evolving development standards, such as SMART on FHIR, and provide operators of hospitals and ambulatory sites with a novel tool kit of usability-tested, workflow-integrated functionality directly aimed at bringing sophisticated evidence-based medicine to the point of care.
Due to the EHR-agnostic nature of EvidencePoint, the development process for these additional forms of CDS will be simpler than traditional and cumbersome IT integration approaches and will benefit from the capabilities provided by the core system. This new functionality will be available to multiple EHRs from various vendors since it will reside in a cloud system as opposed to being built directly into the EHR. As evidenced in the previously detailed NOCOS example, we will be able to quickly modify individual CDS tools to meet the needs of our end users and optimize our tools’ adoption and use. As noted, behavioral nudges will be activated to address barriers to CDS uptake and usage. By providing a library of evidence-based CDS tools that can be integrated directly into the flow of care while also being easy to disseminate to multiple locations on an EHR-agnostics basis, EvidencePoint will help to reduce the “evidence gap” that has traditionally made it difficult to use evidence at the point of care.
Limitations
A few limitations are noted as the results of the ongoing outcomes study are forthcoming. Future publications will discuss ordering behavior, adoption and acceptance by the provider, and the overall effectiveness of the CDS tools in clinical outcomes (ie, decreased hospital-acquired VTE and decreases in unnecessary diagnostic testing for pulmonary embolism based on patient risk stratification). Currently, the IMPROVE-DD tool has been deployed in 4 clinical sites in a clustered randomized controlled trial as part of an impact analysis, and improved outcomes will be reviewed and forthcoming. In addition, while we have attempted to address all of our end-user feedback, the technological limitations of our EHR prevent us from making some of the changes that would further enhance the user experience with the system.
Although the study was limited to the Northwell Health System, it included the implementation of the EvidencePoint platform and CDS tools at academic tertiary centers and community-based hospitals. Given the size of the health system as well as the vast range of differentiated demographics among the patient population during one of the greatest health crises in recent history, this constraint alleviated much of the team’s concern regarding further implementation and dissemination of the system.
Conclusions
Due to the EHR-agnostic nature of EvidencePoint, the development process for additional forms of CDS will be simpler than traditional and cumbersome IT integration approaches and will benefit from the capabilities provided by the core system of EvidencePoint.
Acknowledgments
Financial support for this study was provided by grants from the US Department of Health and Human Services, Agency for Healthcare Research and Quality grant 5R18HS026196 and the National Heart, Lung, and Blood Institute grant K23HL145114. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report.
Abbreviations
- API
application programming interface
- CDS
clinical decision support
- CPR
clinical prediction rule
- ED
emergency department
- EHR
electronic health record
- FHIR
Fast Healthcare Interoperability Resources
- HIE
health information exchange
- IMPROVE-DD
International Medical Prevention Registry on Venous Thromboembolism-D-Dimer
- NOCOS
Northwell COVID-19 Survival
- SMART
Substitutable Medical Applications, Reusable Technologies
- VTE
venous thromboembolism
Data Availability
The data sets generated and analyzed during this study are available from the corresponding author on reasonable request.
Footnotes
Conflicts of Interest: None declared.
References
- 1.McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS. Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. JAMA. 2000;284(1):79–84. doi: 10.1001/jama.284.1.79.jml90005 [DOI] [PubMed] [Google Scholar]
- 2.Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. 1996. Clin Orthop Relat Res. 2007;455:3–5. [PubMed] [Google Scholar]
- 3.McGinn T. Putting meaning into meaningful use: a roadmap to successful integration of evidence at the point of care. JMIR Med Inform. 2016;4(2):e16. doi: 10.2196/medinform.4553. https://medinform.jmir.org/2016/2/e16/ v4i2e16 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.McGinn TG, McCullagh L, Kannry J, Knaus M, Sofianou A, Wisnivesky JP, Mann DM. Efficacy of an evidence-based clinical decision support in primary care practices: a randomized clinical trial. JAMA Intern Med. 2013;173(17):1584–1591. doi: 10.1001/jamainternmed.2013.8980. https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/1722509 .1722509 [DOI] [PubMed] [Google Scholar]
- 5.Cresswell K, Majeed A, Bates DW, Sheikh A. Computerised decision support systems for healthcare professionals: an interpretative review. Inform Prim Care. 2012;20(2):115–128. doi: 10.14236/jhi.v20i2.32. http://hijournal.bcs.org/index.php/jhi/article/view/32 .32 [DOI] [PubMed] [Google Scholar]
- 6.Ghédira C, Maret P, Fayn J, Rubel P. Adaptive user interface customization through browsing knowledge capitalization. Int J Med Inform. 2002;68(1-3):219–228. doi: 10.1016/s1386-5056(02)00078-3. https://www.sciencedirect.com/science/article/abs/pii/S1386505602000783?via%3Dihub .S1386505602000783 [DOI] [PubMed] [Google Scholar]
- 7.Lieberman H, Paternò F, Klann M, Wulf V. End-user development: an emerging paradigm. End User Dev. 2006:1–8. doi: 10.1007/1-4020-5386-x_1. [DOI] [Google Scholar]
- 8.Crunkhorn AE. The philosophical limits of evidence-based medicine. Tonelli MR (University of Washington School of Medicine, Seattle, Wash). acad med. 1998;73:1234-1240. J phys ther educ. 2001;15(3):75–76. doi: 10.1097/00001416-200107000-00017. https://journals.lww.com/jopte/citation/2001/07000/the_philosophical_limits_of_evidence_based.17.aspx . [DOI] [PubMed] [Google Scholar]
- 9.Press A, DeStio C, McCullagh L, Kapoor S, Morley J, Conigliaro J, SBIRT NY-II Team Usability testing of a national substance use screening tool embedded in electronic health records. JMIR Hum Factors. 2016;3(2):e18. doi: 10.2196/humanfactors.5820. https://humanfactors.jmir.org/2016/2/e18/ v3i2e18 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Li AC, Kannry JL, Kushniruk A, Chrimes D, McGinn TG, Edonyabo D, Mann DM. Integrating usability testing and think-aloud protocol analysis with "near-live" clinical simulations in evaluating clinical decision support. Int J Med Inform. 2012;81(11):761–772. doi: 10.1016/j.ijmedinf.2012.02.009. https://www.sciencedirect.com/science/article/abs/pii/S138650561200041X?via%3Dihub .S1386-5056(12)00041-X [DOI] [PubMed] [Google Scholar]
- 11.Press A, McCullagh L, Khan S, Schachter A, Pardo S, McGinn T. Usability testing of a complex clinical decision support tool in the emergency department: lessons learned. JMIR Hum Factors. 2015;2(2):e14. doi: 10.2196/humanfactors.4537. https://humanfactors.jmir.org/2015/2/e14/ v2i2e14 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Feldstein DA, Hess R, McGinn T, Mishuris RG, McCullagh L, Smith PD, Flynn M, Palmisano J, Doros G, Mann D. Design and implementation of electronic health record integrated clinical prediction rules (iCPR): a randomized trial in diverse primary care settings. Implement Sci. 2017;12(1):37. doi: 10.1186/s13012-017-0567-y. https://implementationscience.biomedcentral.com/articles/10.1186/s13012-017-0567-y .10.1186/s13012-017-0567-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Richardson S, Mishuris R, O'Connell A, Feldstein D, Hess R, Smith P, McCullagh L, McGinn T, Mann D. "Think aloud" and "Near live" usability testing of two complex clinical decision support tools. Int J Med Inform. 2017;106:1–8. doi: 10.1016/j.ijmedinf.2017.06.003. https://europepmc.org/abstract/MED/28870378 .S1386-5056(17)30180-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.McCullagh LJ, Sofianou A, Kannry J, Mann DM, McGinn TG. User centered clinical decision support tools: adoption across clinician training level. Appl Clin Inform. 2014;5(4):1015–1025. doi: 10.4338/ACI-2014-05-RA-0048. https://europepmc.org/abstract/MED/25589914 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Evans RS. Electronic health records: then, now, and in the future. Yearb Med Inform. 2016;25(Suppl 1):S48–S61. doi: 10.15265/IYS-2016-s006. http://www.thieme-connect.com/DOI/DOI?10.15265/IYS-2016-s006 .me2016-s006 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Ajami S, Bagheri-Tadi T. Barriers for adopting Electronic Health Records (EHRs) by physicians. Acta Inform Med. 2013;21(2):129–134. doi: 10.5455/aim.2013.21.129-134. https://europepmc.org/abstract/MED/24058254 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Mann DM, Kannry JL, Edonyabo D, Li AC, Arciniega J, Stulman J, Romero L, Wisnivesky J, Adler R, McGinn TG. Rationale, design, and implementation protocol of an electronic health record integrated clinical prediction rule (iCPR) randomized trial in primary care. Implement Sci. 2011;6:109. doi: 10.1186/1748-5908-6-109. https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-6-109 .1748-5908-6-109 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Melnick ER, Hess EP, Guo G, Breslin M, Lopez K, Pavlo AJ, Abujarad F, Powsner SM, Post LA. Patient-centered decision support: formative usability evaluation of integrated clinical decision support with a patient decision aid for minor head injury in the emergency department. J Med Internet Res. 2017;19(5):e174. doi: 10.2196/jmir.7846. https://www.jmir.org/2017/5/e174/ v19i5e174 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA. 1997;277(6):488–494. [PubMed] [Google Scholar]
- 20.Anderson DR, Kovacs MJ, Kovacs G, Stiell I, Mitchell M, Khoury V, Dryer J, Ward J, Wells PS. Combined use of clinical assessment and d-dimer to improve the management of patients presenting to the emergency department with suspected deep vein thrombosis (the EDITED study) J Thromb Haemost. 2003;1(4):645–651. doi: 10.1046/j.1538-7836.2003.00131.x. https://linkinghub.elsevier.com/retrieve/pii/131 .131 [DOI] [PubMed] [Google Scholar]
- 21.Wells PS, Anderson DR, Bormanis J, Guy F, Mitchell M, Gray L, Clement C, Robinson KS, Lewandowski B. Value of assessment of pretest probability of deep-vein thrombosis in clinical management. Lancet. 1997;350(9094):1795–1798. doi: 10.1016/S0140-6736(97)08140-3.S0140-6736(97)08140-3 [DOI] [PubMed] [Google Scholar]
- 22.Wells PS, Anderson DR, Ginsberg J. Assessment of deep vein thrombosis or pulmonary embolism by the combined use of clinical model and noninvasive diagnostic tests. Semin Thromb Hemost. 2000;26(6):643–656. doi: 10.1055/s-2000-13219. [DOI] [PubMed] [Google Scholar]
- 23.Wells PS, Owen C, Doucette S, Fergusson D, Tran H. Does this patient have deep vein thrombosis? JAMA. 2006;295(2):199–207. doi: 10.1001/jama.295.2.199.295/2/199 [DOI] [PubMed] [Google Scholar]
- 24.Levy TJ, Richardson S, Coppa K, Barnaby DP, McGinn T, Becker LB, Davidson KW, Cohen SL, Hirsch JS, Zanos T. Development and validation of a survival calculator for hospitalized patients with COVID-19. medRxiv. doi: 10.1101/2020.04.22.20075416. doi: 10.1101/2020.04.22.20075416. Preprint posted online on April 30 2020. 2020.04.22.20075416 [DOI] [Google Scholar]
- 25.Beckman MG, Hooper WC, Critchley SE, Ortel TL. Venous thromboembolism: a public health concern. Am J Prev Med. 2010;38(4 Suppl):S495–S501. doi: 10.1016/j.amepre.2009.12.017. https://www.ajpmonline.org/article/S0749-3797(09)00946-5/fulltext .S0749-3797(09)00946-5 [DOI] [PubMed] [Google Scholar]
- 26.Spyropoulos AC, Anderson FA, FitzGerald G, Decousus H, Pini M, Chong BH, Zotz RB, Bergmann JF, Tapson V, Froehlich JB, Monreal M, Merli GJ, Pavanello R, Turpie AGG, Nakamura M, Piovella F, Kakkar AK, Spencer FA, IMPROVE Investigators Predictive and associative models to identify hospitalized medical patients at risk for VTE. Chest. 2011;140(3):706–714. doi: 10.1378/chest.10-1944. https://journal.chestnet.org/article/S0012-3692(11)60477-7/fulltext .S0012-3692(11)60477-7 [DOI] [PubMed] [Google Scholar]
- 27.Spyropoulos AC, Lipardi C, Xu J, Peluso C, Spiro TE, De Sanctis Y, Barnathan ES, Raskob GE. Modified IMPROVE VTE risk score and elevated d-dimer identify a high venous thromboembolism risk in acutely ill medical population for extended thromboprophylaxis. TH Open. 2020;4(1):e59–e65. doi: 10.1055/s-0040-1705137. http://www.thieme-connect.com/DOI/DOI?10.1055/s-0040-1705137 .190059 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Gibson CM, Spyropoulos AC, Cohen AT, Hull RD, Goldhaber SZ, Yusen RD, Hernandez AF, Korjian S, Daaboul Y, Gold A, Harrington RA, Chi G. The IMPROVEDD VTE risk score: incorporation of d-dimer into the improve score to improve venous thromboembolism risk stratification. TH Open. 2017;1(1):e56–e65. doi: 10.1055/s-0037-1603929. https://europepmc.org/abstract/MED/31249911 .17006 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Rosenberg D, Eichorn A, Alarcon M, McCullagh L, McGinn T, Spyropoulos AC. External validation of the risk assessment model of the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) for medical patients in a tertiary health system. J Am Heart Assoc. 2014;3(6):e001152. doi: 10.1161/JAHA.114.001152. https://www.ahajournals.org/doi/10.1161/JAHA.114.001152?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub0pubmed .jah3760 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Spyropoulos AC, Cohen SL, Gianos E, Kohn N, Giannis D, Chatterjee S, Goldin M, Lesser M, Coppa K, Hirsch JS, McGinn T, Barish MA, COVID‐19 Consortium Group Validation of the IMPROVE-DD risk assessment model for venous thromboembolism among hospitalized patients with COVID-19. Res Pract Thromb Haemost. 2021;5(2):296–300. doi: 10.1002/rth2.12486. https://linkinghub.elsevier.com/retrieve/pii/S2475-0379(22)01337-1 .S2475-0379(22)01337-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Richardson S, Cohen S, Khan S, Zhang M, Qiu G, Oppenheim MI, McGinn T. Higher imaging yield when clinical decision support is used. J Am Coll Radiol. 2020;17(4):496–503. doi: 10.1016/j.jacr.2019.11.021. https://europepmc.org/abstract/MED/31899178 .S1546-1440(19)31309-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Richardson S, Dauber-Decker KL, McGinn T, Barnaby DP, Cattamanchi A, Pekmezaris R. Barriers to the use of clinical decision support for the evaluation of pulmonary embolism: qualitative interview study. JMIR Hum Factors. 2021;8(3):e25046. doi: 10.2196/25046. https://humanfactors.jmir.org/2021/3/e25046/ v8i3e25046 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.McGinn T, Feldstein DA, Barata I, Heineman E, Ross J, Kaplan D, Richardson S, Knox B, Palm A, Bullaro F, Kuehnel N, Park L, Khan S, Eithun B, Berger RP. Dissemination of child abuse clinical decision support: moving beyond a single electronic health record. Int J Med Inform. 2021;147:104349. doi: 10.1016/j.ijmedinf.2020.104349. https://linkinghub.elsevier.com/retrieve/pii/S1386-5056(20)31885-2 .S1386-5056(20)31885-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Taber P, Radloff C, Del Fiol G, Staes C, Kawamoto K. New standards for clinical decision support: a survey of the state of implementation. Yearb Med Inform. 2021;30(1):159–171. doi: 10.1055/s-0041-1726502. http://www.thieme-connect.com/DOI/DOI?10.1055/s-0041-1726502 . [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The data sets generated and analyzed during this study are available from the corresponding author on reasonable request.