Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2023 Mar 1.
Published in final edited form as: Healthc (Amst). 2021 Dec 16;10(1):100598. doi: 10.1016/j.hjdsi.2021.100598

Collaborative Design and Implementation of a Clinical Decision Support System for Automated Fall-Risk Identification and Referrals in Emergency Departments

Gwen Costa Jacobsohn a, Margaret Leaf b, Frank Liao a,b, Apoorva P Maru a, Collin J Engstrom a,h, Megan E Salwei c,d,i, Gerald T Pankratz e, Alexis Eastman e, Pascale Carayon c,d, Douglas A Wiegmann c,d, Joel S Galang b, Maureen A Smith f,g, Manish N Shah a,e,g, Brian W Patterson a,f
PMCID: PMC8881336  NIHMSID: NIHMS1763469  PMID: 34923354

Abstract

Of the 3 million older adults seeking fall-related emergency care each year, nearly one-third visited the Emergency Department (ED) in the previous 6 months. ED providers have a great opportunity to refer patients for fall prevention services at these initial visits, but lack feasible tools for identifying those at highest-risk. Existing fall screening tools have been poorly adopted due to ED staff/provider burden and lack of workflow integration. To address this, we developed an automated clinical decision support (CDS) system for identifying and referring older adult ED patients at risk of future falls.

We engaged an interdisciplinary design team (ED providers, health services researchers, information technology/predictive analytics professionals, and outpatient Falls Clinic staff) to collaboratively develop a system that successfully met user requirements and integrated seamlessly into existing ED workflows. Our rapid-cycle development and evaluation process employed a novel combination of human-centered design, implementation science, and patient experience strategies, facilitating simultaneous design of the CDS tool and intervention implementation strategies. This included defining system requirements, systematically identifying and resolving usability problems, assessing barriers and facilitators to implementation (e.g., data accessibility, lack of time, high patient volumes, appointment availability) from multiple vantage points, and refining protocols for communicating with referred patients at discharge. ED physician, nurse, and patient stakeholders were also engaged through online surveys and user testing.

Successful CDS design and implementation required integration of multiple new technologies and processes into existing workflows, necessitating interdisciplinary collaboration from the onset. By using this iterative approach, we were able to design and implement an intervention meeting all project goals. Processes used in this Clinical-IT-Research partnership can be applied to other use cases involving automated risk-stratification, CDS development, and EHR-facilitated care coordination.

Keywords: Geriatric Emergency Medicine, Falls, Clinical Decision Support, Human-Centered Design, Implementation, Risk Stratification, Electronic Health Record

Background

Falls are the leading cause of traumatic injury and a common cause of mortality among adults age 65 and older [1,2]. National data indicate that older adults experience 35.6 million falls per year with incidence rates of 27–34%, significantly increasing with age [3,4]. Falls result in approximately $50 billion in medical costs annually, a number that has been rapidly increasing along with the aging population [2,5,6]. Almost 3 million older adults are seen for fall-related injuries in United States emergency departments (EDs) each year [3,7], about one-third of whom had at least one ED visit in the 6 months prior to the fall [8]. Although this demonstrates a key reason why the ED is an ideal site for identifying patients at risk for future falls, it also highlights a missed opportunity to refer high-risk patients for fall-prevention services when they are initially seen in the ED [911].

Guidelines for in-person ED fall screening and risk evaluation have not been widely adopted in practice [1113], and more complex ED-based interventions for screening and referral have not been widely implemented [14]. Other opportunities to evaluate ED patients for modifiable fall risk are routinely missed [15]. As the population ages and EDs continue to expand their role as the primary site for delivery of acute unscheduled care [16,17], the need to create a scalable intervention to assess older adults for fall risk and connect them to appropriate risk reduction interventions following ED discharge has become more urgent and research investigating effective programs has become a priority [13,14].

To create an acceptable and efficient way to identify and refer high-risk patients during ED visits without negatively impacting patient care, our interdisciplinary team has designed an automated clinical decision support (CDS) tool utilizing electronic health records (EHR) data and machine learning-based risk stratification algorithms. CDS systems demonstrate great promise in using patient data to tailor clinical care, but their complexity and perceived lack of transparency can make clinical integration and acceptance a challenge [1821]. Based on previous research and past experience, we knew this intervention would require collaborative design, usability testing, tailored implementation strategies, and continuous evaluation processes to succeed in practice.

Organizational Context

UW Health provides 2.3 million ambulatory visits per year at close to 50 clinical locations throughout Wisconsin and Northern Illinois. The University Hospital ED is a level 1 trauma center with over 60,000 visits per year (including over 10,000 visits by older adults). UW Health has utilized a comprehensive EHR (Epic, Verona, WI) system since 2003 integrating emergency, inpatient, and outpatient care. The BerbeeWalsh Department of Emergency Medicine is also affiliated with the University of Wisconsin-Madison School of Medicine and Public Health, which provides resources actively supporting collaborative healthcare research and the adoption of evidence-based innovations aimed at improving population health outcomes.

The UW Health Mobility and Falls Clinic is a consultative clinic administered by the Division of Geriatrics, funded as a self-sustaining program through the Department of Medicine with visits billed as per standard organizational practice. Clinic visits are multidisciplinary, involving a medical geriatric assessment focused on medications and medical conditions associated with increased fall risk, a physical therapy assessment focused on balance, strength, and mobility, and social work assessment focused on home safety, caregiving, and community resources. Patients and caregivers receive personal instruction regarding fall prevention. Patients’ primary care physicians also receive specific recommendations regarding medications, testing, and suggestions for additional services. Follow up appointments, when indicated, generally occur over the span of 6–12 months. Patients can either be referred for services from providers (within or external to the health system) or can self-refer.

The Falls Clinic provides a clinical intervention similar to that studied in the PROFET trial, which found the percentage of repeat fallers decreased from 52% to 32% in a high risk cohort of patients discharged from the ED after a fall—a relative risk reduction of 38% (95% CI: 21%–52%) [22,23]. Such multifactorial fall-prevention interventions are recommended for high risk older adults by both the US Preventative Services Taskforce and the CDC STEADI initiative [24,25].

Problem

Historically, UW ED providers referred almost no patients to the Falls Clinic even though the clinic had capacity to accommodate additional patients. This includes patients seen in the ED for fall-related injuries, a well-known predictor of future falls in older adults [26]. During the formative stage of project development, ED clinicians explained that they lacked knowledge about Fall Clinic services, referral placement, and how to identify patients who might benefit. They also cited professional barriers, as some were conflicted about the appropriateness of EDs as sources of preventative non-acute care [27,28]. These perspectives are consistent with other recent research on barriers to fall-related screening and intervention among ED providers, with 87% citing lack of time, 51% citing lack of knowledge, and 47% citing lack of fall-prevention resources [29].

Fall-risk assessments commonly used in other healthcare environments (e.g. inpatient, long-term care) have shown poor performance at predicting falls following ED care [8,3032]. This is understandable given differences in patient acuity, clinical environment, care practices, and workflow compared to other clinical and inpatient settings [3338]. A few ED-specific interventions for identifying and referring high-risk patients for fall prevention services have proven efficacy, but their delivery is dependent upon real-time effort by ED providers and staff [22,3942]. In general, in-person screening tools requiring extensive time and human resources are a known barrier to widespread adoption in the ED, regardless of feasibility or efficacy [27,28,43,44]. Fall screening is no exception, for although 84% of ED providers believe all older patients should be screened for falls, and 76% feel those determined to be at risk should receive a preventative intervention, 50% are unwilling to spend 2 or more minutes on the task [29]. As a result, fall risk assessment frequently falls to ED nurses, who are already responsible for administering multiple screening tools during intake, adding significant time to their clinical workflow.

In addition to screening, point of care technologies perceived as particularly inefficient, not integrated into ED workflows, or unrelated to present acute problems are also seen as adding unnecessary burden, resulting in low acceptance and infrequent use [18,19,34,38,45]. In order to address these problems, we needed to develop and pilot test a reliable, accurate, and user-friendly CDS that was (1) specific to the ED context, (2) integrated into existing workflows, and (3) as automated as possible to minimize clinician burden.

Developing such a system requires more than building the technology. Though much is known about human factors and human-centered design (HCD) [46], health IT interventions still suffer from problems with usability (defined as the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [47]), leading to user errors and workflow disruptions that can negatively impact patient safety [4850]. Failure to take usability into account throughout design and intervention implementation makes workflow integration, institutional buy-in, provider acceptability, and sustained clinical use more challenging [18,19,34,37,51]. Similarly, while CDS interventions for EDs have demonstrated some success [38,5254], many either fail to address usability factors in their initial design, or only address them after the intervention has been fully-developed and faced significant barriers to adoption [21,55].

Unfortunately, even CDS tools developed using HCD processes have experienced poor outcomes. Many of these cases have been hindered by inadequate early planning of implementation and maintenance strategies tailored to the specific clinical environment, including longitudinal intervention evaluation and iterative optimization. To address this persistent gap, the application of robust implementation science principles is also needed, e.g., understanding multiple types of context (sources of facilitators and constraints), aligning barriers with appropriate implementation strategies, and adapting interventions both before and after implementation [5659]. Implementation science models, however, typically assume or apply to situations where an evidence-based intervention already exists. As such, they do not directly include processes and principles for system design and evaluation [60]. Recent publications have laid the conceptual foundations for aligning the principles and methods of these disparate disciplines [6063], but applications of such integrated approaches have not yet proliferated within the literature. As such, in order for us to meet the short and long-term objectives of this program (i.e., acceptability, adoption, program sustainability, referral placement, and fall-reduction outcomes), we needed to develop and employ a novel collaborative process seamlessly integrating best practices from both human-centered design and implementation science.

Personal Content

To best address these needs we assembled a design team (with 3 sub-teams) covering multiple areas of expertise, perspectives, and prior experience with health IT development. The interdisciplinary research team included experts in emergency medicine, human factors engineering, implementation science, machine learning, health services research, and patient experience. These members contributed different disciplinary knowledge not only to the intervention design, but also processes and methodologies used throughout the project.

Healthcare IT interventions risk poor outcomes when design teams do not meaningfully involve members of the IT infrastructure (e.g., data analysts, EHR developers, interface designers) from the formative design phase through intervention implementation and optimization [19,20,64]. As the fall-risk and referral CDS could not exist without deep and sustained input from UW Health IT (both in terms of EHR data access and point-of-care system integration), we partnered with the health system’s Applied Data Science Team from project inception. Our data systems sub-team also included research team members involved with predictive algorithm development.

In order to decrease the potential for translation and implementation problems downstream, we also included clinical stakeholders on our design team [65,66]. In addition to establishing user requirements and expectations, they were also crucial for helping establish trust, acceptance, and transparency [18,45]. This sub-team included clinicians, staff, and administrators from the ED and Falls Clinic, providing perspectives of those both making and receiving referral orders. Collectively, members of these three sub-teams had the knowledge and skills necessary to create and implement the CDS, as well as generate the institutional buy-in necessary for sustained use.

Solution

1. Use Case and CDS Concept

The use case for the automated screening and referral process was developed to address the aforementioned problems related to screening patients for fall risk during ED visits and placing outpatient clinic referrals for those at highest risk of falling within the next 6 months. To minimize ED nursing staff burden, the intervention (Figure 1) replaces in-person screening with an EHR-based algorithm to predict risk of future fall-related ED visits. The validated algorithm utilizes data from patients’ medical history, prior healthcare utilization, and initial assessments from the present ED visit. Predictive features include age, use of a mobility device (e.g., walker or cane), history of falls in the prior 6 months, number of ED visits in the prior 6 months, method of transport, acuity, and scores on initial intake assessments. The development and validation of this predictive algorithm, along with sensitivity and specificity, are described in detail in a separate publication [67].

Figure 1:

Figure 1:

Operationalization of the automated fall risk screening and referral process in the ED

The integration of the predictive model with the existing hospital EHR system means all ED patients, age 65 and older, can be screened upon arrival and fall-risk scores can be generated in real time. Once a patient is identified as high fall risk by the predictive algorithm, the CDS fires, generating an interruptive alert recommending the ED clinician place a referral order to the Falls Clinic prior to discharge. Clinicians then have the opportunity to decide whether or not to place that order, as per their discretion.

2. Collaborative Intervention Design Process

It was vital that we employ a collaborative process to engage and leverage the knowledge and skills of design team members, as well as additional clinical and patient stakeholders, throughout the project trajectory. Working in parallel to the development and testing of the predictive algorithm [67], design team members developed all other aspects of the intervention, including referral order processes, user interfaces, integration into clinician workflows, patient-facing materials, provider and nurse training, and evaluation. This iterative intervention design process (Figure 2) covers the period from inception to the start of live pilot testing in the ED.

Figure 2:

Figure 2:

Iterative Process for CDS Development and Implementation Design

This process map incorporates many core elements of HCD (e.g., understanding context of use, defining user requirements and workflows, and iterative design-evaluation cycles) and corresponding processes [36,46,50,68,69]. Activities constituting the iterative development of the CDS tool are pictured across the upper portion of the diagram. The process of designing the implementation of the CDS is detailed across the lower portion of the diagram. Consistent with our approach (integrating HCD principles with models from implementation science research), we believe that the development of any successful CDS intervention must include the design of both the product and the process, in conjunction with the underlying predictive analytics.

Another critical element of this process was the collaboration between IT developers and clinical end users—something often overlooked during CDS builds [19,64]. These interactions facilitated knowledge sharing and perspective-taking that rarely occur during health IT development, increasing the quality of each design-testing cycle. They also kept usability, workflow, and acceptance issues front-and-center throughout the process, closing the translational gap between research and practice.

Process Phase 1: Concept Design and Requirements

Early design sessions focused on user requirements and clinical processes, as well as barriers to successful implementation [70,71]. Design team members participated in sessions related to their role and expertise. For example, a session was held with members of the research and Falls Clinic teams to understand service delivery models, scheduling processes, and clinic availability. This provided context necessary to build an acceptable referral process. We also identified an unanticipated requirement: the ability to adjust the algorithmic threshold flagging “high risk” patients, thereby allowing us to adjust the average number of weekly referrals to reflect changes in appointment availability. This was a critical step in retaining Falls Clinic support of the program and ensuring timely patient access to clinic services.

The health IT and research teams also held multiple design sessions to understand system capabilities and constraints, addressing issues around EHR data access (e.g., navigating organizational barriers/constraints for acquisition, use, and security of hospital data), and requirements for the user interface. Members of these teams continued to meet on a weekly basis throughout the entire design and implementation process.

Core members of the research team, involved in all design sessions, communicated information between groups to facilitate knowledge integration. This high-level perspective allowed us to identify gaps and facilitate solutions. For instance, after incorporating feedback from multiple design sessions, we developed a detailed communication plan, giving ED clinicians structured guidance for explaining the fall-risk assessment to referred patients. The process model was amended to indicate that ED patients would receive written and verbal communication at discharge.

Phase 2: Prototyping and Implementation Strategies

Prototyping is a key aspect of HCD. Thinking across the entire intervention trajectory it became important to develop and iteratively test not only working versions of the CDS tool, but also the intervention implementation plan. This included drafting all materials needed to support use of the tool during ED visits, as well as plans for training, rollout, and evaluation.

CDS Prototype:

Once user and technology systems requirements were established, a core set of IT and research team members held weekly meetings to build a working prototype of the CDS tool within the EHR. Rapid iterative design cycles were employed to continually update and refine all aspects of the back-end functionality, as well as the user interface, based on the results of prior testing (Figure 3). Members working on the fall risk algorithm heavily engaged with the IT team to seamlessly integrate predictive analytics with EHR and data systems, creating a fully-functioning prototype able to score patients in real-time.

Figure 3:

Figure 3:

Prototype for CDS interruptive alert. [©2021 Epic Systems Corporation]

Patient Communication Content Design:

In collaboration with Falls Clinic providers and administrators, members of the research team developed all patient-facing materials in the referral process (using evidence-based health communication approaches and plain language). This included the written information about the referral contained on printed after visit summaries (AVS) and a document containing talking points for ED physicians, APPs, or nurses communicating with referred patients prior to discharge.

Evaluation Planning:

Appropriate selection of outcomes is crucial in building an evaluative framework that rigorously examines implementation from multiple vantage points, levels of analysis, and stakeholder perspectives at multiple points along the implementation trajectory [72]. We selected outcomes and metrics that aligned with the RE-AIM framework, widely-used for planning and evaluating the implementation of health-related interventions [7375]. Examples include measuring clinician acceptability of the CDS, number and characteristics of patients identified as “high risk” per week, perceived barriers to use, proportion of ED clinicians who place referrals, and changes in rates of use over time. Planning evaluation methods in advance was pivotal, as some involved a pre-post design requiring data collection prior to rollout.

Phase 3: User testing and evaluation

Heuristic Usability Evaluation:

Following CDS prototype development, 10 members of the design team (representing emergency medicine, human factors engineering, user experience, and enterprise analytics) participated in a structured heuristic evaluation—a collaborative process in which systems design experts evaluate the prototype of a user interface based upon principles of effective design in order to identify potential usability problems[62,76,77]. Participants walked through the CDS interface navigation using fictionalized patient charts, identifying and discussing usability concerns, and generating ideas for possible solutions. Solutions were successfully generated for the majority of usability errors, including positioning the alert to make it more noticeable, changing the color and language of the alert banner, and simplifying text using bullet points. Data collected from the discussion were then synthesized and evaluated based on feasibility within system constraints (Table 1). All possible changes were made prior to clinical user testing, with some necessitating EHR design modifcations requiring additional time. A small number of identified problems, such as the location of the “order” button, could not be changed. Other elements of the CDS were modified in order to offset the impact of these issues.

Table 1:

Usability Problems Identified During Heuristic Evaluation Session

Problem Heuristic(s) Violated Barriers/Limitations Solution
User Interface Design
Should always be an “active” (pop-up) alert Visibility, Consistency, Match Only certain trigger patterns result in active alerts; Otherwise passively displayed on screen
  • Discuss alternative solutions with EHR programmers

  • Request changes in future system updates to support active alerts

Have to scroll up to see alert once triggered Visibility Dependent upon screen placement of triggering action
  • Move location of alert on screen to appear near triggering action

“High Risk Falls” banner has same placement and color as nurse flow sheet evaluation for falls; Confusing for users. Consistency Visibility Language Match Can’t change the color or language of nurse alert (already part of clinical practice)
  • Change color of our banner

  • Change banner language to “Consider Referral to Falls Clinic”

  • Move alert to right column, away from nurse fall-risk alert; Placed with “Follow Up” and “Instructions”

“Follow-Up Actions” button location is not intuitive or easy to find; Should be closer to instructions Visibility Match Control Can’t move button or generate alternative way to access order screen
  • Unable to change

“Acknowledge Reason” button is not relevant Match Patients not consulted prior to order placement
  • Remove “Patient Declines Referral” button

  • Add “Other” button instead

Placement of “Consult to…” language next to “Do Not Order” is confusing Match Consistency Placement can’t be changed and is what ED providers are used to seeing
  • Unable to change

Functionality
Alert fires only after final diagnosis and disposition have been entered, requiring screen refresh Match Feedback
Visibility
Fall risk algorithm currently requires final ED diagnosis to calculate score
  • Revise algorithm to reliably assess fall risk without using final ED diagnosis as a factor

No feedback telling providers that the order was successfully generated Feedback Closure System does not generate feedback upon placement of orders
  • Unable to change

  • Providers don’t expect automatic feedback following order placement

Content
Too much text on alert screen Minimalist Language Must include all information for both initial alert and Follow-Up Actions screen
  • Simplify language

  • Use bullet points rather than longer sentences

Clicking “Follow-Up Actions” should bring up list of action steps, not same content as initial alert Match Language
Feedback
System requires the exact same text on the initial alert and “Follow-Up Actions” screen
  • Unable to change prior to initial launch

Action steps do not include communicating about order with patient Control Prevent Errors None
  • Add bullet item: “Discuss this order with the patient and/or caregiver”

No easy way to access to supporting documentation from the alert screen Documentation Memory Load None
  • Add link to intranet page containing:
    • Falls Clinic information
    • Guide for communicating with patients about falls risk assessment
    • FAQs for providers answering patient questions

Patient Stakeholder Feedback:

Engaging patients and families has beneficial outcomes for healthcare quality and health IT improvement initiatives [7880]. Even though ED clinicians use the CDS interface, older adult patients experience a different part of the process and are the ultimate end users of the referral. We engaged older adult patients through UW Health’s Patient and Family Advisor Program, a network of volunteers who partner with staff on organizational improvement efforts. All advisors age≥60 (N=65) were solicited by email for feedback.

A total of 25 advisors (38%) completed the survey, offering detailed suggestions about language changes to decrease confusion and increase acceptability across a wider range of older adults. For example, patients suggested avoiding the term “geriatric” and clarifying that the referral was coming from their ED clinician rather than an automated system. They recommended that messages explaining why individuals may be at higher risk of falls be more personalized and less “judgmental”. They also expressed confusion about whether the visit would be at a clinic site or in the patient’s home, resulting in changes to the language describing the appointment in the AVS and talking points. Based on the number and scope of respondents’ questions—many of which were unanticipated—we created an additional “FAQ” document for providers describing how to answer many of the questions raised by these patient stakeholders.

ED Clinician User Testing:

Consistent with best practices of HCD, it was imperative to obtain user feedback during simulated use of the CDS prototype [81]. Having made changes to the CDS interface following the heuristic evaluation, ED clinician volunteers engaged in a facilitated cognitive walkthrough of the intervention. Using simulated patient data, they navigated the EHR discharge process as per their normal practice. The facilitator took structured notes, asking questions about thought processes and decisions.

Users thought providing Falls Clinic referrals was the “right thing to do” for patients and were motivated to use the CDS. They felt its placement in a “discharge navigator” was appropriate, but that an interruptive alert was critical (otherwise they “would miss it for sure”). The biggest discussion topic was the exact action that would trigger the CDS firing (i.e., clicking a particular button). Some even suggested a change to the workflow forcing a hard-stop if the CDS was bypassed.

Clinician Education and Surveys:

ED physicians, advanced practice providers, and nurses received information about the CDS during in-person presentations at multiple regularly-scheduled meetings. This included demonstrations of the new EHR workflow and CDS screenshots (Figure 4). Clinicians were sent a follow-up email with all presentation materials and a link to an online survey. This pre-intervention assessment measured perceptions and attitudes related to program necessity, design, and implementation. Items were adapted from validated scales measuring intervention implementation [82]. Open-ended questions also elicited insights about potential barriers to adoption and implementation-related concerns.

Figure 4:

Figure 4:

Screenshot CDS user interface, prior to formal launch. [©2021 Epic Systems Corporation]

Thirty-two ED clinicians completed the survey. Across roles, respondents believed the intervention was feasible, acceptable, and appropriate for the ED. They reported little comfort referring patients to the Falls Clinic prior to the intervention, but believed it was very important to identify and refer patients there. They also rated the intervention as moderately-to-very useful for preventing fall-related ED visits in the future. Commonly described potential barriers were lack of provider time, high patient volumes, competing priorities at discharge, and feeling overloaded by multiple CDSs for one patient. The biggest concern was their inability to effectively explain the referral so that patients would make Falls Clinic appointments. Based on this feedback, additional information was added to the talking points and FAQ documents.

3. Pilot Implementation: Feedback and Adaptive Redesign

Not all user feedback can be gathered in simulated settings prior to initial rollout. Processes need to be put in place to collect and assess multiple sources of feedback on an ongoing iterative basis. Both HCD and implementation science models impart the need for continually monitoring user data so that interventions can be iteratively and adaptively redesigned for optimization within a particular context over a prolonged post-implementation period [19,46,64,66]. As such, we pre-planned a cyclical process for collecting user data in near-real time throughout our pilot study, developing and testing improvement strategies based on that feedback (Figure 5). This process was initiated immediately following launch of the CDS in the University Hospital ED (go-live date communicated to all ED clinicians via email). Data collection included ED operations metrics (e.g., referral rates, patterns of use) and academic detailing interviews with CDS users about decisions to place or not place referral orders.

Figure 5:

Figure 5:

Post-implementation Feedback, Optimization, and Adaptive Redesign Cycle

This process has already resulted in user-driven changes to the CDS. For instance, hospital data indicated that a larger-than-anticipated number of residents were not seeing the interruptive alert on the discharge navigation screen. Working through this cycle allowed us to gather additional data (via academic detailing interviews with CDS users), determine potential solutions, test options in simulated settings, and gain necessary approvals for making the changes to the workflow. In this case, the approved solution was to institute a “hard stop”, requiring users to interact with the fall risk alert prior to discharging the patient. Following implementation of this change, we continued to monitor CDS activity, assessing provider acceptability and adoption.

Administrative data indicates that this change has improved user adoption of the CDS tool, supported by interview data demonstrating acceptability from ED providers. Prior to instituting the “hard stop”, 25.4% of patients flagged by the algorithm received referrals, averaging 2.7 referrals per week. In the year following the “hard stop”, 30.0% of flagged patients received referrals, averaging 4.3 referrals per week. These rates are now more aligned with Falls Clinic goals. We will continue to collect feedback from multiple sources to monitor sustainability and further optimize intervention delivery (adding feedback from referred patients). We are also collecting program effectiveness data on completed clinic appointments and future incidence of injurious falls.

Lessons for the Field

A major goal of this study is to demonstrate the promise of using information technology to deliver preventative health interventions in the ED setting without diverting limited clinician resources from the core ED mission of providing quality acute illness and injury care. Successful design and implementation required integration of multiple new technologies and processes into existing workflows, necessitating collaborative interdisciplinary design and analysis of potential barriers. By using a collaborative approach combining HCD and implementation science strategies with predictive analytics, we were able to successfully design and implement an intervention meeting all project goals. The processes used in this clinical-IT-research collaboration can be applied to other use cases involving automated risk-stratification, CDS development, and/or EHR-facilitated care coordination. More broadly, our innovative integration of implementation science and HCD approaches can be used as a model for other programs incorporating both the design and implementation of healthcare interventions.

This application of smart automation allows ED clinicians to improve patient care without interruption to existing workflows, constituting a novel use of health IT to help link patients to targeted preventive care following an ED visit. These interventions will become increasingly important as EDs harness technology to improve care trajectories for older adults and adapt to meet the changing role of EDs within the larger context of care. Accomplishment of these larger goals will require further research regarding the scalability and adaptability of these types of interventions, spanning more diverse hospital/ED sites and patient populations.

Financial Support:

This work was supported by the Agency for Healthcare Research and Quality (grant number: K08HS024558) and the National Institutes of Health (grant number: 1K24AG054560).

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Conflict of interest statement:

All authors have no conflicts of interest to disclose

References

  • 1.Burns E; Kakara R Deaths from Falls Among Persons Aged ≥65 Years — United States, 2007–2016. MMWR. Morb. Mortal. Wkly. Rep 2018, 67, 509–514, doi: 10.15585/mmwr.mm6718a1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Houry D; Florence C; Baldwin G; Stevens J; McClure R The CDC Injury Center’s Response to the Growing Public Health Problem of Falls Among Older Adults. Am. J. Lifestyle Med 2016, 10, 74–77, doi: 10.1177/1559827615600137. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Moreland B; Kakara R; Henry A Trends in Nonfatal Falls and Fall-Related Injuries Among Adults Aged ≥65 Years — United States, 2012–2018. MMWR. Morb. Mortal. Wkly. Rep 2020, 69, 875–881, doi: 10.15585/mmwr.mm6927a5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Verma SK; Willetts JL; Corns HL; Marucci-Wellman HR; Lombardi DA; Courtney TK Falls and Fall-Related Injuries among Community-Dwelling Adults in the United States. PLoS One 2016, 11, doi: 10.1371/journal.pone.0150939. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Florence CS; Bergen G; Atherly A; Burns E; Stevens J; Drake C Medical Costs of Fatal and Nonfatal Falls in Older Adults. J. Am. Geriatr. Soc 2018, 66, 693–698, doi: 10.1111/jgs.15304. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Burns ER; Stevens JA; Lee R The Direct Costs of Fatal and Non-Fatal Falls among Older Adults — United States. J. Safety Res 2016, 58, 99–103, doi: 10.1016/j.jsr.2016.05.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Center for Disease Control and Prevention (CDC) Key Data and Statistics|WISQARS|Injury Center|CDC Available online: https://www.cdc.gov/injury/wisqars/overview/key_data.html (accessed on 27 January 2021).
  • 8.Patterson BW; Repplinger MD; Pulia MS; Batt RJ; Svenson JE; Trinh A; Mendonça EA; Smith MA; Hamedani AG; Shah MN Using the Hendrich II Inpatient Fall Risk Screen to Predict Outpatient Falls After Emergency Department Visits. J. Am. Geriatr. Soc 2018, 66, 760–765, doi: 10.1111/jgs.15299. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Patterson BW; Smith MA; Repplinger MD; Pulia MS; Svenson JE; Kim MK; Shah MN Using Chief Complaint in Addition to Diagnosis Codes to Identify Falls in the Emergency Department. J. Am. Geriatr. Soc 2017, 65, doi: 10.1111/jgs.14982. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Shankar KN; Treadway NJ; Taylor AA; Breaud AH; Peterson EW; Howland J Older Adult Falls Prevention Behaviors 60 Days Post-Discharge from an Urban Emergency Department after Treatment for a Fall. Inj. Epidemiol 2017, 4, 18, doi: 10.1186/s40621-017-0114-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Carpenter CR; Lo AX Falling behind? Understanding Implementation Science in Future Emergency Department Management Strategies for Geriatric Fall Prevention. Acad. Emerg. Med 2015, 22, 478–480. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Tirrell G; Sri-on J; Lipsitz LA; Camargo CA, J.; Kabrhel C; Liu SW Evaluation of Older Adult Patients with Falls in the Emergency Department: Discordance with National Guidelines. Acad Emerg Med 2015, 22, 461–467, doi: 10.1111/acem.12634. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Hammouda N; Carpenter C; Hung W; Lesser A; Nyamu S; Liu S; Gettel C; Malsch A; Castillo E; Forrester S; et al. Moving the Needle on Fall Prevention: A Geriatric Emergency Care Applied Research (GEAR) Network Scoping Review and Consensus Statement. Acad. Emerg. Med 2021, acem.14279, doi: 10.1111/acem.14279. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Carpenter CR; Cameron A; Ganz DA; Liu S Older Adult Falls in Emergency Medicine: 2019 Update. Clin. Geriatr. Med 2019, 35, 205–219, doi: 10.1016/j.cger.2019.01.009. [DOI] [PubMed] [Google Scholar]
  • 15.Davenport K; Alazemi M; Sri-On J; Liu S Missed Opportunities to Diagnose and Intervene in Modifiable Risk Factors for Older Emergency Department Patients Presenting After a Fall. Ann. Emerg. Med 2020, 76, 730–738, doi: 10.1016/j.annemergmed.2020.06.020. [DOI] [PubMed] [Google Scholar]
  • 16.Chou SC; Venkatesh AK; Trueger NS; Pitts SR Primary Care Office Visits for Acute Care Dropped Sharply in 2002–15, While ED Visits Increased Modestly. Health Aff 2019, 38, 268–275, doi: 10.1377/hlthaff.2018.05184. [DOI] [PubMed] [Google Scholar]
  • 17.Pitts SR; Carrier ER; Rich EC; Kellermann AL Where Americans Get Acute Care: Increasingly, It’s Not at Their Doctor’s Office. Health Aff 2010, 29, 1620–1629, doi: 10.1377/hlthaff.2009.1026. [DOI] [PubMed] [Google Scholar]
  • 18.Rundo L; Pirrone R; Vitabile S; Sala E; Gambino O Recent Advances of HCI in Decision-Making Tasks for Optimized Clinical Workflows and Precision Medicine. J. Biomed. Inform 2020, 108, 103479, doi: 10.1016/j.jbi.2020.103479. [DOI] [PubMed] [Google Scholar]
  • 19.Khairat S; Marc D; Crosby W; Al Sanousi A Reasons For Physicians Not Adopting Clinical Decision Support Systems: Critical Analysis. JMIR Med. informatics 2018, 6, e24, doi: 10.2196/medinform.8912. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.He J; Baxter SL; Xu J; Xu J; Zhou X; Zhang K The Practical Implementation of Artificial Intelligence Technologies in Medicine. Nat. Med 2019, 25, 30–36, doi: 10.1038/s41591-018-0307-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Mann D; Hess R; McGinn T; Mishuris R; Chokshi S; McCullagh L; Smith PD; Palmisano J; Richardson S; Feldstein DA Adaptive Design of a Clinical Decision Support Tool: What the Impact on Utilization Rates Means for Future CDS Research. Digit. Heal 2019, 5, doi: 10.1177/2055207619827716. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Close J; Ellis M; Hooper R; Glucksman E; Jackson S; Swift C Prevention of Falls in the Elderly Trial (PROFET): A Randomised Controlled Trial. Lancet 1999, 353, 93–97, doi: 10.1016/S0140-6736(98)06119-4. [DOI] [PubMed] [Google Scholar]
  • 23.Close JCT; Hooper R; Glucksman E; Jackson SHD; Swift CG Predictors of Falls in a High Risk Population: Results from the Prevention of Falls in the Elderly Trial (PROFET). Emerg. Med. J 2003, 20, 421–425, doi: 10.1136/EMJ.20.5.421. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Stevens JA The STEADI Tool Kit: A Fall Prevention Resource for Health Care Providers. IHS Prim. Care Provid 2013, 39, 162. [PMC free article] [PubMed] [Google Scholar]
  • 25.Grossman DC; Curry SJ; Owens DK; Barry MJ; Caughey AB; Davidson KW; Doubeni CA; Epling JW; Kemper AR; Krist AH; et al. Interventions to Prevent Falls in Community-Dwelling Older Adults: US Preventive Services Task Force Recommendation Statement. JAMA 2018, 319, 1696–1704, doi: 10.1001/JAMA.2018.3097. [DOI] [PubMed] [Google Scholar]
  • 26.Carpenter CR; Scheatzle MD; D’Antonio JA; Ricci PT; Coben JH Identification of Fall Risk Factors in Older Adult Emergency Department Patients. Acad. Emerg. Med 2009, 16, 211–219, doi: 10.1111/j.1553-2712.2009.00351.x. [DOI] [PubMed] [Google Scholar]
  • 27.Carpenter CR; Griffey RT; Stark S; Coopersmith CM; Gage BF Physician and Nurse Acceptance of Technicians to Screen for Geriatric Syndromes in the Emergency Department. West. J. Emerg. Med 2011, 12, 489–495, doi: 10.5811/westjem.2011.1.1962. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Donaldson MG; Khan KM; Davis JC; Salter AE; Buchanan J; McKnight D; Janssen PA; Bell M; McKay HA Emergency Department Fall-Related Presentations Do Not Trigger Fall Risk Assessment: A Gap in Care of High-Risk Outpatient Fallers. Arch. Gerontol. Geriatr 2005, 41, 311–317, doi: 10.1016/j.archger.2005.04.008. [DOI] [PubMed] [Google Scholar]
  • 29.Davenport K; Cameron A; Samson M; Sri-On J; Liu SW Fall Prevention Knowledge, Attitudes, and Behaviors: A Survey of Emergency Providers. West. J. Emerg. Med 2020, 21, 826–829, doi: 10.5811/westjem.2020.4.43387. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Lovallo C; Rolandi S; Rossetti AM; Lusignani M Accidental Falls in Hospital Inpatients: Evaluation of Sensitivity and Specificity of Two Risk Assessment Tools. J. Adv. Nurs 2010, 66, 690–696, doi: 10.1111/j.1365-2648.2009.05231.x. [DOI] [PubMed] [Google Scholar]
  • 31.Robey-Williams C; Rush KL; Bendyk H; Patton LM; Chamberlain D; Sparks T Spartanburg Fall Risk Assessment Tool: A Simple Three-Step Process. Appl. Nurs. Res 2007, 20, 86–93, doi: 10.1016/j.apnr.2006.02.002. [DOI] [PubMed] [Google Scholar]
  • 32.Oliver D; Britton M; Seed P; Martin FC; Hopper AH Development and Evaluation of Evidence Based Risk Assessment Tool (STRATIFY) to Predict Which Elderly Inpatients Will Fall: Case-Control and Cohort Studies. Br. Med. J 1997, 315, 1049–1053, doi: 10.1136/bmj.315.7115.1049. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Carpenter CR; Avidan MS; Wildes T; Stark S; Fowler SA; Lo AX Predicting Geriatric Falls Following an Episode of Emergency Department Care: A Systematic Review. Acad Emerg Med 2014, 21, 1069–1082, doi: 10.1111/acem.12488. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Gutenstein M; Pickering JW; Than M Development of a Digital Clinical Pathway for Emergency Medicine: Lessons from Usability Testing and Implementation Failure. Health Informatics J 2019, 25, 1563–1571, doi: 10.1177/1460458218779099. [DOI] [PubMed] [Google Scholar]
  • 35.Chisholm CD; Weaver CS; Whenmouth L; Giles B A Task Analysis of Emergency Physician Activities in Academic and Community Settings. Ann. Emerg. Med 2011, 58, 117–122, doi: 10.1016/j.annemergmed.2010.11.026. [DOI] [PubMed] [Google Scholar]
  • 36.Salwei ME; Carayon P; Hoonakker P; Hundt AS; Novak C; Wang Y; Wiegmann D; Patterson B Assessing Workflow of Emergency Physicians in the Use of Clinical Decision Support. Proc. Hum. Factors Ergon. Soc. Annu. Meet 2019, 63, 772–776, doi: 10.1177/1071181319631334. [DOI] [Google Scholar]
  • 37.Patterson BW; Pulia MS; Ravi S; Hoonakker PLT; Schoofs Hundt A; Wiegmann D; Wirkus EJ; Johnson S; Carayon P Scope and Influence of Electronic Health Record–Integrated Clinical Decision Support in the Emergency Department: A Systematic Review. Ann. Emerg. Med 2019, 74, 285–296, doi: 10.1016/j.annemergmed.2018.10.034. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Jun S; Plint AC; Campbell SM; Curtis S; Sabir K; Newton AS Point-of-Care Cognitive Support Technology in Emergency Departments: A Scoping Review of Technology Acceptance by Clinicians. Acad. Emerg. Med 2018, 25, 494–507, doi: 10.1111/acem.13325. [DOI] [PubMed] [Google Scholar]
  • 39.Tan A; Durbin M; Chung FR; Rubin AL; Cuthel AM; McQuilkin JA; Modrek AS; Jamin C; Gavin N; Mann D; et al. Design and Implementation of a Clinical Decision Support Tool for Primary Palliative Care for Emergency Medicine (PRIM-ER). BMC Med. Inform. Decis. Mak 2020, 20, doi: 10.1186/s12911-020-1021-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Goldberg EM; Marks SJ; Ilegbusi A; Resnik L; Strauss DH; Merchant RC GAPcare: The Geriatric Acute and Post-Acute Fall Prevention Intervention in the Emergency Department: Preliminary Data. J. Am. Geriatr. Soc 2020, 68, 198–206, doi: 10.1111/jgs.16210. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Morello RT; Soh SE; Behm K; Egan A; Ayton D; Hill K; Flicker L; Etherton-Beer CD; Arendts G; Waldron N; et al. Multifactorial Falls Prevention Programmes for Older Adults Presenting to the Emergency Department with a Fall: Systematic Review and Meta-Analysis. Inj. Prev 2019, 25, 557–564, doi: 10.1136/injuryprev-2019-043214. [DOI] [PubMed] [Google Scholar]
  • 42.Tiedemann A; Sherrington C; Orr T; Hallen J; Lewis D; Kelly A; Vogler C; Lord SR; Close JCT Identifying Older People at High Risk of Future Falls: Development and Validation of a Screening Tool for Use in Emergency Departments. Emerg. Med. J 2013, 30, 918–922, doi: 10.1136/emermed-2012-201783. [DOI] [PubMed] [Google Scholar]
  • 43.Southerland LT; Slattery L; Rosenthal JA; Kegelmeyer D; Kloos A Are Triage Questions Sufficient to Assign Fall Risk Precautions in the ED? Am. J. Emerg. Med 2017, 35, 329–332, doi: 10.1016/j.ajem.2016.10.035. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Goldberg EM; Gettel CJ; Hayes K; Shield RR; Guthrie KM GAPcare: The Geriatric Acute and Post-Acute Fall Prevention Intervention for Emergency Department Patients – A Qualitative Evaluation. OBM Geriatr 2019, 3, 1–1, doi: 10.21926/obm.geriatr.1904078. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Horsky J; Schiff GD; Johnston D; Mercincavage L; Bell D; Middleton B Interface Design Principles for Usable Decision Support: A Targeted Review of Best Practices for Clinical Prescribing Interventions. J. Biomed. Inform 2012, 45, 1202–1216, doi: 10.1016/j.jbi.2012.09.002. [DOI] [PubMed] [Google Scholar]
  • 46.Marcilly R; Peute L; Beuscart-Zephir MC From usability engineering to evidence-based usability in health IT. In Evidence-Based Health Informatics: Promoting Safety and Efficiency through Scientific Methods and Ethical Policy; IOS Press, 2016; Vol. 222, pp. 126–138 ISBN 9781614996354. [PubMed] [Google Scholar]
  • 47.“International Organization for Standardization” ISO 9241–11:2018; Ergonomics of Human-System Interaction. Part 11: Usability: Definitions and Concepts; Geneva, Switzerland, 2018; [Google Scholar]
  • 48.Carayon P; Hoonakker P Human Factors and Usability for Health Information Technology: Old and New Challenges. Yearb Med Inf 2019, 28, 71–77, doi: 10.1055/s-0039-1677907. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Brown CL; Mulcaster HL; Triffitt KL; Sittig DF; Ash JS; Reygate K; Husband AK; Bates DW; Slight SP A Systematic Review of the Types and Causes of Prescribing Errors Generated from Using Computerized Provider Order Entry Systems in Primary and Secondary Care. J. Am. Med. Informatics Assoc 2017, 24, 432–440, doi: 10.1093/jamia/ocw119. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50.Turner P; Kushniruk A; Nohr C Are We There Yet? Human Factors Knowledge and Health Information Technology - the Challenges of Implementation and Impact. Yearb. Med. Inform 2017, 26, 84–91. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Salwei ME; Carayon P; Hoonakker PLT; Hundt AS; Wiegmann D; Pulia M; Patterson BW Workflow Integration Analysis of a Human Factors-Based Clinical Decision Support in the Emergency Department. Appl. Ergon 2021, 97, 103498, doi: 10.1016/j.apergo.2021.103498. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52.Hoonakker PLT; Carayon P; Salwei ME; Hundt AS; Wiegmann D; Kleinschmidt P; Pulia MS; Wang Y; Novak C; Patterson BW The Design of PE Dx, a CDS to Support Pulmonary Embolism Diagnosis in the ED. Stud. Health Technol. Inform 2019, 265, 134–140, doi: 10.3233/SHTI190152. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 53.Carayon P; Hoonakker P; Hundt AS; Salwei M; Wiegmann D; Brown RL; Kleinschmidt P; Novak C; Pulia M; Wang Y; et al. Application of Human Factors to Improve Usability of Clinical Decision Support for Diagnostic Decision-Making: A Scenario-Based Simulation Study. BMJ Qual. Saf 2020, 29, 329–340, doi: 10.1136/bmjqs-2019-009857. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Bates DW; Kuperman GJ; Wang S; Gandhi T; Kittler A; Volk L; Spurr C; Khorasani R; Tanasijevic M; Middleton B Ten Commandments for Effective Clinical Decision Support: Making the Practice of Evidence-Based Medicine a Reality. J. Am. Med. Informatics Assoc 2003, 10, 523–530, doi: 10.1197/jamia.M1370. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55.Press A; McCullagh L; Khan S; Schachter A; Pardo S; McGinn T Usability Testing of a Complex Clinical Decision Support Tool in the Emergency Department: Lessons Learned. JMIR Hum. Factors 2015, 2, e14, doi: 10.2196/humanfactors.4537. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56.Waltz TJ; Powell BJ; Fernández ME; Abadie B; Damschroder LJ Choosing Implementation Strategies to Address Contextual Barriers: Diversity in Recommendations and Future Directions. Implement. Sci 2019, 14, 42, doi: 10.1186/s13012-019-0892-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 57.Powell BJ; Fernandez ME; Williams NJ; Aarons GA; Beidas RS; Lewis CC; McHugh SM; Weiner BJ Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda. Front. Public Heal 2019, 7, 3, doi: 10.3389/fpubh.2019.00003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58.Powell BJ; Waltz TJ; Chinman MJ; Damschroder LJ; Smith JL; Matthieu MM; Proctor EK; Kirchner JAE A Refined Compilation of Implementation Strategies: Results from the Expert Recommendations for Implementing Change (ERIC) Project. Implement. Sci 2015, 10, 21, doi: 10.1186/s13012-015-0209-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 59.Perry CK; Damschroder LJ; Hemler JR; Woodson TT; Ono SS; Cohen DJ Specifying and Comparing Implementation Strategies across Seven Large Implementation Interventions: A Practical Application of Theory. Implement. Sci 2019, 14, 1–13, doi: 10.1186/s13012-019-0876-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60.Chen E; Neta G; Roberts MC Complementary Approaches to Problem Solving in Healthcare and Public Health: Implementation Science and Human-Centered Design. Transl. Behav. Med 2021, 11, 1115–1121, doi: 10.1093/tbm/ibaa079. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 61.Dopp AR; Parisi KE; Munson SA; Lyon AR Integrating Implementation and User-Centred Design Strategies to Enhance the Impact of Health Services: Protocol from a Concept Mapping Study. Heal. Res. Policy Syst 2019, 17, doi: 10.1186/s12961-0180403-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 62.Dopp AR; Parisi KE; Munson SA; Lyon AR A Glossary of User-Centered Design Strategies for Implementation Experts. Transl. Behav. Med 2019, 9, 1057–1064, doi: 10.1093/tbm/iby119. [DOI] [PubMed] [Google Scholar]
  • 63.Haines ER; Dopp A; Lyon AR; Witteman HO; Bender M; Vaisson G; Hitch D; Birken S Harmonizing Evidence-Based Practice, Implementation Context, and Implementation Strategies with User-Centered Design: A Case Example in Young Adult Cancer Care. Implement. Sci. Commun 2021, 2, doi: 10.1186/s43058-021-00147-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 64.Ray JM; Ratwani RM; Sinsky CA; Frankel RM; Friedberg MW; Powsner SM; Rosenthal DI; Wachter RM; Melnick ER Six Habits of Highly Successful Health Information Technology: Powerful Strategies for Design and Implementation. J. Am. Med. Informatics Assoc 2019, 26, 1109–1114, doi: 10.1093/jamia/ocz098. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 65.Moullin JC; Dickson KS; Stadnick NA; Rabin B; Aarons GA Systematic Review of the Exploration, Preparation, Implementation, Sustainment (EPIS) Framework. Implement. Sci 2019, 14, 1–16. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 66.Chambers DA; Glasgow RE; Stange KC The Dynamic Sustainability Framework: Addressing the Paradox of Sustainment amid Ongoing Change. Implement. Sci 2013, 8, 117, doi: 10.1186/1748-5908-8-117. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 67.Patterson BW; Engstrom CJ; Sah V; Smith MA; Mendonça EA; Pulia MS; Repplinger MD; Hamedani AG; Page D; Shah MN Training and Interpreting Machine Learning Algorithms to Evaluate Fall Risk After Emergency Department Visits. Med. Care 2019, 57, 560–566, doi: 10.1097/MLR.0000000000001140. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 68.Kushniruk A; Nøhr C Participatory design, user involvement and health IT evaluation. In Evidence-Based Health Informatics: Promoting Safety and Efficiency through Scientific Methods and Ethical Policy; IOS Press, 2016; Vol. 222, pp. 139–151 ISBN 9781614996354. [PubMed] [Google Scholar]
  • 69.Carayon P; Wooldridge A; Hose BZ; Salwei M; Benneyan J Challenges And Opportunities For Improving Patient Safety Through Human Factors And Systems Engineering. Heal. Aff 2018, 37, 1862–1869, doi: 10.1377/hlthaff.2018.0723. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 70.Nilsen P Making Sense of Implementation Theories, Models and Frameworks. Implement. Sci 2015, 10, 53, doi: 10.1186/s13012-015-0242-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 71.Moullin JC; Dickson KS; Stadnick NA; Albers B; Nilsen P; Broder-Fingert S; Mukasa B; Aarons GA Ten Recommendations for Using Implementation Frameworks in Research and Practice. Implement. Sci. Commun 2020, 1, 42, doi: 10.1186/s43058-020-00023-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 72.Brownson RC; Colditz GA; Proctor EK Dissemination and Implementation Research in Health; Brownson RC, Colditz GA, Proctor EK, Eds.; Oxford University Press, 2017; Vol. 1; ISBN 9780190683214. [Google Scholar]
  • 73.Kwan BM; McGinnes HL; Ory MG; Estabrooks PA; Waxmonsky JA; Glasgow RE RE-AIM in the Real World: Use of the RE-AIM Framework for Program Planning and Evaluation in Clinical and Community Settings. Front. Public Heal 2019, 7, 345, doi: 10.3389/fpubh.2019.00345. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 74.Glasgow RE; Vogt TM; Boles SM, 1999: Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American Journal of Public Health 89, 1322–1327 10.2105/AJPH.89.9.1322 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 75.Thomas SA; Stefanovska-Petkovska M; Leeman J; Shelton RC; Chambers DA; Glasgow RE An Extension of RE-AIM to Enhance Sustainability: Addressing Dynamic Context and Promoting Health Equity Over Time. Front. Public Heal. | www.frontiersin.org 2020, 1, 134, doi: 10.3389/fpubh.2020.00134. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 76.Zhang J; Johnson TR; Patel VL; Paige DL; Kubose T Using Usability Heuristics to Evaluate Patient Safety of Medical Devices. J. Biomed. Inform 2003, 36, 23–30, doi: 10.1016/S1532-0464(03)00060-1. [DOI] [PubMed] [Google Scholar]
  • 77.Scapin DL; Bastien JMC Ergonomic Criteria for Evaluating the Ergonomic Quality of Interactive Systems. Behav. Inf. Technol 1997, 16, 220–231, doi: 10.1080/014492997119806. [DOI] [Google Scholar]
  • 78.Bombard Y; Baker GR; Orlando E; Fancott C; Bhatia P; Casalino S; Onate K; Denis JL; Pomey MP Engaging Patients to Improve Quality of Care: A Systematic Review. Implement. Sci 2018, 13, 98. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 79.Leung K; Lu-McLean D; Kuziemsky C; Booth RG; Rossetti SC; Borycki E; Strudwick G Using Patient and Family Engagement Strategies to Improve Outcomes of Health Information Technology Initiatives: Scoping Review. J. Med. Internet Res 2019, 21, e14683. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 80.Hertel E; Cheadle A; Matthys J; Coleman K; Gray M; Robbins M; Tufte J; Hsu C Engaging Patients in Primary Care Design: An Evaluation of a Novel Approach to Codesigning Care. Heal. Expect 2019, 22, 609–616, doi: 10.1111/hex.12909. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 81.Harte R; Glynn L; Rodríguez-Molinero A; Baker PM; Scharf T; Quinlan LR; ÓLaighin G A Human-Centered Design Methodology to Enhance the Usability, Human Factors, and User Experience of Connected Health Systems: A Three-Phase Methodology. JMIR Hum. Factors 2017, 4, e8, doi: 10.2196/humanfactors.5443. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 82.Weiner BJ; Lewis CC; Stanick C; Powell BJ; Dorsey CN; Clary AS; Boynton MH; Halko H Psychometric Assessment of Three Newly Developed Implementation Outcome Measures. Implement. Sci 2017, 12, 108, doi: 10.1186/s13012-017-0635-3. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES