A step-by-step roadmap for the development and deployment of an electronic health record sidecar application that tracks patient outcomes: The RA PRO dashboard

Abstract

Objective

Despite interest in optimizing the electronic health record (EHR) to facilitate chronic disease care for conditions like rheumatoid arthritis (RA), progress in this area has been slow. EHR sidecar applications offer one solution, but little guidance exists to facilitate their successful development, deployment, and maintenance in the healthcare setting. We aimed to provide a roadmap for how to develop and deploy an EHR sidecar application based on our experience building a new EHR-integrated, patient-facing visualization tool that displayed disease outcomes to RA patients during a clinical visit (the “RA PRO dashboard”) in a large academic health center.

Methods

We describe the technical design and implementation of the RA PRO dashboard; report clinic workflow adaptations to incorporate this new technology; and discuss the resources required and challenges encountered in maintaining this application.

Results

The RA PRO dashboard required extensive human-centered design work, regulatory approvals, software development, user testing, integration with Epic-based workflows, and maintenance. Key requirements were prioritized based on the anticipated effects on usefulness and ease of use. Implementation science strategies were used to improve use of the dashboard in clinic and included education for patients, staff, and clinicians; reports of actual use of the dashboard and data quality; and regular meetings between the research team and clinicians to discuss and address barriers to use.

Conclusion

Successful development and deployment of an EHR-integrated application are resource-intensive and require technical, operational, and educational innovations. The roadmap presented in this study can serve as a resource for future developers.

Introduction

With an ever-growing body of electronic health record (EHR) data, including patient-reported outcomes (PROs), one important role of health information technology (IT) tools is to help summarize this data for incorporation into real-time clinical decision-making. Graphical displays of structured data (such as blood pressure, BMI, hemoglobin A1C, and other laboratory test results) are commonly available in many EHRs or patient portals. These tools can be used by clinicians or patients to quickly and efficiently visualize important health metrics, facilitating a more informed and collaborative approach to healthcare.^1,2 However, there has been less progress in developing tools to track chronic disease outcomes effectively. For example, even when documented as structured data in the EHR, PROs are not easily tracked over time for seamless interpretation by clinicians or patients during a clinical visit.^3,4

This gap highlights a significant challenge in modern healthcare IT: the need for more sophisticated systems that can not only capture and store data but also present it in a way that supports long-term health management and proactive care strategies.

Optimization of the EHR to facilitate chronic disease care has been slow. Approaches such as working with large EHR vendors to customize foundation software for chronic disease care, or constructing local customizations of existing EHR software, have yielded little progress. Other approaches such as developing parallel systems entirely outside the EHR come with significant drawbacks such as requiring clinicians to work with multiple interfaces, which can be cumbersome and time-consuming. EHR sidecar applications offer a potential solution to some of these challenges because these applications offer more design flexibility and can fully integrate with the EHR, thereby avoiding the complexities and inefficiencies associated with interacting with multiple applications. By seamlessly embedding within the existing EHR infrastructure, sidecar applications can enhance the functionality of the EHR without the need for extensive overhauls or parallel systems. This integration allows for a more streamlined and cohesive user experience, enabling clinicians to focus more on patient care rather than on managing disparate technological tools.

Informed by digital innovation best practices, we present a roadmap for the successful development, integration, and maintenance of an EHR sidecar application for a common chronic condition, rheumatoid arthritis (RA). RA is a chronic inflammatory disease in which individuals develop joint pain, stiffness, and swelling. It is a pertinent case study for development of a sidecar application because the condition has several key disease outcomes, including disease activity, functional status, and pain, which are captured routinely as structured fields in the EHR. However, prior studies have shown that communication around these outcomes, even when documented in the EHR, is limited in real-world practice.^5–8 In this paper, our goal was to describe the processes associated with the development, deployment, evaluation, and maintenance of the RA dashboard in a large health system. We provide a roadmap based on our experience building a new EHR-integrated, patient-facing visualization tool that displayed disease outcomes to RA patients during a clinical visit (the “RA PRO dashboard”) in a large academic health center.

Methods

We used the proposed guidelines for health research involving design to describe building and deploying the RA PRO dashboard, an EHR application to display real-time PROs data. ⁹

Approach

Our overall approach was based on an adaptation of a well-known digital innovation process to describe the development, integration, and maintenance of our EHR sidecar application. ¹⁰ The roadmap includes human-centered design (HCD), Agile product development, and lean/clinic deployment and was divided into a “problem-solving phase” and “execution and solution-building phase” (see Figure 1).

Figure 1.

Roadmap for the development and deployment of an EHR sidecar application to track patient outcomes in rheumatoid arthritis.

Frameworks

We used two frameworks to inform our approach to development and deployment of the RA PRO dashboard: First, we used the technology acceptance model (TAM) as the guiding theoretical framework for feature selection. ¹¹ The TAM is a validated model commonly used in healthcare settings to understand and explain user acceptance, adoption, and use of new technologies. ¹² The model theorizes that a user's intention to use (acceptance) and usage behavior (actual use) of technology are affected by their perceptions of its usefulness (increased benefits and positive outcomes when adopted/used) and ease of use (degree to which using the technology is free of effort). ¹³ Within the TAM framework, we considered both patients and clinicians to be users of the RA PRO dashboard, since the dashboard was designed to support communication about RA outcomes during the clinical visit.

Second, we used the Consolidated Framework for Implementation Research (CFIR) as a guiding framework for clinic deployment of the RA PRO dashboard. CFIR is commonly used in studies of implementation of new processes or tools to highlight key domains that can facilitate or create barriers to actual use (intervention characteristics, outer setting, inner setting, characteristics of individuals, and the process of implementation). ¹⁴

Timeline

Building and deploying the RA PRO dashboard took over 3 years (May 2018 to October 2021), starting with the HCD work, through regulatory approvals, application build, user testing, and deployment (see Appendix 1). The prolonged timeline was partly attributable to the COVID-19 pandemic in 2020–2021, when development stopped, while our technology partners were occupied developing pandemic-related health IT tools for the health system.

Dashboard development team

Our research team, comprised of rheumatologists, qualitative researchers, health educators, informaticists, and research coordinators, partnered with a technology team (the University of California San Francisco (UCSF) School of Medicine technology team or SOM-Tech) that included designers, software engineers, and a project manager. SOM-Tech is a full-service team, including designers, developers, analysts, project managers, and quality assurance engineers. Because SOM-Tech is embedded within UCSF Health, the group was familiar with pros and cons of different application architectures and knowledgeable about the security and other regulatory processes required to implement the project.

Clinical setting and patients

We designed the dashboard to be deployed in the rheumatology clinic of UCSF medical center (a large academic medical center in Northern California), which uses Epic as its EHR vendor. This is a teaching clinic with approximately 12 attending physicians who see their own patient panels on 1–4 half days per week in addition to supervising trainees. Some clinicians attend in subspecialty clinics and hence see very few RA patients. Trainees (rheumatology fellows, occasionally internal medicine residents) have weekly half-day clinics and rotate through the clinic over 1–2-year periods. There is also occasionally a nurse practitioner seeing patients.

Patients come to the clinic from northern California and beyond. There are approximately 500 RA patients followed regularly at the clinic. Of note, at the start of the COVID pandemic, telehealth visits increased dramatically, from <5% of visits to >50% of visits, and this has been sustained through 2023. Although the HCD process was undertaken in both English and Spanish, most patients seen in this clinic are English speakers, so this version of the RA PRO dashboard was developed in English only.

Clinical workflows for collecting chronic disease outcomes (prior to deployment of the RA PRO dashboard)

In the UCSF Health rheumatology clinic, disease activity scores (specifically, CDAI scores ¹⁵ ) are collected during in-person visits with clinicians (Supplemental material 1). Typically, patients check in at the front desk and are provided a paper form that includes a patient global assessment, one of the components of the disease activity score. A medical assistant checks vital signs and enters the patient global assessment score into the EHR. The other components of the CDAI are entered during the clinical visit as part of the documentation of the physical exam, including tender joint count, swollen joint count, and evaluator (physician) global assessment score. The EHR automatically calculates the total CDAI score and creates a “smart data element” which can be pulled into the clinical note. In general, CDAI scores are not collected during telehealth visits.

Physical function scores, specifically Patient-Reported Outcomes Measurement Information System-physical function (PROMIS-PF10a scores), are also collected routinely at the clinic (Supplemental material 1).^16,17 During the check-in process, patients are provided a paper form that includes the PROMIS-PF10a questionnaire. A medical assistant enters a raw score into the EHR. The EHR automatically calculates the PROMIS-PF10a t-score and saves it as a smart data element. For patients who have an active patient portal account, the PROMIS-PF12 can be collected online using a questionnaire module. Patients receive a message sent through the portal in advance of the visit with a link to the relevant survey. In this case, the t-score is also saved as a smart data element.

Pain scores are collected by medical assistants during the check-in process using a Likert scale (0–10) for both in-person and telehealth visits (Supplemental material 1).^18,19

Process for the development and deployment of the RA PRO dashboard

We followed the roadmap laid out in Figure 1, including HCD, Agile product development, and clinic deployment/lean principles.

Design thinking

We undertook a HCD process, including focus groups and interviews with patients and clinicians to define the clinical opportunity for the RA PRO dashboard. A verbal consent was obtained from all study participants prior to participation in the focus group discussions and interviews. The informed consent script was reviewed and approved by the UCSF Institutional Review Board. Findings from this process, along with the TAM framework, guided our identification of key features for the application (a “minimal viable product”).

Agile development

Once we had defined a minimally viable product, we undertook an Agile development process, which involved the following: (a) choosing an application architecture that would allow for EHR integration, (b) obtaining risk assessment and regulatory approvals, (c) Scrum process (multiple 2-week sprints of actual development to build the application), and (d) launching the application to a test environment and user testing.

1. Choosing the application architecture: As described elsewhere, several options exist for how EHR integration can be achieved. ²⁰ We worked with our technical team to understand pros and cons of several approaches, including using native Epic tools, developing an API within Epic, or using a sidecar application (Table 1). We ultimately settled on a sidecar application since this would allow maximal flexibility in designing the user interface (UI). The technical team also used the security and application architecture best practices at our institution to help evaluate various approaches. ²¹

2. Risk assessments and regulatory approvals: At our institution, the Digital Diagnostics and Therapeutics (DD&T) committee oversees and supports the extraction of real-time data from Epic hyperspace via the use of APIs. ²² The role of the DD&T includes coordinating approvals from the Privacy, Legal, Risk, Compliance, and Industry, Technology, and Alliances offices. DD&T also acts as an enterprise architectural review board to ensure proposed applications can be deployed safely. The DD&T process includes an assessment of whether the application qualifies as a medical device. ²³

3. Agile development process to build the application: After the application architecture was planned and approvals obtained, the technical team used a Scrum approach for development. ²⁴ Scrum focuses on the regular delivery of a working software to users and depends on user feedback throughout software development. ²⁵ Scrum uses sprints, which are time-limited development efforts, usually lasting between 1 and 4 weeks. Sprint planning sessions were attended by members of the research and technical team. At the end of each sprint, there was an opportunity to demonstrate new features and provide feedback to designers and developers.

4. Launching the application to a test environment, quality assurance, and user testing: The RA PRO dashboard application was first implemented in a sandbox environment that supported the development of applications that interface with the institution's EHR. The environment permits preproduction testing of the interface once the application is fully developed and facilitates production deployment with oversight from the DD&T committee to ensure the application is functioning correctly.

Table 1.

Pros and cons of possible EHR integration approaches.

Approach	Pros	Cons
EHR foundation software build	–Seamless integration –Automatic updates and maintenance	–Difficult to lobby for changes since broader national user may be impacted –Challenging to customize once implemented
EHR integration via sidecar application	–Possible to integrate –Development focused on UI (not the application itself) –Customization possible	–Design requires API, data flow from multiple sources –Maintenance requires attention to EHR and sidecar upgrades –Cost of licensing sidecar application
EHR integration via home-grown application	–Complete customization possible	–Development requires building entire application as well as designing UI –Maintenance requires attention to EHR
Independent application outside of EHR	–Simple data flow outside of the EHR – often coming directly from patients	–No integration with EHR –Data entry required to include information in EHR

Our development process included usability testing of the application in the development environment to help identify potential issues that needed to be addressed with further development or by training. User testing in the sandbox environment was completed by a UI designer with multiple clinicians. Clinicians were asked to comment on the auto-launch functionality of the sidecar application. They were then observed navigating the UI and performing specific tasks.

Clinic deployment/lean

We used the CFIR to guide our approach to deploying and evaluating the implementation of the RA PRO dashboard in the clinic. Steps included user orientation and continuing education and support, monitoring actual use of the dashboard, and establishing a comprehensive set of strategies to effectively manage and maintain the application over time.

Results

We followed the steps in the roadmap (Figure 1) to develop and deploy our RA PRO dashboard (detailed actions in Table 2).

Table 2.

Specific actions taken in the design, development, and deployment of the RA PRO dashboard.

Phase	Actions taken
Design thinking
HCD process	We worked with patients and clinicians, researchers, and the technology team on a HCD process (described in detail elsewhere)26,27
Proposed clinical workflow	Based on our HCD work, we developed a proposed workflow incorporating the RA PRO dashboard into a clinical visit (Figure 2)
Minimum viable product definition	Key feature selection and prioritization was derived from the HCD process. We used the TAM as guiding framework for feature selection (see Appendix 2 for list of features)
Agile development
EHR integration approach	We selected EHR integration via a sidecar application based on ability to customize the interface and institutional license fees
Risk assessment and approvals	Our proposed architecture needed to be shared with the Digital Diagnostics and Therapeutics Committee before the build could begin. After the application was built and tested, we returned to get a final approval prior to deployment
Scrum	We used storyboards to identify and prioritize key features, followed by 2-week sprints
Launch to test environment and user testing	User testing in the sandbox environment occurred by the UI designer in the clinic with four providers (including two attendings and two fellow physicians)
Launch to production and user testing	After testing and final risk assessments, the application was launched into the production environment
Clinic deployment/Lean
User orientation and education	Study staff developed training and development of educational materials to encourage clinicians, patients, and other clinic personnel to highlight the features of the dashboard
Continuing education and support	We developed training guides and educational resources for patients, clinic staff, and physicians in order to improve their understanding and promote use of PROs during routine clinical care
Monitoring actual use	We distributed reports to clinicians and clinic leadership to track the quality of the data entering the dashboard (i.e. the collection of PROs and RA outcomes) and monitored engagement with the dashboard using user audit logs
Application maintenance	We established a comprehensive set of strategies to effectively manage and maintain the application over time including content updates, software and security updates, and major changes to clinical workflows

Design thinking

Following a HCD process, we developed a proposed clinical workflow and defined the minimum viable product for the dashboard (see Table 2 and Figure 2). Key features required for the RA PRO dashboard are described in Appendix 2. Based on concepts from the TAM framework, features were selected to increase the ease of use of the dashboard, including automatic launch with the EHR and various features to tailor the delivery of the information based on the preferences of the clinician and patient (e.g. toggle buttons to hide PROMIS and Pain graphs; hovering over data points to reveal score values and features designed for low-literacy patients such as smile/frown faces on axes and medication route graphics). Others were selected to maximize the usefulness of the dashboard, including the selection of specific outcomes, display of target ranges for each outcome, and the ability to scroll through historical outcomes and information about medication use over time. We also included a mechanism for assessing actual use of the dashboard through user logs.

Figure 2.

Workflow incorporating the RA PRO dashboard into a clinical visit.

Historical RA outcome data is pulled from the EHR data warehouse and loaded into the dashboard. Outcomes collected during the check-in process (e.g. PROMIS-PF and pain scores) and disease activity scores documented by the clinician are loaded into the dashboard during the visit. The dashboard automatically launches as soon as the patient's chart is opened and can be refreshed to incorporate newly documented information at any time during the visit. Outcomes are displayed in a graphic format to show trends over time. At the end of the encounter, the clinician can print the view of the dashboard that they have discussed, complete with the patient's most recent scores. The patient can take the print-out home to communicate the results of their ongoing treatment with family, friends, or other clinicians in their care team.

Agile development

With the minimum viable product defined, we proceeded with an Agile development process to build the application.

1. Choosing an EHR integration approach: We elected to build the RA PRO dashboard as a sidecar application to the EHR using a Salesforce platform. This allowed the development to be focused on the UI as opposed to the application itself; some customization would still be possible. Potential barriers to using the sidecar application included licensing costs and that maintenance would require attention to both EHR and sidecar application upgrades. However, our technical team had experience with this approach and our institution had a discounted license cost, which made it more feasible.

2. Risk assessments and regulatory approvals: We pursued our institution's established processes to gain approval to deploy the dashboard in a production environment. This included reviewing the application architecture, data flow, and proposed impacts to other systems through an established change control process. ²⁸ The dashboard was deemed not to be a medical device, since it visualizes existing outcomes (CDAI, PROMIS-PF, and arthritis pain scores) and ranges, all of which have been previously established.

3. Application build (Scrum): We used storyboards to identify and prioritize key features, followed by 2-week sprints. We completed the build after seven 2-week sprints, with one last sprint to complete the application after a long COVID-related delay. Technical optimizations included minimizing data load time for historical and real-time (same day) data, automatic launch once the patient's chart was opened to remove extra clicks, and weekly incremental data loads to improve application performance. The final application architecture is described in Appendix 3.

4. Quality assurance and user testing: The RA PRO dashboard underwent a quality assurance process prior to being released into production to ensure that it accurately and reliably performed its intended functions in a real-world clinical setting. Usability testing by four clinicians (two attendings and two fellows) resulted in minor changes to the UI; for example, clinicians and patients commented that the text needed to be larger for optimal viewing and that the Y axis scales for each outcome needed to more clearly highlight target values (e.g. low disease activity or remission). This information was subsequently incorporated into our training materials. Screenshot of the final RA PRO dashboard is shown in Figure 3.

Figure 3.

Final dashboard screenshot. ²⁹

Clinic deployment/lean

The RA PRO dashboard was launched into the production environment in August 2021.

Supporting clinician engagement with the RA PRO dashboard: with the launch of the RA PRO dashboard, we incorporated principles of quality improvement and lean management to encourage the use of the dashboard during clinical encounters. Our approach focused on supporting positive user attitudes and behavioral intentions to use the dashboard according to the CFIR framework (see Table 3). Orientation sessions for the dashboard and clinician discussion groups in the several months after deployment were designed to increase clinicians’ awareness and understanding of the usefulness and usability dashboard and influence their intention to share it with their patients during clinical visits. Study staff also developed targeted educational materials to encourage clinicians, patients, and clinic staff to highlight the useful features of the dashboard. Data streams were already in place to monitor collection of RA outcomes including CDAI and PROMIS scores prior to launch, and these were continued after deployment: for example, we generated monthly reports showing the proportion of visits with these outcomes documented in the EHR, overall and by clinician (see Appendix 4).

Table 3.

Implementation science concepts as applied to the clinic deployment of the RA PRO dashboard.

Strategy	CFIR construct	Description	Explanation
Clinician orientation sessions	Process of implementation	The research team provided individual, in-person orientation sessions lasting between 10 and 15 min at the time each clinician was provisioned to have access to the dashboard	Orientation sessions included a general description of the objectives of the dashboard and its features
Bimonthly conferences to discuss deployment with clinicians	Inner setting	Brief bimonthly conferences were organized to elicit feedback about the dashboard from clinicians	Over the first year that the dashboard was deployed, we held four conferences for all clinicians to attend as part of regularly scheduled faculty meetings. Discussion included successes and challenges in using the dashboard and allowed clinical champions to share best practices for using the dashboard
Clinician educational resources	Characteristics of clinicians	Teaching sessions, educational handouts, and scripts on how to collect and discuss PROs were shared with clinicians	Training on RA outcomes and PROs were incorporated into regular teaching sessions for trainees along with information on how to score PROs with all clinicians. We also provided clinicians scripts for how to discuss PROs with patients
Training for nurses and medical assistants	Inner setting	We developed a guide and trained nurses and medical assistants (MAs) on collecting RA PROs. The training focused on collecting the patient global assessment component of the CDAI, PROMIS-PF, and the pain scores, and entering the data into the EHR	This training provided information about RA PROs and the importance of collecting these measures. The goal was to increase clinical staff knowledge on the importance of collecting PROs, with the hope that this would motivate their collection of PROs—with more data available, this could enhance the use of the dashboard by clinicians
Resources for patients	Inner setting	We developed an information sheet for patients explaining the types, meaning, importance, and use of the different PROs collected before and during visits at the rheumatology clinic	The information sheet increased patient awareness and understanding of the types and meaning of the RA outcomes collected before and during their RA visit and helped motivate them to complete the PRO surveys and measures offered to them in the waiting room
Audit-log and data quality reports share with clinic leaders	Outer setting	We generated reports using EHR data (the same data used for display on the dashboard) to share with clinicians and clinic staff (see sample report, Appendix 2). Second, we tracked the actual use of the dashboard over time using audit-log data from Google Analytics (see sample report, Appendix 5)	These reports were provided to clinic leadership and individual clinicians on a weekly and monthly basis

Because the RA PRO dashboard was deployed as part of a stepped-wedge pragmatic clinical trial, we were able to fund research staff to attend every clinic. Staff provided reminders to clinicians about using the dashboard and provided educational materials and print-outs of the RA PRO dashboard to patients upon arrival. Staff could also offer “at the elbow” support to clinicians who needed help accessing or interacting with the dashboard. Finally, we monitored actual engagement with the dashboard through user audit logs (Appendix 5). These data were shared with clinic leaders (medical director and staff supervisor) to review at weekly and monthly intervals and help address barriers to dashboard use for specific clinicians.

Ongoing maintenance issues

Collection of user log data was an important part of our lean process, since meaningful changes in number of sessions or specific actions on the dashboard heralded maintenance issues with the dashboard: these included content updates such as medication additions, software system updates, and major changes to clinical workflows (see Table 4).

Table 4.

Maintenance issues related to building an EHR-integrated sidecar application.

Maintenance category	Example issue	Potential solutions
Content updates	New medications (or new versions of existing medications) are continually becoming available	Set regular update timelines (monthly, quarterly) Design application to be able to accommodate changes or additions to variable names
Software updates	Because there were so many systems involved in the application, software or security updates to component systems were continual, including updates to the EHR, sandbox environments, Salesforce, and Google Analytics	Plan for continued engagement with technology team to respond to software updates Look ahead and know when software updates will occur to anticipate potential downstream consequences Monitor data streams (e.g. user audit logs) and create alerts for reduced or missing data for early identification of problems
Major changes to clinical workflows	The explosion of telehealth visits during the COVID-19 pandemic resulted in a sharp decrease in the collection of RA outcomes, since assessment of disease activity (tender and swollen joints) requires a physician exam	Routine data quality assessments Stratification of quality assessments based on visit type

Discussion

In this paper, we present a roadmap for the successful development, integration, and maintenance of an EHR sidecar application for RA, a common chronic condition in ambulatory care. Grounded in HCD principles, the development process was led by an internal technology team familiar with established software development practices at our institution. To ensure effective adoption of the dashboard within our clinical setting, we invested significant efforts into clinical implementation, developing orientation and educational materials tailored for clinicians, medical staff, and patients. We also employed systems for monitoring data quality and tracking actual usage of the dashboard.

We found the TAM an especially useful framework for prioritizing tasks to maximize usability and ease of use for clinicians, clinic staff, and patients. As part of our approach, we prioritized certain features to improve the dashboard's functionality. These included personalized axes for graphical ranges, a toggle button to show or hide additional PROs and detailed lab data per physician preferences, and the inclusion of target ranges for disease activity and physical function represented by a “green zone” on each graph. However, during the development process, we had to make several challenging decisions when competing priorities arose. One example of such a trade-off was the choice to implement an auto-launch feature. This feature was designed to simplify the user experience by reducing potential friction when accessing the dashboard during clinical visits. ³⁰ While this user-centric approach was beneficial for ease of use, it introduced challenges in tracking usage, as passive viewing could not be recorded in audit logs. Despite this limitation, we considered it a worthwhile trade-off because it prioritized removing barriers to utilization, ultimately benefiting both clinicians and patients.

Careful attention to human factors and workflows played a pivotal role in the successful deployment of the RA PRO dashboard within the clinic setting. Although our user testing suggested the use of the dashboard would be intuitive for most clinicians and staff, we discovered a few technical quirks during user testing that required repeated troubleshooting. We were also surprised that many clinicians found integrating the dashboard into their clinical routines challenging, particularly conversations with patients about the meaning and implications of functional status. Because the deployment was in the context of a pragmatic clinical trial, we had the advantage of having staff on-site to provide support to clinicians and to develop educational materials, including scripts, to assist them in effectively discussing the content of the dashboard during patient visits; such research resources to study implementation may not be available in purely operational deployments. In addition, systematic collection of survey and qualitative data through the pragmatic trial enabled us to tailor new materials and target clinicians with lower engagement with the dashboard. Finally, implementation of robust data tracking systems served multiple purposes: (1) audit use logs alerted our development team to any issues with the dashboard that required technical attention, and (2) PRO documentation-run chart reports were provided monthly to clinic leadership to inform them about the impact of the project on clinical metrics. This symbiotic relationship between the RA PRO dashboard's development and the pragmatic clinical trial emphasizes the benefits of aligning innovative healthcare solutions with research initiatives.

The development of health IT tools such as the RA PRO dashboard as a research project in an academic health setting comes with several inherent constraints, especially in terms of funding and timelines. The nature of funding for medical researchers requires careful planning and submission of grant proposals to fund the work months or years in advance. Compliance with regulatory standards and ethical review by an institutional review board adds complexity to the development process. Maintenance of such applications poses another significant challenge for physician researchers since budgets are limited once the period of grant funding has ended. In this project, we encountered issues related to numerous security updates, software upgrades, and evolving EHR structures, even over a relatively short, 2-year period. These maintenance challenges are typical for any digital health application, particularly one tightly integrated with the EHR, involving multiple software components, but for physician researchers, sustaining the effort and funding required for long-term sustainability can be problematic. ³¹

From the perspective of the healthcare system, the cost-effectiveness of a new sidecar application will be an additional critical component of a successful long-term deployment. Clearly, institutions must be prudent in selecting investments to avoid committing significant resources to applications or features that do not yield substantial benefits. However, assessing the effectiveness of an application may be challenging. The impact on clinical outcomes may be slow to materialize; improvements in patient satisfaction, self-efficacy, or patient-doctor communication might be more immediate but harder to quantify or may not directly translate to into measurable clinical benefits. Future studies should include a comprehensive cost–benefit analysis to determine the financial viability of maintaining the dashboard in the long-term, considering both direct costs, such as development and maintenance expenses, and indirect benefits, such as potential improvements in chronic disease management that could lower long-term healthcare costs.

In conclusion, we have outlined a roadmap for the successful development and deployment of an EHR-integrated sidecar application, drawing insights from the Gartner model to describe the constraints often encountered in large health systems and academic funding timelines. To facilitate the effective adoption of clinical dashboards like the RA PRO dashboard, we recommend the creation of comprehensive training materials tailored for clinicians, medical staff, and patients. Maintenance challenges, including software and security updates, should be anticipated and addressed proactively. The main limitation of this roadmap is that it may not be generalizable to other institutions or healthcare settings. However, we believe that many of the concepts and phases we have outlined will have broad applicability across different environments.

Future studies with the RA PRO dashboard will focus on assessing patient and clinician perceptions of the dashboard's utility and measuring its impact on patient outcomes and clinician satisfaction within the framework of a stepped-wedge pragmatic trial. Additionally, future work could incorporate results of artificial intelligence simulations into the dashboard to visualize personalized predictions about how patients will respond to different medications. Such a forward-looking approach will be essential in advancing the capabilities of EHR-integrated applications and enhancing patient care.

Appendix

Appendix 1

Study timeline.

Dates	Activity
May–August 2018	HCD process (focus groups, etc.)
September 2019	Initial engagement with technical team
October 2019–September 2021	Risk assessment and regulatory approvals
December 2019	Technical design and planning sessions
January 2019–April 2020	Application build (sprints)
April 2020–July 2021	COVID-19–related delay
August 2021	User testing
September 2021	RA PRO Dashboard go-live
October 2021	Stepped-wedge pragmatic trial begins

Appendix 2

Key feature requirements for the RA PRO Dashboard.

Feature	TAM construct	Explanation
Graphical display of disease outcome data
Outcomes include disease activity (CDAI), physical function (PROMIS-PF10a), and pain scores	Usefulness	These are key outcomes identified by patients and clinicians needed for a “treat-to-target” approach to RA care
Disease activity and physical function outcomes are both oriented such that “up” on the graph indicates a better physical function status and a high disease activity	Ease of use	In focus groups patients felt strongly that better scores should be higher up on the page. Therefore, we “flipped” the direction of the disease activity measure such that lower scores appeared toward the top of the screen
Personalized Y axis ranges based on patient's individual results range	Ease of use	Y axis ranges need to change based on the maximum scores for individual patients in order for changes over time to be apparent when viewing the dashboard
Y axis labeled with smile/frown faces	Ease of use	In focus groups, patients reported that specific scores did not have as much meaning and preferred to have general indicators of improvement vs. worsening of measures
Hovering mouse over each data point shows pop-up label with score value	Ease of use	Other patients preferred to see the “raw data”—a hovering score achieved a balance between data access and an uncluttered look for the dashboard
Goals/benchmarks for each outcome
Target ranges for disease activity and physical function identified as a “green zone” on the graph. Disease activity (CDAI) targets are scores in low disease activity or remission range. Physical function score (PROMIS-PF) targets are t-scores better than 50. Pain score target is zero	Usefulness	Because treatment for RA takes a “treat-to-target” approach, having the graphs display targets help orient patients to the goals of treatment
Longitudinal display of data over time
For all three outcomes, data from last five visits to the rheumatology clinic (including “real-time” data from today's visit) displayed in default view	Usefulness	Since visits happen every 3–4 months, data from the last five visits represents over a year of disease activity and helps patients appreciate trends in their disease trajectory
Scrolling backward through timeline reveals scores for additional prior visits.	Usefulness	Pressing the backward arrow allows patients to see data from earlier visits as well
RA medication information
RA medication information (including disease modifying antirheumatic drugs and glucocorticoids) is displayed graphically, with shaded bars representing initiating dates and periods of use	Usefulness	One goal for the dashboard was to highlight the relationship between treatments and their disease trajectory, with a hypothesis that understanding this connection may increase medication adherence
Icons indicate whether medications are oral or injectable	Ease of use	RA treatments have many difficult-to-pronounce names, so icons were added to make this more easily understandable, especially for limited health literacy patients
Medication information is synchronized with scroll of the timeline	Usefulness	Scrolling backward through the timeline to prior visits also results in the medication information from prior visits to become visible
Laboratory values
Labs include the most recent values for AST, creatinine, and CRP, along with normal ranges	Usefulness	Patients uniformly requested to have information about medication safety and expressed a strong desire to have access to laboratory data. We included only three lab tests that are the most important with regard to medication safety (assessing liver and kidney toxicity) and disease activity (CRP)
Ability to show more/less information
A toggle button can hide a section of the graph (only disease activity and medications are visible)	Ease of use	In order to make the amount of information less overwhelming, we added a toggle button that would hide the functional status and pain graphs; thus, the dashboard would only display disease activity and medications. This also brings the medication information closer to the disease activity graph, making it easier to appreciate the relationship between them
Lab values can be hidden such that only the brief message is visible: “Your labs are normal.” in green text, “Talk to your doctor about your labs.” in red text, or “Your labs are missing.” in yellow text (for labs that have not been recorded in the EHR within the past 2 years).	Ease of use	Patients had mixed opinions on how much information was necessary. The lab section displayed a message about the labs, and we also created a toggle button that revealed the latest lab values for patients wishing to have more information
Access to dashboard information outside of the clinic visit
Default view of the dashboard (last five visits) can be printed for after-visit review by other clinicians or family members	Usefulness	Patients expressed a strong desire to access their data after the visit was over. By pressing the print button, an image of the dashboard prints at the printer in the clinic room and can be handed to the patient at the end of the visit
Screenshot of dashboard can be pasted into a patient portal message for telehealth visits	Usefulness	Alternately, a screenshot of the dashboard would be pasted into a portal message and sent to the patient electronically. This is especially important for patients completing a telehealth visit
Automatic launch within the EHR
RA PRO dashboard launches within the EHR automatically for a subset of patients seen in rheumatology clinic. Criteria for which patients had the dashboard launch included (1) scheduled for a follow-up visit; (2) at least one documented RA outcome score (CDAI, PROMIS, or pain). The dashboard is immediately visible for all provisioned clinicians—no additional sign in is required	Ease of use	The dashboard autolaunched as soon as the chart is opened to nudge clinicians to use it during the clinical visit
User logs/analytics
Launch and actions within the dashboard are logged, including data on clinician, patient, dates, times, and specific actions (clicks, scrolls, toggles)	Assessment of actual use	User logs can be used to monitor engagement with the dashboard

TAM: technology acceptance model

Appendix 3

Application architecture for RA PRO Dashboard.

The application architecture needed to account for our key requirements (see Table 1) to display both historical and “real-time” data (i.e. data from the current visit; see Figure 2). Historical data were brought in via a one-time load from Epic's Clarity reports for all patients ever seen in the UCSF Health rheumatology clinic. Data from the same day's visit came via a mule application programming interface (API) application. The process flow included the clinician opening the patient chart in Epic. The Salesforce-based RA PRO dashboard app was automatically launched for provisioned clinicians (an authentication dance occurs to authenticate the clinician). The medical record number of the patient is passed to the Mule API app. The Mule API app calls the Epic API to retrieve information from the current encounter, including disease outcomes, and most recent lab results. Package information is collected and returned to the Salesforce RA PRO dashboard app and saved. The RA PRO dashboard can display current and historical data for disease outcomes. Incremental data updates (weekly) are completed by a Salesforce refresh process to update lab results: the Mule API app calls the Epic API to retrieve information about lab results, and the data are again saved into Salesforce.

Appendix 4

Example of a run chart shared with clinical leadership to display key RA outcome inputs into the RA PRO dashboard.

Appendix 5

Example of a chart shared with clinical leadership to display clinician engagement with the dashboard