Implementation of an EHR-integrated web-based depression assessment in primary care: PORTAL-Depression

Abstract

Objectives

To integrate a computerized adaptive test for depression into the electronic health record (EHR) and establish systems for administering assessments in-clinic and via a patient portal to improve depression care.

Materials and Methods

This article reports the adoption, implementation, and maintenance of a health information technology (IT) quality improvement (QI) project, Patient Outcomes Reporting for Timely Assessment of Life with Depression (PORTAL-Depression). The project was conducted in a hospital-based primary care clinic that serves a medically underserved metropolitan community. A 30-month (July 2017-March 2021) QI project was designed to create an EHR-embedded system to administer adaptive depression assessments in-clinic and via a patient portal. A multi-disciplinary team integrated 5 major health IT innovations into the EHR: (1) use of a computerized adaptive test for depression assessment, (2) 2-way secure communication between cloud-based software and the EHR, (3) improved accessibility of depression assessment results, (4) enhanced awareness and documentation of positive depression results, and (5) sending assessments via the portal. Throughout the 30-month observational period, we collected administrative, survey, and outcome data.

Results

Attending and resident physicians who participated in the project were trained in depression assessment workflows through presentations at clinic meetings, self-guided online materials, and individual support. Developing stakeholder relationships, using an evaluative and iterative process, and ongoing training were key implementation strategies.

Conclusions

The PORTAL-Depression project was a complex and labor-intensive intervention. Despite quick adoption by the clinic, only certain aspects of the intervention were sustained in the long term due to financial and personnel constraints.

Introduction

The prevalence of depression symptoms has escalated in the United States in recent years.^1,2 But without systematic screening, depression goes unrecognized in about half of the primary care patients,^3–5 and about half of those who are diagnosed receive inadequate treatment and management.^6,7 Strong evidence suggests that measurement-based care reduces delays in depression treatment and improves outcomes.^8–12 However, healthcare systems may not have the capacity to routinely screen or re-assess symptoms. Therefore, it is crucial to identify efficient interventions that can measure depression symptoms in a timely manner.

Health information technology (HIT) has the potential to address gaps in mental healthcare by promoting population-level health and management.^13,14 Increasingly, research has shown that HIT interventions can enhance behavioral healthcare by reaching high-risk populations that are not otherwise engaged in routine care.^15,16 Additionally, health technologies have the flexibility to supplement face-to-face healthcare or can stand alone, which can be advantageous for the time-limited resources in primary care.¹⁷ HIT interventions have the potential to facilitate a health system’s capacity to conduct measurement-based care, but the implementation of HIT interventions that support population-based depression measurement is challenging.

This article reports the rationale, design, and implementation of a HIT quality improvement (QI) project, Patient Outcomes Reporting for Timely Assessment of Life with Depression (PORTAL-Depression, NCT03832283), which sought to improve depression detection rates and measurement-based depression care in a primary care clinic. The goal of the PORTAL-Depression project was to integrate a computerized adaptive test for depression into the electronic health record (EHR) and establish systems for administering this test in a clinic and via the electronic patient portal.

Methods

Setting

The project was undertaken at an adult general internal medicine clinic at an academic, urban tertiary care center. At implementation, the clinic was staffed by 35 internal medicine and medicine-pediatrics attending physicians, 97 internal medicine resident physicians, 16 internal medicine-pediatric resident physicians, 3 advanced practice nurses, 5 registered nurses, 6 licensed practical nurses, 9 medical assistants (MAs), and 1 social worker. The clinic population was about 28 000 patients.

Since 2015, the Primary Care Behavioral Health Integration Program (PC-BHIP) has supported clinical care by providing improved access to behavioral health services, developing clinical decision support tools, coordinating the management of common behavioral health problems, establishing internal and external referral partnerships, and advising on specific patient cases when needed.^18,19

The PC-BHIP team developed a Health Maintenance Activity (HMA) and Best Practice Advisory (BPA) for depression screening in patients 12 years or older without a history of depression in the Epic EHR in 2016.²⁰ Then in 2017, the PC-BHIP team successfully improved depression screening rates by implementing a protocol to have MAs administer a paper-based PHQ-2 during triage.²¹ Primary care physicians (PCPs) and advanced practice nurses were responsible for completing the PHQ-9 in patients who had a positive (≥3) PHQ-2 result. By 2018, however, the screening rate in the clinic had plateaued at around 50%, a lower-than-expected number of patients (1%) were screening positive for depression, and PCPs were unaware of about 10% of positive PHQ-2 results.²¹

In 2019, the PC-BHIP team designed HMAs and BPAs to encourage routine assessment of patients with a history of depression. A depression surveillance BPA advised annual assessment using the PHQ-9 among patients with depression on their problem list or a previous PHQ-9 score of 10 or greater. A depression monitoring BPA advised monthly assessment among patients whose most recent PHQ-9 score was 10 or greater, indicating moderate to severe symptoms until they reached remission (PHQ-9 < 5). The surveillance and monitoring workflows were piloted in the fall of 2019 and added to MA triage responsibilities in January 2020. However, demands on MA and PCP time during visits inhibited the uptake of depression surveillance and monitoring assessments.

Intervention

PORTAL-Depression was designed to improve depression identification and symptom assessment, facilitate measurement-based care, and decrease the in-clinic burden of administering depression assessments. The lead investigator (N.L.) assembled a multidisciplinary team that focused on integrating the PORTAL-Depression project into the clinical workflow. Team members included the primary care clinic medical director (L.V.), the chair of psychiatry (D.Y.), the director of health psychology (N.B.), the associate chief medical information officer (S.S.), the health technology director of clinical research initiatives (J.M.), a senior business system analyst (W.D.), statisticians (R.G., M.Z.), and project managers (M.F., E.S.). The team met throughout the project period to develop key intervention components, discuss barriers and solutions, and strategize intervention implementation.

The PORTAL-Depression project integrated 5 major innovations into the EHR (Figure 1). These included: (1) use of a computerized adaptive test for depression assessment, (2) 2-way communication between cloud-based software and the EHR without the exchange of protected health information (PHI), (3) improved accessibility of depression assessment results in the EHR, (4) enhanced awareness and documentation of positive depression results, and (5) capabilities to send assessments via the portal.

Figure 1.

The PORTAL-Depression health IT innovation framework.

1. Computerized adaptive tests use algorithms to ask the most diagnostically informative questions, adapting to patient responses in real-time to optimize precision and brevity. The Computerized Adaptive Test for Mental Health (CAT-MH) is an external cloud-based system of questions adaptively administered based on multidimensional item response theory.²² The adaptive test for depression draws from a bank of over 400 questions and is comprised of 2 components. The depression diagnostic test (CAD-MDD) reflexes into the depression assessment tool (CAT-DI) in patients that screen positive for depression.^23–26 Additionally, the adaptive test can read questions aloud, thereby reducing literacy issues or problems with question fidelity. The CAT-MH was selected because its sensitivity, compared to the PHQ-2, was judged to be superior in a previous study in our primary care clinic.²⁶

2. To integrate the CAT-MH into the EHR and clinical practice, the PORTAL-Depression team implemented an end-to-end mechanism to communicate with a third-party entity using an Enterprise Service Bus (ESB). The ESB is a middleware solution that can translate a message from one software (CAT-MH) to another software (EHR) without the transmission of any PHI. Linkages between CAT-MH and the ESB and between the EHR and the ESB allowed for mutual 2-way communication. The team developed a process through which placing an order for depression assessments in the EHR generated a unique CAT-MH URL for each patient. The link is directed to the CAT-MH in English or Spanish based on the patient’s preferred language in the EHR. When a patient completed the CAT-MH, results were automatically transmitted to the EHR.

3. The depression screening BPA was modified with a link to the CAT-MH order rather than a link to a PHQ-2/9 smart flowsheet. In the clinic, MAs or PCPs could place the order to generate a unique link and open the CAT-MH. When a patient completed the CAT-MH, results were automatically saved in the patient’s record and sent to their PCP’s EHR in basket. CAD-MDD results and CAT-DI scores were designed as result components to allow these data to be displayed in the Results section of the EHR with other laboratory tests. In the Results section, PCPs could view the CAD-MDD result, CAT-DI score and interpretation (ie, level of severity), the text of the specific questions the patient completed, and the patient’s responses. Figure S1 shows the depression score interpretation that was used. If patients completed the CAT-MH multiple times, PCPs could view all results and compare scores over time.

4. If a patient scored in the moderate or severe range (CAT-DI ≥ 50 or PHQ-9 ≥ 10), a high-priority (red) passive BPA appeared in the encounter. The BPA prompted PCPs to acknowledge the positive assessment result and gave PCPs the option to add depression to the problem list and to open a depression order set with common referrals, patient instructions, and medications to facilitate appropriate follow-up care.

5. To administer assessments outside of clinic visits, the PORTAL-Depression team generated individual links for eligible patients by placing bulk CAT-MH orders. A template invitation letter was created with smart text to insert the unique link for each patient. Letters were sent in bulk. Upon receiving a notification (eg, email, text) that they had a new letter in their portal account, patients could log in, review the letter, and click on the link to open the CAT-MH. Results were automatically saved in the EHR and sent to the PCP as described above.

Study design

PORTAL-Depression was a 30-month QI project (Figure 2) evaluated using the RE-AIM framework.²⁷ Results of 2 randomized trials testing the effectiveness and reach, including patient characteristics, have been previously reported.^28,29 In this article, we report on adoption, implementation, and maintenance (Table 1).

Figure 2.

PORTAL-Depression project timeline. ^aContinuous quality improvement throughout the implementation period included clinic walkthroughs, chart audit and feedback, team challenges, feedback and revising, and ongoing training. Abbreviations: CAT-MH = Computerized Adaptive Test for Mental Health; EHR = electronic health record; IT = information technology; PHQ = Patient Health Questionnaire.

Table 1.

PORTAL-Depression project measures and evaluation.

Domain	Outcomes	Data sources
Adoption	Number and percentage of clinical staff and physicians who participated in delivering the intervention	EHR data
Implementation	Implementation strategies, fidelity, adaptations, and cost of the intervention	Meeting agendas and notes, project documents and emails, IT team hourly rate and effort, CAT-MH data, survey data
Maintenance	Sustained use of intervention in routine practice, interest in continuing intervention	Project documents and emails, EHR data, survey data

Abbreviations: EHR = electronic health record; IT = informational technology; CAT-MH = computerized adaptive test for mental health.

To conduct the evaluation, the PORTAL-Depression team collected data from the EHR and the CAT-MH platform. We tracked project activities and meetings, as well as the hourly effort for the IT build. We administered surveys to attending PCPs and MAs at 2-time points: (1) post-implementation of the clinic- and portal-based screening workflows (June 2019) and (2) post-implementation of the clinic- and portal-based surveillance and monitoring workflows (July 2020). On the surveys, statements regarding knowledge, attitudes, and behaviors related to depression assessment were rated on a 5-point scale (strongly disagree to strongly agree).

Measures

Adoption was defined as the percentage of clinical staff and PCPs who participated. We measured the proportion of MAs who performed screening, surveillance, and monitoring as part of routine care and the proportion of PCPs who had patients screened in the clinic or via the portal.

To assess implementation, we documented strategies, adaptations, and costs. We categorized implementation strategies using the Expert Recommendations for Implementing Change (ERIC) framework,^30,31 tracked adaptations to components of the intervention, and calculated costs of the intervention based on billed hours for the health IT build. The survey asked attending PCPs and MAs about the implementation process, confidence and satisfaction using the assessment tools, and potential benefits and barriers. Resident PCPs were excluded from the survey because the residency program restricts the number of surveys residents are asked to complete.

Maintenance was assessed by the extent to which intervention components were sustained as part of routine practice and PCPs’ willingness to continue the intervention.

Analysis

A descriptive analysis was conducted using summary statistics to address each AIM domain. Analyses were performed in Microsoft Excel, SAS 9.4 (Cary, NC, United States), and R 3.5.1.

Results

Adoption

During the 30-month period, 70% (N = 134/191) of PCPs had patients assessed for depression during a clinical visit (92% [N = 33/36] attending; 65% [N = 101/155] resident). A similar proportion (68%, N = 130/191) had patients assessed via the portal (92% [N = 33/36] attending; 63% [N = 97/155] resident). Three PCPs (2 attending; 1 resident) opted out of portal-based depression assessment. All MAs performed clinic-based screening, surveillance, and monitoring.

Implementation strategies

We used a range of strategies when implementing PORTAL-Depression (Table 2). We designed and tailored the clinic- and portal-based workflows to the needs of the primary care clinic and changed the EHR infrastructure. We cultivated partnerships with stakeholders and consumers. Referral partnerships were established with community mental health providers to ensure mental health care resources were available for patients identified to have depression. We engaged key clinical stakeholders (eg, clinical leadership, community) and individualized outreach to clinic staff and faculty to ask for feedback on system design, patient-facing materials, and protocols for depression assessment. To train and educate the clinic staff, providers, and residents on upcoming changes to depression screening and management workflows, we held in-person (eg, at section meetings, and clinical operations meetings) training sessions and provided self-guided online training materials prior to implementation. The PORTAL-Depression team continued to provide interactive assistance by educating and training the clinic workforce throughout the project. Further, to maintain clinic engagement, the project leader (N.L.) did regular clinic walkthroughs to identify issues and be available as a resource during clinic hours.

Table 2.

PORTAL-Depression project implementation strategies.

Strategy	Operationalized for the PORTAL-Depression project
Adapt and tailor to context	Designed and tailored the clinic- and portal-based depression screening, surveillance, and monitoring system to clinical needs
Change infrastructure	Integrated IT solutions within the electronic health record to conduct clinic- and portal-based screening and assessment (eg, availability to send messages to patient without scheduled visits, high priority alert and positive results sent directly to providers)
Develop stakeholder interrelationships	Engaged and obtained approval from clinic leadership to integrate a new clinic- and portal-based screening and assessment workflow as part of standard care Engaged outside experts to develop and integrate new workflows into standard care and the electronic health record Held meetings with stakeholders to discuss project integration into routine care and solutions to barriers and challenges Sent monthly newsletters updating stakeholders about implementation progress Developed partnerships and referral system with community mental health providers
Engage consumers	Involved community members and clinic leadership, staff, and providers in developing and revising patient-facing materials and designing the clinic- and portal-based depression screening and assessment system System-level awareness of the need for systematic depression screening and assessment
Provide interactive assistance	Availability and accessibility of training materials and project team members to provide additional instructions and guidance on conducing clinic-based depression screening and assessments
Support clinicians	Relayed instant positive results directly to provider’s inbox for awareness Stored all depression screening and assessment results as laboratory results for accessibility
Train and educate stakeholders	Conducted in-person, online, and individualized training on the new clinic- and portal-based depression screening and assessment workflows Developed clinic-specific depression screening and management clinical decision support tools to be placed in clinic rooms Distributed written educational materials to all clinical staff, providers, and leadership on the new clinic- and portal-based depression screening and assessment workflows Provided continuous training in the clinic through individualized training sessions, during medical staff meetings, and technical assistance during clinic walkthroughs
Use evaluative and iterative strategies	Assessed practical and technical barriers and facilitators to clinic- and portal-based screening and assessment workflows Iterative feedback from various health system stakeholders and community members on patient-facing materials Obtained feedback on new clinic- and portal-based workflows via surveys of clinical staff and providers Developed and implemented a medical assistant protocol for clinic-based depression screening and assessment Developed a workflow to conduct portal-based depression screening and assessment Conducted daily chart audits and provided individual and team feedback to clinic leadership and medical assistants on complete and incomplete depression screening, surveillance, and monitoring assessments at the visit and population-level Created weekly team challenges with incentives Piloted clinic-based screening and assessments among a small group of providers Used a slow rollout strategy for portal-based screening and assessments by sending messages at longer time intervals to accommodate resident rotations
Utilize financial strategies	Obtained funding from the Agency for Healthcare Research and Quality to support the IT build needed to integrate a clinic- and portal-based screening and assessment system into the electronic health record

Abbreviation: IT = information technology.

We used several evaluative and iterative strategies to monitor the PORTAL-Depression progress. When implementing CAT-MH screening in the clinic, an initial 2-week pilot was done with a small group of providers to streamline workflows, address IT issues, and answer any questions and concerns before the full launch. A similar approach was made for portal-based screening by sending invitations to a small group of patients prior to the full launch. Daily audits of depression screening, monitoring, and surveillance completion were also conducted, and feedback was provided to clinic staff and leadership weekly. From October 2019 to August 2020, weekly challenges were added to the daily audit and feedback with small incentives (eg, pens, note pads) for MAs when they maintained a benchmark assessment rate for 5 consecutive days. A total of 7 team challenges were completed by the MAs in which the average time to complete each challenge was about 10 days (Table S1).

Adaptations

Patient-facing messages were tailored twice during the project based on patient and provider feedback. The original message was formal and listed the study PI as the sender (Figure S2A). The letter detailed the new depression assessment procedures in the clinic and their importance for emotional well-being. After receiving feedback from patients and community members, messages were refined to be less formal, emphasized the importance of routine assessment for emotional well-being, and listed the patient’s PCP as the sender (Figure S2B).

Time and financial investment

During the project period, 105 meetings totaling 96 hours were held, including community outreach, engagement with leadership and other experts, quality, and compliance, IT build, and personnel training (Tables S2 and S3). Most of the IT team efforts were focused on building the connections that would allow for 2-way PHI-free communication between the systems needed for integration, the CAT-MH and the EHR, and designing portal-based workflows. The total cost to complete the project was $150 716. The effort to complete the IT build for the PORTAL-Depression project was 1074 h costing $102 056 ($95 per hour) (Table 3). During the project, 9732 adaptive tests were completed (8976 CAD-MDD; 756 CAT-DI), which cost about $48 660 ($5 per test). The average time to complete the CAD-MDD was about 30 seconds (SD = 60 s) and CAT-DI was about 2.5 minutes (SD = 3.5 min).

Table 3.

PORTAL-Depression Project August 2018-May 2019 health information technology efforts and cost.

	High-level design		Integration		Clinic-based workflows		Portal-based workflows		Wrap-up		Total
IT build time (h) and cost (USDa)	Hour	$	Hour	$	Hour	$	Hour	$	Hour	$	Hour	$
Enterprise business arch director	1	95	–	–	3	238	–	–	–	–	4	332
Epic ambulatory analyst	–	–	–	–			212	20 093	15	1425	227	21 518
Epic ambulatory research analyst	–	–	–	–	56	5273	31	2945	–	–	87	8218
Epic core analyst	–	–	–	–	–	–	26	2494	–	–	26	2494
Epic data courier programmer	–	–	20	2090	–	–			–	–	20	2090
Epic MyChart analyst	–	–	–	–	–	–	46	4370	–	–	46	4370
Epic security analyst	–	–	3	285	–	–	–	–	–	–	3	285
Information security engineer	–	–	12	1140	–	–	–	–	–	–	12	1140
Integration architect	–	–	90	8550	–	–	–	–	–	–	90	8550
Integration engineer	–	–	105	9928	–	–	–	–	–	–	105	9928
Integration programmer	–	–	320	30 400	–	–	–	–	–	–	320	30 400
Network engineer	–	–	8	760	–	–	–	–	2	190	10	950
Open systems engineer	–	–	6	570	–	–	–	–	–	–	6	570
Solution architect	28	2660	–	–	35	3325	43	4085	–	–	106	10 070
Technical architect	–	–	–	–	–	–	10	950	–	–	10	950
Windows engineer	–	–	–	–	–	–	2	190	–	–	2	190
Total, hours (cost)	29	2755	564	53 723	94	8836	370	35 127	17	1615	1074	102 056

^aThe hourly rate was $95.00 (US Dollars).

Maintenance

In-clinic depression screening, monitoring, and surveillance continued after the project ended. Team challenges, weekly chart audits, and report cards were also continued to help maintain screening, surveillance, and monitoring rates. Post implementation, the average depression screening and surveillance and monitoring rates in the clinic were 65% and 31%, respectively. In March 2021, about 72% of providers acknowledged positive results in the clinic. However, the computerized adaptive test was replaced with the PHQ due to a lack of ongoing investment. All the connections for portal-based screening, surveillance, and monitoring have been built for the health system. Integration of routine surveillance and monitoring care, use of high-priority BPAs, and acknowledgment of positive results have also been implemented in all internal medicine, family medicine, general pediatric, and geriatric clinics, as well as several specialty clinics including infectious disease (among patients with HIV) and oncology. Ongoing QI projects are being evaluated in these clinics to further support the adoption of population health care screening and measurement strategy in the health system.

PCP survey

The PCP response rate was 63% (N = 27/43) in 2019 and 66% (N = 27/41) in 2020; 79% of PCPs responded at least one of the years (N = 34/43). At both time points, the majority of respondents were women (2019: 70%, N = 19/27; 2020: 67%, N = 18/27) and had worked at the clinic for >5 years (2019: 70%, N = 19/27; 2020: 78%, N = 21/27). Gender and years in practice reflected the overall characteristics of attending PCPs in the clinic.

In 2019, 4 months after initial implementation, about half of PCPs (54%, N = 14/26) were satisfied with CAT-MH depression screening and agreed that the changes to depression screening workflows went smoothly (56%, N = 15/27). About three-quarters of PCPs knew where to find CAT-MH results in the EHR (78%, N = 21/27) and 65% (N = 17/26) were confident in interpreting results and were incorporating results into their clinical decision-making. About half (48%, N = 13/27) of PCPs anticipated that responding to the portal-based results would be burdensome.

In 2020, about two-thirds of PCPs agreed that having patients complete portal-based assessments improved case identification (65%, N = 17/26) and care (62%, N = 16/26) of patients with depression. Only 27% (N = 7/26) found reviewing portal-based results burdensome. Importantly, there was an increase from 2019 to 2020 in the percentage of PCPs agreeing that patients were adequately treated for depression (30% vs 59%). About 78% (N = 21/27) and 70% (N = 19/27) of PCPs agreed that the clinic should continue portal-based screening and measurement, respectively. A similar level of support was seen when PCPs were asked if portal-based screening and measurement should be implemented at the health system level (Table S4).

Medical assistant survey

The MA response rate was 80% (N = 4/5) in 2019 and 43% (N = 3/7) in 2020. All respondents were women and the majority had worked in the clinic <3 years (2019: 75%, N = 3/4; 2020: 68%, N = 2/3). In 2019, all MAs who responded knew how to order the CAT-MH depression screening in the EHR, were confident in administering the CAT-MH, and agreed that portal-based screening would improve clinic workflows. Three out of 4 were comfortable screening primary care patients for depression and agreed that the changes to the depression screening process went smoothly. In 2020, MAs who responded were satisfied with the depression screening, surveillance, and monitoring process both in-clinic and virtually (Table S5).

Discussion

The PORTAL-Depression project facilitated improvements to clinic-based workflows and established portal-based processes to carry out a population health approach to depression detection and measurement. We develop and apply a health IT innovation framework to show how to successfully leverage IT solutions and engage key stakeholders to improve depression care. Integration of a computerized adaptive test into the EHR and efforts to establish a system to administer a patient-reported depression assessment tool through a portal were complex; however, it demonstrated several benefits in our clinic which serves a predominantly minority metropolitan community.

Implementation of the PORTAL-Depression project has proven successful due to its ability to improve accessibility to depression assessments and results, provide early intervention, and promote continuity of care and awareness. Most importantly, we were able to complete the IT build needed to implement the PORTAL-Depression Project. We were able to integrate third-party software (CAT-MH) with the EHR that facilitated communication without PHI. Furthermore, when using the CAT-MH, we were able to observe a significantly higher positivity rate compared to the PHQ used in our clinic. Additionally, we were able to implement workflows and IT tools to help providers identify and manage depression. This resulted in increased awareness and accessibility of the results. Providers mentioned that the CAT-MH results were easier to find within a patient's medical record (75%) compared to the PHQ (61%). In general, we see higher rates of detection, surveillance, and monitoring in the clinic from the implementation of these technological innovations.

Implementation of population health portal-based depression assessments using cloud-based software was successful but complex and labor-intensive. Collecting patient-reported outcomes is known to improve patient care, but the integration of patient-reported outcomes has also been slow.^32–35 We demonstrated the feasibility of collecting patient-reported outcomes by integrating assessments based on care portals. The PORTAL-Depression project demonstrated the feasibility of population health portal assessment in reaching patients who would not otherwise be connected to care without additional clinical and time burdens. In addition, we were able to demonstrate that population health portal-based depression screenings were able to detect a higher percentage of patients with depression compared to clinic-based screenings. Due to the exceptional level of commitment from providers and staff, we were able to successfully maintain the use and dissemination of assessment tools and workflows of the PORTAL-Depression Project. We were able to further sustain efforts through continuous QI, adaptation, and ongoing training. A key attribute of the successful implementation of the PORTAL-Depression project was that it was carried out in a highly engaged clinic, which may not be reproducible in other settings.

Despite considerable achievements, the complexity of the project hampered the sustainability and scalability of the intervention. In our case, the IT creation took longer than originally planned because of the complexity of the project. As a result, the build took 540 more hours than originally budgeted. However, this amount of time was an investment in institutional knowledge, and subsequent projects at the institution with a similar framework have been built which greater ease.³⁶ Even though the main components of PORTAL-Depression were successfully implemented, there were some features that could not be implemented. The initial intervention sought to automate the workflows of the portal. This included generating individual links for eligible patients by ordering CAT-MH in bulk and mailing the letter to patients without assistance from the study team. This characteristic of the intervention was essential for the sustainability of the intervention. However, we also wanted the bulk messages to be sent staggered in time to distribute work for PCPs. As a result of the desire to stagger the send date of bulk messages, we were unable to automate the portal messages, which prevented the population health approach to depression assessments through the portal to continue after the clinical trial.

It is important to recognize that although the CAT-MH is a more sensitive assessment, the PHQ was preferred as it is available at no cost. Future implementation efforts should consider the trade-offs between assessment sensitivity and the financial implications for their setting. Additionally, future IT features should consider displaying mental health results in multiple places, including laboratory results (eg, blood work) to improve accessibility. We will continue to advocate for a population health approach to depression screening, monitoring, and surveillance at the institution and gather patient experiences and satisfaction with this approach. The next steps include fostering open discussions about automating essential IT functions to support a population health approach and incorporating patient insights to refine the PORTAL-Depression Project. Overall, the advances in depression care at our institution have demonstrated the potential to extend to other facets of mental health. In response to 2023 recommendations for universal screening for anxiety by the United States Preventive Task Force,³⁷ we have begun efforts to incorporate a population health approach to anxiety screening with the potential to yield significant benefits in mental health screening for patients seen at our institution.

Conclusion

The implementation of the PORTAL-Depression project took a considerable amount of time to build, while also being quickly adopted by the clinic. Given financial and personnel constraints, only certain aspects of the intervention were sustained. Throughout the project, it became clear that it was not feasible to maintain the entire intervention, and therefore the characteristics of the intervention that were influential in improving depression care in the clinic were maintained. In addition, the clinic’s commitment played a crucial role in the implementation and sustainability of the PORTAL-Depression project. We recommend a clinical environment of high commitment and engagement before integrating a complex and labor-intensive intervention.

Author contributions

Melissa I. Franco is the guarantor of this work and had access to the data, contributed to the design, analyzed and interpreted the data, and drafted the manuscript. Neda Laiteerapong had access to the data, designed the study, analyzed and interpreted data, critically reviewed/edited the manuscript, and obtained funding for the study. Erin M. Staab and Mengqi Zhu had access to the data, contributed to the design, analyzed and interpreted data, and critically reviewed/edited the manuscript. William Deehan, John Moses, Robert Gibbons, Lisa Vinci, Sachin Shah, Daniel Yohanna, and Nancy Beckman contributed to the design and data interpretation and reviewed/edited the manuscript.

Supplementary material

Supplementary material is available at JAMIA Open online.

Funding

This work was supported by the Agency for Healthcare Research and Quality (U18 HS26151-01).

Conflicts of interest

Robert Gibbons, PhD developed the CAT-MH which was used as the depression screener and measurement tool in the project. Dr Gibbons received no funding from the grant. The grant paid for the depression screener tool directly to the company. Dr Gibbons receives no funds from the company.

Data availability

The data generated and/or analyzed during the current study are available from the corresponding author upon reasonable request.