Implementing Outcome-based Care in Pediatric Psychiatry: Early Results and Overcoming Barriers

Abstract

Background:

Poor utilization of standardized outcome measures for monitoring patient care and progress in mental health services is evident. The objective of this work was to implement computerized diagnostic and history assessments for outpatient mental health visits in the ambulatory psychiatric clinic of a large pediatric health system.

Methods:

A computerized assessment system was created for the iPad to administer and score a series of validated diagnostics before new patient and follow-up appointments with a psychiatry provider. Outcome measures were a percentage of completed assessments, provider satisfaction, and patient satisfaction.

Results:

Across all outpatient psychiatry clinics, screener completion rate for new patients increased from 0% to 90% within 1 year of implementation and sustained within 3σ process control limits for 2 years. Return visit assessment completion increased from 0% to 80%. The most substantial completion rate increase was related to scheduling assessment time as part of the visit. Assessment of provider and patient satisfaction through surveys before and after the implementation showed 94% of providers felt that visit efficiency had improved, and overall opinion of the system was highly positive. Patients also reported high satisfaction with the assessment process (4.1 on Likert scale 0–5, 5 = very positive).

Conclusions:

This quality improvement program demonstrates strategies for overcoming barriers to the use of standardized assessments in outpatient psychiatry. We show that a large pediatric mental health system can achieve systematic outcome data collection with minimal disruption to routine clinical care.

INTRODUCTION

Since the release of the 2001 Institute of Medicine report Crossing the Quality Chasm,¹ the practice of medicine has made strides to improve the quality of care through systematic measurement and evaluation of clinical outcomes.^2–5 Psychiatry has lagged behind this trend, citing the inherent challenges of identifying objective, quantifiable symptom metrics in the behavioral health field. In recent years, research literature has demonstrated that the measurement-based care (MBC; the use of standard, validated symptom or outcome measures to guide treatment decisions systematically) can be effective at improving outcomes in psychiatry.^6,7 There have been increasing calls to begin adopting such programs within the discipline.^3–5,8 Despite this acknowledgment, the actual implementation of such programs has remained a challenge,^2,3 particularly in child and adolescent psychiatry where both patients and parents become relevant parties to the assessment process. This study describes the early implementation process of an infrastructure to support MBC in the outpatient psychiatry clinic of a major pediatric hospital. We begin by describing the existing literature regarding barriers and challenges to the implementation of such systems and then discuss an implementation process intended to mitigate these challenges. We present the results of this implementation process and the subsequent feedback from providers and patients.

Review of Literature

Restoring patients to their optimal levels of function is the ultimate goal for health care providers in all specialties. Consistent measurement of validated outcomes can help to determine when patients will achieve this optimal level. MBC can be used to help define an ultimate goal of treatment and track patient progress while working toward this goal.^2,3,5,9 Providers can determine more accurately if treatment plans are beneficial or if adjustments should be made using these data.^2,3,10,11

Despite the potential benefits, psychiatrists do not regularly use these techniques due to both perceived and actual barriers such as a preference for their clinical assessment, uncertainty as to the added value of standardized measures, and barriers to the implementation process. Although the most significant barrier is the time it takes to administer assessments, other substantial concerns include financial cost, unfamiliarity with the scales themselves, and the uncertainty about the clinical utility of assessments.^{2,3,5,8,10,12,13} Other pertinent concerns have been described.^2,3,9,12

Although the barriers to implement MBC in psychiatry are undeniable, the potential long-term benefits are also substantial.^5,6 Technology may provide the opportunity to lessen or remove the most significant barriers to MBC in psychiatry and shift the balance toward more widespread adoption.^3,5,8,12 In this study, we aimed to address implementation barriers to MBC and increase completion rates from 0% to 90% by December 2017 in patients presenting for initial and follow-up visits at our outpatient psychiatry clinics.

STUDY DESIGN AND METHODS

Context

We performed this quality improvement (QI) work in the outpatient psychiatry clinics at Nationwide Children’s Hospital in Columbus, Ohio. This clinic is typically staffed by 14 providers and averages 1,095 monthly visits, with 172 monthly new patient visits. The patient population is predominantly urban, Medicaid families. The psychiatry department operates on a typical electronic medical record (EMR) system (Epic; Epic Systems, Verona, Wis.) utilized by the entire hospital system. At the beginning of this QI effort, there was no use of kiosks for data collection or registration during patient visits nor any paper-based systematic data collection. Patients were scheduled, and patient flow was managed through an EMR-based solution (Epic Cadence).

Key Drivers and Interventions

The primary intervention for this QI effort was the creation of an infrastructure for systematic collection of diagnostic rating scales during the initial visit with a psychiatric provider with the goal of achieving completion rates >90%. An important secondary goal was to develop a culture of MBC; overcoming provider resistance to the administration and review of standard outcome measures; and enhancing provider recognition of the value of MBC. The QI effort followed the Institute of Healthcare Improvement Model for improvement.

We began by identifying potential barriers to the use of MBC in psychiatry from available literature and a survey of current providers. Survey responses were highly consistent with the literature, with 67% of providers reporting concerns of time burden and schedule disruption. On the basis of this assessment of likely barriers, we constructed a key driver diagram (Fig. 1). Specific interventions were identified to target key drivers, particularly as related to clinic flow, provider efficiency, and time constraints. We developed a computerized assessment system that administered a standard set of validated, public domain, self-reported mental health assessments to both patients and caregivers on the initial visit, and targeted provider-selected assessments on subsequent visits. This system was built on a web-based infrastructure using JavaScript and standard web technologies. Patients completed this assessment on an iPad (Apple, Inc., Cupertino, Calif.) provided by the clinic while they awaited their scheduled visit time. The system automatically administered and scored each assessment, collated the data between patients and caregivers, and produced a concise report with scoring and interpretation information made available to the provider before the actual clinical encounter. The workflow was organized to maintain the confidentiality of protected health information throughout the process. As there was no available technology solution to push the resulting data directly into the EMR, we established a data transfer system that allowed staff to easily copy and paste the final report into the patient’s formal medical record.

Fig. 1.

Specific aim and key driver diagram developed by the quality improvement team.

Implementation

We implemented the assessment program through a series of Plan-Do-Study-Act (PDSA) cycles. First, we systematically expanded the number of providers included in the assessment process, and then we altered the patient appointment structure to ensure sufficient assessment completion time. The study team consisted of 2 physician leads throughout implementation, 2 physician champions/early adopters, the clinic operations manager and a nursing manager, and members of the front desk and medical assistant staff. This team captured representation from all aspects of the workflow and could quickly identify and respond to implementation problems in the early stages. As the implementation effort grew, this core team served a support function, passing on their experience with the implementation process to other clinics, providers, and staff.

Methods of Evaluation

We monitored 3 primary features of the intervention throughout implementation.

1. Completion—primary outcome measure: Percent of completed previsit computerized assessments completed before provider visits over the total number of visits.

2. Provider satisfaction: Secondary outcome measure as determined by a provider satisfaction survey completed several months after implementation. Data from this survey also served as a proxy measure for provider culture around the use of MBC given the barriers identified in the literature and initial skepticism demonstrated by clinic providers.

3. Patient satisfaction—balancing measure: We administered a patient satisfaction survey on completion of the assessment to gauge the effect of the extra time taken by this process on patient and family satisfaction.

Analysis

We tracked the primary outcome measure on a statistical process control p-chart. Special cause variations were evaluated against ongoing modifications to the intake and assessment process.

Due to the limited sample size, provider response data were evaluated qualitatively. Providers were asked to rate how helpful the assessment had been on a Likert scale, 1 being “not helpful at all” to 5 being “very helpful,” and were also asked a series of further questions to qualify the nature of their impressions.

Patient satisfaction scores were monitored in a qualitative format also as there was no available comparator concerning the level of satisfaction itself. Statistical process control and monitoring techniques verified the stability of scores across time. Because the length of the assessment process changed throughout the implementation period, satisfaction scores versus total assessment completion time were evaluated by Pearson correlation coefficient to determine if the length of assessment affected satisfaction.

Ethical Considerations

The study was deemed QI and not human subject research. Therefore, review and approval by from the institutional review board was not required.

RESULTS

Primary Outcome Measure: Assessment Completion Rates

Figure 2 depicts the rate of successful completion of computerized diagnostic assessments, annotated with key PDSA’s and special cause variation. Notably, despite several strategies to better engage patients, the most significant impact on screener completion rates was achieved by scheduling time before visits for assessment completion (Fig. 2, point C). We met the initial goal of 70% completion of new patient assessments in April 2015, and we revised the goal to 90% completion. A drop in the completion rate in May 2015 coincides with a move of clinic location and a series of technical problems limiting effective assessment completion. These technical issues were resolved by June 2015, resulting in a stable process for the rest of 2015 and the beginning of 2016.

Fig. 2.

Control chart (p-chart) used by the project team to track the percentage of completed screens over time. Control limits denote 3 SDs around the mean value. This chart is also used to track PDSAs and their impact on completion rates. At point A, 1 provider started PDSA using metric screener at 1 clinic. At point B, PDSA expanded to 3 providers at the same clinic. At point C, the scheduling department started PDSA with adding “assessment” appointment times for new patients at the same clinic. At point D, the clinic moved to a new location resulted in fewer completions of the screener. At point E, the metric screener is relaunched with new infrastructure and expanded to 6 clinics for all new and follow-up patients. At point F, 1 of the 6 clinics moved to a new location. The dotted line in 2016 indicates the missing data due to the metric screener system were taken offline to rebuild infrastructure.

Based on the success of implementation in the pilot clinic, the tool was expanded to 5 other outpatient psychiatry clinics. To facilitate this expansion, we updated the data infrastructure and secured additional iPads in 2016. The data infrastructure upgrade led to inconsistencies in data collection and tracking for several months in 2016 (Fig. 2, point E). By December 2016, the updated system was fully online, and we were able to resume assessments and data collection. We quickly expanded the system from one clinic to the next and from new patients to follow-up patients in parallel. Each clinic expansion adopted the processes and methodology proven out in the pilot clinic phase, and in most instances, staff from the pilot clinic provided initial training and support during the launch at a new clinic. By early 2017, the assessment system supported >1,000 patient visits a month, and we were able to sustain 85% completion rate at all 6 clinics through 2017 and first quarter of 2018.

Secondary Measure: Provider Satisfaction

We administered a survey to assess provider buy-in and response several months after achieving the initial goal rate of 90% new visit assessment completion. Of 24 providers in the department, 18 responded to the survey, of whom 13 were involved in the pilot program. Contrary to expectations from the available literature, provider response was highly positive. Average provider response on the 5-point Likert scale was 4.36. Sixty-seven percent of the providers believed that the data changed how they approached their diagnostic visits. Most providers (94%) indicated that direct interaction with the patient was improved or not affected, and 100% of providers express the same regarding the EMR documentation process. In total, 78% of providers indicated that the assessment improved the efficiency of their visits. On average, they reported that 16 minutes of time has been saved of a 90-minute assessment. Providers overwhelmingly reported that this extra time was utilized for improved patient care including more detailed history collection (100% of providers), improved treatment planning (83%), increased patient education (67%), and increased indirect care (61%).

Balancing Measure: Patient/Parent Satisfaction

We measured patient and parent satisfaction with a brief postcompletion Likert survey of 6 questions. Figure 3 shows questions asked in the satisfaction survey and average scores of 1,471 survey responses by parents visiting between June 2014 and January 2016.

Fig. 3.

Average responses in patient satisfaction survey (0 = strongly disagree, 3 = neutral, and 5 = strongly agree; n = 1,471).

To ensure that the average satisfaction scores did not vary over the course of implementation, we also tracked variation in patient satisfaction scores using Xbar and S control charts. There was no significant shift in monthly averages or SDs (Fig. 4). Satisfaction question 3, best represents the overall satisfaction (value identification) with the process, and therefore, we used these data to analyze using Xbar-S control charts.

Fig. 4.

Monthly averages of scores for satisfaction question 3: “I feel the information collected will be useful for my visit.” (n = 1,471).

In addition, we measured “the time taken to complete assessment” and correlated against satisfaction scores to assess the impact of assessment length. A Pearson correlation coefficient comparing satisfaction with actual completion time produced a value of −0.058 (P = 0.285), indicating no correlation between the time taken and patient satisfaction across a completion duration of 20–30 minutes.

DISCUSSION

Summary

A reliable process to convert anecdotal patient reports into measurable, clinically useful data is necessary for implementing MBC and improving patient outcomes in psychiatry.^2,3 This article discusses the implementation of a system enacting this process in an outpatient psychiatry clinic at a large children’s hospital. Through a combination of literature review and direct survey of providers, we identified some significant barriers to deploy such program and structured our implementation strategy accordingly. Through several PDSA cycles, we further refined our process to maximize patient participation and to assess the ultimate impact on both patients and providers. We observe that it is not only possible to address provider concerns, but note that providers and patients are broadly supportive of the benefits of a structured assessment process.

Interpretation

We consider our results in light of the separate domains of patient and provider engagement and systemic impact. Concerning patient engagement, there are 2 critical observations seen in our data. First, it was only possible to achieve the goal completion rate when we scheduled additional time slots for patients to complete the screener. Several efforts to engage patients without formally incorporating this time into the structure of the visit met with only limited success. Second, although we expected that the increased visit time may result in patient dissatisfaction, patients and parents were receptive to the additional time and had positive feedback about the potential benefits of the computerized screener (Fig. 3). Notably, although the length of the new patient assessment increased to up to 30 minutes throughout the implementation, patient satisfaction did not change significantly. This finding demonstrated a high degree of tolerance on the part of patients and families to participate in such an assessment process. Finally, we were unable to achieve completed assessments on 15% of visits (Fig. 2). On review, these missed opportunities were predominantly from urgently scheduled assessments in which families were asked to arrive early but were not explicitly scheduled to do so, further supporting the importance of building assessment time into visit structure.

Concerning provider engagement, our results are both clear and in striking contrast to the concerns raised in the literature.² Our providers expressed many of the same concerns as noted in the literature during pre-implementation, with the greatest perceived barrier being increased time burden (67%). However, postimplementation surveys demonstrated that providers were not only overwhelmingly in support of the screener, but generally found that it reduced their workload rather than increasing it. Several providers also noted anecdotally that some diagnoses might have been missed without the assessment process, contrary to initial concerns that a standardized tool would diminish the value of their clinical assessment. We believe that provider engagement as demonstrated in our results, and the shift in provider perception of the process, speaks to a change in provider culture and demonstrates the potential to improve the quality of patient care and to simultaneously improve the time efficiency and work satisfaction of those providing that care. Such a win–win opportunity is generally not considered in the literature.

At a systems level, consistent collection of standardized assessments has dramatically impacted our ability to organize and shape care. Reviewing the actual clinical data revealed, for example, between January 2017 and April 2018, 533 of 1,210 (45%) patients in our general outpatient clinic presented with self-reported symptoms consistent with depression. Of that group, 408 patients (76.5%) improved throughout treatment, 59 patients (11%) remain unchanged, and 66 patients (13%) worsened. We are developing future QI projects to identify the population of nonresponders and target them for alternate interventions. These future projects would not be possible without the data from this tool.

Limitations

We believe that our results are largely generalizable to most other psychiatric practice settings, but note several specific limitations to this work and its generalizability. Most significantly, we received enormous support from division and hospital leadership in implementing this process. Although we recognize that many other institutions and settings may not have similar support, our results offer a pathway for gaining support both by using technology to reduce barriers with MBC and by offering data to challenge the existing wisdom that provider buy-in cannot be achieved.

Concerning provider buy-in, we recognize that despite the very positive feedback, our sample size was quite small and restricted to providers working in the outpatient clinic of a large institution. In this context, the clinic staff provided a buffer against the technical and process components of the necessary workflow changes. This buffer may not generalize to smaller practice settings in which providers may need to be more directly involved in setting up the infrastructure for MBC.

Second, to collect information on patient satisfaction consistently, this survey was administered at the end of the computerized screener itself rather than at the end of the full visit. We also did not have the baseline data on patient satisfaction before implementation of the screener. Thus, we are unable to comment on the actual impact of computerized screenings on patient satisfaction as a whole, and only that the patients were highly satisfied with the screening process itself (Fig. 3). Baseline data on the individual provider’s use of MBC by administering paper assessments during their visits were also not available. Anecdotally, there was no indication of the widespread use of systematic measures before implementation of this process. However, our reported baseline assumes zero assessments before this effort, which may not be strictly accurate.

Finally, we must recognize that the outcome of this work is truly the completion of a process facilitating the broader outcome of MBC and improving mental health outcomes. Although we can measure completion of the process, and anecdotal data from providers speak to the active use of the information collected, we do not ultimately measure the cultural or cognitive changes that would result in the effective use of MBC as opposed to just effective data collection. Our QI project and results offer only a step toward the ultimate goal of using MBC to improve clinical outcomes in mental health patients by decreasing structural barriers along the way.

CONCLUSIONS

In this article, we discuss early results from the implementation of MBC in pediatric psychiatry. Our results demonstrate that a high rate of compliance with screener use could be achieved with structured visits and systems to achieve provider and patient buy-in. We also achieve these compliance rates while maintaining high provider and patient satisfaction and engagement. These early results demonstrate that MBC can be a viable tool in psychiatry allowing for more clearly defined goals, targeted interventions, and better patient outcomes.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.