Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2023 May 1.
Published in final edited form as: Psychol Serv. 2021 Apr 1;19(2):343–352. doi: 10.1037/ser0000456

Implementation Facilitation Strategies to Promote Routine Progress Monitoring among Community Therapists

Joyce H L Lui 1, Lauren Brookman-Frazee 2,3, Ashley Smith 1, Teresa Lind 2,3, Laura Terrones 2,3, Adriana Rodriguez 1, Mojdeh Motamedi 2,3, Miguel Villodas 3,4, Anna S Lau 1
PMCID: PMC8484366  NIHMSID: NIHMS1655360  PMID: 33793283

Abstract

Despite substantial support for the importance of routine progress monitoring (RPM) as part of evidence-based practice, few providers utilize measurement-based care. This study sought to identify the relative importance of facilitation strategies viewed as most helpful for increasing intention to use RPM among 388 ethnically diverse community therapists serving children and families. Four types of facilitation strategies were examined: Language/Interpretability, Automation, Staffing/Access, and Requirements. Mixed analyses of variance found that therapists’ reported intentions to use RPM were more influenced by strategies of automating assessment administration, provision of clerical assistance, and agency requirements, than by making linguistically appropriate measures available. However, the importance of strategies differed depending on therapist race/ethnicity and current RPM use. Language/interpretability of RPM assessments was less emphasized for non-Hispanic White therapists and therapists who have not yet or only minimally adopted RPM compared to ethnic minority therapists and therapists who regularly use RPM, respectively. Further, therapists who were not current RPM users emphasized Automation more than Staffing/Access. Results may inform prioritization of implementation facilitation strategies for agencies to encourage RPM.

Keywords: Routine progress monitoring, implementation, facilitation strategies, community therapists, children’s mental health

Routine progress monitoring (RPM) involves therapists systematically collecting data from clients using standardized measures to monitor change on a session-by-session basis and using that data to inform clinical care (Lambert & Shimokawa, 2011; Shimokawa et al., 2010). There is empirical evidence supporting the importance of RPM as part of evidence-based practice in children’s mental health. Specifically, in addition to providing valuable data on client progress, RPM improves therapists’ clinical decision-making (Lambert & Shimokawa, 2011), decreases risk of client negative treatment response or failure (Bickman et al., 2011; Brattland et al., 2018; Lambert et al., 2018), and accelerates treatment progress (De Jong et al., 2014). Although the majority of past literature focused on therapists working with adults, there is growing data supporting the utility of RPM in children’s mental health. For example, in a randomized trial of RPM among community therapists serving youth, researchers found that youth demonstrated faster improvements when therapists used RPM compared to youth served by therapists who did not use RPM, and there was suggestion of a dose response (Bickman et al., 2011). Further, data from RPM can be used for benchmarking and quality improvement initiatives across organizations (Boswell, 2019). Although RPM is linked to improvements in clinical practice and outcomes, it is not commonly used in usual care settings, such as community mental health. For example, in a national survey, only 13.9% of clinicians reported using RPM at least on a monthly basis (Jensen-Doss et al., 2018). Understanding factors contributing to this implementation gap can provide insight to facilitate uptake of RPM in community practice. Indeed, researchers call for the timeliness and importance of research on implementation factors that impact the use of RPM (Rye et al., 2019).

Although there is variability between studies on therapist attitudes and perceptions toward the use of RPM in usual care settings (Norman et al., 2014; Jensen-Doss et al., 2018), practical and logistical challenges are consistently cited as barriers to implementation. For example, therapists providing child and adolescent mental health services frequently report barriers such as time burden associated with administering, scoring, and interpreting measures, lack of training on use of progress monitoring tools/systems, and lack of ongoing administrative support and information technology (Hall et al., 2014; Sharples et al., 2017). When the clients’ clinical needs are complex (Baker-Ericzén et al., 2010) and the local resources are limited, such as in community mental health settings, these challenges to adoption are likely amplified. Furthermore, dissatisfaction with the assessment measures themselves (e.g., items that are difficult to understand, scores that are difficult to interpret) reduce perceived utility of RPM among child clinicians (Batty et al. 2013; Norman et al., 2014; Sharples et al., 2017).

Therapist- and organizational-level factors have also been linked to therapist use of RPM. Previous research has found that therapist discipline and orientation impact receptiveness to RPM (Ionita & Fitzpatrick, 2014; Jensen-Doss & Hawley, 2010; Jensen-Doss et al., 2018); with psychologists and therapists with a cognitive-behavioral orientation reporting greater use of RPM. There is evidence that some organizational-level implementation strategies, such as institutional mandates, may negatively impact implementation and use of RPM (Sharples et al., 2017). For example, therapists described perceiving increased ruptures in therapeutic alliance with their clients as a function of mandated use of RPM, particularly when clients expressed resistance or discomfort with the measures (Sharples et al., 2017).

As with any innovation, active and ongoing strategies are likely required to facilitate RPM use in community care (Herschell et al., 2010). As such, it is helpful to examine therapist perception of the potential impact of these facilitation strategies. Facilitation strategies are processes and activities for implementing an innovation by leveraging strengths and problem solving barriers and include multiple evidence-based implementation strategies (Midboe et al., 2018; Ritchie et al., 2017). Though existing studies offer some insight into challenges associated with RPM, less is known about which strategies are needed to effectively overcome these challenges. Previous qualitative studies examining the adoption of RPM among therapists serving youth and adults have identified some of the following facilitation strategies as helpful for adoption: briefer administration of measures, simplifying language and interpretations of measures, increasing therapist knowledge on the use and interpretation of measures, allowing flexibility in the administration and interpretation of measures, ensuring fit of the measures with clients, and adapting perspectives to overcome therapist’s own attitudinal barriers (Borntrager & Lyon, 2015; Ionita et al., 2016; Lucock et al., 2015; Waldron et al., 2018). Though informative, these studies have been limited in sample size due to the qualitative nature of the designs. Researchers have recently highlighted that there is limited quantitative data on clinician’s views on the process of implementing RPM (Kaiser et al., 2018). More quantitative research is needed to understand which types of facilitation strategies would be most helpful in making the adoption of RPM more feasible and acceptable to therapists (Ionita et al., 2016).

The current study builds on previous qualitative exploratory work that have identified potential barriers to RPM adoption by quantitatively identifying types of facilitation strategies to address barriers that may be most related to therapists’ intentions to use RPM in community contexts serving youth. Existing quantitative studies on RPM have focused on therapist attitudes on using RPM and/or therapist characteristics that predict use of RPM more generally. No quantitative study has examined therapists’ views on the types of active facilitation strategies that can help increase their use of RPM and compared which strategies therapist consider most helpful. For example, although organizational support is positively associated with greater use of RPM (Rye et al., 2019), the specific strategies provided by the organization have not been quantitatively identified and compared. The current study also examined whether therapist characteristics (related to professional background, racial/ethnic diversity, workload, and current use of RPM) moderated the types of facilitation strategies that had greatest impact on intentions to use RPM. Therapist discipline, theoretical orientation, and licensure were selected as moderators based on previous findings on differential attitude towards RPM by therapist professional characteristics (Hatfield & Ogles, 2004; Jensen-Doss et al., 2018). Therapist race/ethnicity and language of service delivery were also examined as potential moderators due to previous literature on concerns about cultural and language factors in the interpretability of outcome measures when used in RPM with diverse clients (e.g., Sharples et al., 2017). Number of weekly direct service hours was included in the models as a proxy for therapist workload burden. Overall, this study sought to identify the relative importance of facilitation strategies to inform which may be most crucial to encourage use of RPM among therapists with different characteristics.

Method

Study Context

The current study was part of a larger observational study, Knowledge Exchange on Evidence-based Practice Sustainment (4KEEPS), examining the sustainment of multiple evidence-based interventions (EBIs) for children and youth within a system-driven implementation in the Los Angeles County Department of Mental Health (LACDMH) service system. In 2010, under the Prevention and Early Intervention (PEI) Transformation, community agencies were offered reimbursement for delivering select EBIs (e.g., Trauma-Focused Cognitive Behavioral Therapy, Managing and Adapting Practice, Positive Parenting Program) for youth. Therapists were required to administer outcome monitoring pre- and post-treatment (and every six months if treatment exceeds 6 months) for each EBI claimed to PEI (LACDMH, 2016). RPM consisting of session by session assessment was encouraged but not required as part of PEI implementation. See Lau and Brookman-Frazee (2016) for a description of the PEI initiative and the larger 4KEEPS study.

Recruitment and Procedures

Survey data were obtained through a supplemental survey completed by a subset of 4KEEPS therapist participants. The larger survey, administered in 2015 and described elsewhere (e.g., Barnett et al., 2017; Lau et al., 2017), included 777 therapists from 76 LACDMH agencies who were trained to deliver at least one of six EBIs. The survey had a response rate of 41.5% and participants received $20 for their participation. Therapists were offered the opportunity to complete a supplemental survey after completing the first 4KEEPS therapist survey. Emails were successfully sent to 743 of the original survey respondents. A total of 388 therapists completed this survey, for a response rate of 52.2%. Participation in this supplemental survey took place between January 2016 and March 2016, and therapists received $10 for completion.

Participants

Participants included 388 therapists from 59 mental health agencies in Los Angeles County. Therapists were predominantly female (n = 342, 88.1%) and with a mean age of 37.2 years (SD = 9.6). Therapists most commonly self-identified as Hispanic (n = 170, 43.8%) and reported having a Master’s degree (n = 341, 87.9%), while 44.8% (n = 174) reported being licensed. Therapists primarily worked in outpatient clinic mental health settings (n = 310, 80.1%). The majority of therapists identified their primary discipline as Marriage and Family Therapy (n = 221, 57%), followed by Social Work (n = 122, 31.4%) and Psychology (n = 40, 10.3%). See Table 1 for therapist characteristics.

Table 1.

Descriptives of Study Variables

Entire Sample M (SD) or % (n) Non RPM User M (SD) or % (n) Minimal RPM User M (SD) or % (n) Regular RPM User M (SD) or % (n)
Therapist Demographic Characteristics n = 388 n = 214 n = 97 n = 76
 Age (years) 37.15 (9.60) 37.12 (9.81) 36.38 (8.94) 37.99 (9.97)
 Gender
  Female 88.1% (342) 86% (184) 90.7% (88) 90.8% (69)
  Male 11.9% (46) 14% (30) 9.3% (9) 9.2% (7)
 Race/Ethnicity
  Hispanic 43.8% (170) 40.7% (87) 43.3% (42) 53.9% (41)
  Non-Hispanic White (NHW) 35.8% (139) 39.7% (85) 40.2% (39) 18.4% (14)
  Another Minority Not Listed 20.4% (79) 19.6% (42) 16.5% (16) 27.6% (21)
Professional/Practice Characteristics
 Education
  < Master’s Degree 1.3% (5) 1.9%% (4) 0% (0) 1.3% (1)
  Master’s Degree 87.9% (341) 86.4% (185) 91.8% (89) 88.2% (67)
  Doctoral Degree 10.8% (42) 11.7% (25) 8.2% (8) 10.5% (8)
 Discipline
  Psychologist 10.3% (40) 11.7% (25) 7.2% (7) 9.2% (7)
  Marriage and Family Therapy 57% (221) 56.5% (121) 57.7% (56) 57.9% (44)
  Social Work 31.4% (122) 30.4% (65) 34% (33) 31.6% (24)
 Theoretical Orientation
  Behavioral 3.9% (15) 3.7% (8) 5.2% (5) 2.6% (2)
  Cognitive Behavioral 54.9% (213) 49.5% (106) 53.6% (52) 71.1% (54)
  Humanistic 6.4% (25) 7% (15) 3.1% (3) 9.2% (7)
  Family Systems 13.1% (51) 12.6% (27) 16.5% (16) 10.5% (8)
  Psychodynamic 9.3% (36) 11.2% (24) 9.3% (9) 3.9% (3)
  Eclectic 10.6% (36) 13.6% (29) 10.3% (10) 2.6% (2)
  Other 1.8% (7) 2.3% (5) 2.1% (2) 0% (0)
 Setting
  Outpatient 80.1% (310) 81.8 (175) 76.3% (74) 78.9% (60)
  School 9.8% (38) 8.9% (19) 13.4% (13) 7.9% (6)
  Home 3.4% (13) 2.8% (6) 3.1% (3) 5.3% (4)
  Other Setting 6.7% (27) 6.1% (13) 7.2% (7) 7.9% (6)
 Caseload Composition
  Majority Ethic Minority Clients 97.4% (378) 96.3% (206) 95.9% (93) 97.4% (74)
 Direct Service Hours/Week (range 0–50) 17.30 (8.87) 17.11 (8.87) 16.40 (8.53) 19.18 (9.03)
 Licensed 44.8% (174) 47.2% (101) 42.3% (41) 40.8% (31)
 Delivered Services in Another Language with Child Clients 33.5% (130) 29.4% (63) 33%% (32) 46.1% (35)
 Delivered Services in Another Language with Client’s Caregivers 40.2% (156) 37.9% (81) 44.3% (43) 42.1% (32)
SFRPM - Language/Interpretability (Factor 1)b 3.42 (.98) 3.28 (1.05) 3.41 (.86) 3.80 (.85)
SFRPM - Automation (Factor 2)b 3.80 (1.04) 3.90 (1.03) 3.69 (.98) 3.66 (1.14)
SFRPM - Staffing/Access (Factor 3)b 3.78 (.97) 3.75 (1.03) 3.76 (.83) 3.88 (.95)
SFRPM - Requirements (Factor 4)b 3.71 (.87) 3.61 (.90) 3.76 (.86) 3.92 (.78)
Open in a new tab
a.

Items from the During Treatment phase from the Current Assessment Practice Evaluation measure (Lyon & Dorsey, 2010).

b.

Supports to Facilitate Routine Progress Monitoring, developed by 4KEEPS to assess types of implementation strategies therapists report as helpful for routine progress monitoring.

Measures

Therapist Demographic, Professional, and Practice Characteristics

Therapists reported on their demographic, professional, and practice characteristics (Brookman-Frazee et al., 2012). The background questionnaire, which was part of the larger 4KEEPS survey, included items asking about age, gender, race/ethnicity, level of education, theoretical orientation, primary work setting, direct service hours per week, number of PEI practices they are using, and language(s) of service delivery. Two items assessed how often therapists delivered clinical services in another language with their clients and client’s caregivers. The two items were dichotomized into 0 (never) and 1 (occasionally or more frequently). The majority of therapists reported providing clinical services in another language at least occasionally with their child clients (51.3%) and clients’ caregivers (67%).

Strategies to Facilitate Routine Progress Monitoring (SFRPM)

A 14-item therapist-report measure was developed for the current study to examine the types of facilitation strategies to support RPM. Item development incorporated therapist feedback from 4KEEPS qualitative research and extant literature (Garland et al., 2003; Ionita et al., 2016). Four factors were identified from exploratory and confirmatory factor analyses: 1) Language/Interpretability, 2) Automation, 3) Staffing/Access, and 4) Requirements. One item was removed due to poor loading across two factors (“I could bill for time spent scoring, interpreting or tracking progress”). Multiple group invariance analysis demonstrated configural and metric invariance across therapists who reported current use of RPM and those who did not. See Table 2 for item content and factor loadings.

Table 2.

Strategies to Facilitate Routine Progress Monitoring (SFRPM) Mean Item Scores and Factor Loadings

Factor Loadings
Factor or Item M (SD) % Agree or Strongly Agree EFA CFA
Factor 1: Language/Interpretability
 The measure is available in multiple languages. 3.13 (1.17) 42.2 .60 .55
 The measure is easy to understand (uses simple language) 3.70 (1.06) 67.7 .99 1.00
Factor 2: Quick and Automated
 The measure can be completed in less than 5 minutes 3.92(1.16) 73.2 .61 .77
 No or minimal scoring is required 3.89 (1.07) 70.6 .55 .69
 The measure is computer-administered in the waiting room 3.49 (1.33) 56.4 .83 .74
 Client responses are automatically scored and sent to me 3.86 (1.23) 72.9 .91 .97
 Client progress over time is tracked for me by computer 3.86 (1.23) 71.6 .98 .92
Factor 3: Staffing/Access
 Someone else scores the standardized assessment measures for me 3.53 (1.27) 55.4 .59 .76
 The measures are easily accessible 3.98 (.98) 76.5 .64 .75
 I receive periodic summary reports of client progress for my clients 3.83 (1.11) 69.4 .70 .84
Factor 4: Requirements
 The [evidence-based] practice requires routine progress monitoring 3.61 (1.05) 57.5 .57 .38
 It is requested by my supervisor 3.67 (1.02) 61.4 .95 .92
 It is mandated by my agency 3.84 (1.01) 69.1 .88 .83
Open in a new tab

Note: Therapists rated on a 5-point scale from 1 (Strongly Disagree) to 5 (Strongly Agree) the degree to which they would use standardized assessment measures to routinely monitor progress every 1–2 sessions if these strategies were implemented.

Language/Interpretability (two items; α = .71) assessed the likelihood of using RPM if the measures are easy to understand and are available in multiple languages. Automation (five items; α = .92) assessed the likelihood of using RPM if the process was automatic or quick. Staffing/Access (three items; α = .82) assessed the likelihood of using RPM if additional resources such as administrative/support staff were available. Requirements (three items; α = .81) assessed the likelihood of using RPM if it is required by a supervisor or agency. Therapists rated the extent to which they agreed that a facilitation strategy would make them more likely to use standardized assessment measures to monitor treatment progress every 1–2 sessions. Items were rated on a 5-point Likert scale from 1 (Strongly Disagree) to 5 (Strongly Agree). Mean scores for each factor was used for analyses.

Current Use of RPM

The Current Assessment Practice Evaluation measure (CAPE; Lyon & Dorsey, 2010) asked therapists to report the proportion of their client caseload with whom they use standardized assessment measures. Therapists rated four items on a 6-point Likert scale from 0 (With No Clients) to 5 (With All Clients) for each of three phases of treatment (beginning, during, end). Four items pertaining to the during treatment phase were used to index current use of RPM. Therapist behaviors reflected by these four items need to occur during treatment to be considered RPM per accepted definitions of RPM (Lambert & Shimokawa, 2011; Shimokawa et al., 2010). A previous study demonstrated acceptable inter-item reliability and sensitivity to capturing change in RPM implementation as a result of training and consultation (Lyon et al., 2015). Cronbach’s alpha for the four items was .90 in the current sample. Therapists were categorized into three groups based on their mean scores on the four items pertaining to the during treatment phase. Therapists with mean scores between 0 and 1 were categorized as “non RPM users,” therapists with mean scores between 1 to 2 were categorized as “minimal RPM user,” and therapists with mean scores of 2 or above were categorized as “regular RPM user.” This resulted in 214 (55.2%) non RPM users, 97 (25%) minimal RPM users, and 76 (19.6%) regular RPM users.

Analytic Plan

Patterns of missing data were examined using SPSS v. 24. Results from the missing variable analyses indicated very low rates of missing data (1.71% of all variables). Data were MCAR, χ2 (2358) = 1782.63, p = 1.00. Multiple Imputation (MI) in SPSS v. 24 was used to create 10 imputed datasets (Graham et al., 2007) to generate parameter estimates that were pooled into a single set of results for analyses.

Descriptive statistics examined which strategies therapists indicated as beneficial for supporting RPM. An initial repeated-measures analysis of variance (ANOVA) was conducted to examine whether the four SFRPM factors were rated differentially within therapists. Then, a Mixed ANOVA was used to examine predictors of these differential ratings. The four SFRPM factors were entered as within-subject variables. Seven between-subject factors were included: therapist race/ethnicity (0 = non-Hispanic White; NHW), discipline (0 = non-psychologist), theoretical orientation, licensure (0 = unlicensed), delivered services in another language with children (0 = no), delivered services in another language with caregivers (0 = no), and current RPM use (0 = none, 1 = minimal use; 2 = regular use). Number of direct service hours per week was entered a covariate in the model. Only main effects of the between-subject variables were specified.

Results

Differences Between SFPRM Factors

Table 2 provides descriptive information on item-level responses for the four implementation support factors. Factors were positively correlated, rs ranging from .15 to .54. Therapists’ ratings were generally positive across items, suggesting that inclusion of a variety of strategies may facilitate RPM. To examine differences in SFPRM ratings within therapists, a repeated-measures ANOVA was conducted with the four SFPRM factors as within-subjects variables. Results revealed significant differences in therapist ratings of the four strategies, F(3, 12.075) = 16.68, p < .001. Contrasts revealed that therapists rated Automation, (Factor 2: M = 3.80, SD = 1.04), Staffing/Access (Factor 3: M = 3.78, SD = .97), and Requirements (Factor 4: M = 3.71, SD = .87) as significantly more important than Language/Interpretability (Factor 1: M = 3.42, SD = .98) for encouraging uptake of progress monitoring. Effect sizes for these differences was in the small range (Cohen’s d = .31 to .38; partial eta squared estimates ranged from .06 to .09). Thus, therapists reported being more likely to use RPM if measures were quick and automatic, easy to access and use, and if required by their agency/supervisor, more so than if the measures had simple language and/or were available in multiple languages.

Predictors of Differences in SFPRM Factors

A mixed ANOVA was conducted to explore interactions between therapist demographic and professional characteristics and strategies for supporting RPM. Therapist race/ethnicity, discipline, theoretical orientation, licensure status, language of service delivery with children and caregivers, and current use of RPM were entered as between-subject factors and weekly direct service hours was entered as a covariate in one model. The four factors of the SFRPM were entered as within-subject variables. There were significant main effects of therapist discipline, F(1, 368) = 7.58, p = .01, and number of direct service hours, F(1, 368) = 5.87, p = .02. Therapists who were psychologists reported higher intended use of RPM across the four types of support strategies overall, relative to therapists from other disciplines (e.g., Marriage and Family Therapy and Social Work). Higher number of direct service hours per week was associated with lower intended use of RPM across the four types of support strategies overall.

Therapist characteristics were also associated with differential receptivity to the four types of facilitation strategies. There was a significant interaction between therapist race/ethnicity and the SFRPM factors, F(2.78, 1027.75) = 4.37, p = .01, ηp2 = .01, where the importance of the four types of strategies for supporting RPM differed among NHW versus ethnic minority therapists. Contrasts revealed that relative to ethnic minority therapists, NHW therapists rated Language/Interpretability as less important than Automation, F(1, 368) = 7.32, p = .01, and Requirements, F(1, 368) = 11.51, p = .001. Ethnic minority therapists rated these types of strategies more comparably (Figure 1). There was also a significant interaction between current RPM use and the SFRPM factors, F(5.55, 1021.75) = 4.50, p < .001, ηp2 = .02. The relative importance of the four types of strategies for supporting RPM differed among therapists with varying levels of current RPM use. Therapists who do not currently use RPM and those who minimally use RPM rated Automation, F(1, 368) = 9.22, p < .001, and Staffing/Access, F(1, 368) = 3.39, p = .04, as more important than Language/Interpretability. Furthermore, therapists who do not currently use RPM reported Automation as more important than Staffing/Access, F(1, 368) = 3.17, p = .04. Therapists who regularly use RPM rated all four strategies comparably (Figure 2). Therapist discipline, theoretical orientation, licensure status, and language of service delivery did not have significant main or interactive effects.

Figure 1. Ratings of SFRPM among Non-Hispanic White and Ethnic Minority Therapists.

Figure 1.

Open in a new tab

* p <.05, denotes significant contrasts between Non-Hispanic White and minority therapists’ ratings of the relative importance of the SFRPM factors. For example, Language/Interpretability was rated as less important than Automation far Non-Hispanic White therapists relative to ethnic minority therapists.

Figure 2. Ratings of SFRPM among Therapists with Varying Levels of Current RPM Use.

Figure 2.

Open in a new tab

* p < .05, denotes significant contrasts between how therapists who regularly use RPM (routine progress monitoring), therapists who minimally use RPM, and therapists who do not currently use RPM, rate the relative importance of the SFRPM factors. For example, Language/Interpretability was rated as less important than Automation among therapists who do not currency use RPM and therapists who minimally use RPM.

Discussion

The use of RPM improves treatment outcomes and clinical practice (Bickman et al., 2011; Lambert & Shimokawa, 2011), yet very few therapists implement it in usual care settings (Jensen-Doss et al., 2018). Even within a context where therapists were required to administer outcome monitoring for pre- and post-treatment for EBI as part of the PEI initiative, only 38.4% of therapists reported currently using RPM with any client and only 5.9% reported using RPM with many clients in our sample. That is, even though therapists are required and may already have resources and support to administer standardized assessment tools at the beginning and end of treatment, therapists are not routinely gathering and monitoring treatment progress data during treatment. Further, even though Managing and Adapting Practice (MAP) is one of the EBI in the PEI initiative and involves the use of RPM dashboards, LACDMH does not monitor or require adherence to dashboard use for submitting reimbursement for care. Indeed, data suggests that dashboard use may not be incorporated in a significant portion of MAP cases in LACDMH (Buckingham et al., 2019). This low RPM adoption and penetration rate highlights the need for active facilitation strategies tailored towards supporting RPM. This study sought to understand the most beneficial strategies that can be implemented in community contexts to encourage therapist uptake and continued use of RPM. This study fills a gap by building on qualitative findings and quantitatively identifying actionable facilitation strategies therapist report as being most helpful for using RPM (Garland et al., 2003; Ionita et al., 2016).

Four domains of facilitation strategies were identified to facilitate therapists’ use of RPM. These strategies involved 1) sensitivity to the language and interpretability of measures used as part of RPM, 2) automating the assessment process, 3) having support/administrative staff to help with the process, and 4) mandating the use of RPM. Descriptive statistics indicated that overall, therapists mostly agreed that these strategies would facilitate their use of RPM. These findings are consistent with previous research indicating convenience and reduced burden are key facilitators in adopting and sustaining progress monitoring (Knoll et al., 2016; Norman et al., 2014; Jensen-Doss et al., 2018). Thus, administrators may consider implementing policies or initiatives to require RPM and build in infrastructure to support RPM use and sustainment, such as hiring dedicated support staff (e.g., staff can assume functions such as outcome monitoring vs. billing) or investing in automatic scoring programs. Costs and financial resources must be considered in concert when deciding to implement these facilitation strategies, in conjunction with understanding organizational and clinical benefits that could be realized with widespread RPM. Further, when considering which measures to adopt within the agencies, leaders may wish to select measures that are available in multiple languages, can be easily understood, and can be administered quickly and automatically to enhance therapist acceptability and use. Seeking consumer feedback from clients and families may also be helpful to inform which measurement tool and platform of administration may be most suitable, consistent with principles of human-centered design.

Therapist discipline, but not theoretical orientation or licensure status, was related to intended RPM use overall. Psychologists reported higher intended use of RPM across the four types of support strategies relative to therapists trained in other disciplines, such as marriage and family therapy and social work. Psychologists may be generally more receptive to RPM due to their training emphasizing psychological assessments and nomothetic and idiographic measurement. Consistent with current findings, a previous study found that psychologists valued good psychometric properties of standardized assessment tools more, perceived greater benefit of these tools, and perceived greater feasibility of using these tools in practice relative to non-psychologists (Jensen-Doss & Hawley, 2010). The difference in receptivity to RPM among psychologists and non-psychologists may also reflect variability in roles and activities. Increasingly, psychologists occupy supervisory roles whereas marriage and family therapists and social workers occupy front-line provider roles. It is possible that differences in activities, workflow, or setting of service provision may present different challenges and intent to RPM use. There is some indication in the current sample that non-psychologists spend more time per week traveling for clients than psychologists, suggesting non-office-based care. The need to travel to clients’ homes or schools would pose additional challenges for RPM use relative to office-based care since it would be necessary for therapists to plan ahead, remember to, and bring needed RPM tools with them, which adds burden. Thus, different facilitation strategies may be required such as providing tablets equipped with all necessary RPM tools that are easily transportable. With social workers and marriage and family therapists projected to be the fastest growing segments in the mental and behavioral health workforce (Human Resources and Services Administration Bureau of Health Workforce, 2019), further examining facilitation strategies that are best suited for different types of therapists is an important area for future research. Findings suggest the importance of attending to the demographic of the workforce, as facilitation strategies may need to differ in typology based on therapist training and discipline. Contrary to previous research finding therapists with cognitive-behavioral theoretical orientation tend to have more positive attitudes toward RPM (Jensen-Doss et al., 2018; Kaiser et al., 2018), therapist theoretical orientation was not a significant predictor of intended RPM use in the current study. Both previous studies included therapists who primarily served adult populations. It is possible that difference in client population may account for differences in findings. Perhaps theoretical orientation has less of an influence on RPM use with child clients because RPM is necessitated by youth’s reduced capacity to verbalize their own symptoms and thoughts in sessions.

Higher number of direct service hours per week was associated with lower intended use of RPM across the four types of support strategies overall, consistent with previous research identifying workload and burnout as potential barriers to RPM (Boswell et al., 2015; Lucock et al., 2015). It is possible that when therapists are working at capacity or feel overburdened, simply providing concrete resources and supports such as administrative help or automation, may not be sufficient to promote the use of RPM. Rather, redistributing workload, shifting or revising roles, and/or creating innovative incentive structures, such as reimbursing for RPM and other clinical administrative work, may be needed.

As far as variation in the impact of the factors on intended use of RPM, the within subjects effect suggested that therapists rated the language/interpretability of measures as relatively less important than other types of facilitation strategies. In the context of limited resources, this suggests that agencies should prioritize other facilitation strategies viewed as more important. However, this effect was moderated by therapist race/ethnicity, such that ethnic minority therapists rated all factors as similarly important, whereas non-Hispanic White therapists rated language and interpretability as significantly less important than the other factors. Ethnic minority therapists may be more likely to serve diverse families with varying levels of English fluency, which may result in language issues being more salient to these therapists. Indeed, in the current sample, ethnic minority therapists were significantly more likely to endorse providing services in another language with both their child clients and their clients’ caregivers, than NHW therapists. The increased exposure may facilitate more comfort with incorporating RPM tools into clinical care with linguistically diverse clients. At the same time, NHW therapists may hesitate to add RPM with clients with whom they may be experiencing a language barrier. Alternatively, ethnic minority therapists may be more attuned to linguistic issues for their clients because of their own personal experiences with navigating multiple cultures and languages. Nonetheless, if an agency is primarily staffed by a diverse workforce serving multilingual diverse families, close attention to language and comprehensibility of measures is crucial to optimize uptake of RPM. Considering the input of front-line providers in organizational decisions is established as important in the increasingly diverse workforce of therapists (Lin et al., 2015; U.S. Department of Health and Human Services, 2015). Including ethnic minority therapists in the decision-making processes in planning RPM processes and tools can promote both the cultural competence of systems as well as provider buy-in and adherence to implementation efforts (Clauss-Ehlers et al., 2019; Rodriguez et al., 2019).

Therapist current use of RPM also moderated the relative preference for the four types of facilitation strategies. Therapists who reported regular use of RPM rated the four types of strategies as similarly important, whereas therapists who were not using RPM or were only minimally using RPM reported automation and staffing/access as more important than language/interpretability of measures. Therapists who have not yet adopted RPM particularly emphasized automation. This finding suggests that when therapists first consider adopting RPM, their decision may be more influenced by how RPM impacts their workflow and burden, especially their time, and correspondingly, indicate a higher intention to use RPM if strategies are in place to reduce such time burden. The decision to adopt RPM among therapists who are not yet using RPM may be less influenced by how easily their clients understand the measure. However, with experience and increased use of RPM, factors that influence client receptivity, including the availability of linguistically appropriate measures, may become more salient in influencing therapist’ decision to continue to use RPM.

Limitation and Future Directions

This study relied on therapist self-reports of RPM use intentions. Although the factors on the SFRPM demonstrated good reliability, other psychometric properties of the SFRPM were in the adequate range, which may impact the robustness of the measured construct. Furthermore, effect sizes were in the small range. Experimental designs to examine the effect of facilitation strategies on actual therapist adoption and implementation of RPM, the magnitude of resulting changes, as well as the relative utility of various types of facilitation strategies on actual RPM use, are needed. Further, data on facilitation strategies already implemented in mental health agencies could contribute to an understanding of what enabled some therapists to adopt and use RPM. Findings may not generalize to therapists in other samples. Therapists in this study were primarily unlicensed. Although licensure status did not impact therapists’ ratings, receptivity to RPM use and the facilitation strategies needed to support RPM may nonetheless differ among populations of therapists who are licensed and/or have different professional characteristics. It is possible that supervisors’ attitudes and beliefs about RPM may have influenced the intent and use of RPM in our current sample. Concurrently examining therapist and supervisor’s perceptions and use of RPM and their dyadic relationship would be an important future direction. Therapists were also providing services to diverse children and families within a system-wide implementation of evidence-based practices in which outcome assessments were already required pre- and post-treatment (although session-by-session RPM was not). Therapists’ views on RPM generally and the importance of various facilitation strategies to support RPM may differ in settings where the infrastructure to support any EBIs may be more minimal. Further, findings may not generalize to therapists who primarily serve adult populations rather than youth, or to therapists who provide services in settings apart from community mental health centers. Clinical care with youth typically involves multiple parties and may require outcome monitoring from caregivers, teachers, and other adults in the youth’s life, particularly for younger youth who may lack insight to monitor and report on their own symptomatology. Therapist’s perceived value of RPM may thus differ depending on informants and involving additional informants impact the feasibility and practicality of implementing RPM use that may be absent from clinical care with adult populations. Clients’ clinical presentations also tend to be more complex and severe in community mental health. The low rates of RPM in the current sample may partly reflect difficulty in identifying brief, appropriate tools that are sensitive to session by session changes in complex cases. It is possible that there may be systematic differences between therapists who elected to participate in the supplemental survey from those who did not (survey response rates were 41.5% and 52.2%). Although therapists rated all facilitation strategies positively, the costs and feasibility of these strategies are unknown and will be shaped by system decisions. There may be additional factors for agencies to consider when implementing RPM that were not addressed in this study. It is unknown whether the types of facilitation strategies examined in the current study may be more or less effective in promoting RPM relative to strategies that were not studied. Future studies evaluating these aspects along with therapist attitudes toward RPM more generally in addition to client outcomes can help further inform how agencies can best support RPM use among community therapists.

Conclusion

To facilitate community therapists’ use of RPM, it may be helpful for agencies/leaders to mandate its use, adopt measures/tools that are linguistically appropriate, and provide automation and administrative/logistic staff support. Although these were important strategies for all therapists, the automation, staff support and agency requirement were emphasized by non-Hispanic White therapists and therapists who were not yet regularly using RPM, while language/interpretability of RPM tools was equally important as other strategies for minority therapists and therapists regularly using RPM. This highlights the need for agencies/leaders to attend to their workforce characteristics to ensure the RPM process and tool fit with their therapists.

Impact Statements:

  • Therapists reported higher intention to use routine progress monitoring if facilitation strategies automated administration and scoring, provided clerical assistance, and mandated its use, more so than if strategies provided linguistically appropriate measures.

  • Linguistically appropriate tools were equally emphasized as other types of facilitation strategies by ethnic minority therapists compared to non-Hispanic White therapists.

  • For therapists who were not current RPM users, facilitation strategies that reduce time and automate the RPM process were emphasized more than providing administrative or staff support.

  • When staffed by a diverse workforce, attention to linguistically appropriate tools is important to increase therapist uptake of routine progress monitoring.

Acknowledgments

Funding for this research project, including the study design and data collection, analysis and interpretation of data, was supported by NIMH Grant R01MH100134.

References

  1. Baker-Ericzén MJ, Hurlburt MS, Brookman-Frazee L, Jenkins MM, & Hough RL (2010). Comparing Child, Parent, and Family Characteristics in Usual Care and Empirically Supported Treatment Research Samples for Children with Disruptive Behavior Disorders. Journal of Emotional and Behavioral Disorders, 18(2), 82–99. 10.1177/1063426609336956 [DOI] [Google Scholar]
  2. Barnett M, Brookman-Frazee L, Regan J, Saifan D, Stadnick N, & Lau AS (2017). How intervention and implementation characteristics relate to community therapists’ attitudes toward evidence-based practices: A mixed methods study. Administration and Policy in Mental Health and Mental Health Services Research, 44(6), 824–837. 10.1007/s10488-017-0795-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Batty MJ, Moldavsky M, Foroushani PS, Pass S, Marriott M, Sayal K, & Hollis C (2013). Implementing routine outcome measures in child and adolescent mental health services: From present to future practice. Child and Adolescent Mental Health, 18(2), 82–87. 10.1111/j.1475-3588.2012.00658.x [DOI] [PubMed] [Google Scholar]
  4. Bickman L, Douglas Kelley S, Breda C, Regina de Andrade A, & Riemer M (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatric Services, 62(12), 1423–1429. 10.1176/appi.ps.002052011 [DOI] [PubMed] [Google Scholar]
  5. Borntrager C, & Lyon AR (2015). Monitoring Client Progress and Feedback in School-Based Mental Health. Cognitive and behavioral practice, 22(1), 74–86. 10.1016/j.cbpra.2014.03.007 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Boswell JF (2019). Monitoring processes and outcomes in routine clinical practice: A promising approach to plugging the holes of the practice-based evidence colander. Psychotherapy Research, 1–14. 10.1080/10503307.2019.1686192 [DOI] [PubMed] [Google Scholar]
  7. Boswell JF, Kraus DR, Miller SD, & Lambert MJ (2015). Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions. Psychotherapy Research, 25(1),6–19. 10.1080/10503307.2013.817696 [DOI] [PubMed] [Google Scholar]
  8. Brattland H, Koksvik JM, Burkeland O, Gråwe RW, Klöckner C, Linaker OM, Ryum T, Wampold B, Lara-Cabrera ML, & Iversen VC (2018). The effects of routine outcome monitoring (ROM) on therapy outcomes in the course of an implementation process: A randomized clinical trial. Journal of Counseling Psychology, 65(5), 641–652. 10.1037/cou0000286 [DOI] [PubMed] [Google Scholar]
  9. Brookman-Frazee L, Drahota A, & Stadnick M (2012). Training community mental health therapists to deliver a package of evidence-based practice strategies for school-age children with autism spectrum disorders: a pilot study. Journal of Autism and Developmental Disorder, 42(8), 1651–61. 10.1007/s10803-011-1406-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Brown TA (2015). Confirmatory Factor Analysis for Applied Research (2nd Edition). New York: Guilford. [Google Scholar]
  11. Clauss-Ehlers CS, Chiriboga DA, Hunter SJ, Roysircar G, & Tummala-Narra P (2019). APA multicultural guidelines executive summary: Ecological approach to context, identity, and intersectionality. American Psychologist, 74(2), 232–244. 10.1037/amp0000382 [DOI] [PubMed] [Google Scholar]
  12. De Jong K, Timman R, Hakkaart-Van Roijen L, Vermeulen P, Kooiman K, Passchier J, & Busschbach JV (2014). The effect of outcome monitoring feedback to clinicians and patients in short and long-term psychotherapy: A randomized controlled trial. Psychotherapy Research, 24(6), 629–639. 10.1080/10503307.2013.871079 [DOI] [PubMed] [Google Scholar]
  13. Garland AF, Kruse M, & Aarons GA (2003). Clinicians and outcome measurement: What’s the use? Journal of Behavioral and Health Services Research, 30(4), 393–405. 10.1007/BF02287427 [DOI] [PubMed] [Google Scholar]
  14. Glisson C, Landsverk J, Schoenwald S, Kelleher K, Hoagwood KE, Mayberg S, Green P, & Research Network on Youth Mental Health. Assessing the organizational social context (OSC) of mental health services: implications for research and practice. Administration and Policy in Mental Health and Mental Health Services Research, 25, 98–113. 10.1007/s10488-007-0148-5 [DOI] [PubMed] [Google Scholar]
  15. Graham JW, Olchowski AE and Gilreath TD (2007) How Many Imputations Are Really Needed? Some Practical Clarifications of Multiple Imputation Theory. Prevention Science, 8, 206–213. 10.1007/s11121-007-0070-9 [DOI] [PubMed] [Google Scholar]
  16. Hall CL, Moldavsky M Taylor J Sayal K, Marriott M, Batty MJ, Pass S, & Hollis C (2014). Implementation of routine outcome measurement in child and adolescent mental health services in the United Kingdom: A critical perspective. European Child and Adolescent Psychiatry, 23(4), 239–242. 10.1007/s00787-013-0454-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Hatfield DR, & Ogles BM (2004). The use of outcome measures by psychologists in clinical practice. Professional Psychology: Research and Practice, 35(5), 485–491. https://psycnet.apa.org/doi/10.1037/0735-7028.35.5.485 [Google Scholar]
  18. Herschell AD, Kolko DJ, Baumann BL, & Davis AC (2010). The role of therapist training in the implementation of psychosocial treatments: A review and critique with recommendations. Clinical Psychology Review, 30(4), 448–466. 10.1016/j.cpr.2010.02.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Howard MC (2016) A review of exploratory factor analysis decisions and overview of current practices: What we are doing and how can we improve? International Journal of Human–Computer Interaction, 32, 51–62. 10.1080/10447318.2015.1087664 [DOI] [Google Scholar]
  20. Human Resources and Services Administration Bureau of Health Workforce (2019). Behavioral health workforce projections, 2017–2030. https://bhw.hrsa.gov/health-workforce-analysis/research/projections/behavioral-health-workforce-projections
  21. Ionita G, & Fitzpatrick M (2014). Bringing science to clinical practice: A Canadian survey of psychological practice and usage of progress monitoring measures. Canadian Psychology/Psychologie Canadienne, 55(3), 187–196. https://psycnet.apa.org/doi/10.1037/a0037355 [Google Scholar]
  22. Ionita G, Fitzpatrick M, Tomaro J, Chen VV, & Overington L (2016). Challenges of using progress monitoring measures: Insights from practicing clinicians. Journal of Counseling Psychology, 63(2), 173–182. 10.1037/cou0000122 [DOI] [PubMed] [Google Scholar]
  23. Jensen-Doss A, Haimes EMB, Smith AM, Lyon AR, Lewis CC, Stanick CF, & Hawley KM (2018). Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Administration and Policy in Mental Health and Mental Health Services Research, 45(1), 48–61. 10.1007/s10488-016-0763-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Jensen-Doss A, & Hawley KM (2010). Understanding barriers to evidence-based assessment: Clinician attitudes toward standardized assessment tools. Journal of Clinical Child & Adolescent Psychology, 39(6), 885–896. 10.1080/15374416.2010.517169 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Kaiser T, Schmutzhart L, Laireiter AR (2018) Attitudes of Austrian Psychotherapists Towards Process and Outcome Monitoring. Administration and Policy in Mental Health and Mental Health Services, 45(5), 765–779. 10.1007/s10488-018-0862-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Knoll M, Ionita G, Tomaro J, Chen V & Fitzpatrick M (2016). Progress monitoring measures: The interaction of clinician initial motivation with selection and maintenance issues. Psychology, 7(3), 444–458. 10.4236/psych.2016.73046 [DOI] [Google Scholar]
  27. Lambert MJ, & Shimokawa K (2011). Collecting client feedback. Psychotherapy Research, 48(1). 10.1037/a0022238 [DOI] [PubMed] [Google Scholar]
  28. Lambert MJ, Whipple JL, & Kleinstäuber M (2018). Collecting and delivering progress feedback: A meta-analysis of routine outcome monitoring. Psychotherapy, 55(4), 520–537. 10.1037/pst0000167 [DOI] [PubMed] [Google Scholar]
  29. Lau AS, & Brookman-Frazee L (2016). The 4KEEPS study: identifying predictors of sustainment of multiple practices fiscally mandated in children’s mental health services. Implementation Science, 11, 31. 10.1186/s13012-016-0388-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Lau A, Barnett M, Stadnick N, Saifan D, Regan J, Wiltsey-Stirman, Roesch S, & Brookman-Frazee L (2017). Therapist report of adaptations to evidence-based practices within a system-driven reform of publicly-funded children’s mental health services. Journal of Consulting and Clinical Psychology, 85(7), 664–675. 10.1037/ccp0000215 [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Lin L, Nigrinis A, Christidis P, & Stamm K (2015). Demographics of the U.S. psychology workforce: Findings from the American Community Survey. Washington, DC: American Psychological Association. https://www.apa.org/workforce/publications/16-demographics/report.pdf [Google Scholar]
  32. Los Angeles County Department of Mental Health. (2016). Prevention and Early Intervention Implementation Handbook. http://file.lacounty.gov/SDSInter/dmh/247145_PEIImplementationHandbook-PDFforWebsiterev.7-27-16.pdf
  33. Lucock M, Halstead J, Leach C, Barkham M, Tucker S Randal C, Middleton J, Khan W, Catlow H, Waters E, & Saxton D (2015). A mixed-method investigation of patient monitoring and enhanced feedback in routine practice: Barriers and facilitators. Psychotherapy Research, 25, 633–646. 10.1080/10503307.2015.1051163 [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Lyon AR, & Dorsey S (2010). The Current Assessment Practice Evaluation. [Unpublished measure]. University of Washington. [Google Scholar]
  35. Lyon AR, Dorsey S, Pullmann M, Silbaugh-Cowdin J, & Berliner L (2015). Clinician use of standardized assessments following a common elements psychotherapy training and consultation program. Administration and Policy in Mental Health, 42(1), 47–60. 10.1007/s10488-014-0543-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Midboe AM, Martino S, Krein SL, Frank JW, Painter JT, Chandler M, Schroeder A, Fenton BT, Troszak L, Erhardt T, Kerns RD, & Becker WC (2018). Testing implementation facilitation of a primary care-based collaborative care clinical program using a hybrid type III interrupted time series design: a study protocol. Implementation science, 13(1), 145. 10.1186/s13012-018-0838-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Norman S, Dean S, Hansford L, Ford T 7 (2014). Clinical practitioner’s attitudes towards the use of routine outcome monitoring within child and adolescent mental health services: A qualitative study of two child and adolescent mental health services. Clinical Child Psychology and Psychiatry, 19(4), 576–595. 10.1177/1359104513492348 [DOI] [PubMed] [Google Scholar]
  38. Ritchie MJ, Parker LE, Edlund CN & Kirchner JE (2017). Using implementation facilitation to foster clinical practice quality and adherence to evidence in challenged settings: a qualitative study. BMC Health Services Research, 17, 294. 10.1186/s12913-017-2217-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Rodriguez A, Terrones L, Brookman-Frazee L, Regan J, Smith A, & Lau AS (2019). Associations between cultural identity and attitudes toward routine progress monitoring in a sample of ethnically diverse community therapists. Psychological Services. 10.1037/ser0000327 [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Rye M, Rognmo K, Aarons GA , & Skre I (2019). Attitudes towards the use of routine outcome monitoring of psychological therapies among mental health providers: The EBPAS–ROM. Administration and Policy in Mental Health and Mental Health Services Research, 46, 833–846. 10.1007/s10488-019-00968-5 [DOI] [PubMed] [Google Scholar]
  41. Sharples E, Qin C, Goveas V, Gondek D, Deighton J, Wolpert M, & Edbrooke-Childs J (2017). A qualitative exploration of attitudes towards the use of outcome measures in child and adolescent mental health services. Clinical Child Psychology and Psychiatry, 22, 219–228. 10.1177/1359104516652929 [DOI] [PubMed] [Google Scholar]
  42. Shimokawa K, Lambert MJ, & Smart DW (2010). Enhancing treatment outcome of patients at risk of treatment failure: Meta-analytic and mega-analytic review of a psychotherapy quality assurance system. Journal of Consulting and Clinical Psychology, 78(3), 298–311. 10.1037/a0019247 [DOI] [PubMed] [Google Scholar]
  43. U.S. Department of Health and Human Services. (2015). The Comprehensive Community Mental Health Services for Children with Serious Emotional Disturbances Program, Report to Congress. http://store.samhsa.gov/shin/content//PEP16-CMHI2015/PEP16-CMHI2015.pdf
  44. Waldron SM, Loades ME, & Rogers L (2018). Routine Outcome Monitoring in CAMHS: How Can We Enable Implementation in Practice? Child and Adolescent Mental Health, 23(4), 328–333. 10.1111/camh.12260 [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES