A Systematic Review of External Validity in Pediatric Integrated Primary Care Trials

Abstract

Objective

We used the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework to conduct a systematic review of external validity reporting in integrated primary care (IPC) interventions for mental health concerns.

Methods

We searched Medline, CINAHL, PsycINFO, the Cochrane Center Register of Controlled Trials, and relevant literature to identify publications from 1998 to 2018 reporting on open, randomized, or quasi-randomized trials of IPC interventions that targeted child (ages 0–18 years) psychological symptoms. For each publication, we extracted the information reported in each RE-AIM domain and calculated the proportion of the total studies reviewed.

Results

Thirty-nine publications describing 25 studies were included in the review. Publications rarely reported some indicators of external validity, including the representativeness of participants (12%), rate of adoption clinics or providers (16%), cost of implementation (8%), or evidence of maintenance (16%). Few studies reported on key pragmatic factors such as cost or organizational change processes related to implementation and maintenance. Strengths of some studies included comparisons of multiple active treatments, use of tailorable interventions, and implementation in “real world” settings.

Conclusions

Although IPC interventions appear efficacious under research conditions, there are significant knowledge gaps regarding the degree to which they reach and engage target recipients, what factors impact adoption and implementation of IPC interventions by clinicians, how fidelity can be maintained over time, and cost-effectiveness. Pediatric IPC researchers should embrace dissemination and implementation science methods to balance internal and external validity concerns moving forward.

Spurred by federal healthcare initiatives to increased access to mental health services, the integration of behavioral health providers into primary care settings is proliferating (Richman et al., 2020), stoking a need for research to inform the delivery of mental health interventions in integrated primary care (IPC) settings. Evidence for the efficacy of IPC interventions has emerged: A meta-analysis of 31 randomized clinical trials found IPC interventions produce superior outcomes compared to usual care (Asarnow et al., 2015). However, a number of pragmatic factors beyond efficacy (e.g., time to employ, ease of use, costs) may impact whether behavioral interventions are adopted, implemented, and sustained in primary care settings (Arora et al., 2016), and it is currently unclear how most IPC interventions perform in these domains. Lack of external validity in clinical research is an oft cited contributor to the gap between healthcare evidence and practice (Green, 2008), so it is important that IPC research is conducted and reported in a manner that allows for inferences about applicability in a spectrum of settings and representative populations in order to maximize “real world” impact.

The RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework, a prominent set of criteria for assessing the generalizability of clinical research (Glasgow et al., 1999), provides a useful lens through which to evaluate IPC research. Reach refers to the absolute number, proportion, and representativeness of individuals who participate in a given initiative, intervention, or program. Effectiveness is the impact of an intervention on important outcomes, including quality of life, economic outcomes, and negative effects. Adoption is defined as the absolute number, proportion and representativeness of settings and staff who are willing to initiate a program. Implementation refers to the extent to which the intervention was implemented as intended, including consistency of delivery, time, and cost of the program. Maintenance is the extent to which the program or policy becomes part of a clinic’s routine and the long-term effects on outcomes 6 or more months after the intervention. Designing research to evaluate these domains is thought to facilitate dissemination of interventions and maximize the public health impact of clinical research (Klesges et al., 2005).

The RE-AIM framework has not previously been used to systematically evaluate the pediatric IPC literature, but other work has called into question the generalizability of some IPC interventions. For example, Brown et al. (2018) found that IPC prevention programs often fail to attract high levels of participation, which may indicate suboptimal reach. Brown et al. also noted that IPC trials are often conducted in potentially unrepresentative settings, including academic medical centers or large health systems (<10% of all child primary care visits take place at such clinics, whereas 79% take place at physician-owned practices; Rui & Okeyode, 2016). These setting likely differ from where most primary care is delivered with regard to organizational structures, operating budgets, workflows, staffing, and patient populations. Consideration of these elements is essential, because the relative generalizability of IPC interventions to different settings may be impacted by the costs and benefits of the models of integration that are required to successfully deploy them. For example, co-located models of integration involve behavioral health providers delivering care in primary care clinics, but do not include a common treatment plan or concurrent delivery with medical care (Asarnow et al., 2017). Such models generally require separate clinical space and administrative procedures (e.g., scheduling, payment authorization) for behavioral health services. In contrast, interventions delivered in integrated models, in which behavioral services are available to all patients as part of routine primary care (i.e., in the context of well-child or acute medical care), require a different set of conditions (e.g., shared clinical space, integrated workflows) to be successfully implemented. Depending on the constraints and resources of a given practice, certain approaches may be more feasible than others.

With exception of Brown et al. (2018), who reviewed early childhood prevention programs, external validity has not been a focus of previous IPC literature reviews. We used RE-AIM to conduct a systematic review in a manner similar to reviews of other health intervention literatures (e.g., Harden et al., 2015). Whereas Brown and colleagues previously focused on prevention programs in primary care, we focused on IPC interventions targeting children and adolescents with elevated mental health symptoms at baseline. Our goals were to characterize the degree to which external validity indicators have been reported in trials of pediatric IPC mental health interventions, summarize the status of such interventions in key external validity domains, and identify methodological gaps for future emphasis.

Methods

Search Strategy and Study Identification

We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines (Moher et al., 2009) in conducting the review. The lead (A.C.B.) and senior (A.R.) authors queried Medline, CINAHL, PsycINFO, and the Cochrane Center Register of Controlled Trials with the assistance of a medical librarian. Each database was queried for publications from 1998 to present from July 20 to 25, 2018. We also manually searched the reference lists of included studies and other relevant works. We last identified eligible publications via manual search April 28, 2020. Inclusion/exclusion criteria, data extraction procedures, and primary synthesis methods were established in a protocol a priori, but the protocol was not registered.

Studies were eligible for review if they: were available in English; were published in a peer-reviewed journal; described open, randomized, or quasi-randomized trials assessing a mental health intervention delivered in a primary care setting; included patients 0–18 years (studies with older participants were included if the minimum age ≤ 17); and the study’s parent publication (i.e., first publication describing outcomes) was published from 1998 to July 2018. Because we focused on interventions targeting existing mental health problems rather than prevention programs, we required that included studies utilized a measure of child behavioral or mental health symptoms as a primary dependent measure, and that mental health diagnosis or elevated symptoms was an inclusion criterion for the study. Finally, we focused on interventions that utilized an integrated behavioral specialist. We defined “behavioral specialist” broadly to include any individual whose primary role was to improve mental health through psychosocial means, including mental health professionals and trainees, medical professionals with specialized training, or lay persons with specialized training. Trials of interventions that were exclusively delivered by digital technology or other multimedia were excluded, as were studies that focused primarily on physical health outcomes (e.g., obesity).

Two authors (A.C.B. and K.R.) independently reviewed article titles and abstracts to assess inclusion and exclusion criteria. Each abstract was initially classified as “yes,” “no,” or “maybe” for inclusion, resulting in an initial agreement rate of 91%. Discrepancies were resolved in consultation with the senior author until consensus. Abstracts identified as “maybe” underwent full-text review to determine eligibility.

Data Extraction

We created a data extraction form and coding guide based on previous RE-AIM systematic reviews to collect information in each of the five RE-AIM domains. Table I summarizes the data that were extracted as RE-AIM domain indicators. Use of qualitative methods across RE-AIM domains has been encouraged (Holtrop et al., 2018), so this was included as a potential indicator in each domain. For each indicator, we recorded both whether it was reported and the value of that indicator when reported. We also categorized each intervention as integrated, co-located, collaborative, or other based on the definitions of IPC models provided by Asarnow et al. (2017) and recorded the target symptoms/diagnoses of each intervention.

Table I.

RE-AIM Domains and Indicators Extracted for this Review

Domain	Extracted Indicators
Reach	Method to identify target population Inclusion criteria Exclusion criteria Eligible for screening Screened Eligible for participation Recruitment rate	Participant characteristics (sex, age, race, and SES) Non-participant characteristics (sex, age, race, and SES) Representativeness (comparison of participants and non-participants) Qualitative methods to assess reach
Effectiveness	Study design Intent-to-treat analysis Missing data/imputation procedures Attrition Negative effects	Participant satisfaction Quality of life measures Provider rating of effectiveness Qualitative methods to assess effectiveness
Adoption	Description of intervention location Description of staff delivering intervention Method to identify delivery agent Level of expertise of delivery agent Rate of adoption	Inclusion/exclusion criteria for setting or staff Characteristics of adoption/non-adoption Start-up costs Qualitative methods to assess adoption
Implementation	Intervention type, frequency, and intensity Extent protocol delivered as intended Cost of delivery	Participant engagement in intervention Qualitative methods to assess implementation
Maintenance	Follow-up of at least 6 months post-treatment Any indication of maintenance	Cost of maintenance Program modifications Qualitative methods to assess maintenance

Each author completed readings about the RE-AIM framework and received training on the data extraction process from the senior author. To calibrate coding practices, each author independently assessed the same three sample articles. We discussed results in group meetings until discrepancies were resolved and definitions were well understood. Two authors then independently reviewed each article. We compared extracted data in order to identify and resolve any discrepancies. Consensus was required to successfully resolve any disagreement.

Data Synthesis

For each study, we calculated the percentage of possible RE-AIM indicators reported in each domain. We also calculated the proportion of studies reporting each RE-AIM domain indicator by dividing the number of studies reporting by the total number of studies. For studies with multiple publications, we examined both the parent publication individually and all publications in sum (i.e., if only one publication reported the outcome of interest, the overall study was coded “yes”). We calculated rates of interest (e.g., enrollment) across studies by pooling the relevant numerators and denominators. We conducted several analyses to examine reporting trends over time. First, we plotted the mean percentage of RE-AIM indicators by year of publication to visually inspect the data. We then calculated Pearson’s r to test whether there was a significant linear correlation between year of publication and percentage of indicators reported in each RE-AIM domain. Finally, we grouped studies into 4-year periods and conducted a series of Kruskal–Wallis tests to determine if the distribution of percentage of indicators reported varied significantly across groups.

Quality Assessment

Three authors (A.C.B., K.R., and A.R.) independently reviewed risk of bias for each trial using the Cochrane Collaboration tool (Higgins et al., 2011), which includes selection bias, performance bias, detection bias, attrition bias, and reporting bias. Trials were categorized into low, high, and unclear risk. All non-randomized trials were considered high risk for selection bias. Discrepancies were resolved via discussion until consensus was made.

Results

The systematic review included 39 publications reporting on 25 studies. Supplementary Figure 1 displays the PRISMA flow diagram and detailed search terms for database queries. We initially identified 1,012 records identified from the database search and 72 records from other sources. After removing duplicates, non-trials, and publications outside the target date range, 481 records were eventually screened for eligibility. Subsequently, 435 and 7 records were excluded by abstract and full-text review, respectively. Table II summarizes the characteristics of the reviewed studies. A total of 3,670 participants with an age range of 2–21 years participated in the 25 trials. The most common trial type was randomized effectiveness trial (68%). Target diagnoses and symptoms included disruptive behavior/oppositional defiance (N = 13), attention deficit and hyperactivity (N = 7), depressed mood (N = 6), anxiety (N = 4), substance use/risk behaviors (N = 1), eating disorder (N = 1), and autism spectrum disorders (N = 1). Several studies (N = 7) targeted comorbid symptoms. Co-located (N = 11) and collaborative (N = 8) integration models were most common. One study used an integrated model to intervene entirely in the context of well-child care (Mason et al., 2011).

Table II.

Characteristics of the Reviewed Studies

	Study population					Intervention			% Indicatorsa
Publication(s)	N	Age, years	% Male	Location	Symptoms	Integration model	Description (Name)	Comparison	R	E	A	I	M
Asarnow et al. (2005,   2009),Ngo et al. (2009), Rapp et al. (2017), and Wells et al. (2012)	418	13–21	22	US	Dep	Collaborative	CM + choice of individual CBT and/or MM	EUC	59	78	44	40	20
Berkovits et al. (2010) and Harwood et al. (2009)	30	3–6	70	US	DB	Co-located	Group PT (Primary Care PCIT)	Written handouts	53	33	33	60	20
Borowsky et al. (2004)	224	7–15	58	US	Agg	Other	PT (Positive Parenting) via telephone consultation and educational multimedia	UC	65	33	33	40	00
Boyle et al. (2010)	10	3–7	40	US	DB	Co-located	Individual PT (PCTP)	WS	35	56	33	60	0
Clarke et al. (2005)	152	12–18	22	US	Dep	Collaborative	Individual CBT + SSRI	UC (SSRI)	53	44	33	60	00
Gillham et al. (2006)	271	11–12	47	US	Dep and Anx	Co-located	Group CBT (PRP)	UC	59	44	33	60	20
Gomez et al. (2014)	21	1–17	62	US	DB	Co-located	Individual PT	WS	53	22	33	40	00
Kolko et al. (2010)	163	6–11	65	US	ADHD, ODD, and CD	Collaborative	Individual and family CBT + MM + PT + school consultation + crisis management (PONI)	EUC	59	44	44	60	40
Kolko et al. (2012)	78	5–12	72	US	Anx, ODD, and ADHD	Collaborative	Individual and family CBT (Alternatives for Families) + CM + MM (DOCC)	EUC	59	56	44	60	00
Kolko et al. (2014), Shaffer et al. (2017), and Yu et al. (2017)	321	5–12	65	US	DB, Anx, and ADHD	Collaborative	Individual and family CBT (Alternatives for Families) + CM + MM (DOCC)	EUC	59	78	67	80	60
Lavigne et al. (2008a, b, and 2010)	117	3–7	53	US	ODD	Co-located	Group PT (Incredible Years)	Written materials	53	44	33	60	20
Linville et al. (2015)	66	13–17	0	US	ED	Collaborative	Group dissonance-based therapy (Body Project)	Educational video	41	44	44	40	40
Mason et al. (2011)	28	14–18	0	US	SU	Integrated	MI + social network counseling	NT	59	33	33	40	00
McMenamy et al. (2011)	23	2–3	n.r.	US	ADHD and ODD	Co-located	Group PT (Incredible Years)	WS	41	78	33	40	20
Mockford and Barlow (2004), Patterson et al. (2002), and Stewart-Brown et al. (2004)	116	2–8	“just over half”	UK	DB	Co-located	Group PT (Incredible Years)	NT	76	78	33	60	20
Perrin et al. (2014)	273	2–4	62	US	DB, ODD, and ADHD	Co-located	Group PT (Incredible Years)	WL	47	44	67	60	20
Reid et al. (2013)	178	2–5	56	Can	DB	Other	Written PT materials (Parenting Matters) + telephone coaching	UC	47	56	33	60	20
Richardson et al. (2009)	40	12–18	10	US	Dep	Collaborative	CM + choice of CBT (Problem Solving Treatment in Primary Care) and/or MM	WS	59	56	33	40	00
Richardson et al. (2014) and Wright et al. (2016	101	13–17	28	US	Dep	Collaborative	Individual CBT + CM + MM	EUC	59	78	33	60	20
Silverstein et al. (2015)	155	6–12	69	US	ADHD	Collaborative	Individual MI + PT (PCTP) + CM	CM	59	44	33	40	00
Sonuga-Barke et al. (2004	89	3	n.r.	UK	ADHD	Other	Individual PT	WL	41	56	44	40	00
Ristkari et al. (2019) and Sourander et al. (2016, 2018)	464	4	62	FIN	DB	Other	Web-based PT (Strongest Families Smart Website) + phone coaching	EUC	47	67	33	40	60
Spijkers et al. (2013	93	9–11	56	NL	Mild problems	Unclear	Individual PT (PCTP)	UC	47	44	33	40	20
Tellegen and Sanders (2014)	64	2–9	86	AU	ASD	Co-located	Individual PT (PCTP)	UC	53	44	22	60	20
Weersing et al. (2017)	185	8–17	42	US	Dep and Anx	Co-located	Individual behavior therapy	Assisted referral	59	44	33	60	00

Note: US = United States; UK = United Kingdom; Can = Canada; FIN = Finland; NL = The Netherlands; AU = Australia; Dep = depression; DB = disruptive behavior; Anx = anxiety; Agg = aggression; ADHD = attention deficit hyperactivity disorder, ODD = oppositional defiant disorder, ED = eating disorder, SU = substance abuse; ASD = autism spectrum disorders; CBT = cognitive behavioral therapy; CM = care management, referring to services such as care coordination, logistical problem solving, referral coordination, etc.; DOCC = Doctor Office-Collaborative Care; MM = medication management, referring to strategies to determine whether and how medication is administered; PT = parent training; PCTP = Primary Care Triple P; PRP = Penn Resiliency Program; PONI = protocol for on-site, nurse-administered intervention; MI = motivational interviewing; EUC = enhanced usual care; UC = usual care; WS = within subjects; SSRI = selective serotonin reuptake inhibitor; NT = no treatment; WL = waitlist.

^aNumber of possible indicators: Reach = 16; Effectiveness = 9; Adoption = 9; Implementation = 5; Maintenance = 5.

Quality Assessment

Results of the risk of bias assessment for each study are reported in Supplementary Table 1. The percentage of studies at each level of risk were as follows: Random sequence generation (selection bias): 68% low, 20% high, 12% unclear; allocation concealment (selection bias): 24% low, 20% high, 56% unclear; reporting bias: 96% low risk, 4% unclear; performance bias: 100% high; detection bias: 36% low, 36% high, 28% unclear; attrition bias: 80% low, 4% high, 16% unclear.

Reporting of RE-AIM Domains

Table III summarizes reporting in each of the RE-AIM domains. Figure 1 shows the percentage of RE-AIM indicators reported over time, indicating little fluctuation in reporting practices. There were no significant correlations between year of publication and percentage of indicators reported in any domain, nor were there any significant differences in the distribution of indicators reported over time. Detailed results for each reviewed publication are included in Supplementary Tables 2–6.

Table III.

Reporting of RE-AIM Domain Indicators in Pediatric Integrated Primary Care Trials

	Number reporting (%)
RE-AIM domain indicator	Parent publication	Any publication
Reach
Method to identify target population	25 (100%)	25 (100%)
Inclusion criteria	24 (96%)	24 (96%)
Exclusion criteria	23 (92%)	23 (92%)
Eligible for screening	14 (56%)	14 (56%)
Screened	21 (84%)	21 (84%)
Eligible for participation	24 (96%)	24 (96%)
Participant characteristics
Race/Ethnicity	23 (92%)	23 (92%)
Age	23 (92%)	24 (96%)
Sex	22 (88%)	23 (92%)
SES	23 (92%)	23 (92%)
Non-participant characteristics
Race/Ethnicity	1 (4%)	1 (4%)
Age	0 (0%)	0 (0%)
Sex	1 (4%)	1 (4%)
SES	1 (4%)	1 (4%)
Representativeness	3 (12%)	3 (12%)
Qualitative methods	0 (0%)	0 (0%)
Effectiveness
Study design	25 (100%)	25 (100%)
Intent-to-treat analysis	19 (83%)	19 (83%)
Missing data/imputation proceduresa	11 (46%)	12 (50%)
Attrition	22 (92%)	22 (92%)
Negative effects	6 (24%)	7 (28%)
Participant satisfaction	15 (60%)	17 (68%)
Quality of life measure	5 (20%)	6 (24%)
Provider rated effectiveness	3 (12%)	3 (12%)
Qualitative methods	5 (20%)	5 (20%)
Adoption
Description of intervention location	24 (96%)	24 (96%)
Description of staff delivering intervention	25 (100%)	25 (100%)
Method to identify deliver agent	2 (8%)	2 (8%)
Level of expertise of delivery agent	25 (100%)	25 (100%)
Inclusion/exclusion criteria for setting/staff	1 (4%)	2 (8%)
Rate of adoption	4 (16%)	4 (16%)
Characteristics of adoption/non-adoption	1 (4%)	1 (4%)
Start-up costs	0 (0%)	1 (4%)
Qualitative methods	1 (4%)	1 (4%)
Implementation
Intervention type, frequency, and intensity	25 (100%)	25 (100%)
Fidelity of intervention delivery	12 (48%)	12 (48%)
Participant engagement in intervention	23 (92%)	23 (92%)
Cost of delivery	1 (4%)	2 (8%)
Qualitative methods	2 (8%)	3 (12%)
Maintenance
Follow-up of at least 6 months	14 (58%)	14 (58%)
Any indication of maintenance	3 (12%)	4 (16%)
Cost of maintenance	0 (0%)	1 (4%)
Program modifications	1 (4%)	2 (8%)
Qualitative methods	0 (0%)	0 (0%)

^aOne study reported no missing data and was not included in the denominator.

Figure 1.

Mean percentage of RE-AIM domain indicators in integrated primary care trials over time. Displayed data reflect parent publications only. Number of possible indicators in each domain: Reach = 16; Effectiveness = 9; Adoption = 9; Implementation = 5; Maintenance = 5.

Reach

Methods used to identify the target population, inclusion criteria, exclusion criteria, number of potential participants screened, and number deemed eligible for participation were all commonly reported (>80% of studies), whereas reporting the number of potential participants eligible for screening was less common (59%). Across 12 studies that reported both the number eligible for screening and the number screened, a combined total of 30% of those eligible were screened. The mean of screening rates was 60% (range = 10–100%). Studies that identified the largest pools of potential participants (Gillham et al., 2006; Kolko et al., 2010; Richardson et al., 2014) via broad screening criteria (e.g., all patients in the target age range) generally reported the lowest rates of screening, whereas studies with higher rates of screening usually employed more restrictive screening eligibility criteria (e.g., primary care provider [PCP] referral prior to screening; Kolko et al., 2012; Richardson et al., 2009; Silverstein et al., 2015). Across 25 studies identifying 7,201 eligible participants, 3,670 (51%) were enrolled (range = 18–100%). Nearly all studies reported on participant demographics. Single studies reported non-participants’ sex (Linville et al., 2015), race/ethnicity (Patterson et al., 2002), or socioeconomic status (SES; Patterson et al., 2002). Three studies (Borowsky et al., 2004; Mason et al., 2011; Patterson et al., 2002) compared participants and non-participants to examine representativeness. None used qualitative methods to assess reach.

Efficacy/Effectiveness

Most study designs included a randomization element (N = 17), often to compare two or more active treatments (including usual care). A majority of studies specified an intent-to-treat analysis, and half specified a missing data approach. Twenty-three studies reported attrition rates ranging from 0% to 30% at the shortest post-intervention assessment points and 9–35% at the longest assessment points. Most studies included measures of participant satisfaction, but fewer reported on quality of life, clinicians’ perceptions of interventions, or any negative effects of interventions. Five studies used qualitative methods to evaluate effectiveness.

Adoption

Nearly all studies provided some description of the intervention setting, including locations, number of clinics involved, and type of clinic (private, public, or academic sector). All studies provided some descriptions of the delivery agents, but only two (Perrin et al., 2014; Sonuga-Barke et al., 2004) reported how the delivery agent was identified and one (Perrin et al., 2014) reported specific inclusion/exclusion criteria for setting or staff. Delivery agents in the behavioral specialist role included doctoral level psychologists, graduate psychology trainees, master’s-level social workers, registered nurses, nurse practitioners, and lay persons. Training of behavioral specialist delivery agents ranged from 6 hr to several days, generally followed by ongoing supervision from senior clinicians. Two studies reported the rate of invited clinics that agreed to participate, including 2/2 (Linville et al., 2015) and 12/43 (Perrin et al., 2014). Rates of PCP participation were also reported by two studies, with 30/30 (Kolko et al., 2010) and 74/75 (Kolko et al., 2014) agreeing to participate. One study reported start-up costs (Yu et al., 2017), and one study (Kolko et al., 2012) used qualitative methods to assess adoption.

Implementation

Interventions commonly included components of cognitive behavioral therapy, parent management training, motivational interviewing, and care coordination. Interventions targeting depression or inattention/hyperactivity commonly included a medication management component. Single-patient/family therapy was the most common modality (64%), followed by group formats (28%). Three studies delivered interventions remotely via telephone or website. The number of intervention sessions ranged from 1 to 14, with a mean intended duration of 82 min per session (range = 15–120 min). The maximum intended total dosage ranged from 20 min to 18 hr of intervention. Several interventions were designed to be flexibly delivered such that both the content and dose of intervention could be tailored to child and family needs. Thirteen studies reported fidelity to treatment protocol. It was difficult to synthesize data on patient engagement because of the variability of metrics used across studies, but it was common that significant proportions of participants did not receive intended doses of interventions. For example, while only one intervention was intended to be single-session (Leslie et al., 2016), multiple studies reported that 20% or more of enrolled participants received one or fewer sessions (Asarnow et al., 2005; Berkovits et al., 2010; Borowsky et al., 2004; Gillham et al., 2006; Gomez et al., 2014; Tellegen & Sanders, 2014). Only Richardson et al. (2014) and Yu et al. (2017) reported on costs of implementation. Three studies (Kolko et al., 2012; Linville et al., 2015; Mockford & Barlow, 2004) utilized qualitative methods to assess implementation.

Maintenance

Most studies reported participant outcomes at least 6 months post-intervention. Four studies (Kolko et al., 2010, 2014; Linville et al., 2015; Ristkari et al., 2019) reported indicators of program-level maintenance, including continued referrals to the implemented program, sustained changes in PCP attitudes and practices, and hiring of staff to maintain behavioral specialist roles. Only Linville et al., (2015) noted modifications to the intervention. One study reported effects of the intervention on costs beyond 6 months (Yu et al., 2017). No studies used qualitative methods to assess maintenance.

Discussion

This is the first systematic review of external validity factors in IPC trials targeting child mental health concerns. The results indicate that trials commonly do not report outcomes pertinent to assessing external validity across the RE-AIM domains, and reveal important knowledge gaps about the scalability of existing IPC interventions. With some exceptions, IPC trials have been designed and reported in a manner that allows for strong inferences about internal validity, but do not robustly inform dissemination of interventions in non-research populations and settings. We found no evidence of changes in reporting practices over time.

Little is known about whom IPC interventions fail to reach in the context of clinical trials, or how that group differs from those who successfully contact interventions. Average screening and enrollment rates of 30% and 51%, respectively, indicate IPC trials often engage a minority of prospective participants, and it is unclear how sampling bias and other sources of error impact the representativeness of samples. Given several trials in our sample required identification of problems and referral by PCPs as an eligibility criterion, and that psychosocial problems are under-identified in absence of systematic screening (Sheldrick et al., 2011), our findings may actually overestimate the permeation of interventions into their intended populations. Determining who IPC interventions fail to reach and why is important, as integration is a key strategy for reducing racial/ethnic and socioeconomic health disparities (Hodgkinson et al., 2017). Integrated care may outperform usual care in this respect, but still be lacking. Research focused on understanding and improving patient engagement is imperative.

Researchers appear to underutilize family and provider perspectives in the evaluation of IPC interventions. Most trials reported a measure of participant satisfaction, but few attempted to measure changes in quality of life or provider perceptions of effectiveness. Capturing stakeholder perceptions, as well as understanding the factors that drive those perceptions, is an important goal for establishing the impact of IPC interventions beyond statistical significance.

With regard to adoption, settings and staff involved in IPC trials are described, but often with little detail about how they were identified or funded. Little is known about non-adopters; an important gap in knowledge and potential barrier to the scalability of IPC interventions. Evidence from other fields indicates physicians who engage in research differ from those who do not on characteristics such as time spent in patient care and practice location (Galliher et al., 2009). It is possible practices engaged in IPC research are not representative with regards to resources, constraints, and attitudes towards integration, and may often be “best case” settings for adoption. Valuable information can be gleaned from such settings and providers, but wide scale dissemination of IPC interventions will require adoption in a broad spectrum of clinical settings, so a more sophisticated understanding of barriers and facilitators of adoption is needed.

Similar to Moon et al.’s (2018) review of IPC parenting interventions, we found that few studies reported extensively on implementation processes. Intervention content and intended doses were generally well-described and delivered with fidelity, but organizational change processes (e.g., quality improvement) required to implement those programs were not well described, nor were the costs of implementation. Several interventions were designed in a manner in which current payment systems are usually not designed to reimburse (e.g., 90–120 min sessions), at least in the United States where Current Procedural Terminology codes are the apparent dominant approach to IPC billing (Riley et al., 2018). Accumulating effectiveness data for innovative delivery formats is an important part of widening the spectrum of reimbursable services, but greater emphasis should be placed on studying interventions that are both pragmatically and financially feasible for IPC clinicians in the current healthcare landscape.

When implemented, there is some question as to how well IPC interventions engage and retain patients and their families. Much like Brown et al. (2018), we observed highly variable rates of treatment engagement and noted that interventions with fewer sessions tended to produce higher rates of participation, though not in all cases. More study of factors that influence participation in available IPC interventions is needed.

Studies commonly evaluate the maintenance of treatment effects on participants, but reporting related to organizational maintenance is rare and largely anecdotal. Only one study reported on the cost-effects of interventions beyond 6 months of implementation (Yu et al., 2017), and it focused on societal level costs, not costs incurred by practices. Considering researchers typically provide financial, logistical, and clinical supports during the acute phases of intervention implementation, it is reasonable to question whether and how treatment services persist in the absence of such supports.

Recommendations for IPC Researchers

We have identified a number of important knowledge gaps in the IPC intervention literature in each of the RE-AIM domains. To begin to fill those gaps, researchers should consider RE-AIM and other dissemination and implementation (D&I) science frameworks in the design, execution, analysis, and dissemination of IPC trials. For example, Breitenstein et al. (2016) used RE-AIM to develop a study protocol for testing a tablet-based parenting program in primary care. Price et al. (2019) recently described how the Consolidated Framework for Implementation Research (Damschroder et al., 2009), another prominent D&I framework, can be applied to pediatric psychology research, and their suggestions are duly applicable to IPC.

Although D&I is not the focus of many trials, even studies primarily focused on efficacy likely produce more information related to external validity than what is typically reported. In particular, researchers may have access to valuable information about the children, families, clinics, and practitioners who do not participate in IPC trials. There are limitations to how much can be gleaned about non-participants, but certain non-identifiable data may be readily available in aggregate via health records or other sources. Patient sex, age, and insurance status at a minimum are likely ascertainable in most electronic health record systems, and can be used to determine the representativeness of study samples. Researchers should report how they identify potential research sites and what proportion agreed to participate. For example, Perrin et al. (2014) reported inviting all 43 practices within 60 min of their location with at least 6 PCPs, resulting in 22 responses and 12 agreements to participate. Such data are known to researchers in most cases and provide important context for evaluating the potential scalability of interventions. Comparing the relevant characteristics of adopting and non-adopting clinics (e.g., number of PCPs, operating budget, payer mix, etc.) should be encouraged. When reporting on the primary results of trials, researchers should consider including descriptive data on external validity factors in Materials and Methods section or as Supplementary Material. More robust findings (e.g., analysis of clinic characteristics in relation to the decision to participate in a trial) may merit stand-alone secondary publications in which external validity outcomes are the main results reported.

In addition to collecting and reporting on “low hanging” external validity data, pediatric IPC research would benefit from research designs that explicitly balance internal and external validity factors, particularly effectiveness-implementation hybrid designs (Curran et al., 2012). These designs place dual emphases on clinical outcomes and how interventions are best implemented in real world settings to varying degrees. Generally, focus on implementation methods should increase as clinical effectiveness is established, but as Curran et al. suggested, collecting information on adoption and implementation in the early stages of clinical effectiveness research is low risk and can provide crucial information.

Only a handful of the reviewed studies (Boyle et al., 2010; Kolko et al., 2012; Linville et al., 2015; McMenamy et al., 2011; Patterson et al., 2002; Richardson et al., 2009) incorporated qualitative methods to assess any of the RE-AIM domains, and in most cases qualitative procedures were ancillary or minimally described. Qualitative and mixed methods are thought to hold advantages for studying primary care, because they capture the complexity and nuance of real-world environments better than quantitative methods alone (Borkan, 2004). Use of mixed methods has grown in the broader primary care literature (Kaur et al., 2019), but remains rare in pediatric IPC trials. There are numerous reasons to employ mixed methods (Bryman, 2006), one of which is nuanced assessment of key stakeholders’ perceptions and experiences of interventions. Mockford and Barlow (2004) provide an excellent example: Via qualitative interview, they identified increased spousal conflict as an unintended negative consequences of an IPC parenting intervention. Such findings have important implications for the refinement of interventions, but may go undiscovered by pre-determined quantitative measures alone. Pediatric IPC researchers should consider mixed method designs at each stage of the intervention development process in order to uncover such insights.

Implications for IPC Clinicians

Although this review focused on research methodology, the findings hold relevance for psychologists and other professionals practicing IPC. The overall finding that IPC trials often fail to provide information that would inform successful adoption and implementation of specific interventions means that IPC clinicians must be critical consumers and translators of research for their particular practice. For example, this review calls into question the degree to which IPC interventions successfully reach their intended recipients, and also indicates that a significant portion of those who initiate treatment do not receive intended doses. Engaging patient and family stakeholders to inform the adaption of interventions to specific settings and patient populations may be an important role for IPC clinicians. Recent research has provided methods of assessing parents’ consumer preferences for the delivery of behavioral interventions in primary care (Riley et al., 2020), and engaging patient populations in this way may inform strategies to increase the reach of interventions. Limited reach and significant attrition in the reviewed studies also points to the need for IPC clinicians to address logistical and motivational barriers at the point of care via strategies like motivational interviewing and care coordination services (Godoy et al., 2019; Ingoldsby, 2010).

As Stancin (2016) noted, pediatric psychologists are well suited to lead quality improvement efforts in primary care. Our review found little evidence to guide the implementation and maintenance of child mental health interventions in primary care. In the absence of such information, IPC clinicians would likely benefit from knowledge of the broader factors that drive organizational change in primary care (Cohen et al., 2004) and specific models of quality improvement (e.g., The Model for Improvement; U.S. Department of Health and Humans Services, 2011).

Limitations

We reviewed a specific set of IPC studies, trials with participants who displayed elevated psychological symptoms, and our findings are exclusive to those studies. Interventions that are designed to prevent mental health problems, address physical health issues, and otherwise promote optimal child development are important parts of IPC practice that were outside the scope of our review. Other non-experimental research has examined factors relevant to the RE-AIM framework (e.g., Berkel et al., 2020), but did not meet the inclusion criteria for this review. We did not attempt to contact the authors of the reviewed studies, as we were primarily interested in how IPC trials are reported. It is possible that doing so would have yielded additional information about data that were collected but not reported. Although we followed recommended practices by using multiple independent reviewers, each of the reviewers was trained on the data extraction process by the senior author, who also resolved any discrepancies between coders, so it is possible the results are influenced by his biases. Our data extraction and coding system was not exhaustive, and some trials possessed important external validity features that were not coded as specific indicators. For example, a majority of studies compared two or more active treatment approaches, which generally informs clinical decision making more effectively than use of inert controls. Some studies (e.g., Asarnow et al., 2005) used sophisticated statistical methods to account for effects of selection bias. A number of studies were conducted in “real life” settings, and several utilized tailorable interventions that were designed to incorporate patient preferences and choices (e.g., Asarnow et al., 2005; Clarke et al., 2005; Kolko et al., 2014; Richardson et al., 2009). This approach better approximates clinical practice than strict adherence to treatment manuals, and provides an excellent example of sacrificing some internal validity to enhance the clinical impact of findings. So, while this review identifies some knowledge gaps, it also identifies exemplary studies on which future work should be modeled.

Conclusions

Although pediatric IPC trials demonstrate multiple strengths, most fail to report factors that are critical for determining external validity. It is not our intention to impugn research focused on efficacy; such work is critical. However, given that IPC interventions are intended to be employed in primary care settings by definition, more attention to the factors that influence the scalability of those interventions is merited. This is particularly true considering that IPC interventions are often adaptations of well-established therapies, which are known to be efficacious when delivered under research conditions. The scientific and pragmatic value of demonstrating effectiveness of IPC interventions is therefore partially yoked to demonstrating that such interventions can be widely disseminated, implemented, accessed, and maintained. The field should embrace scientific methods and reporting practices to better capture and communicate that value.

Supplementary Data

Supplementary data can be found at: https://academic.oup.com/jpepsy.

Funding

This work was supported by the Agency for Healthcare Research and Quality [K12HS022981], Health Resources and Services Administration [D40HP26865], and National Institutes of Health [UL1GM118964 and RL5GM118963]. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funders.

Conflicts of interest: None declared.