Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2024 Feb 1.
Published in final edited form as: Am Psychol. 2023 Feb-Mar;78(2):82–92. doi: 10.1037/amp0001077

Optimizing ATTAIN Implementation in a Federally Qualified Health Center Guided by the FRAME-IS

Kassandra Martinez 1,2, Elizabeth Lane 2,3, Vannia Hernandez 4, Elizabeth Lugo 4, Fatima A Muñoz 4, Timothy Sahms 4,5, Sarabeth Broder-Fingert 6, Miya Barnett 7, Nicole A Stadnick 1,2,3,8
PMCID: PMC10071441  NIHMSID: NIHMS1864179  PMID: 37011161

Abstract

Introduction:

Implementation strategies are methods or techniques used to adopt, implement, and sustain evidence-based practices (EBPs). Implementation strategies are dynamic and may require adaptation to fit implementation contexts, especially in low resource settings, which are most likely to serve racially and ethnically diverse patients. The Framework for Reporting Adaptations and Modifications to Evidence-Based Implementation Strategies (FRAME-IS) was used to document adaptations to implementation strategies to inform an optimization pilot of ATTAIN (Access to Tailored Autism Integrated Care; an integrated care model for children with autism and co-occurring mental health needs) in a federally qualified health center (FQHC) near the U.S./Mexico border.

Methods:

Quantitative and qualitative data were collected from 36 primary care providers who participated in the initial ATTAIN feasibility pilot to inform adaptations. Adaptations were mapped to the FRAME-IS through an iterative template analysis to inform an optimization pilot at a FQHC 1-year from the start of the COVID-19 pandemic.

Results:

Four implementation strategies (training and workflow reminders, provider/clinic champions, periodic reflections, and technical assistance) were employed during the feasibility pilot and were adapted for the optimization pilot to fit the needs of the FQHC and service delivery changes prompted by the pandemic.

Conclusions:

Findings demonstrate the utility of using the FRAME-IS to systematically inform EBP optimization in a FQHC providing care to underserved communities. Findings will inform future research studies implementing integrated mental health models in low resourced primary care settings. Implementation outcomes and provider perceptions of ATTAIN at the FQHC are also reported.

Keywords: autism spectrum disorder, co-occurring mental health, integrated care, adaptations, implementation strategies, health equity

School-aged children on the autism spectrum (clinically referred to as Autism Spectrum Disorder or ASD) experience high rates of co-occurring mental health needs (Brookman-Frazee et al., 2018; Joshi et al., 2010; Tye et al., 2019). However, this population faces challenges in accessing and engaging in specialty mental health services (Maddox et al., 2021). Ethnic disparities in access-to and quality-of specialized treatment services for Latino autistic youth, specifically, are well documented (Magaña et al., 2012, 2013). An integrated care approach, such as primary care providers and mental health specialists collaborating to identify and facilitate access to mental health services, is a potential solution to this challenge. Pediatric primary care is well-suited to address these concerns, as most youth visit their primary care physician at least annually (Furuta et al., 2012; Malow et al., 2012).

There are several challenges to consider when implementing integrated care for autistic youth in primary care settings. First, the structure of primary care workflows limits the opportunity to address mental health concerns appropriately, as most pediatric primary care visits only last between 11–20 minutes (Halfon et al., 2011). Second, while healthcare providers and parents of autistic youth have demonstrated support for pediatric primary care as an appropriate place for management of ASD-associated conditions, they report that primary care settings lack ASD-specific practices and policies to facilitate this type of care (Morris et al., 2019). Third, primary care providers report limited knowledge about efficient referral pathways and effective mental health referral sources for their autistic patients and have stated difficulty identifying symptoms beyond ASD that require attention from mental health providers (e.g., anxiety, depression, attention difficulties; Stadnick et al., 2022b).

To address these challenges, Access To Tailored Autism INtegrated Care (ATTAIN) was co-designed with implementation practitioners and researchers for use in pediatric primary care settings (Stadnick et al., 2019). ATTAIN is a customizable integrated care model designed to enhance mental health screening and linkage to services for autistic youth at-risk for co-occurring mental health conditions. ATTAIN was informed by a formative mixed-methods contextual assessment that highlighted the need for a tailored approach to facilitating the connection to mental health services for autistic youth and proactively incorporating organizational capacity considerations to support implementation (Stadnick et al., 2020).

ATTAIN includes 8–9 general steps that are initiated during pediatric primary care visits for children with an ASD diagnosis, ages 4–16 years old, and whose caregiver speaks Spanish or English. The process involves administering, scoring, and interpreting results from the Pediatric Symptom Checklist-17 (PSC-17; Gardner et al., 1999), offering a referral and specific pathway to mental health services (if PSC-17 score is elevated and family expresses interest), and documenting the referral in the electronic medical record using standardized language. An identified staff member, such a case manager or call center representative, subsequently contacts the family on a set frequency to support scheduling the child’s mental health appointment and updates the referring primary care provider about the status of a scheduled or attended mental health appointment. (Stadnick, Aarons, Martinez, Sklar, et al., 2022a).

Opportunity to Optimize ATTAIN for a Federally Qualified Health Center

A mixed-method concurrent exploratory quantitative + qualitative design (Palinkas, 2014) was used to assess the feasibility of ATTAIN in six pediatric primary care clinics in Southern California across two large primary care organizations (i.e., feasibility pilot; Stadnick, Aarons, Martinez, Sklar, et al., 2022a). The feasibility pilot of ATTAIN was conducted in 2020 during the start of the COVID-19 pandemic. A clinic from a Federally Qualified Health Center (FQHC) serving a predominantly Latino population at the U.S./Mexico borer was originally enrolled in the feasibility pilot but needed to temporarily discontinue their participation due to COVID-19 pandemic impacts.

At the start of 2021, and after the conclusion of the feasibility study, our team re-engaged our FQHC partners to conduct an optimization pilot of ATTAIN. This re-engagement presented the opportunity to capitalize on the feasibility pilot findings regarding implementation barriers and facilitators to refine implementation strategies for this FQHC setting.

Informing and Documenting Implementation Strategies: A Stakeholder-Engaged Process

The growing field of implementation science examines how programs and interventions are used in real-world settings and seeks to develop tools or strategies to facilitate the implementation and sustainment of evidence-based practices in routine care settings (Lobb & Colditz, 2013). Implementation strategies are methods or techniques used to adopt, implement, and sustain evidence-based practices; (Powell et al., 2015). Implementation strategies are dynamic and may require adaptation to fit implementation contexts (Miller et al., 2021). This is especially true for low-resourced settings, such as FQHCs. FQHCs are a vital source of healthcare for underserved communities, such as those who are insured via Medicaid, are of low socioeconomic status, and/or have limited English proficiency or health literacy (National Association of Community Health Centers, 2016). The COVID-19 pandemic highlighted the exacerbated needs and disparities that communities accessing services at FQHCs encounter, thus necessitating greater attention to implementation strategy selection and use (Centers for Disease Control and Prevention, 2021). Given the unique implementation context factors specific to FQHCs and the downstream effects of the COVID-19 pandemic on ATTAIN implementation during the feasibility pilot (2020), there was a need to take a systematic and stakeholder-engaged approach to modifying ATTAIN implementation strategies for the FQHC optimization pilot (2021).

In implementation science, guidelines, methods, and frameworks are used to systematically track implementation processes and to inform adaptations (i.e., thoughtful and deliberate changes; Wiltsey Stirman et al., 2017) to evidence-based practices and/or implementation strategies to improve fit and ultimately improve implementation efforts. Increased attention has been given to documenting and reporting adaptations to evidence-based practices (EBPs), with frameworks being utilized in the literature, such as the Framework for Reporting Adaptations and Modifications to Evidence-Based Practices (FRAME; Wiltsey Stirman et al., 2019). However, less work has focused on documenting and reporting adaptations to implementation strategies (Miller et al., 2021).

The Framework for Reporting Modifications to Evidence-Based Implementation Strategies (FRAME-IS) is a novel framework for systematically informing and documenting adaptations to implementation strategies and it is an ideal framework for tracking these adaptations given that is flexible and practical. For example, the FRAME-IS includes core and supplementary modules to document adaptations that can be mapped onto the FRAME-IS to classify what is being adapted, the nature of the specific adaptations, and the goals for each adaptation (Miller et al., 2021). See Figure 1 for a visual representation of the FRAME-IS. Please note that the current study only utilized the core modules of the FRAME-IS.

Figure 1.

Figure 1.

Open in a new tab

Case Example: Utilizing the FRAME-IS Core Modules

Note. Figure adapted from Miller et al., 2021 (published in Open Access Journal).

ATTAIN Feasibility Pilot (2020) Implementation Strategies and Outcomes

Three primary implementation strategies were used to facilitate the adoption of ATTAIN during the feasibility pilot (2020). Implementation strategies were selected at the outset of the feasibility pilot study (2020). For the first strategy, Provider/Clinic Champions, a physician or clinic staff member was identified as the primary point of contact within the participating clinic and agreed to be in contact with the research team every two weeks to discuss barriers, facilitators, and adaptations. For the second strategy, Periodic Reflections (Finley et al., 2018), a member of the research team contacted the provider/clinic champion every two weeks via phone or email to assess the implementation process and to identify technical assistance needs. For the third strategy, Technical Assistance, the research team provided technical assistance via bi-weekly group provider emails with fidelity tips and answers to commonly asked questions.

Results of the feasibility pilot indicated significant gains in provider perceived knowledge and confidence caring for autistic children with co-occurring mental health needs in primary care, though no changes in comfort were demonstrated. Ratings of feasibility and acceptability of ATTAIN were high and providers reported slight to moderate agreement with intentions to continue ATTAIN implementation. Reach (i.e., percentage of eligible patients who were screened) was between 53–55%. Several opportunities for refinement of ATTAIN were documented and reported in Stadnick et al., 2022a.

Study Objectives: ATTAIN Optimization Pilot (2021)

Given the unique implementation context factors specific to the FQHC and the downstream effects of the COVID-19 pandemic on ATTAIN implementation during the feasibility pilot (2020), the current study’s objectives were to (1) engage in a community-partnered process to systematically adapt ATTAIN implementation strategies for the optimization pilot of ATTAIN using the FRAME-IS, and to (2) examine implementation outcomes and provider perceptions of ATTAIN at the FQHC.

Method

Study Context

The ATTAIN optimization pilot (2021) was conducted at a FQHC serving culturally and linguistically diverse communities along the US/Mexico border. The FQHC did not have an existing integrated mental health care model for autistic youth, but they were building their care coordination program for pediatric patients to maintain their patient-centered medical home certification. PCPs at the FQHC completed a 1-hour training on ATTAIN before the start of the optimization pilot period, which lasted 5 weeks. In addition, medical assistants were trained to identify and flag eligible patients prior to or at the start of patient visits.

Objective 1: Systematically Adapt ATTAIN Implementation Strategies for the FQHC

The primary data source for the adaptations used in the FQHC optimization pilot (2021) were from qualitative data collected from PCP exit interviews in the feasibility pilot study (2020). Group and individual semi-structured interviews were conducted by a doctoral research assistant (KM) and staff research assistant with a subset (n=16) of PCPs who participated in the ATTAIN feasibility pilot (2020) to gather targeted recommendations for ATTAIN implementation adaptations (i.e., implementation determinants and recommendations for refinement). The interview guide used for these interviews was developed to probe for implementation determinants and recommendations for ATTAIN refinement based on relevant domains within the Implementation and Sustainment phases of the Exploration, Preparation, Implementation, Sustainment Framework (EPIS; Aarons et al., 2011; Moullin et al., 2019). The EPIS framework facilitates evaluation of efforts to implement and sustain an EBP or innovation within an implementation context. Examples of interview questions included: “With regards to ATTAIN, what has not been going well so far?” and “What kinds of changes or work-arounds have you had to make to ATTAIN so it could work effectively in your practice?”

Interviews were conducted using on Zoom and were audio recorded and transcribed using a professional service. Rapid qualitative assessment methods (Hamilton, 2013; Hamilton, 2020; Nevedal et al., 2021) were used to identify themes. Specifically, a templated matrix (Averill, 2002) was developed that included summary responses from each interview divided by the question posed from the interview guide. Next, two members of the research team (KM & NS) iteratively reviewed the summaries across interviews and by interview question to identify themes. In addition, the implementation team (NS, KM, EL) met to review the team’s process notes to identify additional themes. All themes were consolidated to identify needed adaptations that were mapped onto the FRAME-IS core modules to classify what was proposed to be adapted, the nature of the specific adaptation, and the goals for each adaptation.

Objective 2: Examine Implementation Outcomes and Provider Perceptions of ATTAIN

Survey data were collected post-pilot from participants (n=4 PCPs) in the optimization pilot (2021). Additionally, participating PCPs self-reported mental health screening, referral, and linkage of their ATTAIN-eligible patients (i.e., 4–16 years old, ASD diagnosis documented in Electronic Health Record [EHR]) and these data were triangulated with aggregated electronic health record data. Specifically, research staff sent weekly reports to participating providers to record the number of eligible patients seen, screened, and their PSC-17 results. The research team verified these numbers during meetings with the Care Coordination and Behavioral Health partners.

Demographics and Professional Background.

Participants completed a demographic and professional background form that included questions about age, sex, race, provider type, years of healthcare experience, and number of patients seen per day. Fifty percent of participating PCPs identified as female (n = 2), while the other 50% identified as male (n = 2). Most PCPs (n = 3; 75%) were white and none of them identified as Hispanic/Latino. Table 2 provides additional demographic and professional background information for participating PCPs.

Table 2.

Demographic and Professional Characteristics of Participating Providers

PCPs
(N=4)
Characteristic n %
Sex
 Female 2 50
 Male 2 50
Race
 Asian American 1 25
 White/Caucasian 3 75
Provider Type
 Nurse Practitioner 1 25
 Medical Doctor 3 75
Years at Organization
 3–10 years 1 25
 >10 years 3 75
Patients Per Day
 11–15 1 25
 16–20 3 75
Open in a new tab
Feasibility and Acceptability.

The Feasibility of Intervention Measure and the Acceptability of Intervention Measure (FIM & AIM, respectively; Weiner et al., 2017) each include 4-items that measure the extent to which participants perceive an intervention as acceptable, appropriate, and feasible in their care setting. Respondents rate their agreement with each item on a Likert scale ranging from 0 (“disagree”) to 4 (“completely agree”). A score for each measure is calculated by averaging responses within each measure.

The Perceived Characteristics of Intervention Scale (PCIS; Cook et al., 2015) was used to assess providers perspectives regarding feasibility and acceptability of ATTAIN post optimization pilot. The PCIS is an 18-item scale that assesses attitudes towards a specific intervention including relative advantage, compatibility, and complexity. The PCIS yields an average score and ten subscale scores. Participants rate the extent to which they agree with each item on a 5-point Likert scale, with 0 indicating “not at all” and 4 indicating “to a very great extent”. For the current study, only the average score was used due to the small sample size and because the internal consistency across items was high (α= .94).

Provider Intentions to Continue ATTAIN.

The Measure of Innovation-Specific Implementation Intentions (MISII; Moullin et al., 2018) was used to assess providers’ intentions to continue using ATTAIN post-pilot. The MISII includes three items; participants are asked to rate their agreement to each statement on a 5-point Likert scale, with 0 indicating “not at all,” and 4 indicating “to a very great extent.” Items include: (1) “I plan to use ATTAIN with my patients”, (2) “Using ATTAIN is a high priority for me”, and (3) “I will use all aspects of ATTAIN with my patients.” An average score was obtained from the three items.

Reach, Adoption, and Service Outcomes.

ATTAIN reach and adoption were measured by extracting data from the EHR and by reviewing provider’s self-reported screening rates. Reach was defined as the percentage of ATTAIN-eligible patients (i.e., children with a documented ASD diagnosis) who were screened using the PSC-17. Adoption was defined as the percentage of trained ATTAIN providers who used ATTAIN with at least one patient. Service outcomes were assessed via documentation of successful completion of a mental healthcare appointment.

Data Analysis.

ATTAIN reach, adoption, and post-pilot provider ratings of implementation experiences were analyzed using descriptive statistics.

Results

Objective 1: Systematically Adapt ATTAIN Implementation Strategies for the FQHC

Qualitative analyses of provider focus groups and interviews conducted after the feasibility pilot (2020) outlined areas for implementation refinement. Providers reported needs for additional support in identifying and flagging eligible patients, multiple clinic champions and the integration of the PSC-17 into the EHR as a way of facilitating the administration, scoring, and tracking of mental health screening. Additionally, the implementation team had challenges with the broader engagement of the clinic team (e.g., front office staff, medical assistants, nurses), coordinating communication across provider teams, and engaging individual providers via weekly, group emails.

To optimize ATTAIN for the FQHC (i.e., optimization pilot), and guided by the FRAME-IS, original implementation strategies were modified to include: (1) a multidisciplinary team of clinic champions, (2) weekly periodic reflections with the champion team, (3) virtual training, (4) weekly personalized outreach to participating providers to provide workflow reminders and technical assistance, and (5) automation of workflow procedures into the EHR.

Champions were expanded to include a wider team, which included medical assistants, nurses, and care coordinators based on initial pilot findings that indicated the need to engage the broader care team, particularly due to changes in workflows and communications due to the pandemic. The structure of periodic reflections changed in the form of scheduled, weekly 15-minute Zoom calls with the expanded champion team to expedite coordinating communication across provider teams (i.e., pediatric, care coordination, and psychiatry providers) and to identify implementation problems. Training was provided to the wider champion team via Zoom rather than in person, due to COVID-19 safety guidelines. Technical assistance changed to weekly, personalized outreach to PCPs about their pilot implementation progress (e.g., number of patients screened, screening results), replacing mass provider emails with fidelity tips. Finally, automating workflow procedures in the EHR was added as an implementation strategy to facilitate administration, scoring, and tracking of mental health screening. Specifically, providers had the ability to administer and score the PSC-17 in the EHR, and automated language (i.e., a Smart Phrase) was created for the provider to include in the visit note and After Visit Summary with information about the PSC-17, the score, interpretation of the score, and recommended next steps for the patient. Additionally, the use of ATTAIN was expanded to include telehealth, in-person, and phone visits (well-child and sick visits) rather than solely in-person, well-child visits. This was requested by the FQHC team due to the increased delivery of telehealth and to better reach eligible children. Table 3 summarizes adaptations, reasons for the adaptations, and the goals of the adaptations.

Table 3.

Summary of Implementation Strategy Modifications Guided by the FRAME-IS

Implementation Strategy Adaptation and Reason Goal of Adaptation
1. Provider/Clinic Champions Tailored/tweaked/refined implementation strategy context/personnel to include a wider group of clinic champions which included PCPs, medical assistants, nurses, and care coordinators. To increase adoption and reach of ATTAIN by facilitating organizational engagement.
2. Periodic Reflections Tailored/tweaked/refined implementation strategy population and setting to include 15-minute, virtual meetings with wider champion team. To increase adoption and reach of ATTAIN by facilitating and expediting communications across provider teams and identifying implementation problems.
3. Training & Workflow Reminders Substituted in-person training for virtual training and provided training to wider champion team. To increase acceptability, appropriateness, and fit by adhering to COVID-19 safety guidelines.
To increase adoption and reach of ATTAIN by facilitating wider provider and staff engagement.
4. Technical Assistance Substituted mass provider emails for weekly personalized outreach to PCPs with added data collection templates and individualized progress reports. To increase adoption and reach by facilitating engagement.
5. Automating Workflow Procedures Integrated PSC-17 administration and scoring onto EHR.
Expanded use of ATTAIN to include telehealth, in-person, and phone visits.		 To increase adoption and reach by facilitating administration, scoring, and tracking of mental health screening.
To increase acceptability, appropriateness, and fit by adhering to COVID-19 safety guidelines, increases in telehealth delivery, and to expand reach of eligible patients.
Open in a new tab

Note. Strategies 1–3 were adapted to enhance implementation in the FQHC; Strategies 4 and 5 were added to enhance implementation in the FQHC. PCPs = primary care providers; EHR = electronic health record.

Objective 2: Examine Implementation Outcomes and Provider Perceptions of ATTAIN at the FQHC

Eighteen eligible patients were identified via provider-report and confirmed with EHR records review. All eligible patients (n = 18) were screened using the PSC-17, which indicates a reach rate of 100%. Of the patients screened, 17 (95%) screened positive (i.e., cutoff Total score of 15 or higher). Of those who screened positive, 9 (53%) were already receiving care through the FQHC’s Behavioral Health department. The remaining 8 patients (47%) were referred for care coordination, which resulted in 100% of these patients scheduling and attending a behavioral health appointment. Figure 2 provides a visual representation of ATTAIN screening, uptake, and service outcomes. Adoption (i.e., percentage of trained providers who used ATTAIN with at least one patient) was 100%.

Figure 2.

Figure 2.

Open in a new tab

ATTAIN Uptake, Screening Results, and Service Outcomes

Note. PSC-17 = Pediatric Symptom Checklist-17; CC = care coordination; BH = behavioral health.

Post-pilot, providers rated ATTAIN as feasible (FIM; M=3.6 out of 4, SD=0.4) and acceptable (AIM: M=3.7 out of 4, SD=0.5). Providers rated their intentions to continue using ATTAIN as high (MISII; M=3.0 out of 4, SD=0.3) and reported agreement towards specific components of ATTAIN to a “moderate extent” (PCIS; M=1.8 out of 4, SD=1.1). The five highest-rated items (i.e., indicating higher agreement) on the PCIS were: “ATTAIN is aligned with my clinical judgement” (M = 2.25, SD = 0.50); “ATTAIN is more effective than other care approaches I have used” (M = 2.00, SD = 0.82); “ATTAIN is easy to use” (M = 2.00, SD = 0.82); “The knowledge required to learn ATTAIN can be effectively taught” (M = 2.00, SD = 0.82); and “The knowledge required to implement ATTAIN can be effectively taught” (M = 2.00, SD = 0.82). The three lowest-rated items (i.e., indicating lower agreement) on the PCIS were: “ATTAIN has helpful supportive materials for patients and their families” (M = 0.50, SD = 1.00); “ATTAIN is more convenient than other care approaches I have used” (M = 0.75; SD = 0.96); and “It is easy to tell whether patients are benefiting from ATTAIN” (M = 0.75, SD = 0.96).

Discussion

This study utilized a novel implementation framework, the FRAME-IS, to systematically inform adaptations to implementation strategies of ATTAIN, a customizable integrated mental health care model for autistic youth at-risk for co-occurring mental health conditions. The reasons for these adaptations were to facilitate an optimization pilot conducted in a FQHC that was previously unable to complete their participation in the ATTAIN feasibility study due to COVID-19 impacts on their service delivery.

Using qualitative data from an earlier feasibility pilot study (2020), we were able to use the FRAME-IS to identify and subsequently implement modified implementation strategies for the FQHC optimization pilot (2021). Modified implementation strategies used for the optimization pilot included: (1) a multidisciplinary team of clinic champions, (2) weekly periodic reflections with the champion team, (3) virtual training, (4) weekly personalized outreach to participating providers to provide workflow reminders and technical assistance, and (5) automation of workflow procedures into the electronic health record. Reasons for adaptations centered around engaging a wider team of primary care staff (i.e., primary care providers, medical assistants, care coordinators), adhering to COVID-19 safety guidelines, and enhancing workflow procedures. The goals of the adaptations were generally to increase the adoption and reach of ATTAIN, as well as to enhance acceptability, appropriateness, and fit at the FQHC during the COVID-19 pandemic.

In addition, the current study reports findings from PCPs regarding the use of ATTAIN in the FQHC. Overall, post-pilot ratings indicated favorable perceptions of the feasibility, acceptability, and intentions to continue using ATTAIN. ATTAIN reach was high in this FQHC setting, with 100% of identified patients screened using the PSC-17 and 100% of the patients who screened in the clinical range referred to care coordination or behavioral health services.

The PCP perceptions of ATTAIN are similar to those of PCPs in the original ATTAIN pilot (Stadnick, Aarons, Martinez, Sklar, et al., in press). What is striking is the descriptively higher reach of ATTAIN in this FQHC context (100%) compared to the adoption rates from participating clinics in the feasibility pilot. Although several limitations that we note below must be acknowledged and may have impacted this finding, our process of using the FRAME-IS to optimize for this FQHC setting is promising.

FQHCs are often described as having limited resources (e.g., low staffing to patient ratios, smaller budgets for clinical operations) that make it difficult to participate in research or consider integrating new innovations or EBPs (Friedberg et al., 2017; Kramer et al., 2017). FQHCs represent a critical point of care for safety net populations and require thoughtful, stakeholder-engaged, and strengths-based approaches to facilitate EBP implementation and sustainment. For example, within this study’s FQHC implementation context, we were able to leverage the ongoing and focused initiatives to expand care coordination to pediatric patients. This FQHC has successfully maintained their National Committee for Quality Assurance recognition as a patient-centered medical home due, in part, to their care coordination program. ATTAIN was a natural and fortuitous opportunity to facilitate implementation and uptake of pediatric care coordination. Next steps include scaling and sustaining ATTAIN at this FQHC through ongoing refresher trainings and bi-weekly implementation review meetings with the clinic champions.

This study has several strengths. First, it offers real-world evidence of the acceptability and feasibility of using ATTAIN in a FQHC. Second, findings demonstrate that implementation frameworks, such as the FRAME-IS, can serve to optimize future research studies implementing integrated health models in diverse pediatric primary care settings, thereby enhancing equity for patients, while also engaging the key stakeholders in the adaptation process.

Balanced with these strengths are limitations. First, the sample size of our optimization pilot was small, and information about provider’s screening practices before the optimization pilot were not available, precluding definitive conclusions about ATTAIN implementation at this FQHC. Second, it is possible that the delayed start of this optimization pilot that occurred one year after the start of the COVID-19 pandemic offered an advantage for implementation because providers and staff had adapted to workflow and service delivery changes that the pandemic required. Third, the data sources were primarily provider self-report, which has inherent bias limitations. However, we were able to confirm the screening data based on EHR record review, which allowed for triangulation of the adoption rate data.

FQHCs represent an important healthcare safety net for underserved communities nationally (National Association of Community Health Centers, 2016). Research partnerships with FQHCs to adapt and tailor implementation efforts for their unique, and often under-resourced environments, can meaningfully impact the quality of care that vulnerable families receive. However, there have been concerns that implementation efforts could increase disparities (e.g., underrepresentation of marginalized communities in clinical trials informing the evidence-based interventions used in implementation studies [Santiago et al., 2014], implementation effort fails due to limited implementation resources [Yapa & Bärnighausen, 2018]). The current study demonstrated the importance and utility of tailoring implementation strategies for equity in low-resourced settings, such as a FQHC.

Future directions for the field suggest practical utility in using the FRAME-IS for documenting, reporting, and applying needed adaptations to implementation strategies in FQHC settings. Future work should identify how best to tailor implementation strategies to enhance reach, adoption, and effectiveness in FQHs. Additionally, work may be needed to understand the timing and frequency of applying the FRAME-IS to support likely evolution and changing resources of context in FQHC settings that may affect implementation activities.

Table 1.

Access to Tailored Autism Integrated Care (ATTAIN) Workflow

Step Completed By Description
1 MA Confirm patient eligibility (ASD diagnosis and 4–16 years old) in medical record/chart.
2 MA and caregiver Accompanying caregiver completes the PSC-17 to determine whether clinically concerning mental health symptoms are present.
3 MA Score PSC-17 and document in patient’s electronic medical record.
4 PCP For patients with an elevated PSC-17 score (≥15), discuss the elevated score using “ATTAIN (ASD + MH) Information Sheet” (a 1-page psychoeducational communication aid)
5 PCP Offer a referral to mental health services.
6 PCP Document referral and family’s acceptance or declination of referral.
7 Pediatric Care Coordinator Contact the family on a set frequency to support scheduling mental health appointment.
8 Pediatric Care Coordinator and Referred Family Schedule and attend mental health appointment.
9 Pediatric Care Coordinator Send a confirmation to referring provider when mental health appointment is scheduled or attended.
Open in a new tab

Note. MA=medical assistant; PCP = primary care provider; ASD = autism spectrum disorder; PSC-17 = Pediatric Symptom Checklist; MH = mental health.

Public Significance Statement.

Implementation of evidence-based practices in low-resourced settings often fail. Engaging stakeholders to fit implementation strategies for context can enhance the uptake of evidence-based practices in low-resourced settings. This study used a newly enhanced implementation science framework (FRAME-IS) to systematically adapt implementation strategies for an optimization pilot of ATTAIN, an integrated mental health care model for children with autism, receiving care at a Federally Qualified Health Center. Results suggest favorable provider perceptions of ATTAIN, high screening rates, and linkage to mental health services. The FRAME-IS served as a pragmatic, stakeholder-engaged tool that may facilitate optimization and implementation of integrated health models in diverse pediatric primary care settings.

Funding Information:

National Institute of Mental Health

K23MH110602

Awarded to Nicole A. Stadnick

References

  1. Aarons GA, Hurlburt M, & Horwitz SM (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23. 10.1007/s10488-010-0327-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Averill JB (2002). Matrix analysis as a complementary analytic strategy in qualitative inquiry. Qualitative Health Research, 12(6), 855–866. [DOI] [PubMed] [Google Scholar]
  3. Brookman-Frazee L, Stadnick N, Chlebowski C, Baker-Ericzén M, & Ganger W (2018). Characterizing psychiatric comorbidity in children with autism spectrum disorder receiving publicly funded mental health services. Autism, 22(8), 938–952. 10.1177/1362361317712650 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Brookman-Frazee L, Roesch S, Chlebowski C, Baker-Ericzen M, & Ganger W (2019). Effectiveness of training therapists to deliver an individualized mental health intervention for children with ASD in publicly funded mental health services: A cluster randomized clinical trial. JAMA Psychiatry, 76(6), 574–583. 10.1001/jamapsychiatry.2019.0011 [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Centers for Disease Control and Prevention. (n.d.). Health equity considerations and racial and ethnic minority groups. Centers for Disease Control and Prevention. Retrieved December 3, 2021, from https://www.cdc.gov/coronavirus/2019-ncov/community/health-equity/race-ethnicity.html. [Google Scholar]
  6. Cook JM, Thompson R, & Schnurr PP (2015). Perceived characteristics of intervention scale: development and psychometric properties. Assessment, 22(6), 704–714. 10.1177/1073191114561254 [DOI] [PubMed] [Google Scholar]
  7. Finley EP, Huynh AK, Farmer MM, Bean-Mayberry B, Moin T, Oishi SM, Moreau JL, Dyer KE, Lanham HJ, Leykum L, . & Hamilton AB (2018). Periodic reflections: A method of guided discussions for documenting implementation phenomena. BMC Medical Research Methodology, 18(1), 1–15. 10.1186/s12874-018-0610-y [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Friedberg MW, Reid RO, Timbie JW, Setodji C, Kofner A, Weidmer B, & Kahn K (2017). Federally qualified health center clinicians and staff increasingly dissatisfied with workplace conditions. Health Affairs, 36(8), 1469–1475. 10.1377/hlthaff.2017.0205 [DOI] [PubMed] [Google Scholar]
  9. Furuta GT, Williams K, Kooros K, Kaul A, Panzer R, Coury DL, & Fuchs G (2012). Management of constipation in children and adolescents with autism spectrum disorders. Pediatrics, 130(Supplement_2), S98–S105. [DOI] [PubMed] [Google Scholar]
  10. Gardner W, Murphy M, Childs G, Kelleher K, Pagano M, Jellinek M, … & Chiappetta L (1999). The PSC-17: A brief pediatric symptom checklist with psychosocial problem subscales. A report from PROS and ASPN. Ambulatory Child Health, 5, 225–236. [Google Scholar]
  11. Ghaderi G, & Watson SL (2019). “In medical school, you get far more training on medical stuff than developmental stuff”: Perspectives on ASD from Ontario physicians. Journal of Autism and Developmental Disorders, 49(2), 683–691. 10.1007/s10803-018-3742-3 [DOI] [PubMed] [Google Scholar]
  12. Halfon N, Stevens GD, Larson K, & Olson LM (2011). Duration of a well-child visit: association with content, family-centeredness, and satisfaction. Pediatrics, 128(4), 657–664. [DOI] [PubMed] [Google Scholar]
  13. Hamilton A (2020). Rapid qualitative analysis: Updates/developments. Health Services Research & Development. https://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/video_archive.cfm?SessionID=3846 [Google Scholar]
  14. Hamilton A (2013). Qualitative methods in rapid turn-around health services research. Health Services Research & Development. https://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/780-notes.pdf [Google Scholar]
  15. Joshi G, Petty C, Wozniak J, Henin A, Fried R, Galdo M, Kotarski M, Walls S, & Biederman J (2010). The heavy burden of psychiatric comorbidity in youth with autism spectrum disorders: A large comparative study of a psychiatrically referred population. Journal of Autism and Developmental Disorders, 40(11), 1361–1370. 10.1007/s10803-010-0996-9 [DOI] [PubMed] [Google Scholar]
  16. Kramer TL, Drummond KL, Curran GM, & Fortney JC (2017). Assessing culture and climate of federally qualified health centers: A plan for implementing behavioral health interventions. Journal of Health Care for the Poor and Underserved, 28(3), 973–987. 10.1353/hpu.2017.009 [DOI] [PubMed] [Google Scholar]
  17. Lobb R, & Colditz GA (2013). Implementation science and its application to population health. Annual Review of Public Health, 34, 235. 10.1146/annurev-publhealth-031912-114444 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Maddox BB, Dickson KS, Stadnick NA, Mandell DS, & Brookman-Frazee L (2021). Mental health services for autistic individuals across the lifespan: Recent advances and current gaps. Current Psychiatry Reports, 23(10), 1–9. 10.1007/s11920-021-01278-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Malow BA, Byars K, Johnson K, Weiss S, Bernal P, Goldman SE, … & Glaze DG (2012). A practice pathway for the identification, evaluation, and management of insomnia in children and adolescents with autism spectrum disorders. Pediatrics, 130(Supplement_2), S106–S124. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Magaña S, Lopez K, Aguinaga A, & Morton H (2013). Access to diagnosis and treatment services among Latino children with autism spectrum disorders. Intellectual and Developmental Disabilities, 51(3), 141–153. 10.1352/1934-9556-51.3.141 [DOI] [PubMed] [Google Scholar]
  21. Magaña S, Parish SL, Rose RA, Timberlake M, & Swaine JG (2012). Racial and ethnic disparities in quality of health care among children with autism and other developmental disabilities. Intellectual and Developmental Disabilities, 50(4), 287–299. 10.1352/1934-9556-50.4.287 [DOI] [PubMed] [Google Scholar]
  22. Miller CJ, Barnett ML, Baumann AA, Gutner CA, & Wiltsey-Stirman S (2021). The FRAME-IS: A framework for documenting modifications to implementation strategies in healthcare. Implementation Science, 16(1), 1–12. 10.1186/s13012-021-01105-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Morris R, Greenblatt A, & Saini M (2019). Healthcare providers’ experiences with autism: a scoping review. Journal of Autism and Developmental Disorders, 49(6), 2374–2388. [DOI] [PubMed] [Google Scholar]
  24. Moullin JC, Ehrhart MG, & Aarons GA (2018). Development and testing of the Measure of Innovation-Specific Implementation Intentions (MISII) using Rasch measurement theory. Implementation Science, 13(1), 1–10. 10.1186/s13012-018-0782-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. National Association of Community Health Centers (n.d.). A sketch of community health centers: Chartbook, August 2016. National Association of Community Health Centers. Retrieved December 3, 2021, from http://nachc.org/wp-content/uploads/2016/08/Chartbook16.pdf. [Google Scholar]
  26. Nevedal AL, Reardon CM, Opra Widerquist MA, Jackson GL, Cutrona SL, White BS, & Damschroder LJ (2021). Rapid versus traditional qualitative analysis using the Consolidated Framework for Implementation Research (CFIR). Implementation Science, 16(1), 1–12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Palinkas LA (2014). Qualitative and mixed methods in mental health services and implementation research. Journal of Clinical Child and Adolescent Psychology, 43(6), 851–861. 10.1080/15374416.2014.910791 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Santiago CD, & Miranda J (2014). Progress in improving mental health services for racial-ethnic minority groups: A ten-year perspective. Psychiatric Services, 65(2), 180–185. [DOI] [PubMed] [Google Scholar]
  29. Stadnick NA, Aarons GA, Martinez K, Sklar M, Coleman KJ, Gizzo DP, Lane E, Kuelbs CL, & Brookman-Frazee L (2022a). Implementation outcomes from a pilot of “Access to Tailored Autism Integrated Care” for children with autism and mental health needs. Autism. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Stadnick NA, Brookman-Frazee L, Mandell DS, Kuelbs CL, Coleman KJ, Sahms T, & Aarons GA (2019). A mixed methods study to adapt and implement integrated mental healthcare for children with autism spectrum disorder. Pilot and Feasibility Studies, 5 (51). 10.1186/s40814-019-0434-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Stadnick NA, Martinez K, Aarons GA, Lee DA, Van Cleave J, & Brookman-Frazee L (2020). Pediatric primary care perspectives on integrated mental health care for autism. Academic Pediatrics, 20(8), 1140–1147. 10.1016/j.acap.2020.03.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Stadnick NA, Martinez K, Coleman KJ, Gizzo DP, Lane E, Lee N, Kuelbs CL, Aarons GA, & Brookman-Frazee L (2022b). Short report: Mental health screening in pediatric primary care for children with autism. Autism. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Stadnick NA, Penalosa MG, Martinez K, Brookman-Frazee L, Gizzo DP, Sahms T, Kuelbs C, & Aarons GA (2021). Pre-implementation organizational environment associated with pediatric integrated care readiness in primary care. Evidence-Based Practice in Child and Adolescent Mental Health, 1–7. 10.1080/23794925.2021.1875344 [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Stirman SW, Baumann AA, & Miller CJ (2019). The FRAME: An expanded framework for reporting adaptations and modifications to evidence-based interventions. Implementation Science, 14(1), 1–10. 10.1186/s13012-019-0898-y [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Tye C, Runicles AK, Whitehouse A, & Alvares GA (2019). Characterizing the interplay between autism spectrum disorder and comorbid medical conditions: An integrative review. Frontiers in Psychiatry, 9. 10.3389/fpsyt.2018.00751 [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, Boynton MH, & Halko H (2017). Psychometric assessment of three newly developed implementation outcome measures. Implementation Science, 12(1), 108. 10.1186/s13012-017-0635-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Wiltsey Stirman S, Gamarra JM, Bartlett BA, Calloway A, & Gutner CA (2017). Empirical examinations of modifications and adaptations to evidence-based psychotherapies: Methodologies, impact, and future directions. Clinical Psychology: Science and Practice, 24(4), 396. 10.1111/cpsp.12218 [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Yapa HM, & Bärnighausen T (2018). Implementation science in resource-poor countries and communities. Implementation Science, 13(1), 1–13. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES