Abstract
Aim:
Family interventions are a core component of first-episode psychosis (FEP) treatment; however, low implementation rates are consistently reported. As such, work is needed to understand the factors impacting real-world treatment delivery. The present paper describes the implementation of the McFarlane-model multifamily psychoeducational groups (MFG) in established FEP early intervention programs within a single state. The aims were to examine: (1) training participation and implementation of MFG, (2) barriers and facilitators to implementation, and (3) modifications made to MFG.
Methods:
Practitioners from six established FEP early intervention programs received in-person training and ongoing consultation in MFG. Training participation data were obtained via attendance and implementation outcomes were obtained from practitioner reports. Fifteen months following the initial training, practitioners reported on clinic-specific barriers, facilitators, and modifications across four categories (context, intervention, practitioner, and recipient).
Results:
Twenty-three practitioners across six clinics received in-person training and were offered ongoing consultation to support implementation. Difficulties in starting MFG were salient as the earliest group was run seven months after the initial training, thereby resulting in low overall frequency of groups. A number of barriers spanning context, intervention, practitioner, and recipient domains were noted, the majority of which were clinic-specific. Despite challenges, practitioners identified several facilitators and made modifications to the intervention and its delivery in service of implementation.
Conclusions:
Results from this implementation case study highlighted the challenges of delivering MFG in real-world FEP early intervention programs. Further, this paper emphasizes the value in identifying and addressing clinic-specific factors when implementing MFG.
Introduction
Family interventions for persons diagnosed with schizophrenia havewell-established efficacy in reducing relapse and hospitalizations(Pharoah et al., 2010; McFarlane, 2016). Further, the Schizophrenia Patient Outcomes Research Team (PORT) treatment recommendations endorse family interventions and specify that treatment should be offered for six to nine months (Dixon et al., 2010). Such interventions are especially important in the treatment of persons with first-episode psychosis (FEP), which explains why they are considered a core component of coordinated specialty care programs for this population (Dixon et al., 2015; Mueser et al., 2015). Moreover, a recent meta-analysis of 14 studies (11 randomized controlled trials)found that family intervention for FEP resulted in lower relapse rates, shorter duration of hospitalization, less severe psychotic symptoms, and increased functioning up to 24 months of follow-up (Camacho-Gomez et al., 2020). In light of the recognized value and recommended length of family interventions, effective implementation (i.e., uptake into routine practice; Eccles & Mittman, 2006)is critical for promoting recovery.
Despite the well-established benefits of family interventions for persons with schizophrenia and FEP, implementation has remained a substantial challenge. In fact, reported rates of implementation are highly variable ranging from 0% to 53% (Bucci et al., 2016; Ince et al., 2016), suggesting that salient barriers have not been comprehensively and consistently addressed. Organizational (e.g., limited time, lack of administrative support, billing structures, high caseloads), provider (e.g., lack of confidence in delivering intervention), and family/client (e.g., lack of interest in intervention, privacy concerns) factors have been identified as common obstacles (Bucci et al., 2016; Cohen et al., 2008; Ince et al., 2016; Kim & Salyers, 2008; Lucksted et al., 2012; Murray-Swank et al., 2007).Cohen and colleagues’ (2010) study, which reported zero referrals to their family psychoeducation program, exemplifies the detriment these issues can have on uptake. Therefore, it is necessary for current implementation efforts to target germane barriers in a manner that promotes both initial uptake and sustained use of family interventions.
The use of modifications and adaptations (i.e., planned modifications) to interventions can support implementation of evidence-based treatments (Stirman, Baumann, & Miller, 2019; Stirman et al., 2012; Stirman et al., 2017; Stirman, Calloway et al., 2013). Such changes can be made to overcome specific barriers faced by organizations, providers, and families/clients, thereby facilitating more effective delivery of treatments. For example, a clinic may choose to shorten the number of sessions specified in an intervention to address transportation challenges faced by their clients. Although these types of changes may be beneficial in promoting delivery of an intervention, they raise questions about the role of fidelity. Establishing standards of fidelity and regularly monitoring them when delivering evidence-based treatments is generally recommended (McHugh & Barlow, 2010) to ensure the integrity of the intervention. The importance of fidelity has been prominent in the delivery of FEP care (Essock et al., 2015; Mueser et al., 2019), particularly as many new coordinated specialty care programs have recently been developed in the United States (Dixon, 2017). Balancing fidelity to established evidence-based practices with modifications that support sustainable implementation of EBPs in diverse clinical settings is a critical conundrum that has only recently begun to receive attention in the early psychosis practice literature (Addington et al., 2018, 2020).
The principal goal of the present paper is to describe a coordinated effort to implement an evidence-based family intervention, the McFarlane-model multifamily psychoeducational groups (MFG; McFarlane, 2004), in established FEP early intervention programs within a single state. MFG was selected as it has been shown to be an effective component of family psychoeducation in reducing symptom relapses and rehospitalizations for individuals with schizophrenia (Jewell, Downing, & McFarlane, 2009). In this paper, we report on (1) training participation and implementation of MFG, (2) barriers and facilitators to implementation, and (3) modifications made to MFG in service of implementation.
Methods
Participants and Implementation Procedure
Providers from six FEP early intervention programs across a single state took part in an MFG implementation initiative supported by the state’s Department of Mental Health. The six programs were selected due to the fact that they received either partial or full funding for their FEP treatment services from the state’s Department of Mental Health. MFG, the focus of this initiative, is comprised of: (1)initial “joining” sessions in which a clinician meets with individual families three times to orient them to the intervention, (2) a psychoeducational workshop during which clinicians present information about symptoms, causes, and treatments for psychosis to families, and (3) multifamily groups, designed to occur every other week for at least nine months, during which clinicians teach problem-solving skills to participants (McFarlane, 2004). Of the six participating FEP teams, one had prior experience conducting MFG groups and two had prior experience conducting parent support groups that were not consistent with the MFG model.
In September 2018, an initial three-day (24-hour) training was conducted by PIER training institute faculty to teach the rationale for offering MFG in FEP early intervention programs and to develop and practice skills in delivering this MFG. A day-long, in-person booster session was conducted nine months later to refresh providers on intervention delivery skills, offer training to new clinic providers, and problem-solve implementation issues. The booster session was provided as a large group and was extended to all providers who participated in the first training as well as any new providers that joined their teams after the initial training. Ongoing consultation with MFG trainers was offered monthly via phone over an “implementation period” of12months after the initial three-day training; however, due to initial start-up delays, consultation was extended an additional three months. Consultation calls comprised one MFG trainer and practitioners from 1–3 clinics (per call) to allow for each clinic to receive sufficient support and to be flexible to different schedules. Consultation calls were offered monthly to providers from all six clinics.
Data Collection and Analysis
Data on training participation were obtained via attendance at in-person trainings. Implementation data (e.g., number of MFG groups held and timeline of implementation) were obtained from reports from trained providers across the participating clinics. Fifteen months following the initial training, providers representing each of the six FEP clinics were asked to report on the status of MFG implementation at their site as well as specific barriers, facilitators, and modifications across four categories: the context/organization, the intervention, the practitioners, and the intervention recipients (i.e., clients and families).These four categories were chosen as they comprise: (1) the intervention being implemented into a health care system and (2)the multiple levels of a health care systems that are involved in the implementation process (Ferlie & Shortell, 2001). Additionally, practitioners were asked to specify which modifications were completed and which were aspirational (i.e., that would have been helpful but were not completed). For the purposes of this paper, only completed modifications are described.
The first, second, and senior authors reviewed qualitative responses and completed three primary tasks on these data. First, they discussed unclear responses and determined a plan for clarification. If they could not clarify the response on their own, the second author followed up with providers by email and/or phone. In cases when clarification was not possible (e.g., could not reach providers), unclear responses were removed from the data. Second, they determined whether a response was placed in the most salient category or whether it would be better suited in a different category and subsequently, made appropriate changes (e.g., barrier was listed under context but was better suited under intervention). Third, responses that reflected similar themes were combined.The present evaluation did not require Institutional Review Board (IRB) approval as it was conducted as a quality improvement initiative and did not meet the IRB’s definition of human subjects research.
Results
Training Participation and Implementation of MFG
Sixteen providers from six FEP early intervention clinics participated in the initial three-day training with 88% having attended all three days. Nine of these providers attended the booster session nine months later along with seven additional providers who did not attend the initial training. In total, 23 providers participated in at least one of the two MFG training offerings (initial and/or booster).Following the initial training, 16 total consultation calls were held over the 12-month implementation period with five additional consultation calls provided after the 12 months had elapsed. One clinic did not attend any consultation calls while the remaining five clinics participated in at least four, highlighting the variability in participation across clinics (M=5.8, SD = 4.1, range: 0–11).
Data on the frequency of MFG groups conducted during the implementation period were obtained from four out of the six participating clinics. One clinic opted not to implement MFG shortly after attending the training and did not participate in further supervision calls or training. A second clinic had already been running MFG groups for ten years prior to the training, and reported no changes in their practice resulting from the trainings. As such, the implementation data reflect the remaining four clinics. Within the 12-month period, there was a range in the total number of MFG groups completed across these four clinics (M=4.3, SD=3.9, range: 0–8). The low frequency was likely due, in large part, to start-up delays as the earliest MFG group was held seven months following the initial training.
Qualitative Findings of MFG Implementation
Ten providers across five (of the six) participating clinics provided qualitative data on barriers and facilitators to implementation as well as modifications.
Barriers and facilitators to MFG implementation.
Barriers and facilitators spanned all four categories (context, intervention, practitioner, recipient). Though there were mostly clinic-specific barriers reported, several factors emerged as relevant to multiple clinics. In terms of barriers, several clinics were challenged by billing and reimbursement such that they either could not bill for the group at all or the amount of time required to implement the group was not fully reimbursable. Further, limited parking and difficulties in traveling to the clinic due to its physical location were noted as impediments to implementation. Practitioners reported that both the duration of the intervention and the length of the psychoeducation workshop component were too long, which contributed to difficulties with implementation. Some practitioners also noted that “joining” sessions felt redundant with ongoing family therapy and they were unsure how to resolve this tension. Regarding practitioner factors, limited time and scheduling was commonly cited as a barrier as well as leaders not speaking the same language as the families that they served. The most notable recipient barrier, endorsed by four (out of five) clinics, was limited availability to attend groups due to time constraints and other competing demands (e.g., work schedules). Recipients were also impacted by limited resources (e.g., financial, childcare) and transportation difficulties, which likely contributed to their limited availability. Finally, some clinics noted that families that were already engaged in their programs were not interested in adding MFG to the services they received (Table 1).
Table 1.
Barriers to MFG Implementation
| Context (Agency culture/policies, billing restrictions) |
Intervention (Qualities/components of the practice itself) |
Practitioner (Knowledge, priorities, personality, schedule) |
Recipient (Cultural factors, interest, personality, access) |
|
|---|---|---|---|---|
|
Barriers (What has gotten in the way?) |
|
|
|
|
Note. Bolded were endorsed by two or more clinics. K refers to the number of clinics.
Regarding facilitators, multiple clinics endorsed contextual factors as aiding implementation including access to appropriate space, ability to bill for MFG, and having the support of the treatment team. Further, practitioners both noted that adapting the model facilitated implementation and that MFG’s inherent emphasis on enhancing connections was a valuable facilitator in its own right. Practitioners commented on the importance of the MFG trainings and the ongoing supervision in promoting implementation. Moreover, practitioners who had flexible scheduling options (e.g., evening availability) endorsed this aspect as beneficial. Finally, recipient factors were mostly clinic-specific with the exception of families’ interest in support either from clinicians and/or from other families (Table 2).
Table 2.
Facilitators of MFG Implementation
| Context (Agency culture/policies, billing restrictions) |
Intervention (Qualities/components of the practice itself) |
Practitioner (Knowledge, priorities, personality, schedule) |
Recipient (Cultural factors, interest, personality, access) |
|
|---|---|---|---|---|
|
Facilitators (What has enabled this to work?) |
|
|
|
|
Note. Bolded were endorsed by two or more clinics. K refers to the number of clinics.
Completed modifications.
Completed modifications spanned only the intervention and practitioner domains. Multiple clinics made changes to the workshop component of MFG including offering it to the entire clinic (as opposed to only those enrolled in and committed to the full MFG program) and offering it in multiple, smaller groups (rather than in a single, larger and longer group). Additionally, two clinics broadened the content covered in MFG including allowing time for recipients to process other’s experiences when shared during groups. The only practitioner modification made was increasing availability to run groups (e.g., late/evening appointments) in order to accommodate recipients’ schedules (Table 3).
Table 3.
Completed Modifications to MFG Implementation
| Intervention (Qualities/components of the practice itself) |
Practitioner (Knowledge, priorities, personality, schedule) |
|
|---|---|---|
|
Modifications (That were completed to help address barriers) |
|
|
Note. Bolded were endorsed by two or more clinics. K refers to the number of clinics.
Discussion
The overarching purpose of this paper was to describe the real-world implementation of MFG across six established FEP early intervention programs within a single state. In service of this goal, we reported on the training participation and implementation of MFG, the barriers and facilitators to implementation, and modifications made to MFG in service of implementation.
Practitioners across these programs received in-person trainings and ongoing phone consultation with MFG trainers, which are recommended aspects of implementation of family interventions for this population (Berry & Haddock, 2008).Twenty-three practitioners across six clinics received in-person training; however, participation in ongoing consultation calls occurred less than the monthly goal with substantial variability across clinics. Although practitioners’ availability was considered when scheduling consultation calls, it is possible that competing demands may have impacted attendance. Clinics experienced difficulties in initiating MFG after the initial training and as such, the total number of completed groups was far lower than the biweekly recommendation over the 12-month implementation period. Overall, these low rates of uptake are consistent with prior reports of family intervention implementation in FEP programs (Selick et al., 2016) and suggest that general training and consultation do not address the salient barriers unique to each clinic.
Clinics endorsed a number of barriers to implementing MFG, the majority of which were specific to the particular program, thereby highlighting that challenges can be distinctive even among established FEP clinics within the same state. Yet, there were consistent obstacles that spanned clinics, most notably the limited availability of families to attend groups due to work schedules and personal circumstances. This primary barrier is consistent with those reported in two prior studies of family intervention for FEP in Australia (Petrakis, Bloom, & Oxley, 2014) and in the United Kingdom (Riley et al., 2011), thus, highlighting the pervasiveness of this issue across countries. Relatedly, the clinic’s location, limited scheduling flexibility among practitioners, and families’ limited access to transportation and resources impeded participation. Practitioners also noted reimbursement and billing issues with MFG, which reflect a larger financing concern with FEP services in the United States (Dixon, 2017). Multiple clinics reported that the length of the overall intervention and the psychoeducation workshop in particular, negatively impacted families’ participation in MFG, which have previously been cited as challenges to engaging families (Nilsen et al., 2015). Yet, Nilsen and colleagues (2014) discussed that clearly identifying an end date for the program can be a valuable strategy to manage families’ concerns about duration. Finally, in line with Petrakis and colleagues (2014), practitioners did not speak the same language as many families they served, thus limiting the reach of MFG.
Despite the slow pace of uptake and the identified barriers, practitioners across all clinics reported multiple factors that facilitated the progress they made in implementing MFG. Similar to the barriers noted, the majority of facilitators were clinic-specific with fewer reflecting consistency across programs. Across clinics, the trainings, ongoing supervision, and having buy-in from the rest of the treatment team were viewed as valuable, which are consistent with past work (Selick et al., 2016) and reflect the usefulness of the overall implementation procedure. Finally, MFG’s emphasis on enhancing connections in tandem with families’ interest in gaining support from others facing similar challenges (Nilsen et al., 2014)were viewed as facilitators and highlight how this intervention can reduce family distress (Addington et al., 2003).
To address implementation challenges, clinics made modifications to the intervention content and the context in which it was delivered (Stirman, Miller, Toder, & Calloway, 2013) and noted that these changes were helpful. With the exception of adding flexibility to practitioners’ schedules when possible, only MFG-specific modifications were made. Specifically, multiple clinics reported altering the workshop component of MFG such that it was offered to the entire clinic and/or could be delivered in multiple, smaller groups. Further, two clinics described broadening the content of MFG, in part to allow for more time for families to process other families’ experiences that were shared in the groups. Though clinics were not instructed to make specific modifications, it is important to recognize that these types of unplanned changes are commonly performed when evidence-based practices are implemented in real-world settings (Stirman et al., 2017; Stirman, Miller et al., 2013). In our case, these modifications may have allowed for more families to receive exposure to MFG; however, it can be difficult to determine whether the benefits of such changes outweigh potential reduced fidelity to the established protocol (Nilsen et al., 2015; Petrakis et al., 2014).
The present report is limited in that it represents the implementation of MFG ina small number of clinics in one state. Data were obtained only from practitioners, which precluded exploration of family members’ and clients’ experiences. Further, we were unable to confirm all clinics’ implementation data (e.g., number of groups and timeline) via medical record review, which may limit its validity. Despite these limitations, this paper offers a case study in real-world implementation of MFG for persons with FEP. Results demonstrated that even established FEP programs in the same state have clinic-specific factors that are not readily addressed by a general implementation strategy. As such, future research should include ongoing formative evaluation of barriers and facilitators at the clinic level as well as completed modifications to inform an appropriately tailored implementation strategy (Stetler et al., 2006). Moreover, it would be important to evaluate the effectiveness of different types of implementation strategies and intervention modifications to better understand how best to provide effective treatments in FEP early intervention settings. Finally, the current COVID-19 crisis might drive a new wave of modifications, notably the use of virtual care (Torous & Keshavan,2020), which will need to be examined for both feasibility and effectiveness.
Acknowledgments:
The authors would like to thank the multifamily group trainers and the staff at the first-episode psychosis early intervention programs.
Source of Funding:
This work was funded by the Massachusetts Department of Mental Health (SCDMH822018082610000;SCDMH822019083980000) and the United States National Institute of Mental Health (1K23MH118373-1A1).
Footnotes
Disclosure of Interest: The authors do not have any conflicts of interest to disclose.
References
- Addington D, Birchwood M, Jones P, Killackey E, McDaid D, Melau M, … & Nordentoft M. (2018). Fidelity scales and performance measures to support implementation and quality assurance for first episode psychosis services. Early Intervention in Psychiatry, 12(6), 1235–1242. [DOI] [PubMed] [Google Scholar]
- Addington D, Cheng CC, French P, Killackey E, Melau M, Meneghelli A, … & Smith J. (2020). International application of standards for health care quality, access and evaluation of services for early intervention in psychotic disorders. Early Intervention in Psychiatry. [DOI] [PubMed] [Google Scholar]
- Addington J, Coldham EL, Jones B, Ko T, & Addington D (2003). The first episode of psychosis: The experience of relatives. Acta Psychiatrica Scandinavica, 108(4), 285–289. [DOI] [PubMed] [Google Scholar]
- Berry K, & Haddock G (2008). The implementation of the NICE guidelines for schizophrenia: Barriers to the implementation of psychological interventions and recommendations for the future. Psychology and Psychotherapy: Theory, Research and Practice, 81(4), 419–436. [DOI] [PubMed] [Google Scholar]
- Bucci S, Berry K, Barrowclough C, & Haddock G (2016). Family interventions in psychosis: A review of the evidence and barriers to implementation. Australian Psychologist, 51(1), 62–68. [Google Scholar]
- Camacho-Gomez M, & Castellvi P. (2020). Effectiveness of family intervention for preventing relapse in first-episode psychosis until 24 months of follow-up: a systematic review with meta-analysis of randomized controlled trials. Schizophrenia Bulletin, 46(1), 98–109. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cohen AN, Glynn SM, Hamilton AB, & Young AS (2010). Implementation of a family intervention for individuals with schizophrenia. Journal of General Internal Medicine, 25(1), 32–37. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cohen AN, Glynn SM, Murray-Swank AB, Barrio C, Fischer EP, McCutcheon SJ, Perlick DA, Rotondi AJ, Sayers SL, Sherman MD, & Dixon LB (2008). The family forum: Directions for the implementation of family psychoeducation for severe mental illness. Psychiatric Services, 59, 40–48. [DOI] [PubMed] [Google Scholar]
- Dixon L (2017). What it will take to make coordinated specialty care available to anyone experiencing early schizophrenia: getting over the hump. JAMA Psychiatry, 74(1), 7–8. [DOI] [PubMed] [Google Scholar]
- Dixon LB, Dickerson F, Bellack AS, Bennett M, Dickinson D, Goldberg RW, … & Peer J. (2010). The 2009 schizophrenia PORT psychosocial treatment recommendations and summary statements. Schizophrenia bulletin, 36(1), 48–70. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Dixon LB, Goldman HH, Bennett ME, Wang Y, McNamara KA, Mendon SJ, … & Essock SM. (2015). Implementing coordinated specialty care for early psychosis: the RAISE Connection Program. Psychiatric Services, 66(7), 691–698. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Eccles MP, & Mittman BS (2006). Welcome to implementation science. Implementation Science, 1. [Google Scholar]
- Essock SM, Nossel IR, McNamara K, Bennett ME, Buchanan RW, Kreyenbuhl JA, … & Dixon LB. (2015). Practical monitoring of treatment fidelity: Examples from a team-based intervention for people with early psychosis. Psychiatric Services, 66(7), 674–676. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Ferlie EB, & Shortell SM (2001). Improving the quality of health care in the United Kingdom and the United States: A framework for change. The Milbank Quarterly, 79(2), 281–315. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Ince P, Haddock G, & Tai S (2016). A systematic review of the implementation of recommended psychological interventions for schizophrenia: Rates, barriers, and improvement strategies. Psychology and Psychotherapy, 89, 324–50. [DOI] [PubMed] [Google Scholar]
- Jewell TC, Downing D, & McFarlane WR (2009). Partnering with families: Multiple family group psychoeducation for schizophrenia. Journal of Clinical Psychology, 65(8), 868–878. [DOI] [PubMed] [Google Scholar]
- Kim HW, & Salyers MP (2008). Attitudes and perceived barriers to working with families of persons with severe mental illness: mental health professionals’ perspectives. Community Mental Health Journal, 44(5), 337–345. [DOI] [PubMed] [Google Scholar]
- Lucksted A, McFarlane W, Downing D, & Dixon L (2012). Recent developments in family psychoeducation as an evidence-based practice. Journal of Marital and Family Therapy, 38, 101–21. [DOI] [PubMed] [Google Scholar]
- McFarlane WR (2004). Multifamily groups in the treatment of severe psychiatric disorders. Guilford Press. [Google Scholar]
- McFarlane WR (2016). Family interventions for schizophrenia and the psychoses: A review. Family Process, 55(3), 460–482. [DOI] [PubMed] [Google Scholar]
- McHugh RK, & Barlow DH (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65(2), 73–84. [DOI] [PubMed] [Google Scholar]
- Mueser KT, Meyer-Kalos PS, Glynn SM, Lynde DW, Robinson DG, Gingerich S, … & Kane JM. (2019). Implementation and fidelity assessment of the NAVIGATE treatment program for first episode psychosis in a multi-site study. Schizophrenia research, 204, 271–281. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Mueser KT, Penn DL, Addington J, Brunette MF, Gingerich S, Glynn SM, … & Cather C. (2015). The NAVIGATE program for first-episode psychosis: rationale, overview, and description of psychosocial components. Psychiatric Services, 66(7), 680–690. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Murray-Swank A, Glynn S, Cohen AN, Sherman M, Medoff DP, Fang LJ, Drapalski A, & Dixon L (2007). Family contact, experience of family relationships, and views about family involvement in treatment among VA consumers with serious mental illness. Journal of Rehabilitation Research & Development, 44(6), 801–812. [DOI] [PubMed] [Google Scholar]
- Nilsen L, Frich JC, Friis S, & Røssberg JI. (2014). Patients’ and family members’ experiences of a psychoeducational family intervention after a first episode psychosis: A qualitative study. Issues in Mental Health Nursing, 35(1), 58–68. [DOI] [PubMed] [Google Scholar]
- Nilsen L, Norheim I, Frich JC, Friis S, & Røssberg JI. (2015). Challenges for group leaders working with families dealing with early psychosis: A qualitative study. BMC Psychiatry, 15(1), 141. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Petrakis M, Bloom H, & Oxley J (2014). Family perceptions of benefits and barriers to first episode psychosis carer group participation. Social Work in Mental Health, 12(2), 99–116. [Google Scholar]
- Pharoah F, Mari JJ, Rathbone J, & Wong W (2010). Family intervention for schizophrenia. Cochrane Database of Systematic Reviews, (12). [DOI] [PMC free article] [PubMed] [Google Scholar]
- Riley G, Gregory N, Bellinger J, Davies N, Mabbott G, & Sabourin R (2011). Carer’s education groups for relatives with a first episode of psychosis: An evaluation of an eight-week education group. Early intervention in Psychiatry, 5(1), 57–63. [DOI] [PubMed] [Google Scholar]
- Selick A, Durbin J, Vu N, O’Connor K, Volpe T, & Lin E (2017). Barriers and facilitators to implementing family support and education in Early Psychosis Intervention programmes: A systematic review. Early Intervention in Psychiatry, 11(5), 365–374. [DOI] [PubMed] [Google Scholar]
- Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, … & Smith JL. (2006). The role of formative evaluation in implementation research and the QUERI experience. Journal of General Internal Medicine, 21(2), S1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Stirman SW, Baumann AA, & Miller CJ (2019). The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implementation Science, 14(1), 58. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Stirman SW, Miller CJ, Toder K, & Calloway A (2013). Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implementation Science, 8(1), 65. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Stirman SW, Calloway MA, Toder MK, Miller MCJ, DeVito MAK, Meisel MSN, … & Crits-Christoph P. (2013). Modifications to cognitive therapy by community mental health providers: Implications for effectiveness and sustainability. Psychiatric Services, 64(10). [DOI] [PMC free article] [PubMed] [Google Scholar]
- Stirman SW, Kimberly J, Cook N, Calloway A, Castro F, & Charns M. (2012). The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implementation Science, 7(1), 17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Stirman SW, Gamarra JM, Bartlett BA, Calloway A, & Gutner CA (2017). Empirical examinations of modifications and adaptations to evidence-based psychotherapies: Methodologies, impact, and future directions. Clinical Psychology: Science and Practice, 24(4), 396–42. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Torous J, & Keshavan M (2020). COVID-19, mobile health and serious mental illness. Schizophrenia Research. Ahead of Print. [DOI] [PMC free article] [PubMed] [Google Scholar]