Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Feb 1.
Published in final edited form as: Prof Psychol Res Pr. 2014 Feb;45(1):57–66. doi: 10.1037/a0035301

A case for modular design: Implications for implementing evidence-based interventions with culturally-diverse youth

Aaron R Lyon 1, Anna S Lau 2, Elizabeth McCauley 1, Ann Vander Stoep 1, Bruce F Chorpita 2
PMCID: PMC4199229  NIHMSID: NIHMS516595  PMID: 25328279

Abstract

Community-based therapists are frequently faced with the complex task of applying existing research knowledge to clients who may differ markedly from those enrolled in controlled outcome studies. The current paper examines the utility of modular psychotherapy design as one method of facilitating the flexible delivery of evidence-based mental health services to ethnically and culturally diverse children and families. Modularity complements existing approaches to the provision of culturally-sensitive, empirically-informed treatment through its ability to balance the prioritization of research evidence and local practitioner cultural knowledge. Specific applications of modular principles to clinical work with diverse youth are highlighted. Special considerations and limitations relevant to modular psychotherapy and the overall mental health services research literature are discussed, as well as the continued importance of individual clinicians’ cultural competence and use of treatment progress monitoring, both of which should be combined with identified treatment modules to support the delivery of high-quality care.

Keywords: evidence-based practice, ethnicity, cultural adaptation, children and families, modular psychotherapy

Although much discussion has focused on the relevance of “evidence-based” psychotherapy interventions to the diverse demographic and clinical landscapes of practicing therapists (e.g., Bernal & Scharrón-Del-Río, 2001; Kazdin, 2008), insufficient resources are available to guide clinicians as they navigate the complexities of applying empirical research in their work. This is particularly true for therapists who routinely provide services to individuals from ethnic or cultural minority groups, populations that may differ in significant and meaningful ways from those included in rigorous clinical trials. The emergence of more flexible conceptual and practical approaches to the integration of the empirical literature into psychotherapy practice may offer new options to help overcome some of the traditional barriers to utilizing clinical research in practice. Although these types of flexible service delivery models (including “transdiagnostic” approaches) have begun to emerge in the adult literature (e.g., Barlow, Allen, & Choate, 2004; Roy-Byrne et al., 2010), they have been more extensively developed for children and adolescents; which is the primary focus of the current paper.

Culturally-Adapted and Culturally Grounded Empirically-Supported Treatments (ESTs)

Until recently, the psychological field has focused almost exclusively on highly-structured, empirically-supported treatment packages (ESTs; Chambless et al., 1996) as the primary vehicle for dissemination and implementation of research findings. Despite advances, there remain relatively few studies to inform the systematic and effective use of existing ESTs with ethnic and cultural minority clients, limiting the conclusions that can be drawn about their utility and generalizability (Bernal & Scharron-del-Rio, 2001; Huey & Polo, 2008). As a result, some investigators have sought to adapt ESTs “to consider language, culture, and context in such a way that is compatible with the client’s cultural patterns, meanings, and values” (Bernal, Jimenez-Chafey, & Domenech-Rodriguez, 2009, p. 362). Although a number of models exist to facilitate cultural adaptation (e.g., Bernal, Bonilla, & Bellido, 1995; Hwang, 2009; Nicolas, Arntz, Hirsch, & Schmiedigen, 2009), progress toward the development of a comprehensive set of adapted interventions has been slow. Furthermore, while culturally-adapted interventions have generally produced medium effect sizes in meta-analytic reviews (Griner & Smith, 2006; Huey & Polo, 2008; Smith, Rodríguez, & Bernal, 2011), there is conflicting evidence surrounding whether culturally-adapted ESTs are any more effective than non-adapted ESTs (Benish, Quintana, & Wampold, 2011; Cardemill, 2010; La Roche & Christopher, 2009; Huey, 2012).

Separate from the adaptation of ESTs for use with new populations, some authors have advocated for a “culturally-grounded” approach to intervention development, which places values, beliefs, practices, and socio-historical perspectives at the center of treatment design (e.g., Marsiglia & Kulis, 2009; Resnicow, Baranowski, Ahluwalia, & Braithwaie, 1999). The goal of this approach is create an effective and contextually appropriate intervention for a specific local population. Although they have the advantage of being culturally-appropriate from the outset, interventions resulting from the culturally-grounded approach are likely to suffer from longer intervention development timelines and limited scalability (Holleran Steiker et al., 2008).

Evidence-Based Practice

In an alternative paradigm to culturally-specific EST development (through adaptation or culturally-grounded design), increasing attention has been focused on the identification and integration of the best evidence-based practices (EBP) in mental health care, with particular emphasis placed on the individual patient context. The American Psychological Association has defined EBP as “the integration of the best available research with clinical expertise in the context of patient characteristics, culture, and preferences” (American Psychological Association [APA], 2006, p.273). Use of EBP requires therapists to think systematically about integrating multiple sources of information, such as scientific research studies, experiential knowledge (e.g., expertise, insight), and key contextual evidence (e.g., intervention acceptability to a community) (APA, 2006; Puddy & Wilkins, 2011). In support of this perspective, Daleiden and Chorpita (2005) have articulated four distinct evidence bases to inform data-driven clinical decision making: general services research evidence (information drawn from the existing empirical literature), case history evidence (individualized, case-specific data derived from direct client interactions), local aggregate evidence (a.k.a., “practice based” evidence, which summarizes case history evidence across caseloads, agencies, etc.), and causal mechanism evidence (a general and comprehensive understanding of etiological and treatment processes, based on theory and experience). Daleiden and Chorpita’s model also frames data-driven decision making as a social process that incorporates both information sources (evidence bases) and the expertise and input of key decision makers, such as providers, clients, and supervisors.

Due to their individual limitations, all of the evidence bases need to be integrated to inform treatment planning and clinical decision-making for a given case. For instance, although causal mechanism evidence is the least standardized, it includes essential experiential and theoretical contextual knowledge. This source of evidence is likely to be particularly relevant when providing services to culturally diverse individuals, where research evidence is more limited. Consideration of causal mechanism evidence may allow a clinician to attend to a diverse range of information sources such as ethnographic research documenting how presenting problems and intervention approaches are experienced within a given sociocultural context (e.g., Lopez & Guarnaccia, 2000) or specific theory-driven guidelines for structuring the work of multicultural therapists (e.g., Gallardo, Parham, Trimble, & Yeh, 2012).

Perspectives on Services Research Evidence Generalizability to Culturally-Diverse Youth

As described above, there are multiple ways to conceptualize the development and delivery of effective treatment technologies within contexts characterized by a high degree of diversity. The different approaches typically stem from key assumptions about the extent to which existing information from the general services evidence base (which is characterized by a high degree of internal validity) generalizes to clinical practice across populations and settings (external validity). Below, we briefly describe these assumptions and note how each influences the way general services research knowledge may be used in the treatment of diverse youth. Table 1 displays additional detail about the assumptions and their implications.

Table 1.

Assumptions about the generalizability of services research evidence to cultural minority groups and resulting approaches

Assumption Resulting Approach Strengths Weaknesses
1. Interventions do not generalize unless there is specific evidence to support it (a) Development of culturally-specific and empirically-tested interventions (culturally-grounded or culturally-adapted)

  • Attends both to local cultural knowledge and empirical evidence

  • Holds interventions to a high standard to demonstrate their relevance/applicability

  • Inefficient: endless adaptations or novel treatments required to serve all groups effectively

  • Even adapted interventions may not match the cultural values of the entire target population
(b) Reliance on local experts to design interventions

  • Places a high value on local cultural knowledge/expertise

  • Discards entire bodies of evidence for intervention efficacy
2. Interventions generalize to all groups, unless there is specific evidence to the contrary Dissemination and implementation of existing empirically-supported intervention protocols (sometimes by state-level mandate; e.g., Jensen-Doss, Hawley, Lopez, & Osterberg, 2009

  • Relies on existing research evidence for efficacy (robust findings suggest that the effects of many treatments do generalize)

  • Simple, allows for greater ease of dissemination, irrespective of effectiveness

  • Ignores the (known or unknown) possibility of differential intervention effectiveness across groups

  • May structure or frame intervention in a way that is culturally incongruent
3. Some aspects of interventions will generalize to most groups, but local adaptation may be needed (a) Intervention “co-design” by laboratory and local experts

  • Makes use of existing research evidence (allows for greater utilization of research than 1)

  • Is enhanced by a MFS, which allows adaptations to be tied to outcome

  • Doesn’t assume that all interventions generalize Difficult to know when adaptations are truly necessary or what adaptations are ideal

  • May require a MFS (which can be cumbersome to implement)
(b) Quality improvement within existing intervention models

  • Offers the greatest practitioner flexibility and ease of integration

  • Uses research knowledge (although more limited than 3a)

  • Is enhanced by a MFS, which allows adaptations to be tied to outcome

  • Subject to many drawbacks of usual care

  • Will likely result in smaller treatment effects that 3a.

  • Focuses primarily on intervention practices

  • May require a MFS (which can be cumbersome to implement)
Open in a new tab

MFS = measurement feedback system

  1. Existing interventions do not generalize sufficiently to other groups, unless there is specific evidence to support their appropriateness and effectiveness. The assumption that research knowledge about particular interventions can only be applied to populations with which they have been tested adequately has two potential outcomes: (a) If the primary goal is to stay within the evidence base, the solution could involve developing and testing culturally-grounded or culturally-adapted interventions, as described previously. These are specific to selected target populations (e.g., Hurdle, 2002; Nicolas et al., 2009), yielding a unique protocol for each group intended to be served (e.g., a parenting program for urban, Asian immigrant families). (b) Alternatively – or simultaneously – one could assume that therapists with detailed local knowledge of a particular treatment context (e.g., culture, neighborhood, etc.) and understanding of relevant causal mechanism evidence should be wholly responsible for designing treatments for each individual case.

  2. Existing research-supported interventions generalize to all groups, unless there is specific evidence to the contrary. This assumption suggests that research findings are sufficiently robust to generalize widely. In practice, this typically results in the application of interventions that match a child’s presenting problem and age, without requiring that the treatment has been tested successfully with research participants who have similar characteristics on other dimensions of human diversity (e.g., ethnicity, country of origin).

  3. Some aspects of interventions will generalize to most groups, but local adaptation may be needed. This assumption leads to at least two distinct, but overlapping, approaches to intervention design: (a) A “co-design” of treatment delivery in which the protocol authors specify the major therapeutic elements and a boundary of structure, within which a therapist could then determine specific aspects in real time to maximize its relevance to their clients (Chorpita et al., 2011). An example of an intervention derived from this assumption would be a protocol that requires exposure for anxiety, but allows the therapist to determine how much time is needed to prepare the child and family based on their particular beliefs, mental health literacy, etc., (b) A quality-improvement approach in which specific components of interventions are incorporated into an existing model that otherwise may not have empirical support. These two approaches differ from those categorized under the first assumption because they do not result in a single, culturally-appropriate intervention for a specific group, but a more general treatment package that can be flexibly applied to a wide range of groups.

Each of the perspectives listed above represents a worthwhile area of pursuit. Table 1 displays the assumptions along with a number of pros and cons. For instance, although culturally-specific interventions resulting from the first assumption have a high likelihood of being appropriate for a given population or context, the time and resources required to develop such interventions for all populations and presenting problems makes this approach impractical as the sole method of ensuring that culturally-diverse youth and families receive high-quality services. Each resulting approach can also be categorized according to the extent to which it prioritizes general services research evidence versus local practitioner knowledge, including existing clinical practices and causal mechanism evidence. Figure 1 places each approach along these two dimensions. To date, the greatest time and resources have been devoted to research activities that follow the first two assumptions, which generally lead to approaches at polarized ends of the internal and external validity spectrum. Nevertheless, it is the third assumption that is most compatible with contemporary thinking about evidence integration and the application of EBP with cultural minority groups (APA, 2006). Although the following sections will focus primarily on intervention approaches that stem from the third assumption, a multifaceted approach that also includes culturally-grounded and culturally-adapted interventions (assumption #1) is warranted to continue to improve the quality of services available to ethnic and cultural minority youth.

Figure 1.

Figure 1

Open in a new tab

Approaches to research utilization in the treatment of ethnically and culturally minority youth and the degree to which they prioritize research evidence and local knowledge.

Although the third assumption necessitates evidence integration, the two resulting approaches differ with regard to their start points for adaptation. Co-design (3a) assumes that therapies are built together by a laboratory-based design team and a local agent who makes adjustments and adaptations to the therapy protocol in real time on a case-by-case basis. In this approach, a crucial task for the lab designer is to create an architecture that places appropriate constraints on practice (e.g., limits flexibility when it could be harmful), but allows for in-the-field adaptations that are systematic and easy to coordinate. The quality improvement approach (3b) begins instead with clinicians’ existing practices, which may already be optimized for work with some cultural minority youth, and then builds in elements of practice for which there is research support. To varying extents, both approaches avoid assumptions about the complete generalization of research findings across groups, but also retain much of the information contained within the extensive scientific knowledge base. The co-design approach is also best conducted in the context of measurement feedback systems (MFS; Bickman, 2008) that allow some objective estimate of when particular criteria for adaptation are met using case history evidence derived from routine outcome monitoring (discussed further below). The following sections are intended to illustrate how a modular approach, driven by the third assumption and based on literature aggregation, is underutilized when addressing our field’s current failure to apply scientific findings to human suffering on a sufficient scale (Kazdin & Blase, 2011). Intervention co-design and quality improvement approaches will both be considered.

Modular Psychotherapy and the Distillation and Matching Model (DMM)

Modular Psychotherapy

Modular approaches to psychotherapy represent one way to construct interventions that retain the content and logic of therapy models but, within a co-design approach, provide an explicit framework for adaptation in real time. At a basic level, modular design implies that treatments can be subdivided into meaningful units, which can then be implemented independently or in complement with one another to bring about a specific treatment outcome (Chorpita, Daleiden, & Weisz, 2005b). Modular psychotherapy design separates therapy content or procedures from the logic model for coordinating those procedures. For example, a module on relaxation would not have information within it that would dictate its position in the sequence of a larger therapy program. Instead, if used within the co-design approach, that procedure would be implemented according to a particular flowchart or logic model, which could indicate that relaxation is only delivered when a certain condition is met. When the rules for organizing the content are fully (or mostly, as is more realistically the case) separated from the content itself, it becomes possible to design thousands of variations from the same content.

Distillation and Matching Model (DMM)

Recently, a distillation and matching model (DMM), was developed to facilitate intervention co-design and quality improvement by laboratory and field experts, thereby more effectively applying research knowledge in practice (Chorpita, Daleiden, & Weisz, 2005a). The DMM provides a framework for aggregating data from the general services research evidence base and involves two primary steps: (1) distilling effective psychosocial treatments into sets of content elements and (2) matching those elements to client problems and other characteristics. Although research has indicated that problem area (e.g., anxiety, depression) is generally the most meaningful variable when distinguishing among sets of treatment elements (Chorpita & Daleiden, 2009), the model also allows for the inclusion of information about various diversity dimensions, such as client gender, ethnicity, and other demographic variables. The DMM is a general framework for deriving commonalities from different interventions and linking them to client characteristics and not an intervention approach. Nevertheless, it may be used to inform modularized service delivery in either therapy co-design or quality improvement scenarios (as in the examples below). Evidence for the effectiveness of modular designs informed by the DMM is beginning to emerge. For instance, in a recent randomized trial, a modular co-design approach to psychotherapy for anxiety, depression, and conduct problems demonstrated superior outcomes to both standard-arranged treatment manuals (i.e., traditional ESTs) and usual care conditions when used with an ethnically-diverse group of youth (Chorpita et al., in press; Weisz et al., 2012).

PracticeWise Decision-Support Tools

One example application of the DMM and its principles comes in the form of a website which provides a regularly-updated, meta-analytic tool: the PracticeWise Evidence-Based Services (PWEBS) database, drawn from youth psychotherapy outcome studies (PracticeWise, 2009). By entering user-selected demographic and clinical data for a particular client of interest (i.e., problem type/diagnosis, age, sex, ethnicity) and indicating the desired “strength of evidence” (PWEBS users can indicate the level of evidence provided by different studies on a 1 to 5 scale; Level 1: “Best Support”), one can retrieve a report that aggregates the findings from all randomized trials matching the child’s characteristics, including a rank ordered frequency count of the practice elements comprising those treatments. In this way, PWEBS is intended to inform clinicians about what treatments are relevant to the youth in question. Among its accompanying tools, clinicians are furnished with brief, written practitioner guides to facilitate the implementation of each indicated practice element, as well as an electronic clinical dashboard (a MFS) to track practice element implementation, facilitate routine treatment target monitoring (e.g., via standardized assessments), and integrate case history evidence into practice.

Applying Modularity to Culturally-Sensitive EBP

An individualized, modular application of treatments has the potential to integrate various sources of evidence (i.e., local practitioner cultural knowledge/causal mechanism evidence, services research evidence, case history evidence) to support clinical decision making processes rather than making a priori assumptions about the role of cultural variables in individuals’ lives. In this way, the modular approach may allow for relatively straightforward adaptation of practices to different cultural scenarios without necessarily threatening fidelity or requiring a new manual to be developed, tested via a randomized controlled trial, and disseminated to practitioners; a process that can easily take over 20 years to complete (Balas & Boren, 2000; Rotheram-Borus, Swendeman, & Chorpita, 2012). Instead, the approach allows for micro-level adaptations to deal with different cultural phenomena as they arise. Nevertheless, because it is based on the information available in the empirical treatment literature, output from the DMM is limited to the extent that it does not provide specific information about how to facilitate a good fit between well-researched practices and particular groups who receive an intervention. Fortunately, additional sources of causal mechanism information, derived from theory and provider experience, can help to obtain a successful fit and maximize the clinical utility of the practice elements identified. For instance, an Asian American youth struggling with depression, who may be more likely to try avoiding troubling thoughts instead of dwelling on them (Cheng, Leong, & Geist 1993), could potentially be a better match for behavioral activation or activity selection modules, which focus on increasing positive experiences, rather than modules devoted to cognitive restructuring. Modules can also be written to have broad applicability across situations, clients, and treatment goals to facilitate their integration into the existing rapport-building, engagement, and intervention practices of culturally-competent therapists.

Indeed, culturally-sensitive practices are themselves local technologies that often develop from local aggregate or “practice-based” evidence (Jensen et al., 2012) within particular service-delivery contexts. When utilizing a modular approach, these technologies can be combined with information from the services research evidence base on a local level to create more relevant and accessible therapies for a broad range of youth. For example, an urban community mental health center that commonly uses neighborhood outreach workers to facilitate initial therapy engagement can incorporate a psychoeducation module into an initial session that involved the family, the outreach worker, and the therapist. Prior literature indicates that the ease with which new intervention strategies can be incorporated into clinicians’ existing practices is one of the most important determinants of the adoption of new practices (Cook, Schnurr, Biyanova, & Coyne, 2009; Hays, 2009). In the recent randomized trial of a modular approach, clinicians in the modular condition also utilized more non-evidence-based treatment components than did those in the standard-arranged manuals condition, suggesting that a modular approach carries new opportunities for such integration (Weisz et al., 2012).

Clinical Illustrations

Below, two clinical vignettes have been constructed to provide examples of how modular practices, supported using information sources such as the PracticeWise website, can be integrated into culturally-sensitive treatment with ethnic minority youth. In the first, the PWEBS database is able to provide some specific direction to a practitioner treating a family from a particular cultural minority group and facilitate intervention co-design. In the second, such detailed information is unavailable and the practitioner must rely more heavily on other information related to culturally-competent care in a quality improvement scenario.

Case #1

Xavier, a 5-year-old, second-generation Mexican-American boy, and his family are referred for services at a local community mental health center (CMHC) in an urban area. Xavier has been exhibiting high levels of oppositional and noncompliant behavior at home and at school (e.g., defiant responses when told to perform age-appropriate tasks, losing his temper, and occasionally fighting with peers). Although these behaviors have been ongoing for over a year, they have recently worsened in the school context. At home, Xavier is reportedly better behaved when his father is present and more disruptive during his absence. After receiving the referral, Xavier’s therapist first follows her local CMHC standards for family engagement by making initial phone contact to clarify the purpose of the referral and to identify and troubleshoot potential barriers. Consulting some existing cultural resources available at her CMHC (books, colleagues) and recognizing that fathers from a variety of Latino backgrounds may be less likely to engage in family psychotherapy (Falicov, 2005) (causal mechanism evidence), she makes a particular effort to encourage Xavier’s father to attend the initial session by remarking on the notable impact of his presence on Xavier’s behavior and eliciting his feelings of responsibility for well-being of the family. Following an initial meeting, the therapist assesses Xavier using standardized behavioral rating scales administered to Xavier’s parents and teacher as well as a structured diagnostic interview, arrives at a primary diagnosis of Oppositional Defiant Disorder (ODD), and searches the PWEBS database by entering Xavier’s presenting problem, age, gender, and ethnicity into the search fields, a process that takes 15–20 seconds. Search results are instantaneous and produce findings based on three papers that contained relevant samples and produced positive results (Bagner & Eyberg, 2007; Barrera et al., 2002; Reid, Webster-Stratton, & Hammond, 2007). Although 24 separate practice elements comprise the three protocols, the elements common to all protocols include: working with caregivers to increase (1) attention to positive behavior, (2) use of effective commands, (3) ignoring and differentially reinforcing behavior, (4) use of praise, and (5) use of time out, as well as (6) therapists’ use of in-session praise and reinforcement. The therapist accesses brief practitioner guides for each of these elements and, when discussing the treatment plan with Xavier’s parents, frames them as a skill-building program to further reduce stigma (McCabe, Yeh, Garland, Lau, & Chavez, 2005). Based on their discussion, a decision is made to begin treatment by focusing on parental attention to positive behavior, using praise, and differential reinforcement of behavior. The therapist takes care to frame each component based on her understanding of how Xavier’s parents currently express approval for positive behavior. Once parents show mastery of positive attention principles, differential reinforcement is introduced and the therapists takes time in clarifying the purpose of active ignoring (often viewed as culturally dystonic) to reduce misbehavior in a way that builds on rather than disrupts family relationships. Ongoing progress monitoring occurs using a standardized rating tool to evaluate Xavier’s disruptive behavior over time. Weekly ratings are entered into a MFS, which allows outcomes to be linked to the application of different practice modules used in Xavier’s treatment (case history evidence).

Case #2

Jane is a 14-year-old Native Hawaiian female referred to a community clinic by her primary care provider. She and her immediate family have recently moved from the island of O’ahu to the west coast of the mainland United States. Shortly before their move, Jane developed symptoms of post-traumatic stress disorder (PTSD) in response to being raped by an 18-year-old male cousin. She sleeps poorly, has frequent nightmares related to the event, reports difficulty concentrating, and avoids being alone at night (when the attack occurred). Her parents also note that she has seemed distant and detached since the incident but are unsure if this is due to the trauma or “just being a teenager.” Jane’s parents have also been hesitant to sever ties with the cousin and noted that they believe in maintaining all family connections, despite wrongdoing, something that has caused conflict between them and Jane. Searched in a manner similar to that described above, the PWEBS database indicates that no studies have provided evidence at the “good support” evidence level for female native Hawaiians of a similar age. In this situation, where no specific studies match the exact combination of ethnic, gender, and presenting problem characteristics possessed by the client, the clinician is faced with implementing treatment elements “off-label.” Removing ethnicity as a search parameter on a subsequent query yields five papers that provide evidence for the effectiveness of psychoeducation (100% of the studies), cognitive restructuring (83%), and exposure (67%) in the treatment of traumatic stress. Although these have not been empirically tested with youth from Jane’s cultural background, when paired with careful progress monitoring (case history evidence) to ensure that the treatment is having its desired effect, they nevertheless represent a promising starting point for building a treatment plan.

Although the therapist had no previous experience with Native Hawaiian youth, she, carefully and directly assesses Jane’s personal and cultural values and treatment preferences. Despite her dissatisfaction with her parents over their continued contact with her cousin, she has expressed a desire for ho‘oponopono, a traditional intervention “to make right” (Kanuha, 2005). Through consultation with colleagues, the therapist identifies relevant readings on the concept (causal mechanism evidence) and identifies points of connection to family communication training. Making use of the ho‘oponopono structure, Jane’s therapist meets with her and her parents simultaneously and, with the consent of the family, assumes the directive, mediator role of a haku. All statements by family members are addressed to the mediator, rather than to one another. During this time, the therapist incorporates multiple components of psychoeducation about trauma, its impact, and the potentially damaging effects of remaining in contact with Jane’s cousin into ho‘oponopono. The therapist also attempts to help each family member to identify and evaluate their cognitions surrounding Jane’s trauma. However, this meets with limited success, based on symptom progress monitoring (case history evidence), as many of their explanations of the event are rooted in strong spiritual beliefs which the therapist is hesitant to challenge directly. Instead, having already secured family buy in, she transitions to more individual sessions with Jane where she conducts a systematic exposure regimen. Ongoing monitoring of trauma symptoms and sleep using a MFS show a decrease over the subsequent eight weeks (case history evidence), indicating that the exposures appear to be effective for this particular case. After these therapeutic gains have been maintained for an additional two weeks, the therapist decides to introduce other relevant modules, including relaxation and communication skills for Jane to use when interacting with her parents.

Special Considerations when Applying the Modular Approach to Culturally-Diverse Youth

Despite showing promise for increasing the availability of EBP for ethnically and culturally diverse children and adolescents, the modular approach and the decision support tools described above are not a panacea and should be evaluated in the context of a number of notable considerations and limitations. Issues discussed below include those that apply to the specific decision support tools described in this paper as well as those that relate to the interface between modularity and diversity factors more generally.

Modular approaches supported by the DMM are restricted first and foremost by the boundaries of the services research evidence base and its tendency to prioritize internal over external validity. Interventions based on this source of evidence still suffer from large amounts of missing information about how cultural minority youth respond to treatment, a limitation that is clearly reflected in the studies included in the PWEBS database. Table 2 displays ethnic group-specific findings, drawn from the PWEBS database, for different presenting problems. As can be seen in the table, up to 63% of the studies that have been conducted did not report ethnicity data and reports of ethnic representation differ by presenting problem. Among the more commonly researched problems (i.e., those with at least 20 studies), depression studies appear to have the most inclusive ethnic representation. These findings are consistent with other studies of the cultural relevance of evidence-based interventions and suggest that more inclusive studies are needed (e.g., Woidneck, Pratt, Gundy, Nelson, & Twohig, 2012). As a consequence, it is possible to encounter a combination of client characteristics for which the literature provides no data to guide treatment. This “empty cell problem” was illustrated in the case example involving Jane and traumatic stress. The process of going “up a node” to a level where more evidence is available provides an opportunity to make clinical decisions based on related information even if specific findings are not available or sufficient for the client’s ethnic group (Chorpita, Daleiden, & Weisz, 2005b). In many ways, this is similar to the “leap of faith” required when implementing full EST protocols with novel groups for which there is no empirical support (i.e., approach #2; Table 1).

Table 2.

Percentage of studies within diagnostic categories representing different ethnic groups in the PracticeWise Evidence-Based Services (PWEBS) database

Amer. Indian or Alaskan Native Asian African American Caucasian Hispanic/Latino Multiethnic Native Hawaiian or Pacific Islander None reported
Anxiety (n = 70) 4% 11% 20% 36% 11% 4% 3% 63%
Attention Problems (n = 25) 0% 4% 8% 24% 8% 4% 0% 76%
Autism Spectrum (n = 6) 0% 17% 33% 50% 17% 0% 0% 50%
Depression (n = 22) 5% 18% 23% 68% 27% 9% 0% 23%
Disruptive Behavior (n = 83) 5% 6% 37% 45% 17% 4% 0% 47%
Eating Problems (n = 4) 0% 0% 25% 100% 25% 0% 0% 0%
Substance Use (n = 11) 9% 27% 46% 64% 55% 9% 9% 18%
Suicidality (n = 2) 0% 0% 100% 100% 0% 0% 0% 0%
Traumatic Stress (n = 9) 0% 0% 56% 67% 33% 11% 0% 33%
Total (n = 232) 4% 9% 29% 45% 18% 5% 1% 50%
Open in a new tab

Note. The PWEBS database allows the user to indicate the level of evidence provided by different studies on a 1 to 5 scale, based on their design. Table 2 represents reported ethnicity for studies of interventions rated at Level 1 (“Best Support”) or Level 2 (“Good Support”). See Chorpita & Daleiden (2007) for specific criteria. Level 2 is similar to the American Psychological Association’s operationalization of a “probably efficacious” intervention (Chorpita et al., 2002). The database uses an inclusive “at least one” strategy in which treatment components are coded as effective for a particular ethnic minority group if at least one participant in a study was a member of that group (Chorpita & Daleiden, 2009); therefore, the actual ethnic representativeness of the outcome literature is likely substantially lower.

Furthermore, the emphasis of the modular approach is primarily on psychotherapy content, but some cultural adaptations of interventions attend to both content and process (e.g., Hwang, 2009). Although the modular framework allows for the inclusion of nearly limitless culturally-specific elements of the therapeutic process (e.g., taking longer to orient to therapy or particular modules, relationship factors), it does not provide specific recommendations for what those changes might look like. As discussed previously, existing decision support tools only provide a “jumping off point” for the use of EBP with cultural minority youth and must be combined with other knowledge at both the group (i.e., local aggregate) and individual (case history evidence) levels within a larger decision-making framework to be most useful (Chorpita & Daleiden, in press). For example, knowledge about relationship building with diverse populations, a key element of effective interventions (Lambert & Barley, 2001), could be derived from theory and personal or agency clinical expertise (causal mechanism evidence) as well as from the empirical literature on psychotherapy’s common factors (general services research evidence) and, perhaps, data gathered from previous clients in the treatment setting of interest (local aggregate). As in any psychotherapy scenario, all available and relevant information sources should be integrated at all points of the psychotherapy process, as determined by key decision makers. In this way, the modular approach is not unique and carries the same limitations and opportunities inherent in any treatment approach. The value of modularity in psychotherapy practice in this context is that it requires that the coordination processes (e.g., logic models) and information sources and resources (e.g., codified psychotherapy procedures, measurement feedback) be more explicit, which fosters efficient and visible methods to manage the delivery of clinical care.

The possibility also exists that some modules/modular therapy content might be inappropriate for particular cultural groups (Hays, 2009). Although the external validity of existing modules may be evaluated by formally testing them with samples of the cultural group for which they are being considered, part of this can also be achieved in the context of case history and local aggregate evidence where practitioners may select (or deselect) certain elements in treatment and test any assumptions about differential applicability. Nevertheless, determining whether a module is truly applicable to a given case can be difficult. Based on findings that the efficacy of existing evidence-based interventions is not necessarily reduced when they are used unadapted with ethnic minority youth (Huey & Polo, 2008), cautious application of any module that is indicated by a client symptom presentation may be appropriate when no other evidence (case history, local aggregate, or causal mechanism) clearly suggests it would be ineffective (Miranda, Nakamura, & Bernal, 2003). In such cases, careful weekly standardized assessment of client progress (case history evidence) – which can be tracked using an electronic dashboard/MFS or simpler paper-and-pencil methods – is all the more essential. Multiple authors (e.g., Borckardt et al., 2008; Jensen et al., 2012), have advocated for using such a case study approach, which allows for ongoing clinical hypothesis testing, to expand and evaluate the reach of empirical findings in the practice setting. Borckardt and colleagues (2008) suggest using continual assessment within the context of an A–B design to examine change from baseline (A) to intervention (B). When first applying new techniques to the treatment of specific groups of clients or presenting problems, case history and local aggregate evidence can allow practitioners to begin to examine for themselves which modules are effective or ineffective for which clients, under what conditions, thereby developing important practice-based evidence. This is advisable within a co-design or a quality-improvement framework.

Perhaps most importantly, the claim that the implementation of modules can be effectively tailored at the level of the individual client presupposes a “culturally-competent” clinician. Although APA (2003) guidelines have underscored the importance of cultural competence, there is still debate about the actual meaning of cultural competence as well as how best to develop and maintain the competence of clinicians. Sue, Zane, Hall, and Berger (2009) identified that definitions of cultural competence typically involve one or more of the following dimensions: (a) provider characteristics (e.g., cultural sensitivity, knowledge), (b) collections of skills or tactics to be utilized (or not) depending on the circumstances, and (c) processes involved in the interactions among clients, therapists, and interventions. Each of these conceptualizations is compatible with the modular approach and an individualized tailoring of therapy content. For instance, providers who have the necessary characteristics to function effectively when treating youth from particular cultural groups may already be adequately equipped to make initial determinations about the appropriateness of particular modules.

Finally, as research on modular approaches to psychotherapy continues, careful attention should be paid to the parameters of the concept of modularity, especially as applied with diverse populations, to determine what characteristics of modular psychotherapy approaches are most predictive of therapist acceptability and client outcomes. As a starting point for this work, we maintain that a “modular” approach should conform to the principles of modularity introduced by Chorpita et al. (2005), which include (a) partial decomposability, (b) proper functioning, (c) standardized interface, and (d) information hiding; although future research should evaluate the validity of this assertion.

Conclusion

The goal of the current paper was to describe and evaluate ways in which modular psychotherapy design may help to bridge the gap between treatment approaches that prioritize research evidence and those that prioritize local knowledge when delivering interventions to ethnic and cultural minority youth. In many ways, modularity helps to address the impossible task of establishing sufficient treatment evidence for every psychosocial treatment package with every cultural (and subcultural) group, and may open the door for more effective and efficient dissemination and implementation. However, this does not imply that efforts to build an evidence base for individual ESTs with specific treatment populations should be curtailed. The representativeness of research samples must also be continually improved.

Furthermore, although it facilitates flexible treatment, the modular approach – whether applied within a co-design or quality improvement framework – complicates the clinical and research landscape relative to manualized ESTs by acknowledging the complexity of real-world cases, introducing a wider array of variables and variable combinations, and placing increased responsibility for culturally-competent care and evaluation on clinicians. For this reason, further dissemination of the modular approach should occur alongside improved efforts to provide clinicians with more general training in cultural competence and quality assessment practices.

Acknowledgments

This publication was made possible in part by funding from grant numbers F32 MH086978 and K08 MH095939, awarded to the first author from the National Institute of Mental Health (NIMH).

Dr. Lyon is an investigator with the Implementation Research Institute (IRI), at the George Warren Brown School of Social Work, Washington University in St. Louis; through an award from the National Institute of Mental Health (R25 MH080916) and the Department of Veterans Affairs, Health Services Research & Development Service, Quality Enhancement Research Initiative (QUERI).

Dr. Chorpita is the President of PracticeWise, LLC, which maintains the Evidence Based Services (PWEBS) database discussed in this article.

References

  1. American Psychological Association. Guidelines on multicultural education, training, research, practice, and organizational change for psychologists. American Psychologist. 2003;58:377–402. doi: 10.1037/0003-066x.58.5.377. [DOI] [PubMed] [Google Scholar]
  2. American Psychological Association. Evidence-based practice in psychology. American Psychologist. 2006;61:271–285. doi: 10.1037/0003-066X.61.4.271. [DOI] [PubMed] [Google Scholar]
  3. Bagner DM, Eyberg SM. Parent-Child Interaction Therapy for disruptive behavior in children with mental retardation: A randomized controlled trial. Journal of Clinical Child and Adolescent Psychology. 2007;36:1–12. doi: 10.1080/15374410701448448. [DOI] [PubMed] [Google Scholar]
  4. Balas EA, Boren SA. Managing clinical knowledge for healthcare improvements. In: Schattauer V, editor. Yearbook of med informatics. New York: Stuttgart; 2000. pp. 65–70. [PubMed] [Google Scholar]
  5. Barlow DH, Allen LB, Choate ML. Toward a unified treatment for emotional disorders. Behavior Therapy. 2004;35:205–230. doi: 10.1016/j.beth.2016.11.005. [DOI] [PubMed] [Google Scholar]
  6. Barrera M, Castro FG. A heuristic framework for the cultural adaptation of interventions. Clinical Psychology: Science and Practice. 2006;14:311–316. [Google Scholar]
  7. Barrera M, Biglan A, Taylor TK, Gunn BK, Smolokowski K, Black C, et al. Early elementary school intervention to reduce conduct problems: A randomized trial with hispanic and non-hispanic children. Prevention Science. 2002;3:83–94. doi: 10.1023/a:1015443932331. [DOI] [PubMed] [Google Scholar]
  8. Benish SG, Quintana S, Wampold BE. Culturally adapted psychotherapy and the legitimacy of myth: A direct comparison meta-analysis. J of Counsel Psych. 2011;58:279–289. doi: 10.1037/a0023626. [DOI] [PubMed] [Google Scholar]
  9. Bernal G, Bonilla J, Bellido C. Ecological validity and cultural sensitivity for outcome research: Issues for the cultural adaptation and development of psychosocial treatments with Hispanics. Journal of Abnormal Child Psychology. 1995;23:67–82. doi: 10.1007/BF01447045. [DOI] [PubMed] [Google Scholar]
  10. Bernal G, Jimenez-Chafey MI, Domenech-Rodriguez MM. Cultural adaptation of treatments: A resource for considering culture in evidence-based practice. Professional Psychology: Research and Practice. 2009;40:361–68. [Google Scholar]
  11. Bernal G, Scharrón-Del-Río MR. Are empirically supported treatments valid for ethnic minorities? Toward an alternative approach for treatment research. Cultural Diversity and Ethnic Minority Psychology. 2001;7:328–342. doi: 10.1037/1099-9809.7.4.328. [DOI] [PubMed] [Google Scholar]
  12. Bickman L. A measurement feedback system (MFS) is necessary to improve mental health outcomes. J of the Amer Acad of Child & Adolescent Psychiatry. 2008;47:1114–1119. doi: 10.1097/CHI.0b013e3181825af8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Borckardt JJ, Nash MR, Murphy MD, Moore M, Shaw D, O’Neil P. Clinical practice as a natural laboratory for psychotherapy research: A guide to case-based time-series analysis. American Psychologist. 2008;63:77–95. doi: 10.1037/0003-066X.63.2.77. [DOI] [PubMed] [Google Scholar]
  14. Chambless DL, Sanderson WC, Shoham V, Bennett-Johnson S, Pope KS, et al. An update on empirically validated therapies. Clinical Psychology. 1996;49:5–18. [Google Scholar]
  15. Chambless DL, Hollon SD. Defining empirically supported therapies. Journal of Consulting and Clinical Psychology. 1998;66:7–18. doi: 10.1037//0022-006x.66.1.7. [DOI] [PubMed] [Google Scholar]
  16. Cheng C, Leong FT, Geist R. Cultural differences in psychological distress between Asian and Caucasian American college students. Journal of Multicultural Counseling and Development. 1993;21:182–190. [Google Scholar]
  17. Chorpita BF, Becker KD, Daleiden EL. Understanding the common elements of evidence-based practice: Misconceptions and clinical examples. Journal of the American Academy of Child and Adolescent Psychiatry. 2007;46:647–652. doi: 10.1097/chi.0b013e318033ff71. [DOI] [PubMed] [Google Scholar]
  18. Chorpita BF, Bernstein AD, Daleiden EL. Empirically guided coordination of multiple evidence-based treatments: An illustration of relevance mapping in children’s mental health services. Journal of Consulting and Clinical Psychology. 2011;79:470–480. doi: 10.1037/a0023982. [DOI] [PubMed] [Google Scholar]
  19. Chorpita BF, Daleiden EL. Mapping evidence-based treatments for children and adolescents: Application of the Distillation and Matching Model to 615 treatments form 322 randomized trials. Journal of Consulting and Clinical Psychology. 2009;77:566–579. doi: 10.1037/a0014565. [DOI] [PubMed] [Google Scholar]
  20. Chorpita BF, Daleiden EL. Structuring the collaboration of science and service in pursuit of a shared vision. Journal of Clinical Child and Adolescent Psychology. doi: 10.1080/15374416.2013.828297. in press. [DOI] [PubMed] [Google Scholar]
  21. Chorpita BF, Daleiden EL, Weisz JR. Identifying and selecting the common elements of evidence-based interventions: A distillation and matching model. Mental Health Services Research. 2005a;7:5–20. doi: 10.1007/s11020-005-1962-6. [DOI] [PubMed] [Google Scholar]
  22. Chorpita BF, Daleiden EL, Weisz JR. Modularity in the design and application of therapeutic interventions. Applied & Preventive Psychology. 2005b;11:141–156. [Google Scholar]
  23. Chorpita BF, Weisz JR, Daleiden EL, Schoenwald SK, Palinkas LA, Miranda J, Higa-McMillan CK, Nakamura BJ, Austin AA, Borntrager C, Ward AM, Wells KC, Gibbons RD the Research Network on Youth Mental Health. Long term outcomes for the Child STEPs randomized effectiveness trial: A comparison of modular and standard treatment designs with usual care. Journal of Consulting and Clinical Psychology. doi: 10.1037/a0034200. in press. [DOI] [PubMed] [Google Scholar]
  24. Cook JM, Schnurr PP, Biyanova T, Coyne JC. Apples don’t fall far from the tree: Influences on psychotherapists’ adoption and sustained use of new therapies. Psychiatric Services. 2009;60:671–676. doi: 10.1176/appi.ps.60.5.671. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Daleiden E, Chorpita BF. From data to wisdom: Quality improvement strategies supporting large-scale implementation of evidence based services. Child and Adolescent Psychiatric Clinics of North America. 2005;14:329–349. doi: 10.1016/j.chc.2004.11.002. [DOI] [PubMed] [Google Scholar]
  26. Falicov CJ. Mexican Families. In: McGoldrick M, Giordano J, Garcia-Preto N, editors. Ethnicity and Family Therapy. 3. New York: Guilford; 2005. [Google Scholar]
  27. Gallardo ME, Parham TA, Trimble JE, Yeh CJ. Understanding the skills identification stage model in context. In: Gallardo M, Yeh CJ, Trimble JE, Parham TA, editors. Culturally Adaptive Counseling Skills: Demonstrations of Evidence-Based Practices. Thousand Oaks, CA: Sage; 2012. [Google Scholar]
  28. Griner D, Simth TB. Culturally adapted mental health interventions: A meta-analytic review. Psychotherapy: Theory, Research, Practice, Training. 2006;43:531–548. doi: 10.1037/0033-3204.43.4.531. [DOI] [PubMed] [Google Scholar]
  29. Hays P. Integrating evidence-based practice, cognitive-behavior therapy, and multicultural therapy. Professional Psychology: Research and Practice. 2009;40:354–360. [Google Scholar]
  30. Holleran Steiker LK, Castro FG, Kumpfer K, Marsiglia FF, Coard S, Hopson LM. A dialogue regarding cultural adaptation of interventions. Journal of Social Work Practice in the Addictions. 2008;8:154–162. [Google Scholar]
  31. Huey SJ. BRISC Research Summit. Seattle, Washington: 2012. Aug, What we know and don’t know about culturally responsive treatments for ethnically minority youth. [Google Scholar]
  32. Huey SJ, Polo AJ. Evidence-based psychosocial treatments for ethnic minority youth: A review and meta-analysis. J of Clin Child & Adoles Psychology. 2008;37:262–301. doi: 10.1080/15374410701820174. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Hurdle DE. Native Hawaiian traditional healing: Culturally-based interventions for social work practice. Social Work. 2002;47:183–192. doi: 10.1093/sw/47.2.183. [DOI] [PubMed] [Google Scholar]
  34. Hwang W. The Formative Method for Adapting Psychotherapy (FMAP): A community-based developmental approach to culturally adapted therapy. Professional Psychology: Research and Practice. 2009;40:369–377. doi: 10.1037/a0016240. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Jensen DR, Abbott MK, Beecher ME, Friner D, Golightly TR, Cannon JAN. Talking the pulse of the group: Utilization of practice-based evidence in group psychotherapy. Professional Psychology: Research and Practice. 2012;43:388–394. [Google Scholar]
  36. Jensen-Doss A, Hawley KM, Lopez M, Oserberg LD. Using evidence-based treatments: The experiences of youth providers working under a mandate. Professional Psychology: Research and Practice. 2009;40:417–424. [Google Scholar]
  37. Kanuha VK. Nā ‘Ohana: Native Hawaiian Families. In: McGoldrick M, Giordano J, Garcia-Preto N, editors. Ethnicity and Family Therapy. 3. New York: Guilford; 2005. [Google Scholar]
  38. Kazdin AE, Blase SL. Rebooting psychotherapy research and practice to reduce the burden of mental illness. Perspectives on Psychological Science. 2011;6:21–37. doi: 10.1177/1745691610393527. [DOI] [PubMed] [Google Scholar]
  39. Kendall PC, Chu B, Gifford A, Hayes C, Nauta M. Breathing life into a manual: Flexibility and creativity with manual-based treatments. Cognitive and Behavioral Practice. 1998;5:177–198. [Google Scholar]
  40. La Roche M, Christopher M. Changing paradigms from empirically supported treatments to evidenced based practice: A cultural perspective. Professional Psychology: Research and Practice. 2009;40:396–402. [Google Scholar]
  41. Lambert MJ, Barley DE. Research summary on the therapeutic relationship and psychotherapy outcome. Psychotherapy: Theory, Research, Practice, Training. 2001;38:357–361. [Google Scholar]
  42. López SR, Guarnaccia PJJ. Cultural psychopathology: Uncovering the social world of mental illness. Annual Review of Psychology. 2000;51:571–598. doi: 10.1146/annurev.psych.51.1.571. [DOI] [PubMed] [Google Scholar]
  43. Marsiglia FF, Kulis S. Diversity, oppression, and change: Culturally grounded social work. Lyceum Books; Chicago: 2009. [Google Scholar]
  44. McCabe KM, Yeh M, Garland AF, Lau AS, Chavez G. The GANA program: A tailoring approach to adapting Parent-Child Interaction Therapy for Mexican Americans. Education & Treatment of Children. 2005;28:111–129. [Google Scholar]
  45. Miranda J, Nakamura R, Bernal G. Including ethnic minorities in mental health intervention research: A practical approach to a long-standing problem. Culture, Medicine, and Psychiatry. 2003;27:467–486. doi: 10.1023/b:medi.0000005484.26741.79. [DOI] [PubMed] [Google Scholar]
  46. Nicolas G, Arntz DL, Hirsch B, Schmiedigen A. Cultural adaptation of a group treatment for Haitian American adolescents. Prof Psych : Research & Prac. 2009;40:378–384. [Google Scholar]
  47. PracticeWise. Evidence-Based Services Database. Satellite Beach, FL: Author; 2009. [Google Scholar]
  48. Puddy RW, Wilkins N. Understanding Evidence Part 1: Best Available Research Evidence. A Guide to the Continuum of Evidence of Effectiveness. Atlanta, GA: Centers for Disease Control and Prevention; 2011. [Google Scholar]
  49. Reid MJ, Webster-Stratton C, Hammond M. Enhancing a classroom social competence and problem-solving curriculum by offering parent training to families of moderate- to high-risk elementary school children. Journal of Clinical Child and Adolescent Psychology. 2007;36:605–620. doi: 10.1080/15374410701662741. [DOI] [PubMed] [Google Scholar]
  50. Resnicow K, Baranowski T, Ahluwalia J, Braithwaie RL. Cultural sensitivity in public health: Defined and demystified. Ethnicity & Disease. 1999;9:10–21. [PubMed] [Google Scholar]
  51. Roy-Byrne P, Craske MG, Sullivan G, Rose RD, Edlund MJ, Lang AJ, et al. Delivery of evidence-based treatment for multiple anxiety disorders in primary care. JAMA. 2010;303:1921–1928. doi: 10.1001/jama.2010.608. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Rotheram-Borus MJ, Swendeman D, Chorpita BF. Disruptive innovations for designing and diffusing evidence-based interventions. Amer Psychologist. 2012;67:463–476. doi: 10.1037/a0028180. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Schulte D, Kunzel R, Pepping G, Schute-Bahrenberg T. Tailor-made versus standardized therapy of phobic patients. Adv in Beh Research and Therapy. 1992;14:67–92. [Google Scholar]
  54. Smith TB, Rodríguez MD, Bernal G. Culture. Journal of Clinical Psychology and Psychiatry: In Session. 2011;67:166–175. doi: 10.1002/jclp.20757. [DOI] [PubMed] [Google Scholar]
  55. Sue S, Zane N, Hall N, Berger LK. The case for cultural competency in psychotherapeutic interventions. Annual Review of Psychology. 2009;60:525–548. doi: 10.1146/annurev.psych.60.110707.163651. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Weisz JR, Chorpita BF, Palinkas LA, Schoenwald SK, Mirandea J, Bearman SK, et al. Testing standard and modular designs for psychotherapy treating depression, anxiety, and conduct problems in youth. Arch of Gen Psych. 2012;69:274–282. doi: 10.1001/archgenpsychiatry.2011.147. [DOI] [PubMed] [Google Scholar]
  57. Woidneck MR, Pratt KM, Gundy JM, Nelson CR, Twohig MP. Exploring cultural competence in Acceptance and Commitment Therapy outcomes. Professional Psychology, Research and Practice. 2012;43:227–233. [Google Scholar]

RESOURCES