Modified Therapeutic Community for Co-Occurring Disorders: Single Investigator Meta Analysis

Abstract

This paper presents the results of a meta-analysis for a single investigator examining the effectiveness of the modified therapeutic community (MTC) for clients with co-occurring substance use and mental disorders (COD). The flexibility and utility of meta-analytic tools are described, although their application in this context is atypical. The analysis includes four comparisons from three studies (retrieved N=569) for various groups of clients with COD (homeless persons, offenders, and outpatients) in substance abuse treatment, comparing clients assigned either to an MTC or a control condition of standard services. An additional study is included in a series of sensitivity tests. The overall findings increase the research base of support for the MTC program for clients with COD, as results of the meta-analysis indicate significant MTC treatment effects for 5 of the 6 outcome domains across the four comparisons. Limitations of the approach are discussed. Independent replications, clinical trials, multiple outcome domains and additional meta-analyses should be emphasized in future research. Given the need for research-based approaches, program and policy planners should consider the MTC when designing programs for co-occurring disorders.

Background

The modified therapeutic community (MTC)¹ treatment model, based on the theoretical framework of the standard TC model (e.g., ¹), was adapted in the early 1990s for those with co-occurring substance use and mental disorders (2, 3, 4, 5). Core principles of the TC that remain pertinent to the treatment of co-occurring disorders include: providing a highly structured daily regimen; fostering personal responsibility and self-help; using peers as role models and the peer community as the healing agent within a strategy of community-as-method (the community provides both the context for and mechanism of change); regarding change as a gradual, developmental process and moving clients through treatment stages; stressing work and self-reliance through the development of vocational and independent living skills; and promoting prosocial values within healthy social networks to sustain recovery. Within this framework, three key alterations are needed to respond to the individual needs and impairments of the client with co-occurring disorders: more flexibility, less intensity, and more individualization. (For a more complete description of the MTC for clients with co-occurring disorders, including treatment manuals and guides to implementation, see 2, 3, 4, 5)

Goals

This paper presents a meta-analysis of data from three studies examining the effectiveness of MTC treatment for clients with co-occurring disorders. Data from a fourth study are included in a separate sensitivity analysis. Since co-occurring disorders are associated with multiple problems requiring multi-faced interventions, measures from six outcome domains were assessed (substance abuse, mental health, crime, HIV-risk, employment, and housing). Meta-analysis was used to determine the consistency of effect sizes across studies and, if consistent, to pool effect sizes for concise, quantitative estimates of MTC treatment effectiveness. As an analytic tool, meta-analysis is flexible and has been put to novel uses, including integrating findings in a multi-site study (6) and generating new hypotheses (7). While meta-analysis is frequently used to synthesize just a few studies (e.g., 8), it has not, to the authors’ knowledge, been used elsewhere to synthesize the work of a single investigator; such innovative use could, perhaps, illuminate critical choices of future design and methods more directly than a qualitative, narrative review.

The paper begins with descriptions of the four studies conducted in a variety of settings and with differing populations of clients with co-occurring disorders (e.g., homeless, offenders, outpatients in substance abuse treatment, and those with HIV/AIDS). Next, the paper describes the meta-analytic techniques used to bring together the results of three of these studies (homeless, offenders, and outpatients) in four comparisons. Various sensitivity analyses, which include the fourth study (HIV/AIDS) are also described for their utility in determining whether conclusions are dependent on a small percentage of all effect sizes and in reducing the appearance of bias. The paper concludes with a discussion of the findings and proposes an agenda for future research.

Descriptive Synthesis of Four Studies

The three studies included in the meta-analysis represent three different MTC settings and co-occurring disorders populations: (Study 1, Homeless) homeless persons in the New York City region (n=232); (Study 2, Offender) male offenders in the Colorado criminal justice system (n=139); and (Study 3, Outpatient) an outpatient setting in Philadelphia (n=198). A fourth study (Study 4, HIV/AIDS), a residential program for people with HIV/AIDS, was included in the analysis as part of the sensitivity testing. Each study compared an MTC program to a control group; Study 1 investigated two experimental conditions (i.e., “MTC-Moderate,” a fully developed MTC model; “MTC-Low,” a less demanding version of the MTC program). The sample size for the control group in Study 1 was divided in half to insure that power was not overestimated. All studies employed random assignment except Study 1, which used sequential assignment (9). A more complete narrative description of these studies, including program and subject characteristics, can be found in a recent publication (10), which presented a synthesis of the interventions and the methodological procedures of the three studies, study-by-study.

Study 1 — Homeless

This series of studies began in 1991 with Study 1, which initiated the development of the MTC, a comprehensive residential program with a planned stay of 12 months. The interventions for this study constituted the core set of MTC interventions either “as is” or “as adapted” (for a full description of the program see 3–5). Homeless men and women (n = 342) with MICA disorders were sequentially assigned (9) to one of two experimental (E) MTC groups (MTC-Moderate or MTC-Low) or to a control (C) group that provided treatment-as-usual. The two MTC programs were similar in planned duration of stay, structure, and array of interventions, but the MTC-Low program was modified to have lower demands and more staff guidance.

In an intent-to-treat analysis of all study entrants, follow-up interviews were obtained at 12 months post baseline for 65% of MTC-Moderate, 70% of MTC-Low, and 73% of C group clients. As compared to the C group, outcomes for MTC-Low showed significant improvements on all measures of substance abuse and employment, while MTC-Moderate differed significantly only on employment. Although differences from the C group were not significant for either of the MTC groups on measures of mental health, crime, or HIV risk behavior, the pattern of findings (indicating an advantage for the MTC group) was maintained. Findings also showed greater treatment effects for MTC-Low in comparison to the more rigorous MTC-Moderate. While the retention rates for both E groups compared favorably to those reported in the literature (11, 12), residents assigned to MTC-Low were more likely to be retained for 12 months compared to subjects in MTC-Moderate (56% vs. 34%, respectively). The results of this study provide some evidence of the comparative effectiveness of the MTC approach and, more particularly, of a less demanding version of the MTC model (13).

Study 2 — Offender

Upon completion of Study 1, the investigators studied the effectiveness of the MTC approach for offenders with MICA disorders. Study 2, conducted within the Colorado Department of Corrections, adapted the TC for offenders with MICA disorders, including: an emphasis on criminal thinking and behavior; recognition and understanding of the interrelationship of substance abuse, mental illness, and criminality (triple recovery); operational adjustments to comply with facility security guidelines; and expansion of the treatment team to include security personnel and other Department of Corrections staff.

The study randomly assigned 185 male inmates with MICA disorders either to the MTC program (E) or to a mental health treatment program (C). The planned duration of stay was 12 months for the MTC program; the duration of the C group program was considered to be variable (for a full description of the program see 14). One year after being released from prison, retrieval rates for follow-up interviews were 82% for the MTC group and 69% for the C group; the prison MTC alone was somewhat effective in reducing reincarceration..

An intent-to-treat analysis of all study entrants showed that, one-year post prison release, those who received MTC treatment both in prison and in aftercare (MTC+Aftercare) had significantly lower rates of reincarceration than did those in the C group. Significant differences from the C group were also found for those who received MTC treatment both in prison and in aftercare (MTC+Aftercare) across a variety of crime measures (i.e., any criminal activity, and alcohol or drug related criminal activity); these differences persisted after an examination of various threats to validity (e.g., initial motivation, duration of treatment, exposure-to-risk) (15). Significant differences favoring the MTC+Aftercare group were likewise observed for substance abuse outcomes (i.e., drinking to intoxication, using illegal drugs) (16). Overall, Study 2 findings were positive and consonant with other studies of integrated prison and aftercare TC programs for substance abusing (non-MICA) offenders (e.g., ^{17, 18}), although qualified by a potential for selection bias (i.e., differences in motivation on entry into aftercare).

Study 3 — Outpatient

While the investigative team continued to work in criminal justice settings, it became apparent that services were needed to bolster treatment for clients with co-occurring disorders in outpatient substance abuse programs. Study 3 sought to evaluate the effectiveness of an enhanced treatment track, Dual Assessment & Recovery Track, or DART, which added MTC features (e.g., community meetings) and three targeted MTC interventions (i.e., psycho-educational seminar, trauma-informed addictions treatment, and case management). DART programming was delivered as part of the outpatient program, with DART elements replacing some standard individual and group activities, and remained within the structure of 9 hours per week of program activities (3 hours on each of 3 days) for the 12-week program duration.

Study 3 (19) screened male and female clients on their admission to the outpatient addictions treatment facility to identify those with psychological symptoms (n=240), who were then randomly assigned either to the E group (the enhanced DART treatment) or to the C group (basic program services without enhancements). An intent-to-treat analysis of all study entrants at one-year post treatment follow-up retrieved 85% of E group clients and 80% of C clients. Compared to C, the E group, as expected, had significantly better outcomes on measures of psychological symptoms and on one key measure of housing stability, “lived where paid rent.” No significant differences between the groups were evident for measures of substance abuse, crime, HIV-risk behavior or employment, although these findings generally showed greater improvements for the E group. The results are qualified because the study lacked an untreated or low treatment control group and findings were obtained for only a few variables. Still, improvements in substance abuse and trauma that were detected for the E group were similar to other reported studies, and group differences were found in outcome domains targeted by the additional interventions. The study provides some support for the effectiveness, on specific outcomes, of outpatient substance abuse treatment programs that add MTC features and selected MTC interventions to strengthen their capacity to treat co-occurring disorders (19 Sacks et al., 2008).

Study 4 — HIV/AIDS

In the late 1990s individuals triply diagnosed with HIV/AIDS and co-occurring disorders were the focus of a special government initiative (20, 21), underscoring the importance of determining both effectiveness and cost of treatment models for this population. The participating site, a Gaudenzia, Inc. facility, provided six months of core residential MTC treatment before referring clients to aftercare services at outpatient sites operated either by Gaudenzia or by other agencies. To accommodate HIV/AIDS and co-occurring disorders, the core residential program altered the structure of the TC model to deliver a unique combination of fully integrated medical/psychiatric/nursing care. The adaptations ensured that clients with co-occurring disorders, who were AIDS symptomatic (program eligibility criteria) and physically ill when they entered the program, were medically stabilized as rapidly as possible, and that their physical and mental health care was integrated within the residential substance abuse treatment program. One hundred thirty-five Study 4 participants entered the core MTC residential treatment; 57% completed the six-month program. The retrieval rate at one-year was 67%; follow up interviews occurred at 12-months post baseline for dropouts and 12-months post residential treatment for subjects who completed the residential phase of treatment.

Study 4 randomly assigned men and women who had completed the core residential MTC program (n = 77) to either MTC-A1 or MTC-A2 for aftercare. The MTC-A1 group was referred to community-based agencies for post residential MTC treatment, as had been established prior to the study, but Gaudenzia staff continued to monitor these clients for a full 90 days to encourage adherence to their medical and service plan. The clients assigned to MTC-A2 received an integrated MTC aftercare program of outpatient activities that were delivered in the residential facility. The MTC aftercare program, which had a planned duration of six months, incorporated case assistance and skills development, peer community meetings and activities, family/significant other support groups, a peer advocacy group and activities, a Re-Entry Group using tools for self-management, and a Health and HIV/AIDS Self-Management Group.

Analysis found improvements occurring predominantly during the residential phase for the subjects randomized into the two aftercare conditions (n=77). Clients continued to make smaller improvements and their gains were stabilized during aftercare; however, no differences between the groups were evident. These findings provide some support for the MTC model in treating triply diagnosed clients, particularly for the residential phase of treatment (22).

Summary

In summary, the series of MTC programs studied the effectiveness of the MTC approach in comparison with alternative treatments for different populations with co-occurring disorders in a variety of treatment settings. In the course of these studies, the MTC program was altered to accommodate both the population and the setting while retaining core TC elements. In every study, significantly better outcomes emerged for the MTC group; however, the measures and domains in which differences were evaluated and detected varied from study to study. The availability of data across the studies enabled the examination of uniformity of the studies’ findings.

Quantitative Synthesis

Rationale

The quantitative analysis determines the consistency of the results from 4 comparisons of MTC treatment across six outcome domains and, if consistent, examines the size of the effects. The assessment of multiple measures and outcome domains is of interest in the field because: (1) clients with co-occurring disorders have multiple problems in multiple domains; (2) the MTC intervention is meant to address multiple problem areas; and (3) multiple measurements and domains provide a more comprehensive picture of treatment effectiveness.

Heterogeneity

The Cochrane Handbook specifically advises reviewers to consider applying meta-analysis only “when a group of trials is sufficiently homogeneous in terms of participants, interventions and outcomes to provide a meaningful summary” (23; section 8.7.1). Higgins and Green (23) identified three broad types of heterogeneity — clinical, methodological, and statistical. Clinical heterogeneity refers to differences among studies in participant characteristics, intervention components and implementation. Methodological heterogeneity refers to differences in research design, such as the use of random assignment to study conditions, and the particular ways in which outcomes are defined and measured. Statistical heterogeneity refers to study differences in the size of effects observed. A degree of clinical heterogeneity among studies to be synthesized is reasonable and appropriate. Some conditions are, by their nature, clinically heterogeneous, as is the case with co-occurring disorders, where different individuals may have different mental health diagnoses, use different substances, and present for treatment or services in different settings and under different terms of referral. Some of this heterogeneity may be more apparent than real; for example, individuals with co-occurring disorders may cycle through different systems, moving from criminal justice, to homelessness prevention, to outpatient substance abuse treatment, such that the same people are being observed in multiple contexts. In addition to clinical heterogeneity in the condition itself, heterogeneity in treatment is a factor because treatments are necessarily complex and require adaptation to local circumstances. A review that is overly concerned about clinical heterogeneity may render itself less relevant to the clinical reality of multiple settings, populations and adaptations of an intervention, and be too narrowly drawn to permit broad application of its findings. The key is to take into account the complexity of the condition and the treatment approaches commonly employed, striking an appropriate balance between heterogeneity, on the one hand, and generalizability, on the other.

In examining the research of a single investigator, the meta-analytic techniques in this paper were used to answer two essential questions. The first question was whether the effect sizes were consistent for each outcome domain across the studies reviewed. If the effect sizes in a given domain were consistent, meta-analysis could answer the second question, and determine how effective the MTC was in producing positive outcomes in that domain. In short, meta-analysis was used to assess the consistency of results in each domain and, where consistent, results of different projects were combined to test the effectiveness of the MTC.

Operations

Sample of Studies

Each of the 3 studies included in the meta-analysis compared an MTC treatment condition with a treatment services as usual control condition (13, 15, 19, 22). Because Study 4, described above, had two MTC conditions, and did not have a treatment-as-usual control condition, data from the study were not included in the meta-analysis, but the effect of that exclusion was explored in sensitivity analysis described below.

Selecting Measures

Some aspects of syntheses of studies conducted by a single investigator are less commonly encountered in other applications of meta-analysis. One such aspect is access to the raw data from the primary studies, creating a need to select those measures to be synthesized from among all potential outcome measures. For example, in an outcome domain such as criminal behavior, an investigator may have data on the number and timing of offenses, arrests, and convictions, as well as the number and timing of any periods of incarceration. Offenses, arrests, and convictions each could be summarized in terms of their number, their timing, or whether they occurred at all during the follow-up period, but generally, it would not be useful to include effects captured in all of these related ways in the same synthesis. For this reason, some system for selecting from among potential measures is needed.

Because a single investigator usually has a stake in the outcome of the synthesis, it is important that measures be selected logically and transparently to reduce the appearance of bias; therefore, a comprehensive description of data collected and a detailed presentation of the criteria used to include or exclude measures are required. In this instance, several points were considered to form the basis for inclusion or exclusion of any given measure in this synthesis. First, measures that were cited in previous published reports of the primary studies were included to enable the reader to follow the development of an emerging research area from the first primary study to the results of the synthesis. Second, in all studies, multiple measures (if possible), and an equal number of measures, were selected to maximize the data being assessed for each domain. Further, in the outcome domains of interest, as yet, specific single measures have not emerged to represent the standard for the field. Third, similar data across studies were included whenever possible; however, if one or more studies had alternative measures of greater relevance to the domain, these data were used. The number of outcomes included in each domain ranged from one to four measures, and was equal to that of the fewest outcomes reported for any study. Fourth, measures were chosen without regard to the direction of statistical significance. This protected against bias that would have resulted from selecting only those measures that favored the MTC. To avoid bias, specific reasons for excluding each measure are presented along with information on the size and direction of excluded effects.

More specifically, this meta-analysis assessed data at 12-month follow-up for four comparisons, drawn from three of the four studies (Studies 1, 2 and 3), which contrasted a group that received MTC treatment with a group that received another treatment approach.² The six outcome domains of interest, which were measured across all studies, included substance use, mental health, criminality, HIV risk behavior, employment and housing. A crosswalk of studies was conducted to assess data collected for each domain (for a complete listing, see the earlier publication¹⁰).

Meta Analytic Procedures

The data were analyzed using Comprehensive Meta-Analysis Version 2 (24). Within each domain, outcomes were coded to insure a consistent direction of treatment effects; specifically, odds ratios were scored so that an odds ratio below the value 1.00 indicates greater improvement for the MTC treatment group relative to the comparison group. Similarly, an odds ratio above 1.00 indicates more improvement for the comparison group relative to the treatment group. This reporting convention is a conservative way of presenting modest effects.

Comprehensive Meta Analysis was used to analyze data to detect differential treatment effects (MTC vs. comparison). Data entered into the software program included sample sizes for each treatment group, Hedges g effect sizes with their standard errors for continuous outcomes (e.g., the frequency of drug use, the number of crimes committed), or, for dichotomous outcomes, the log odds ratios with their standard errors (e.g., any time spent in jail/prison, any needle use). Irrespective of the type of data entered, an overall, or pooled, effect size was calculated using the relative weight of each comparison, and an estimate of the heterogeneity of the results across the studies, the I², the Cochran Q statistic, and the corresponding p-value for similarity of the results across the studies. Results are reported more conservatively using a random effects model, which assumes a greater between-comparison variance than does a fixed-effects model. When effect sizes are consistent, the pooled effect size (e.g., an odds ratio) is a statistic to measure the magnitude (size) of the treatment effect in the combined data; whereas, the P value indicates the significance. For statistically significant results, the 95% confidence interval for a significant odds ratio should only contain values less than one, since an odds ratio of one indicates no treatment effect.

Sensitivity Analyses

Sensitivity tests examine the limits of the significant findings by assessing measurement sensitivity and study sensitivity. In testing the limits of the main synthesis, sensitivity analyses indicate the degree to which findings are “robust” and minimize the appearance of bias attributable to the synthesis being conducted by an individual investigator who has a stake in the outcome.

Measurement sensitivity tests are used to explore the effect of measure inclusion; namely, does measure selection influence the findings or, in other words, are the effects attributable primarily to one measure? One such sensitivity test, which can be undertaken only for domains with two or more measures, is to remove the measure with the largest treatment effect. A second sensitivity test is to add a measure with limited effects to the original outcomes; i.e., half the strength of the average of the observed measures. The purpose of this sensitivity test, like the first, is to investigate potential bias due to measure selectivity by adding measures with limited effects.

Study sensitivity tests assess potential influences resulting from the inclusion of studies; namely, are the pooled effects attributable primarily to one study? A third sensitivity test is to add a hypothetical study of average size (relative to the actual studies), containing the maximum number of measures in each domain, none of which indicate treatment effects (e.g., an odds ratio of 1.00) to the meta-analysis. A fourth sensitivity test is to include other actual studies (one by one) that differ from the included studies in some important respect to determine the effect of each study's exclusion on the results of the synthesis.

Results

Differential Treatment Effects — MTC vs. Comparison

All Outcome Domains

Magnitude of Effects

Table 1 reports the main findings of the meta-analysis across four comparisons and all domains. The magnitude of the pooled effect — odds ratios and 95% confidence intervals—indicated moderate and significant MTC treatment effects for five of six outcome domains. Table 1 shows that the MTC was associated with significantly greater improvements for five of the six outcome domains (significant p value shown in bold typeface): substance use, mental health, crime, employment, and housing. The effects were of moderate size and similar, ranging from 0.40 (employment) to 0.68 (mental health) with most of the other measures (substance abuse, crime and housing) having odds ratios close to that of mental health. Significant differences were not observed for the remaining domain, HIV-risk behavior.

TABLE 1.

Summary of meta-analysis combined study comparisons — random effects analysis (differential treatment effects: MTC vs. Comparison)

Domain	Effect Size Odds Ratio†	95% CI	p	Q (p)	I2
Substance abuse	0.650	(0.428 – 0.986)	.043*	4.998 (0.172)	39.977
Mental health	0.679	(0.478 – 0.966)	.031*	2.026 (0.567)	0.000
Crime	0.662	(0.454 – 0.966)	.032*	2.573 (0.462)	0.000
HIV-risk behavior	1.007	(0.659 – 1.539)	.974	3.068 (0.381)	2.225
Employment	0.404	(0.251 – 0.651)	.000***	6.351 (0.096)	52.761
Housing	0.634	(0.420 – 0.958)	.030*	0.370 (0.946)	0.000

^*p<0.05;

^***p<0.001

^†An odds ratio less than one indicates a greater improvement for clients in the MTC group than in the comparison group.

Consistency of Effects

The I² statistic was used to measure heterogeneity along with the p value computed for the Q statistic that also identifies the significance of the I² value. Consistent MTC effects across studies are indicated by I² statistics at or near zero. If the effects are in opposite directions, or if the magnitude of the effect sizes differs greatly, then the I² will be greater than zero. None of the I² statistics reported in Table 1 reach significance, which indicates consistency among the studies included in the analysis.

Each Outcome Domain

Table 1 presents results representing the magnitude, significance, and consistency of MTC effects for each domain; study-level results for each domain of interest are shown in Figure 1.

Figure 1.

^† Squares represent effects at the study level and diamonds symbolize effect sizes across studies.

The size of each square varies according to the weight attributed to the individual study.

Substance abuse

Moderate, significant (OR=0.65, p=0.05) and consistent (I²=40.00, p<0.18) effects across the studies favoring the MTC emerged for the substance abuse domain. The resulting odds ratios (pooled measures), which varied from 0.42 to 0.92, consistently favored the MTC in the four comparisons. Significant MTC effects were observed for two of the four comparisons: the MTC-Low comparison of Study 1, Homeless, and Study 2, Offender.

Mental Health

Moderate, significant (OR=0.68, p=0.04) and consistent (I²=0.00, p=0.57) MTC effects across the studies were also detected for the pooled effect size for the domain of mental health. The range of mental health treatment effects for the four comparisons was 0.53 to 1.03 with one study (Study 3, Outpatient) producing significant effects. Although not significant, the direction of treatment effects for one comparison (Study 1, Homeless, the MTC-Moderate comparison, OR=1.03) was in opposition to the other comparisons, which favored the MTC condition.

Crime

Similar effects emerged from the analysis of the crime domain, revealing MTC treatment effects that were moderate, significant (OR=0.66, p=0.04), and consistent (I²=0.00, p=0.47) across the studies. The MTC effect was positive across all four comparisons with odds ratios ranging from 0.38 to 0.91; one comparison (Study 2, Offender) reached statistical significance.

HIV risk behavior

The analysis of HIV risk behavior, including needle use and HIV-related sexual behavior, found little or no MTC treatment effect. Across studies, MTC treatment produced similar (I²=2.23, p=0.39) and non-significant effects (OR=1.01, p=0.98) in HIV risk behavior. In the four individual comparisons, the MTC effect varied from 0.26 to 1.13 and was never significant.

Employment

Across the studies, the employment domain indicated that MTC treatment produced consistent (I²=52.76, p<0.10), and statistically significant effects (OR=0.40, p<0.001). Treatment effects favoring MTC, ranging from 0.26 to 0.71, emerged for all comparisons, three of which were statistically significant (Study 1, Homeless, MTC-Moderate and -Low comparisons, and Study 2, Offender).

Housing

Results from the housing domain showed that MTC treatment produced moderate, significant (OR=0.63, p=0.03) and consistent (I²=0.00, p=0.95) effects across the comparisons. MTC treatment effects for the four individual comparisons ranged from 0.56 to 0.86, none of which was significant.

Sensitivity testing

This section discusses the results of four sensitivity tests, displayed in Table 2, which were designed to test the limits of the significant findings by either excluding measures with significant effects, or by adding studies whose measures showed limited, or no, treatment effects. Sensitivity test results (Table 2) have been organized to show tests assessing measurement sensitivity (Tests 1–2; Column 3–4) and study sensitivity (Tests 3–4; Columns 5–6). Apart from the addition of the HIV study, the treatment effects resulting from the sensitivity tests were expected to be less than those from the original findings.

TABLE 2.

Summary of sensitivity tests for the meta-analysis—random effects analysis (differential treatment effects: MTC vs. Comparison)

		Measurement Sensitivity		Study Sensitivity
Domain	Original Findings Effect Size Odds Ratioe(p)	Test 1a Effect Size Odds Ratioe (p)	Test 2b Effect Size Odds Ratioe (p)	Test 3c Effect Size Odds Ratioe (p)	Test 4d Effect Size Odds Ratioe (p)
column 1	2	3	4	5	6
Substance use	0.650 (.043*)	0.667 (.043*)	0.673 (.021*)	0.716 (.065)	0.666 (.027*)
Mental health	0.679 (.031*)	0.714 (.061)	0.685 (.035*)	0.710 (.042*)	0.705 (.048*)
Crime	0.662 (.032*)	0.692 (.048*)	0.735 (.105)	0.712 (.052*)	0.681 (.038*)
HIV-risk behaviorf	1.007 (.974)	-	-	-	-
Employment	0.404 (.000***)	n/a	0.521 (.000***)	0.480 (.004**)	0.495 (.014*)
Housing	0.634 (.030*)	n/a	0.711 (.104)	0.676 (.044*)	0.719 (.102)

^*p<0.05;

^**p<0.01;

^***p<0.001

^aTest 1 — The measure showing the largest MTC treatment effect was treated as an outlier and removed in every domain with 2 or more measures (i.e., substance use, mental health, crime).

^bTest 2 — A measure with limited treatment effect (i.e., half the strength of the average of the observed measures) was added to each study in every domain.

^cTest 3 — A hypothetical study with no treatment effects was added. This study had the average number of subjects (92 MTC & 51 comparison=143), and the same number of measures for each domain. An odds ratio of 1.00 and the mean standard error from all other studies were assigned to the hypothetical study.

^dTest 4 — All HIV cases were included.

^eAn odds ratio less than one indicates a greater improvement for clients in the MTC group than in the comparison group.

^fSensitivity tests were not conducted on the HIV-risk behavior domain since results for original findings were not significant.

Measurement Sensitivity

The measures that were included in each domain all are commonly used to assess problems among individuals with co-occurring disorders. As described in the section on Operations, the primary meta-analysis selected measures carefully to reduce bias and to produce a fair representation, neither overestimating nor underestimating the effects; statistical significance was not a criterion for inclusion. Bias due to measure selectivity was investigated in Tests 1 and 2, which removed measures with the strongest effects (Test 1) or added measures with limited effects (Test 2). These two sensitivity tests created extreme conditions to evaluate the robustness of the observed findings.

The first sensitivity test treated the measure with the largest MTC treatment effect as an outlier and removed it from the analysis; Test 1 could only include domains with two or more measures (i.e., substance use, mental health, and crime). This sensitivity test examines measure inclusion to reveal if measure selection influenced the findings, or if the effects were primarily attributable to one measure. Sensitivity Test 1, which omitted one measure from each domain, or 3 of 36 measures (8%), produced odds ratios between 0.67 and 0.71, with significant results retained in two domains: substance use and crime (Column 3).

Test 2 added a measure with limited effects to (half the strength of the average of the observed measures) to the original 36 outcomes. Across all domains, three of the five original significant findings remained significant after the addition of the limited effect measure: substance use, mental health and employment (Column 4); however, when a measure with no MTC treatment effects (odds ratio=1.00) was added to each comparison in each domain, a significant MTC effect emerged only for employment (data not shown).

Study Sensitivity

Tests 3 and 4 were conducted to investigate study sensitivity. Test 3 (Column 5) added a hypothetical study to the meta-analysis that was of average size (92 MTC and 51 Control) and that contained the maximum number of measures in each domain, none of which indicated MTC treatment effects (odds ratio 1.00). When this hypothetical study was included, significant results were sustained in three (mental health, crime, employment and housing) of the five domains that showed significant effects in the original analyses, as shown in Table 2 (Column 5).

The last sensitivity test, Test 4 (Column 6), added data from an “HIV study”; a study of MTC aftercare for HIV positive clients with co-occurring disorders who had completed residential MTC treatment. This study was excluded from the primary meta-analysis because its design did not include a C group comparable to those of the other studies. Treatment effects for the experimental (MTC aftercare) and control (treatment-as-usual) comparisons in the HIV study were limited in magnitude and significance compared to the other studies, yet, when added to the meta-analysis in Test 4, four of five domains remained significant (Column 6; substance use, mental health, crime, employment). In aggregate, these tests of study sensitivity imply that more than one additional study with no differential MTC effects would need to be included to alter the basic picture of effectiveness that emerged from the primary meta-analyses.

Discussion

Effectiveness of the MTC

The meta-analysis revealed significant MTC treatment effects for 5 of the 6 outcome domains across the four comparisons, which encompassed a large and varied population of clients with co-occurring disorders. Moderate and similar effects were evident in substance abuse, mental health, crime, employment, and housing; I² values indicated consistency of effects across domains. Specifically, significant MTC treatment effects emerged for substance abuse and employment in Study 1 (Homeless), for substance abuse, crime, and employment in Study 2 (Offender), and for mental health in Study 3 (Outpatient).

In sensitivity analyses, the effects remained strong when the best measures were excluded, when measures with no effects were added, and when either a hypothetical study with no effects or an actual study without a comparable C group was included. These results have considerable clinical relevance, since individuals with co-occurring disorders are typically perceived to have extensive needs and the MTC provides a comprehensive treatment model intended to meet those needs.

A non-significant positive pooled treatment effect for MTC emerged for HIV risk when the analysis was limited to measures of sexual risk. The four comparisons produced MTC effects, ranging from 0.08 to 1.30, one of which was significant (Homeless MTC-Low). This analysis found more heterogeneous effects (I²=60.79, p=0.06) for MTC treatment across the studies. While the analysis produced some evidence of positive MTC effects on HIV risk related to sexual behavior, none were evident for injection drug use. This finding is consistent with another meta-analysis (25) which evaluated HIV/AIDS risk-reduction interventions for clients enrolled in drug abuse treatment, reporting an overall odds ratio of 0.57; specific categories included odds ratios of 0.57 for knowledge/attitudes/beliefs, 0.63 for sexual behavior, 0.33 for risk-reduction skills, but 0.93 for injection practices (25). In view of the suggested benefit of MTC treatment to HIV outcomes, additional work is warranted to improve the capacity of MTC programs to reduce HIV risk for a range of behaviors, and to assess the utility of such efforts.

These findings add to, and strengthen, those reported previously (10) and have clinical relevance since individuals with co-occurring disorders are typically perceived to have multifaceted needs requiring multidimensional interventions (26). The earlier paper stated that significant MTC effects were detected in each outcome domain and for every comparison; however, meta-analytic techniques enabled effect size odds ratios to be computed and tested across the comparisons. Comparison- and domain-level results were explored in the meta-analysis, which allowed specific values to be assigned to the average size of MTC treatment effects. Furthermore, inconsistencies in effects across studies could be tested, producing a clear picture, rather than merely an impression, of consistency. Both of these results, derived from the meta-analysis, provide more specificity than could be achieved without meta-analysis.

A more concrete measure of effectiveness can be derived from the odds ratio of 0.65 for substance abuse. First, consider a post treatment plausible rate of relapse to substance use of 30% for a treatment-as-usual group (based on data from the foundation studies). Together with the estimated odds ratio, this suggests that, while 30% of the treatment-as-usual group will relapse, 22% of the MTC group will likewise fail to maintain abstinence³. Similar odds ratios (i.e., around 0.50), as in the employment and crime domains, suggest similar clinical impacts. Thus, clients completing MTC treatment displayed measurable and observable positive change in areas that are both critical in their own right and essential to sustaining recovery and to integrating with mainstream society.

Core Elements/Fidelity

The four studies demonstrated the effectiveness of the MTC program with different co-occurring disorders populations and in different settings. The course of these studies reflects the shifting focus of co-occurring disorders as the condition became identified in different populations, and as specialized populations became an area where research efforts were encouraged and supported. Core elements of the model were present in each of the MTC programs studied (i.e., community meetings, psycho-educational classes, dual recovery groups) and the delivery of core elements was demonstrated. For example, in examining program fidelity to the MTC model (Study 1), the investigators reported that clients endorsed items 87% of the time (3), indicating that the delivery of core program elements was consistent with the program manual description (4). However, the programs included in this meta-analysis differed in the extent to which the core elements were present and in the specific enhancements provided for each particular population. For example, in Study 2 (Offender), the MTC program had a far greater emphasis on the reduction of criminal thinking and behavior than did the other program models and, in Study 3 (Outpatient), many fewer core MTC elements were offered than were included in the other study programs, and certain of those elements were only partially delivered. The data offered some evidence that those programs providing specific enhancements (e.g., the criminal thinking curriculum in the prison MTC) produced better outcomes in the targeted area (e.g., of criminal behavior). Further work is needed to identify the essential core MTC elements and the nature of specific enhancements with the potential to improve MTC treatment for any given population and setting.

Strengths of the Meta-Analytic Approach

This application of meta-analysis is potentially useful in at least two ways. First, it allows investigators to plan their own future research with greater insight into the size and consistency of previous effects. When an intervention under study has had either weak or inconsistent effects, changes in study design or intervention components and delivery can be considered. Pooled effect sizes and effect size consistency also can be considered when determining the number of participants required to detect effects in a new study. Second, this type of meta-analysis allows investigators to present a quantitative summary of their work, which may help identify promising treatment approaches needing a broad research base and spur other investigators to undertake related studies. In the latter application, a high standard of transparency is required. Although this type of meta-analysis has the potential to appear biased when presented to outside audiences, several approaches (e.g., a priori establishing of criteria for selection of measures, sensitivity testing) can be used to enhance transparency and reduce the potential for bias. This paper describes a particularly challenging instance of applying meta-analytic tools to synthesize the work of a single investigator or research team, in terms of the heterogeneity of specific implementations of the MTC, the populations and contexts studied, and the range of outcome measures. Other applications of the methodology will not necessarily contain all of these complications simultaneously.

Limitations

Important considerations that should be kept in mind when interpreting these results stem from the research sample, the MTC treatment as delivered, and the meta-analytic techniques employed.

While meta-analysis helps to increase statistical power relative to individual studies, none of the other study-specific issues are affected, including the design (e.g., studies designed with rigorous assignment methods that take on quasi-experimental characteristics), fidelity, validity of self-report data, treatment dose, sample attrition, and length of follow-up. In this sense, apart from sample size, meta-analysis retains the limitations of the primary studies, which should be considered when drawing conclusions from any quantitative synthesis. For example, data from Study 1, which compared two MTC programs to one comparison group, may have over- or under-estimated the effects of MTC treatment. Although the sample size for the control group was divided in half, insuring that power was not overestimated, control group means were used in comparisons with both MTC groups, so interdependency may have occurred.

Although an array of outcomes was included in the meta-analysis, the mental health domain was limited to measures of symptomotology and use of psychotropic medication. Mental health is a multidimensional domain and, in addition to symptom change and medication use, should include measures such as avoidance of hospitalization, improved self-esteem or sense of psychological well-being, compliance, and use of services. A reduction in hospitalization is key, both because it is a critical treatment goal for clients with mental disorders, and because of the cost savings that accrue from reducing the number of days an individual or group uses inpatient services. Differential change of psychological symptoms may be difficult to achieve due to the widespread use of medication for clients in both experimental and comparison conditions and the effectiveness of recent (and some older) medications in reducing and controlling symptoms (27).

The interactive relationship between various mental health measures and between mental health outcomes and those in other domains may be useful to consider. For example, the investigators observed a significant relationship between medication compliance and reductions in criminal behavior in Study 2 (28), but a corresponding relationship between symptom reduction and improved crime outcomes was not apparent. Finally, it may be equally constructive to measure positive psychological change (e.g., self esteem, psychological well being) as these effects may be the target of certain interventions and may well be fundamental to broader behavior change.

While meta-analysis has advantages in terms of similarities of methods and procedures, its use by a single investigator is complicated by the fact that those most identified with the development of a treatment model are both conducting the key studies and interpreting the results. Although it is usual for key investigators to report initial findings of studies, as the research advances it becomes necessary for the approach and findings of the research to be examined by a investigators not involved in the development of the model.

In the results presented above, effects were generally consistent, indicating treatment effects favoring the MTC across the studies for each outcome domain. In general, in the application of meta-analysis to a small number of studies by a single investigator, when effects are not consistent, the power will not be sufficient to examine effect size moderators quantitatively. In such cases, it may be difficult to identify the reasons for inconsistency. Nonetheless, merely being able to identify inconsistencies via meta-analysis provides an advantage and, at the very least, suggests that effects may be more sensitive to the context in which they are investigated than previously anticipated. Armed with this knowledge, investigators could plan changes in context more cautiously in their subsequent work.

Future Directions

Study findings have implications for both methodological initiatives and additional clinical research. The MTC approach, although shown to be effective with various populations of people who abuse drugs, including those with co-occurring disorders, encounters difficulty in achieving more widespread use. To achieve greater receptivity to the MTC and application of its methods and to strengthen the research base, several developments are necessary:

With regard first to methodological considerations, —

Others should consider using the tools of meta-analysis for synthesizing a small number of studies in a specific research area conducted by a single investigator. While these applications of meta-analysis must overcome the potential for the appearance of bias, when approached carefully, ensuring transparency, meta-analysis could increase the objectivity of a review relative to a narrative approach. As new studies of MTC for the treatment of co-occurring disorders by the same investigator, or by investigators new to the area, are completed, meta-analysis can provide an updated snapshot of the research base supporting the approach. This cumulative application of meta-analysis has the potential to guide the design of new studies by identifying methodological and clinical gaps in knowledge and to incorporate the knowledge gained from new studies into revised overall summaries of the evidence.

With regard to clinical research, —

(1) The results of the meta-analysis confirm the capacity of the MTC to achieve positive outcomes with a number of co-occurring disorders populations. At the same time, it is recognized that the employment of residential treatment is costly and that resources are limited. In that context, two kinds of studies are of particular significance: (a) it is important to identify the clients who are more likely to benefit from MTC treatment Simpson and colleagues have concluded that clients “with more severe problems at intake were more likely to benefit from longer care in residential services [TC and MTC], affirming the importance of maintaining long-term intensive care as a treatment option” (29:513); (b) it is important to clarify whether components of the residential MTC can be exported to outpatient settings such that the benefits of the MTC can be injected into outpatient care, thereby increasing the capacity to deliver effective services.

Summary

The findings from this single investigator meta analysis add to the research base supporting the effectiveness of MTC treatment for clients with co-occurring substance use and mental disorders. Results included significant MTC treatment effects for 5 of the 6 outcome domains (substance use, mental health, crime, employment, and housing) across the four comparisons; one domain, HIV-risk behavior, showed little or no treatment effect. Outcomes favoring the MTC group emerged in every study; however, the domains in which differences were detected varied from study to study. A single investigator (or research team) application of meta-analytic tools is not intended to provide a definitive determination of the effectiveness of a particular approach using a review of results accumulated from some number of large-scale clinical trials. Instead, a single investigator meta-analysis is designed to consolidate the results of a few studies to produce a quantitative synthesis of the current status of the research and to encourage others to undertake additional and related studies.

Findings suggest that MTC programs should be strengthened to increase their impact on HIV risk behaviors, and research should be designed to measure the effects of these efforts on outcomes. Future research should emphasize independent replications, clinical trials, multiple outcome domains and additional meta-analyses. Nevertheless, findings from this research synthesis hold considerable clinical relevance since individuals with co-occurring disorders are typically perceived to have multifaceted needs requiring multidimensional interventions (26). Given the need for research-based approaches, program and policy planners should consider the MTC when designing programs for individuals with co-occurring disorders.