Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2024 Oct 29.
Published in final edited form as: J Subst Abuse Treat. 2007 Oct 23;35(1):36–40. doi: 10.1016/j.jsat.2007.08.005

Analyzing Data from Open Enrollment Groups: Current Considerations and Future Directions

Antonio A Morgan-Lopez 1, William Fals-Stewart 2
PMCID: PMC11521140  NIHMSID: NIHMS54541  PMID: 17936551

Abstract

Difficulties in modeling turnover in treatment group membership have been cited as one of the major impediments to ecological validity of substance abuse and alcoholism treatment research. In this review, our primary foci are on (a) the discussion of approaches that draw on state-of-the-science analytic methods for modeling open enrollment group data and (b) highlighting emerging issues that are critical to this relatively new area of methodological research (e.g., quantifying membership change, modeling “holiday effects”, modeling membership change among group members and leaders). Continuing refinement of new modeling tools to address these analytic complexities may ultimately lead to the development of more federally-funded open enrollment trials. These developments may also facilitate the building of a “community-friendly” treatment research portfolio for funding agencies which support substance abuse and alcoholism treatment research.

Keywords: treatment groups, open enrollment, data analysis

Although psychosocial interventions for alcoholism and drug abuse are delivered in a variety of formats, including one-on-one counseling, partner- and family-involved therapy, and so forth, by far the most commonly used is the provision of intervention to a collection, or group, of patients simultaneously (e.g., Price, Burke, D’Aunno, Klingel et al., 1991; Stinchfield, Owen, & Winters, 1994). The term ‘group therapy’ most precisely describes an intervention delivery format, although it is often used in common clinical parlance to characterize a therapeutic approach in and of itself. In fact, many kinds of therapies (twelve-step, cognitive behavioral, marital and family) can, and often are, delivered in a group format.

Regardless of type of intervention being delivered, therapy groups are very often broadly categorized in terms of how they form and sustain membership. In closed enrollment, groups are formed with a core set of members and typically are conducted for a predetermined, circumscribed period of time; the typical length of the closed treatment group corresponds directly to the length of the prescribed treatment (e.g., if the treatment protocol calls for 12 weeks of treatment, the group only runs for 12 weeks). Once the group is formed, members are not added, although there is almost always a certain degree of membership dropout from the group over time. Once the group has run its predetermined course, a new group can be formed and started anew. Conversely, in open enrollment, members can join or leave the group at any point in time; as such, membership in these groups is constantly evolving. Because members can be added at any time and thereby replenish membership when it becomes low, these groups usually have no clearly defined endpoint and can continue indefinitely, regardless of the length of the prescribed treatment (e.g., a particular group member may complete the group treatment after 12 weeks of attendance, even if the group has been running for many years).

Of these types, open enrollment groups appear to be far more common in community-based substance abuse treatment settings. In a survey of substance abuse treatment programs (N = 57) conducted by the second author (Fals-Stewart, 2005), the majority of treatment directors (84%) reported that most of the therapy groups running programs used open enrollment. The primary reasons reported for the preference of rolling admissions were economic (e.g., groups are ongoing; thus, there is no wait period before patients can enter and be billed for services and there is no point where the group ceases) and clinical (e.g., rolling admissions dramatically reduced wait times for group entry as compared with closed groups, where new members are not allowed entry after the group starts).

In spite of the popularity of open enrollment groups in community settings, capturing and quantifying the dynamic processes of open enrollment groups, and how it may affect treatment outcomes, presents a unique set of challenges for investigators who wish to appropriately model the interrelated changes of therapy groups and their participants. Specifically, difficulties in modeling the dynamics of the open enrollment process in the analysis of outcome data from treatment groups have, in part, led to avoidance of their use in substance abuse and alcoholism clinical trials (Morgan-Lopez & Fals-Stewart, 2006a; Morgan-Lopez & Fals-Stewart, in press; Weiss, Jaffe, de Menil & Cogley, 2004).

The purpose of this paper is to (a) summarize recent progress in modeling approaches that draw on state-of-the-science analytic methods that can be used to analyze data generated from open enrollment groups; and (b) highlight new interrelated issues that are critical to this emerging area of research (e.g., quantifying membership change, issues of statistical power) and are presently the subject of ongoing investigation. We present these ideas with an eye towards highlighting approaches that will eventually help bridge the current gap between treatment research and treatment in community settings.

Ongoing Work on Approaches in Modeling Open Enrollment Groups

Based on experiences from our limited empirical work on models for capturing the turnover process in open enrollment therapy groups (Morgan-Lopez & Fals-Stewart, in press), there are four primary questions that should be considered when selecting an analytic framework for modeling longitudinal data from open enrollment therapy groups: (a) Is treatment group attendance related to the response that the patient would have given had they showed up for treatment?, (b) Does the treatment effect differ across patterns of treatment attendance?, (c) Is the proportion of patient subtypes (as defined by subtypes of attendance patterns) not consistent from session-to-session and/or across time?, and (d) Does the proportion of patient subtypes differ systematically as a function of the time (in the calendar year) that the patient enters treatment? If the answer is ‘yes’ to any one of these questions, the choice of analytic strategy becomes critical, as different strategies may lead to different inferences regarding overall treatment potency (Morgan-Lopez & Fals-Stewart, 2006a; 2006b; Morgan-Lopez & Fals-Stewart, in press). Standard analytic approaches have been deemed sub-optimal for handling the changes over time in treatment group membership (particularly with continual membership additions) that occur within open enrollment groups; these criticisms of standard analytic approaches are not uncommon in the review of NIH substance abuse treatment-related grant submissions which incorporated open enrollment structure (Morgan-Lopez & Fals-Stewart, 2006a; Morgan-Lopez & Fals-Stewart, in press).

Earlier theoretical work in this area suggested that pattern mixture models (Hedeker & Gibbons, 1997; Little, 1993), originally intended for handling non-ignorable missingness, also had the potential for handling the process of membership turnover (Morgan-Lopez & Fals-Stewart, 2006a). However, we have since found that latent class pattern mixture models ((LCPMMs; Lin, McCulloch & Rosenheck, 2004; Muthén, Jo & Brown, 2003) provide a framework that more closely represents the process of turnover in group membership than traditional methods (e.g., group-clustered latent growth models) or even conventional pattern mixture models (Morgan-Lopez & Fals-Stewart, in press). LCPMMs allow researchers to model (a) variability in the treatment effect across a finite number of latent attendance classes among individuals in the same treatment group and (b) the point of treatment entry. The core rationale for linking LCPMMs to the analysis of data from substance abuse treatment trials with open enrollment protocols is that, within any point of the trial, the proportions of different types of attendance patterns (and, therefore, different subtypes of patients) are allowed to vary at any given slice in time (consistent with changes over time/turnover in group composition) at which the trial is running (Morgan-Lopez & Fals-Stewart, in press; Morgan-Lopez & Fals-Stewart, 2006b). Moreover, as group composition and norms change over time, in concert with changes in the membership of the group (e.g., the proportion of dropouts decreases over time as the group becomes more cohesive and efficacious), it may affect the efficacy of the treatment during particular periods of the history of the group.

The LCPMM approach may be critical in increasing the accuracy of inferences made from treatment trial data with rolling groups, as preliminary simulation work has shown that conventional methods may increase the likelihood that significant treatment effects are detected in a sample when there are no differences in the population in analytic frameworks where turnover is not explicitly modeled (Morgan-Lopez & Fals-Stewart, 2006b).

Emerging Analytic and Methodological Issues in Group Therapy Research: Future Directions

With NIDA and NIAAA pushing to make their respective behavioral treatment portfolios more community-friendly and more ecologically-valid (NIAAA/NIDA, 2002), there has been considerable interest in (a) understanding the methodological barriers that exist in group therapy research (NIDA, 2003; Weiss et al., 2004) and (b) developing solutions to many of these methodological barriers. Our recent work has focused on the barrier of session-to-session changes in treatment group membership in group-based open enrollment trials (Morgan-Lopez & Fals-Stewart, 2006a, 2006b, in press), though there are several other methodological considerations that make group therapy research a difficult enterprise (for a thorough review, see Weiss et al., 2004).

Stability index for change in group membership

There are several other subtopics that have emerged from our work on open enrollment groups. One such topic involves the development of approaches to quantify the (in)stability of the group during the period that each individual is in the group and whether such a measure can be used to model the open enrollment process in data from ecologically-valid trials. We have developed a measure (Morgan-Lopez & Fals-Stewart, in press) called the Percentage of Group Change Index (PGCI) to capture change in group membership composition from session-to-session. It is calculated using the following equation:

([1−[aa+b+c]])(100) (1)

Where a is the number of group members who remained the same from the previous week, b is the number of members who were present in the current week that were not present the previous week and c is the number of members who were present in the previous week who were absent in the current week. The value can range from 0% (i.e., the exact same membership from one session to the next) to 100% (i.e., complete turnover in membership).

To this point, we have only used this measure as a descriptor to illustrate that the level of group turnover in the open enrollment context is non-trivial (Morgan-Lopez & Fals-Stewart, in press); however, there are many potential uses for this measure. First, the predictive validity of this measure must be established in analyses where this turnover index is used as a predictor of treatment outcomes (i.e., Is group (in)stability related to treatment efficacy?). Second, group (in)stability may be a moderator of treatment efficacy, such that differences in the efficacy between two or more treatment conditions may depend on the stability of the group. Finally, it has been suggested1 that the turnover measure may be used as a weighting variable in the context of Hierarchical Linear Modeling of treatment outcome data from therapy groups.

For example, in behavioral genetics research, the strength of genetic dependencies will vary within a family (e.g., monozygotic twins versus first cousins) and weights are incorporated into the model to handle differences in the relative proportion of shared and unshared genotypic information (Guo & Wang, 2002; McArdle & Prescott, 2005). In this case, within-family dependencies will be weaker among family members that have less common genetic information. We have proposed elsewhere (Morgan-Lopez & Fals-Stewart, 2006a; Morgan-Lopez & Fals-Stewart, in press) that an analog to this weighting approach in group therapy research, using the PGCI measure, may be to weight individuals in the therapy group based on a) the length of time individuals remain in the treatment group and/or b) the level of turnover occurring in the group during the period the individual is a member of the group.

Holiday Effects

Emerging approaches to handling turnover in therapy group membership, regardless of the analytic framework (e.g., LCPMM, weighted HLM), are geared towards the possibility that different individuals within the same treatment group can have a differential response to treatment as a function of their attendance patterns during treatment and the timing of treatment entry. These types of models, particularly LCPMMs, have raised the possibility of examining “holiday” effects: treatment effects among sub-types of individuals in treatment who differ in their treatment outcomes and attendance patterns which appear to occur mainly during two periods: the Winter holiday season and the end-of summer (Morgan-Lopez & Fals-Stewart, in press).

Morgan-Lopez & Fals-Stewart (in press) found that, among male patients in treatment for alcoholism with erratic patterns of treatment attendance (as opposed to consistent attenders or “classic” dropouts), a) they were more likely to be in treatment during these two critical holiday periods than any other time of year and b) they had statistically significant reductions in alcohol use under group therapy compared to patients in individual treatment with erratic attendance patterns. While several epidemiologic studies have established the Winter holiday season and the latter part of summer as the points of peak prevalence, particularly for alcohol use (Carpenter, 2003; Helzer, Badger, Rose, Mongeon & Searles, 2002) very few studies have examined whether there are seasonal impacts on treatment outcomes; it appears that advances in modeling of longitudinal turnover in therapy group membership have, as a by-product, introduced opportunities to examine such seasonal effects among treatment outcomes.

Changes in Group Leaders over Time

Research on turnover in therapy groups is still very much in its infancy (Morgan-Lopez & Fals-Stewart, 2006a; Morgan-Lopez & Fals-Stewart, in press). Thus far, there has been an exclusive focus on approaches to modeling the impact of turnover among group members on treatment outcomes, with members implicitly defined as patients or clients. What has remained absent from any of this work is modeling the impact of turnover among group leaders on treatment outcomes. Group leaders are also group members of who bring with them to the group their own unique style and typically bring a more complete history of the group than do many of the group members. In many ways, group leaders are more of a constant than the group members, as they typically lead the group for a time period that is much longer than members will be in the group for the prescribed treatment and may arguably contribute more in their impact on group history and norms over time than any single member. Nevertheless, turnover among group leaders is notoriously high, with yearly turnover among addictions counselors exceeding 50% (McLellan, Carise & Kleber, 2003). As such, there remains room for research on modeling the impact of both types of turnover, that of patients or clients and leaders, within ecologically-valid substance abuse and alcoholism treatment trials.

A potential solution to modeling turnover among group leaders may come from the multiple membership modeling framework (Goldstein, 2003) which has been used, for example, to model clustering of patients in hospitals who have more than one physician attending to them; patients are said to be “members” of multiple physicians, as opposed to the “classic” nesting of patients being nested within groups based on treatment by one-and-only-one physician. This framework may eventually be combined with at least one of the two proposed frameworks for modeling turnover among group members (i.e., latent class pattern mixture models, turnover-weighted HLM) to simultaneously handle turnover among group leaders and members.

These developments may take a considerable amount of time to come to fruition. For example, no current structural equation modeling software has multiple membership modeling facilities (Muthén, 2005); as a result, recent approaches for handling turnover among group members based on a special case of the finite mixture SEM model (Morgan-Lopez & Fals-Stewart, in press) cannot yet be combined with multiple membership models, at least not without advanced software (e.g., S+, R). It is more likely that multiple membership models can be combined with turnover-weighted HLM approaches, of the types that originated in behavioral genetics research (Guo & Wang, 2002; McArdle & Prescott, 2005) within current random coefficient modeling software packages, but as we noted earlier, the utility of our group turnover measure has not been sufficiently examined.

Statistical Power

As discussed in Morgan-Lopez & Fals-Stewart (2006a), differences in the assumptions concerning differential treatment efficacy across patterns of treatment attendance, (in)consistency in the proportion of patient subtypes from session-to-session and across time and timing of treatment entry can have major implications for treatment effect estimation. In fact, there is limited empirical evidence suggesting that different inferences (and thus, very different sets of effect sizes) concerning treatment efficacy are possible across models that make different assumptions about therapy group turnover (Morgan-Lopez & Fals-Stewart, in press).

The issue of differences in effect sizes across analytic frameworks has critical implications for statistical power in the design of new ecologically-valid substance abuse and alcoholism treatment trials with turnover built into the group structure. Of course, differences in the effect sizes derived from competing analytic frameworks (e.g., LGM for group-clustered data versus Latent Class Pattern Mixture Modeling for group-clustered data) will have implications for differences in the required sample size to achieve acceptable statistical power in a new trial under each analytic framework, which may lead investigators to a significant crossroad: (a) plan analyses that maximize statistical power while sacrificing the linkage between the analytic framework and the conceptual assumptions that underlie the generation of rolling group data, or (b) plan analyses that more closely resemble the conceptual process that underlies how rolling groups work but increase the required sample size for treatment effect detection. And even this decision makes the presumption that the smaller effect size is at an acceptable level of clinical significance to further justification for studying the treatment at all (Morgan-Lopez & Fals-Stewart, in press).

Based on limited work with both simulated (Morgan-Lopez & Fals-Stewart, 2006b) and real data (Morgan-Lopez & Fals-Stewart, in press), it does appear that larger samples than those commonly observed in behavioral treatment trials (e.g., N = 200 to 400) may generally be necessary when modeling therapy group turnover in the context of LCPMM. However, there are several parameters that will ultimately play into (a) how large the sample size differential would be between adequate power for LCPMMs versus LGMs, and (b) under which conditions would the same inference be made under either framework. For example, in trials where the differences in treatment effects across attendance patterns is not as large as they were in Morgan-Lopez & Fals-Stewart (in press), the difference between the (weighted-averaged) treatment effect in LCPMM and LGM may not differ as dramatically. In fact, the smaller the difference in treatment effects across patterns of missingness, the closer we would be to meeting the missing-at-random assumption and not require LCPMM modeling at all. There remains much work to be done in identifying and quantifying the point at which these methods will have a high likelihood of yielding different results (and thus have different sample size requirements to achieve equal statistical power), though ultimately, the issue of effect size may be the most critical component in the discussion on statistical power in open enrollment designs.

Conclusion

In this paper, we focused on the data analytic issues of open enrollment groups, with an emphasis on highlighting the complexities of data generated from such groups, promising approaches that can be used to model these data, and areas that need to be examined as this general programmatic line of research evolves. It is our hope to facilitate research on open enrollment groups by addressing data analytic barriers that we believe has impeded, to a certain extent, the rigorous empirical evaluation of interventions that mirror those typically provided in community-based treatment programs.

It should not be inferred from our emphasis on data modeling that this is the only, or even the primary, barrier that investigators confront when considering or undertaking trials that use open enrollment groups. Other issues include, but are not limited to, those that are logistic (e.g., feasibility of sufficient recruitment for a group therapy trial, problems of random assignment of participants to groups, scheduling issues that have to accommodate many members and group leaders) and clinical (e.g., who are appropriate candidates for groups versus other delivery modalities, following a manualized sequence of intervention delivery when new members are continually being added) (Weiss et al., 2004). However, it is critical to develop approaches to best address all of these barriers (versus sidestepping them by eschewing research on open enrollment groups) to enable funded intervention research to increase its ecological validity.

Acknowledgments

Antonio A. Morgan-Lopez, Ph.D., RTI International, Behavioral Health and Criminal Justice Research Division, Research Triangle Park, North Carolina 27709. William Fals-Stewart, Ph.D., School of Nursing, University of Rochester, Rochester, New York, 14642. This project was supported by grants R21DA021147 and R21AA016543 (Antonio A. Morgan-Lopez, PI) and by grants R01DA12189, R01DA014402, R01DA014402-SUPL, R01DA015937, R01DA016236, R01DA016235-SUPL and the Alpha Foundation (William S. Fals-Stewart, PI). We acknowledge the comments of Keith C. Klostermann on earlier drafts of this manuscript.

Footnotes

1

We acknowledge the suggestions of an anonymous reviewer of our in-press article in Journal of Consulting and Clinical Psychology for this potential alternative framework for modeling therapy group turnover.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Baldwin SA, Murray DM, Shadish WR. Empirically supported treatments or Type-1 Errors?: A revaluation of group-administered treatments on the empirically supported treatments list. Journal of Consulting and Clinical Psychology. 2005;73:924–935. doi: 10.1037/0022-006X.73.5.924. [DOI] [PubMed] [Google Scholar]
  2. Blalock HM. Auxiliary measurement theories revisited. In: Hox JJ, De Jong-Gierveld J, editors. Operationalization and research strategy. Amsterdam: Swets & Zeitlinger; 1990. pp. 33–48. [Google Scholar]
  3. Carpenter C. Seasonal variation in self-reports of recent alcohol consumption: Racial and ethnic differences. Journal of Studies on Alcohol. 2003;64:415–418. doi: 10.15288/jsa.2003.64.415. [DOI] [PubMed] [Google Scholar]
  4. Demirtas H, Schafer JL. On the performance of random-coefficient pattern-mixture models for non-ignorable drop-out. Statistics in Medicine. 2003;22:2553–2575. doi: 10.1002/sim.1475. [DOI] [PubMed] [Google Scholar]
  5. Fals-Stewart W. Use of group therapy in outpatient substance abuse treatment programs: A survey of randomly selected programs in the Northeastern United States. 2005 Unpublished raw data. [Google Scholar]
  6. Flores P. Group psychotherapy with addicted populations. New York: The Haworth Press; 1988. [Google Scholar]
  7. Goldstein H. Multilevel statistical models. 3rd ed. London: Arnold; 2003. [Google Scholar]
  8. Guo G, Wang J. The mixed or multilevel model for behavior genetic analysis. Behavior Genetics. 2002;32:37–49. doi: 10.1023/a:1014455812027. [DOI] [PubMed] [Google Scholar]
  9. Hedeker D, Gibbons RD. Application of random-effects pattern-mixture models for missing data in longitudinal studies. Psychological Methods. 1997;2:64–78. [Google Scholar]
  10. Hedeker D, Mermelstein R. Analysis of longitudinal substance use outcomes using ordinal random-effects regression models. Addiction. 2000;95:S381–S394. doi: 10.1080/09652140020004296. [DOI] [PubMed] [Google Scholar]
  11. Hedeker D, Rose JS. The natural history of smoking: a pattern-mixture random-effects regression model. In: Rose JS, Chassin L, Presson CC, Sherman SJ, editors. Multivariate Applications in Substance Use Research. Hillsdale, NJ: Erlbaum; 2000. pp. 79–112. [Google Scholar]
  12. Helzer JE, Badger GJ, Rose GL, Mongeon JA, Searles JS. Decline in alcohol consumption during two years of daily reporting. Journal of Studies on Alcohol. 2002;63:551–558. doi: 10.15288/jsa.2002.63.551. [DOI] [PubMed] [Google Scholar]
  13. Lin HQ, McCulloch CE, Rosenheck RA. Latent pattern mixture model for informative intermittent missing data in longitudinal studies. Biometrics. 2004;60:295–305. doi: 10.1111/j.0006-341X.2004.00173.x. [DOI] [PubMed] [Google Scholar]
  14. Little RJA. Pattern-mixture models for multivariate incomplete data. Journal of the American Statistical Association. 1993;88:125–134. [Google Scholar]
  15. McArdle JJ, Prescott CA. Mixed-effects variance components models for biometric family analyses. Behavior Genetics. 2005;35:631–652. doi: 10.1007/s10519-005-2868-1. [DOI] [PubMed] [Google Scholar]
  16. McLellan AT, Carise D, Kleber HD. Can the national information treatment infrastructure support the public’s demand for quality care? Journal of Substance Abuse Treatment. 2003;25:117–121. [PubMed] [Google Scholar]
  17. Morgan-Lopez AA, Fals-Stewart W. Analytic methods for modeling longitudinal data from rolling therapy groups with membership turnover. Appearing in Volume 75(4; August 2007) of Journal of Consulting and Clinical Psychology. doi: 10.1037/0022-006X.75.4.580. (in press) [DOI] [PubMed] [Google Scholar]
  18. Morgan-Lopez AA, Fals-Stewart W. Analytic complexities associated with group therapy in substance abuse treatment research: Problems, recommendations, and future directions. Experimental & Clinical Psychopharmacology. 2006a;14:265–273. doi: 10.1037/1064-1297.14.2.265. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Morgan-Lopez AA, Fals-Stewart W. The utility of latent class pattern mixture modeling in handling data from substance abuse treatment trials with rolling admissions. Paper presented at the UCLA Center for Advancing Longitudinal Drug Abuse Research (CALDAR) Summer Research Institute; August 14, 2006; Los Angeles. 2006b. [Google Scholar]
  20. Muthén BO. Comment on multiple membership modeling in Mplus (May 9, 2005) 2005. Available via World Wide Web at http://www.statmodel.com/discussion/messages/12/609.html#POST5004.
  21. Muthén B. Latent variable modeling of longitudinal and multilevel data. In: Raftery A, editor. Sociological methodology. Boston, MA: Blackwell; 1997. pp. 453–480. [Google Scholar]
  22. Muthén BO, Jo B, Brown CH. Comment on the Barnard, Frangakis, Hill & Rubin article, Principal stratification approach to broken randomized experiments: A case study of school choice vouchers in New York City. Journal of the American Statistical Association. 2003;98:311–314. [Google Scholar]
  23. Muthén LK, Muthén BO. Mplus Users Guide. Fourth Edition. Los Angeles: Muthén & Muthén; 1998–2006. [Google Scholar]
  24. National Institute on Drug Abuse. Group therapy research. Workshop sponsored by the NIDA Behavioral Treatment Branch. 2003 April; Meeting summary available via World Wide Web at http://www.drugabuse.gov/whatsnew/meetings/grouptherapy.html.
  25. National Institute on Alcoholism and Alcohol Abuse & National Institute on Drug Abuse. Request for applications for group therapy for individuals in drug abuse and alcoholism treatment (RFA-DA-04-008) Washington, DC: Department of Health and Human Services; 2003. [Google Scholar]
  26. Price RH, Burke AC, D’Aunno TA, Klingel DM, McCaughrin WC, Rafferty JA, Vaughn TE. Outpatient drug abuse treatment services, 1988: Results of a national survey. In: Pickens RW, Leukefield CG, Schuster CR, editors. Improving drug abuse treatment. Rockville, MD: National Institute on Drug Abuse; 1991. pp. 63–92. [PubMed] [Google Scholar]
  27. Roy J. Modeling longitudinal data with nonignorable dropouts using a latent dropout class model. Biometrics. 2003;59:829–836. doi: 10.1111/j.0006-341x.2003.00097.x. [DOI] [PubMed] [Google Scholar]
  28. Rubin DB. Inference and missing data. Biometrika. 1976;63:581–592. [Google Scholar]
  29. Schafer JL. Analysis of incomplete multivariate data. London: Chapman & Hall; 1997. [Google Scholar]
  30. Schafer JL. Multiple imputation in multivariate problems where the imputer's and analyst's models differ. Statistica Neerlandica. 2003;57:19–35. [Google Scholar]
  31. Schafer JL, Graham JW. Missing data: Our view of the state of the art. Psychological Methods. 2002;7:147–177. [PubMed] [Google Scholar]
  32. Stinchfield RD, Owen PL, Winters KC. Group therapy for substance abuse: A review of the empirical literature. In: Fuhriman A, Burlingame GM, editors. Handbook of group psychotherapy: An empirical and clinical synthesis. New York: John Wiley and Sons; 1994. p. 459. [Google Scholar]
  33. Weiss RD, Jaffe WB, de Menil VP, Cogley CB. Group therapy for substance use disorders: What do we know? Harvard Review of Psychiatry. 2004;12:339–350. doi: 10.1080/10673220490905723. [DOI] [PubMed] [Google Scholar]
  34. Yalom I. The theory and practice of group psychotherapy. 4th ed. New York: Basic Books; 1995. [Google Scholar]

RESOURCES