Level of analysis issues in assessing treatment beliefs in substance abuse clinics

Abstract

The current study applies the growing literature in the organizational sciences regarding levels of analysis issues to the analysis of substance abuse treatment beliefs. Research on clinicians' beliefs in substance abuse treatment is often based on the assumption that the beliefs are sufficiently shared by clinicians within a clinic and sufficiently vary across clinics that they can be treated as a group-level phenomenon. Further, efforts to introduce new innovations are often focused at the group or clinic level without testing this assumption, which can lead to failure to adopt or to successfully implement the innovation. We tested the assumption of sharedness by examining if there was sufficient agreement about treatment beliefs within clinics, within groups of clinics or within groups of clinicians to justify treating these aggregations as meaningful groups. Using three statistical approaches to examining level of analysis (Within and Between Analysis (WABA I), Intraclass Correlation Coefficients (ICC(1)), and r_wg), we found that variability in treatment beliefs largely occurred at the individual rather than at the tested aggregate levels of analysis. These findings serve as an example of the importance of testing the assumption of shared perceptions in future research.

1. Introduction

Recent research has demonstrated the importance of shared perceptions or beliefs within an organization as a determinant of the appropriateness of innovations. Pearce and Ensley (2004), for example, present evidence suggesting that the extent to which members of an organization share a vision about what the organization stands for is predictive of the organization's successful adoption and implementation of innovations congruent with that shared vision. Further, when organization members do not share a vision, even well-planned innovations that have been successful in other similar settings are likely to fail. As such, it is important to understand the extent to which such sharedness of treatment beliefs may exist.

A critical issue to consider, however, is whether these perceptions are sufficiently shared amongst clinicians within a clinic and sufficiently varied across clinics to justify treating them as organizational-level phenomena, or whether they vary both within and across organizations such that they need to be considered at the individual level. In other words, it is important to understand whether clinicians can be grouped together in meaningful ways based on the similarity of their beliefs about treatment – whether clinicians' beliefs can be meaningfully aggregated to a group level of analysis. The overall goal of the present study is thus to take a more detailed consideration of clinicians' perceptions by examining if they are shared, using generally accepted statistical approaches commonly applied in the organizational sciences.

By focusing on level of analysis, we are determining whether it is most meaningful to focus on variability between individuals, variability between programs/clinics, or variability between other groupings of individuals. As Klein, Dansereau and Hall (1994) explain, “A theorist predicts that group members are sufficiently similar with respect to the construct [or variable] in question that they may be characterized as a whole” (p. 199). While researchers recognize group members are not identical, if analysis supports a group level of analysis, the group members are believed to be sufficiently similar that they can be treated as identical. Thus, the variance between these group members may be treated as error variance on the construct or variable of interest, and analysis can proceed using the aggregated data. If it is found that group membership plays little or no role in predicting the degree of variability in clinicians' treatment beliefs, then these beliefs are operating at an individual level of analysis and aggregation could lead to erroneous interpretations of the results, or what is often called an “ecological fallacy” (Robinson, 1950) or the “fallacy of the wrong level” (Dansereau, Cho, & Yammarino, 2006).

But what constitutes a group of clinicians who may experience shared perceptions or beliefs? One theoretical model suggests that those individuals comprising a workplace tend to be relatively homogeneous. The Attraction-Selection-Attrition (ASA) model describes the attraction of similar potential employees to an organization, selection of individuals deemed by the organization to share the organization's values and common personality, and attrition of those employees who join the organization but find themselves not “fitting in” with the organization (Schneider, 1987; Schneider, Goldstein, & Smith, 1995). Specifically, the ASA model proposes that, through the dynamic interaction between the processes of attraction, selection and attrition, along with organizational socialization processes, organizations become increasingly homogeneous over time in terms of the type of people that comprise the organization (Schneider et al., 1995). Of course, various job-related issues (e.g., the economy and accompanying labor market, the interests of potential employees) may influence the degree to which ASA processes occur. Applying ASA theory in the present context, however, suggests that it is likely that clinicians within an organization or clinic would share similar treatment beliefs. Based on this rationale, it makes sense to examine whether clinicians' treatment beliefs vary between organizations but are similar within organizations (i.e., treatment beliefs vary meaningfully at the organizational/clinic level of analysis).

However, organization or clinic membership is not the only potentially meaningful grouping of clinicians. Other potential groupings for which there could be variability between groups and similarity within groups include various clinician characteristics, such as clinicians who are themselves in recovery. Additionally, the type of organization in which clinicians work could form a meaningful aggregate. For example, clinicians at methadone programs may differ in beliefs from clinicians at programs that do not use medically-approved substitute therapy.

Finally, another approach to grouping clinicians would be between those clinicians whose organizations have a research affiliation and those who work in organizations not affiliated with research. One network of research affiliation is the National Institute on Drug Abuse's Clinical Trials Network (CTN; Hanson, Leshner & Tai, 2002; www.nida.nih.gov/CTN). The CTN was created to lessen the research and practice gap by forming a network of researchers and practitioners to encourage collaborative efforts to evaluate the effectiveness of substance abuse treatment innovations in community settings. The main focus of the CTN is to test treatment innovations using clinical trials at community treatment programs as opposed to the traditionally specialized research settings. This suggests another subgroup of clinicians who may have distinguishable treatment beliefs: those clinicians in CTNs (research affiliated) and those clinicians not in the CTN (non-research affiliated).

In summary, the goal of the current study is to draw on the growing literature in the organizational sciences on levels of analysis issues, and to use these approaches to illustrate the possibilities and pitfalls related to levels issues in substance abuse treatment research. To do this, we will employ a commonly-used measure of clinicians' treatment beliefs, and use a variety of analytical techniques to determine the most appropriate level of analysis for understanding clinicians' beliefs about substance abuse treatment. We will specifically assess the following hypothesis, suggested by the various approaches to aggregation described above:

• H1a: Clinicians' treatment beliefs vary meaningfully at the individual level of analysis. We propose the following alternative hypotheses in a competing hypotheses framework:

• H1b: Clinicians' treatment beliefs vary meaningfully at the organization/clinic level of analysis.

• H1c: Clinicians' treatment beliefs vary meaningfully between the group of clinicians in recovery and those not in recovery.

• H1d: Clinicians' treatment beliefs vary meaningfully between the group of clinicians at methadone clinics and the group of clinicians at non-methadone clinics.

• H1e: Clinicians' treatment beliefs vary meaningfully between the group of clinicians at research-affiliated clinics and the group of clinicians at non-research affiliated clinics.

In other words, our goal is to assess whether the variability in clinician treatment beliefs is primarily at the individual level of analysis, such that each clinician's beliefs tend to be separate and unique from other clinicians in proximity, or whether the variability in clinician treatment beliefs is at a group level of analysis, such that clinicians who routinely come into contact with each other (i.e., same clinic), who have similar life histories (e.g., similar recovery status), or whose organizations share similar characteristics (e.g., availability of methadone treatment; research affiliation) tend to have similar beliefs. To minimize chance findings based upon multiple comparisons, we will first test the hypotheses using data from a first wave of data collection, and then assess if they replicate using data from a 1-year follow-up. Our goal is to use these assessments as an example to show the importance of testing assumptions about the appropriate levels of analysis for various measures employed in substance abuse treatment research.

2. Method

2.1 Participants and participating organizations

The current study was part of a larger longitudinal study of organizational characteristics that may affect adoption of evidence-based substance abuse treatment innovations. Both the Wayne State University and University of Michigan Institutional Review Boards approved the study. The participants and participating organizations were from both CTN research-affiliated and non-research-affiliated substance abuse treatment centers. The research-affiliated programs were from the Great Lakes Node (all located in Michigan) of the CTN. The Great Lakes Node was one of the 17 nodes in the CTN and included five community treatment programs. The five programs comprised two residential programs, and one each of a drug-free outpatient program, a methadone outpatient program, and a women's outpatient program. The non-research affiliated programs were specifically chosen to match the five CTN programs in a 2:1 ratio on criteria of treatment modality, special population served, and geographic location.

For year 1 data, two of the non-research affiliated programs were not included in the current analysis because only three clinicians per program took part in the study, bringing the total number of programs in the current study to 13 (5 CTN programs and 8 non-CTN affiliated programs) for year 1. For year 2 data, these same clinics participated with one of the two non-research affiliated programs excluded in year 1 having sufficient responses and as such, was included for year 2. However, two other clinics were not included because one non-research affiliated program had only four clinicians respond to all items under consideration and a CTN program withdrew from the study due to concerns about closure of the clinic, bringing the total number of programs to 12 (4 CTN programs and 8 non-CTN affiliated programs) for year 2. Nine of the 12 programs from year 2 were part of the year 1 sample.

2.2 Procedure

As part of the on-going longitudinal study, data collection was repeated each year at each participating site. For the data collection utilized in the present study, a research assistant visited each site and held a group presentation. During this presentation, the research assistant described the self-administered survey and gathered signed consent forms. The respondents chose to either complete the survey at that time or mailed the completed survey at a later date. The survey took approximately 30 minutes to complete and either the respondent or the clinical program was reimbursed for participating in the confidential survey. Nearly all clinicians completed the survey, giving a response rate approaching 100% for both year 1 and year 2.

2.3 Measures

Among others in the larger study, measures include demographic and professional characteristics, as well as beliefs about substance abuse treatment. The measurement of beliefs about treatment was based on a longer instrument developed by Forman, Bovasso, and Woody (2001). Items with acceptable variability in a pilot test and/or particular relevance to the clinics in our sample were selected (see Table 1 for a listing of all selected items). Item wordings of those selected belief items were not changed from the original survey. Response options for these items were originally on a 4-point agreement scale (1=strongly disagree, 2=somewhat disagree, 3=somewhat agree, 4=strongly agree), with additional response options of “depends on the person” and “don't know.” For the current analysis, these additional response options were re-coded into the agreement scale as “neutral” (i.e., the midpoint of a re-calculated 5-point scale) in an effort to reduce missing data points.

Table 1. Means, standard deviations, and WABA I results for beliefs about treatment by clinic, recovery status, methadone/non-methadone availability and research affiliation for year 1.

Belief Items	Meana	S.D.a	By Clinica		By Recovery Statusb		By Methadone Availabilityc		By Research Affiliationd
			F ratio (within)	E ratio	F ratio (within)	E ratio	F ratio (within)	E ratio	F ratio (within)	E ratio
Confrontation should be used in addiction treatment	3.83	1.10	.30	.54‡	.41	.13‡	7.37	.03‡	1.71	.06‡
Methadone maintenance should be used with heroin addicts	2.93	1.44	.19	.68†	580.59**	.00‡	.06	.32‡	.97	.08‡
Spirituality should be emphasized in addiction treatment	4.57	.78	.26	.58†	.49	.12‡	7.89	.03‡	.26	.16‡
Research findings should be used in addiction treatment	4.51	.88	1.47	.24‡	28.63	.02‡	.28	.15‡	1.31	.07‡
Psychiatric medications should be used in addiction treatment	3.75	1.10	.55	.40‡	6.35	.03‡	.29	.15‡	27832.94***	.00‡
New approaches should be used in addiction treatment	4.49	.85	.98	.30‡	.70	.10‡	.28	.15‡	.45	.12‡
Working in the addiction treatment field has been a rewarding experience	4.69	.76	1.86	.22‡	11.26	.03‡	8.18	.03‡	13.41	.02‡
People who are unsuccessful in treatment don't want it bad enough	2.55	1.49	.92	.31‡	.08	.29‡	5.61	.03‡	109.84*	.01‡
Addiction is really a disease	4.50	1.13	.43	.45‡	.56	.11‡	19.89	.02‡	21.01	.02‡
Recovering counselors make the best therapists	3.18	1.28	.37	.49‡	.02	.57‡	10.11	.03‡	.47	.12‡
Noncompliant clients should be discharged	3.16	1.25	.44	.45‡	99.40*	.01‡	.48	.12‡	.33	.14‡

NOTES.

^aN = 150, J - 1 = 12, N - J = 137;

^bN = 142, J - 1 = 1, N - J = 140;

^cN = 156, J - 1 = 1, N - J = 154;

^dN = 150, J - 1 = 1, N - J = 148.

Practical significance criteria:

^†parts - 15⁰, E< or = 0.77;

^‡parts - 30⁰, E< or = 0.58

^*p<.10;

^**p<.05;

^***p<.01

2.4 Analyses

To determine the appropriate level of analysis to consider these treatment beliefs in substance abuse treatment programs, we employed three statistical approaches: Within and Between Analysis (WABA I), Intraclass Correlation Coefficients (ICC(1)) and r_wg. Of the three, WABA is the more conservative, and so we first describe analyses using this technique.

Within and Between Analysis I (WABA; Dansereau, Alutto & Yammarino, 1984) compares the variance within groups as well as between groups (or in the present context, within and between clinics). Through this comparison, WABA I determines at what level of analysis the variables or treatment beliefs are operating (i.e., at the group level or individual level of analysis). From this analysis it can be determined whether it is appropriate to aggregate treatment belief data to a group level of analysis or if data should be interpreted at the individual level of analysis, or if the results are ambiguous or “equivocal.” In our study, WABA I was conducted using the DETECT program (Dansereau, Chandrasekaran, Dumas, Coleman, Ehrlich & Bagchi, 1986).

An advantage of using the WABA I procedure is the ability to determine traditional statistical significance (using F ratios) as well as what the technique's developers refer to as practical significance (using an E ratio, which equals the between-eta correlation divided by the within-eta correlation). The difference between the two is that the test of practical significance is not as influenced by sample size as is the test of statistical significance (Dansereau et al., 1986).¹ The E ratio is considered practically significant for a parts condition (i.e., support for the individual level of analysis) as it approaches zero and significant for a wholes condition (i.e., support for the group level of analysis) as it approaches or exceeds 1.0. Based upon various geometric properties, an E ratio inference at the 30° condition represents a stronger induction than at the 15° condition. This inference is somewhat conceptually similar to the p < .05 and p < .01 inferences more commonly used in social science statistics, with the 30° condition similar to the more conservative p < .01 inference and the 15° condition similar to the more liberal p < .05. However, unlike the probability distributions used in statistics, the E value is calculated using tangents (Dansereau et al., 2006), thus the use of angles versus probabilities. If an E ratio is not deemed significant, the results are interpreted as either equivocal, having meaningful individual and group level variation, or null/ambiguous, not having enough meaningful variance to interpret (see Dansereau et al., 1984 for a more detailed description of these geometric properties of the E ratio).

In the current study, we consider both statistical and practical significance to assess the appropriate level of analysis for the following group entities using WABA I: between all clinics, between clinicians in recovery versus not in recovery, between methadone clinics versus non-methadone treatment clinics, and between research affiliated versus non-research affiliated clinics. For these analyses, significant results at either the individual or group levels of analysis provide support for the hypothesis being tested, while null results can be interpreted as an indication that the treatment belief is not focused on either an individual or a group level of analysis. Specifically, failure to reject the null hypothesis can represent an equivocal condition or an inexplicable condition. In an equivocal condition, either an individual or group level of analysis may apply. In an inexplicable condition, there is not enough variability to warrant a valid focus and any observed variability is considered error (Dansereau et al., 1984).

In addition to using WABA I, we consider Intraclass Correlation Coefficients or ICC(1) for the following groups at both year 1 and 2: between all clinics, between clinicians in recovery versus not in recovery, between methadone clinics versus non-methadone treatment clinics, and between research affiliated versus non-research affiliated clinics. Like WABA I, ICC(1) is based on data from a one-way ANOVA, yet is a more traditional measure. ICC(1) is considered a measure of reliability or degree to which raters are “substitutable” (James, 1982). It has also been described as a measure of the variability in individuals' responses that can be explained by or related to group membership (Bryk & Raudenbush, 1992). The ICC(1) presented here may range from -1 to +1.

While the WABA I and ICC(1) analyses both allow for an omnibus test of whether or not each treatment belief should be considered a group or individual level variable, neither approach considers which specific clinics or groupings of clinicians may have sufficient agreement within them to be considered a group level phenomena. For example, it is possible that an omnibus test would show that a specific belief should be considered an individual level variable, yet at a number of specific clinic sites there could be justification for aggregating that particular belief to the group level of analysis. WABA I and ICC(1) are thus a particularly conservative approach to assessing level of analysis.

To supplement our understanding of the appropriate level of analysis at which treatment beliefs should be considered, we assessed the amount of agreement for each individual clinic, by clinicians' recovery status, by methadone clinics/non-methadone treatment clinics, and at the research/non-research affiliated level of analysis. These less conservative analyses are useful for determining whether a “shared vision” exists, and if so, at what level of grouping (i.e., organization, recovery status, methadone clinic, or research affiliation) it exists. To make these assessments, we relied on the r_wg index, which is a measure of agreement or consensus within a group (Kozlowski & Hattrup, 1992; James, Demaree, & Wolf, 1993). In the current study, r_wg represents the degree of similarity among clinics, in recovery/not in recovery clinicians' ratings of treatment beliefs, methadone/non-methadone clinic, and research/non-research affiliated clinic, or the extent to which these groups of clinicians give the same or very similar ratings to the treatment belief items. Unlike both the WABA I and ICC(1) procedures, which analyze data according to a contrast of within group variance to between group variance, the r_wg index does not depend on between group variance. The r_wg index considers the within group variability and contrasts this variability to the variability expected due to random responding.

In addition, it is important to note that, unlike ICC(1), r_wg is not a measure of reliability among clinicians' ratings of treatment beliefs. The r_wg index is also not a statistical measure of similarity in the rank order of items across clinicians; instead, it measures whether or not clinicians are giving essentially the same ratings on treatment belief items. Specifically, the r_wg index represents the proportion of observed variance compared to the proportion of variance one may expect with random responding on the item (Kozlowski & Hattrup, 1992). Typically, an r_wg of .70 or greater is treated as evidence that there is sufficient agreement among raters to justify aggregating responses to a group level, i.e., that a particular perception can be treated as shared at the group level of analysis within a specific aggregation of individuals (Klein & Kozlowski, 2000).

3. Results

Our set of hypotheses relates to the appropriate level of analysis to consider treatment beliefs in substance abuse treatment clinics. In the current study, we consider the individual or clinician level of analysis (H1a) versus the group levels of clinic (H1b), recovery status (H1c), methadone availability (H1d), and research affiliation (H1e) for both year 1 and year 2.

3.1 WABA I: Statistical Significance Results

With clinic as the aggregate level of analysis being tested in WABA I, none of the 11 treatment beliefs can be supported as functioning at an aggregated or group level of analysis (Table 1). When recovery status is the group level of analysis, “methadone maintenance should be used with heroin addicts” (F = 580.59, p<.05) and “noncompliant clients should be discharged” (F = 99.40, p<.10) clearly appear to be individual (rather than group) level phenomena, and the results are equivocal for the other items (i.e., there is no evidence for either the group or individual level when considering statistical significance). With methadone availability as the aggregate level of analysis being tested in WABA I, none of the 11 treatment beliefs can be supported as functioning at an aggregated or group level of analysis. When research affiliation is the target group level of analysis, “psychiatric medications should be used in addiction treatment” (F = 27,832.94, p<.01) and “people who are unsuccessful in treatment don't want it bad enough” (F = 109.84, p<.10) clearly appear to be individual (rather than group) level phenomena, and the results are equivocal for the other items.

3.2 WABA I: Practical Significance Results

All substance abuse treatment beliefs are better represented at the individual level of analysis as opposed to the group level of analysis at year 1 when examining the E ratio measure of practical significance (Table 1). A majority of these findings are practically significant at the part - 30° induction, which is a more stringent level than the 15° induction. These findings support Hypothesis 1a in favor of Hypotheses 1b – 1e.

In year 2, only one treatment belief is statistically significant, indicating it is an individual level phenomena (data not shown but is available upon request). This treatment belief, “recovering counselors make the best therapists” (F = 104.88, p<.10), is for the group level distinction of methadone maintenance treatment program. The E ratios suggest that all but one of the substance abuse treatment beliefs are better represented at the individual level of analysis as opposed to the group level of analysis at year 2. Again in year 2, the majority of these findings are practically significant at the part - 30° induction, which is a more stringent level than the 15° induction. The one belief that does not reach practical significance, “methadone maintenance should be used with heroin addicts,” occurs when clinic is the group-level variable. This lack of both statistical and practical significance suggests the appropriate level of analysis is inconclusive (Dansereau et al., 1986). Taken together, these findings replicate year 1 data and support Hypothesis 1a, which states that clinicians' treatment beliefs vary meaningfully at the individual level of analysis, rather than at any aggregated level of analysis.

3.3 ICC(1) Results

In general, the ICC(1) results also support Hypothesis 1a, which states that clinicians' treatment beliefs vary meaningfully at the individual level of analysis, rather than at any aggregated level of analysis (see Table 2). However, there was limited evidence that clinicians' treatment beliefs vary meaningfully at the aggregated level of analysis for some specific beliefs and groupings. By clinic and in year 2, the ICC(1) of .40 for “methadone maintenance should be used with heroin addicts” indicates that 40% of the variability in individuals' ratings may be explained by or related to clinic membership. With recovery status as the grouping variable in year 2 an ICC(1) of .37 for “recovering counselors make the best therapists” indicates that 37% of individuals' response variance may be related to recovery status group membership. Also, with methadone availability as the grouping variable and in year 2 ICC(1) of .42 for “methadone maintenance should be used with heroin addicts” indicates that 42% of variability in individuals' ratings may be explained by or related to clinic membership. However, none of these findings were consistent over two years, and only three of the 88 item by aggregation assessments (11 items by two years by four aggregations) appeared to indicate a group level effect.

Table 2. Intraclass correlation coefficients for beliefs about treatment by clinic, recovery status, methadone/non-methadone availability, and research affiliation for years 1 and 2.

Belief Items	By Clinica		By Recovery Statusb		By Methadone Availabilityc		By Research Affiliationd
	Year 1	Year 2	Year 1	Year 2	Year 1	Year 2	Year 1	Year 2
Confrontation should be used in addiction treatment	-.03	.26	.02	.04	.00	.04	.01	.04
Methadone maintenance should be used with heroin addicts	.12	.40	-.01	-.01	.20	.42	-.01	-.01
Spirituality should be emphasized in addiction treatment	-.02	.04	-.01	.04	-.01	.03	.00	.00
Research findings should be used in addiction treatment	-.04	.01	-.01	-.01	.03	-.01	-.01	-.01
Psychiatric medications should be used in addiction treatment	.05	.07	.00	-.01	.02	-.01	.00	-.01
New approaches should be used in addiction treatment	.00	-.03	-.01	-.01	.01	-.01	.01	.02
Working in the addiction treatment field has been a rewarding experience	-.01	.09	-.01	.02	-.01	.03	-.01	.04
People who are unsuccessful in treatment don't want it bad enough	-.01	.02	.03	.02	-.01	.01	.00	-.01
Addiction is really a disease	.02	.17	.01	-.01	-.01	.00	-.01	-.01
Recovering counselors make the best therapists	-.03	.14	.08	.37	-.01	-.01	-.01	.01
Noncompliant clients should be discharged	-.02	.02	-.01	-.01	-.01	.00	-.01	-.01

Notes. Year 1: ^aN = 153 - 156; ^bN = 144 - 146; ^cN = 153 - 156; ^dN = 153 - 156. Year 2: ^aN = 153 - 154; ^bN = 150 - 151; ^cN = 153 - 154; ^dN = 153 - 154.

3.4 r_wg Results

The index r_wg (James, Demaree and Wolf, 1984), a less conservative analysis, was calculated to assess the appropriateness of aggregating each of the items within each aggregate unit (see Table 3). Using the standard .70 criterion as indicating sufficient levels of agreement to justify aggregation (Klein & Kozlowski, 2000), we tested whether a sufficient level of agreement would exist to justify aggregation within clinic, by recovery status, by methadone clinic/non-methadone clinic, and by research affiliation for both year 1 and year 2. The results by clinic for year 1 ranged from .00 to 1.0. Over 80% of clinics had sufficient within-clinic agreement to justify aggregation for the belief “spirituality should be emphasized in addiction treatment” and over 60% of clinics had sufficient within-clinic agreement to justify aggregation for the belief “working in the addiction treatment field has been a rewarding experience.” For these items, the r_wg results indicate acceptable degrees of homogeneity in treatment beliefs, which provides justification for the aggregation of these items to the group level of analysis for these clinics only, and for this year only.

Table 3. Levels of agreement using r_wg on beliefs about treatment by clinic for year 1.

Belief Items	Clinic													Mean rwg	rwg>.70
	CTN					NonCTN
	1	2	3	4	5	6	7	8	9	10	11	12	13
Confrontation should be used in addiction treatment	.75	.62	1.00	.15	.64	.65	.47	.16	.25	.72	.31	61	.17	.50	23%
Methadone maintenance should be used with heroin addicts	.30	.00	.00	.19	.17	.75	.36	.22	.67	.18	.62	00	.02	.27	8%
Spirituality should be emphasized in addiction treatment	.72	.95	.74	.07	.84	.90	.92	.61	.94	.89	.88	78	.82	.77	85%
Research findings should be used in addiction treatment	.44	.95	.49	.01	.50	.90	.87	.73	.66	.61	.77	92	.49	.64	46%
Psychiatric medications should be used in addiction treatment	.54	.35	.44	.22	.68	.40	.78	.16	.25	.15	.51	59	.55	.43	8%
New approaches should be used in addiction treatment	.74	.26	.90	.04	.43	.60	.87	.80	.94	.61	.87	69	.79	.66	54%
Working in the addiction treatment field has been a rewarding experience	.91	.26	.90	.67	.53	1.00	.32	.84	.90	.89	.20	90	.92	.71	62%
People who are unsuccessful in treatment don't want it bad enough	.56	.00	.00	.06	.08	.90	.06	.00	.00	.00	.00	00	.00	.13	8%
Addiction is really a disease	.44	.26	.94	.00	.51	1.00	.92	.00	1.00	.44	.22	32	.95	.54	38%
Recovering counselors make the best therapists	.32	.17	.57	.22	.28	.60	.15	.30	.21	.08	.00	59	.32	.29	0%
Noncompliant clients should be discharged	.40	.49	.00	.41	.00	.10	.41	.38	.44	.37	.00	42	.17	.28	0%
Mean rwg	.56	.39	.54	.19	.42	.71	.56	.38	.57	.45	.40	53	.47

NOTES. Clinic N ranges from 5 to 17 respondents. r_wg of greater than .70 is interpreted as sufficient agreement to interpret aggregation to that level of analysis as justified.

The year 2 results by clinic varied slightly from those in year 1 (data not shown but is available upon request). Specifically, there was sufficient within-clinic agreement to justify aggregation for 92% of clinics on the belief “new approaches should be used in addiction treatment,” for 75% of clinics on “working in the addiction treatment field has been a rewarding experience,” and for 75% of clinics on the belief that “addiction really is a disease.” Again, for these items, the r_wg results indicate acceptable degrees of homogeneity in treatment beliefs, which provides justification for the aggregation of these items to the group level of analysis for these clinics only, and for this year only.

For clinicians in recovery, item by clinic r_wgs ranged from .00 to .79 while r_wgs for those clinicians not in recovery ranged from .00 to .76 for year 1. The mean r_wg results were .44 and .36 respectively for the group of clinicians in recovery and not in recovery. For year 2, the item by clinic r_wgs range from .02 to .81 for those in recovery and .00 to .80 for those not in recovery. The treatment belief “spirituality should be emphasized in addiction treatment,” had r_wgs indicating acceptable agreement for those in recovery but not acceptable levels for those not in recovery for both year 1 and year 2. Surprisingly, “recovering counselors make the best therapists,” had little agreement in recovery/non-recovery groups for both years studied.

For methadone maintenance treatment, the various belief items ranged from .00 to .72 for year 1 for those offering methadone and .00 to .82 for those not offering it. Year 2 were similar, with methadone clinics' r_wg indices ranging from .13 to .84 and indices for clinics not offering methadone treatment ranging from .13 to .82. Interestingly, the general lack of agreement by this grouping of clinics even applies to the treatment belief of “methadone maintenance should be used with heroin addicts,” with all r_wg indices at .02 or less.

For research affiliated clinics r_wg indices for the various belief items ranged from .00 to .66 (year 1), while the indices for non-research affiliated clinicians ranged from .00 to .82. For year 2, we see similar findings with r_wg indices for the belief items ranging from .06 to .75 for research affiliated clinicians and .00 to .85 for non-research affiliated clinicians. Further, for each of the belief items separately, the r_wg indices for the research affiliated and non-research affiliated groups were strikingly similar, suggesting that the two groups are similarly diverse. The treatment belief “research findings should be used in addiction treatment,” had an increase in agreement from year 1 (r_wg = .47) to year 2 (r_wg = .75) for research affiliated clinics and a decrease in agreement for non-research affiliated clinics, from year 1 (r_wg =.74) to year 2 (r_wg = .67). For the belief “new approaches should be used in addiction treatment,” r_wg indices indicate sufficient agreement to justify aggregation in year 1 and 2 for non-research affiliated clinics, but not sufficient agreement in year 1 among those research-affiliated clinics.

4. Discussion

The results of this study highlight the importance of assessing the assumed level of analysis and of testing that the aggregated data do represent a specific, often higher level of analysis. This is particularly important for data sampling strategies that have individuals embedded in groups, who are embedded in clinics or organizations, etc. When data are collected from individuals embedded in higher level groups, it is important to recognize that the higher level group may have an influence on the variables or effects of interest. It is in these situations that the opportunity for researchers to make a fallacy of the wrong level is possible (Klein, Dansereau, & Hall, 1994). The “fallacy of the wrong level” is when researchers attribute something, such as an effect or variable, to one level of analysis when that effect or variable is actually properly attributable to another level (Dansereau et al., 2006).

Dansereau et al. provide the example of training individuals in hopes of increasing performance. If, however, the low performance is actually attributable to group norms (i.e., to a group rather than an individual phenomenon), then training the individual will not increase the low performance. In their example, increasing performance would require changing the group norms surrounding performance levels, rather than increasing individuals' skill levels. A more specific case in point would be the case where new clinicians are sent on an off-site team-building program where they learn techniques for interacting with colleagues, but upon their return to the clinic find that the use of those techniques is ridiculed by more senior clinicians. In this case, the lack of effective teamwork behavior would not be properly attributed to individuals' lack of knowledge or skill, but rather to the prevailing group norms against those behaviors. The same effect could of course be found with training in new treatment techniques that are not supported by clinic leaders, or are not reimbursed by major insurers – again, the proper attribution would be to group level effects rather than to individuals.

The reverse is also possible – survey data about clinician beliefs or values might be collected from clinicians in several clinics, with the goal of comparing clinics on particular variables such as satisfaction, support for a certain treatment, or knowledge of a certain technique. If there is insufficient agreement among clinicians within a clinic to justify the aggregation, misleading conclusions can be drawn (such as when a few outliers pull a group mean significantly away from results that would better characterize the group as a whole).

Turning now to the findings from this specific study, we found strong evidence that the individual level of analysis (parts) is a better representation of the substance abuse treatment beliefs considered than is the groupings of clinic, clinicians' recovery status, treatment modality, and research affiliation (wholes). Dansereau et al. (2006) conceptualize the higher level or group level of analysis by saying, “Effects that we call wholes are compatible with notions such as norms, culture or climate type or variables that are thought to apply in a similar way within groups, divisions or organizations but that vary between them” (p. 560). In some ways, this is encouraging news for those interested in influencing clinicians' treatment beliefs. Because we found clinicians' treatment beliefs operating at the individual level of analysis, clinicians' treatment beliefs are not being driven by the clinics' treatment norms, clinicians' recovery status, or the culture regarding treatment modality or research affiliation. In other words, there are no norms or cultural barriers to influencing individual clinicians' treatment beliefs. In this case, even though clinicians are in the same group, they are likely to respond to information designed to change, influence or expand their treatment beliefs differently. These idiosyncratic treatment beliefs allow for more diversity and treatment options to those seeking substance abuse treatment at clinics.

4.1 Limitations and Future Research

A limitation in the present study, especially when considering the clinic level of analysis, is the small sample size available (ranging from 5 to 17 clinicians per clinic for year 1 and 6 to 24 for year 2). However, the group sample sizes when dividing the sample into in-recovery/not-in-recovery, modality and into research affiliated/non-research affiliated groupings were sufficient for assessing the appropriate level of analysis; thus, we are more confident in our assertions about the lack of agreement at those levels. Another limitation to the current study is that treatment beliefs are measured using individual items instead of a multi-item scale for each treatment belief. However, the goal of this study was to better understand shared agreement, not reliability.

A contributing factor to the lack of homogeneity in clinicians' treatment beliefs may be high staff turnover. Because turnover of staff has been identified in previous research to be a concern in substance abuse treatment facilities (e.g., McLellan, Carise, & Kleber, 2003), this explanation of lack of homogeneity is plausible and may explain why we found clinicians within the same clinic to be more different than they are similar. It may also explain the difficulty in replicating the findings from year 1 to year 2.

For the specific context presented in the current study, future research may consider reasons for why clinicians may appear to not share beliefs. Some of these may include autonomy of clinicians, heterogeneity in prior training, rapid changes in expectations of funding sources may prevent emergence of consensus, perceptions and beliefs other than those concerned with treatment may have higher priority and/or a leadership focus on other areas than those treatment beliefs considered in the current study. While the current study suggests clinicians' treatment beliefs are individual level phenomena, theory and hypotheses may suggest multiple levels or cross-levels of analysis. The current study does not preclude the possibility that multiple levels may be warranted based on theory or that a group level phenomena may moderate these individual level treatment beliefs (George & James, 1993).

4.2 Implications for Substance Abuse Treatment Research

While the current study found clinicians' treatment beliefs to be operating at an individual level of analysis, recent research has begun to consider the multilevel and organizational level effects on issues surrounding substance abuse treatment. In their introduction of the organizational functioning and readiness for change measure, Lehman, Greener, and Simpson (2002) drew attention to the role of organizational climate in the adoption of innovations and technology. Timko and Moos (1998) found that treatment clinics with climates that were perceived as supportive had higher patient functioning, more active engagement in the program and community and increased use of program services. Using a cross level design, Knudsen, Ducharme, and Roman (2006) found that clinicians' endorsed higher emotional exhaustion scores and turnover intentions in clinics that emphasized centralized decision-making (as rated by the administrator and/or director) and lower emotional exhaustion scores and turnover intentions in clinics that emphasized distributive and procedural justice (again as rated by the administrator and/or director). Finally, Knudsen, Roman, Ducharme, and Johnson (2005) found that hospital based treatment programs, as opposed to freestanding programs were more likely to adopt the use of the pharmacological treatment innovation of disulfiram. Other important organizational factors were clinician education level, clinician certification level, and a program treatment philosophy emphasizing medical models of addiction, all associated with increased use of disulfiram. Those programs emphasizing confrontational group therapies and spirituality were less likely to endorse using disulfiram.

As these studies demonstrate, consideration of more group or organizational level variables show promise to furthering our understanding of the effectiveness of substance abuse treatment, which has also been noted by Roman, Ducharme, and Knudsen (2006). However, as the use of embedded data increases (Knudsen et al., 2006; Ball et al, 2002; Lehman et al., 2002), consideration of the appropriate level of analysis of variables gathered as self-report, individual level data may be a growing concern. This has been a concern in the area of organizational studies (Klein, Dansereau, & Hall, 1994) and the current study is an example of how the statistical and conceptual tools applied in other areas of organizational research can be effectively applied in the area of substance abuse treatment.

As noted by Dansereau et al. (2006), the statistical approaches used here can be an appropriate complement and first step to the use of more advanced statistical techniques designed to test multilevel theory and data, such as Structural Equation Modeling, Latent Growth Modeling, and Hierarchical Linear Modeling. These advanced statistical approaches allow for testing theoretical models that include higher levels of analysis as well as multiple levels of analysis. These approaches, in particular Hierarchical Linear Modeling, allow for the simultaneous consideration of both individual level and group level variables and their relationships. With the increasing use of samples of individuals embedded in groups or organizations, these statistical approaches would allow researchers to determine how valuable it may be to consider the individuals' group membership when examining relationships of interest in substance abuse treatment. However, like Dansereau et al., we recommend researchers test the assumption of the level of analysis of their data using the procedures presented in this study, before moving to these more advanced statistical approaches. This will ensure that a fallacy of the wrong level is avoided and the conclusions drawn are appropriate.

Based on the research presented here, we recommend future research utilize these analyses to provide a more thorough investigation of the appropriate level of analysis for the variables of interest. Future research may consider the level of analysis presented in the theoretical rationale of the study and should use proper measures to ensure a group level entity exists prior to aggregation. By providing both a more conservative omnibus test (such as WABA I or ICC(1)) as well as a consideration of specific clinics or groupings of clinicians (using r_wg), we have provided an example of how researchers can assess the evidence about levels of analysis assumptions regarding the appropriate level of analysis for their constructs.

Table 4. Levels of agreement using r_wg on beliefs about treatment by recovery status, methadone treatment availability, and research affiliation for year 1 and year 2.

Belief Items	By Recovery Status				By Methadone Treatment Availability				By Research Affiliation
	Year 1		Year 2		Year 1		Year 2		Year 1		Year 2
	Yes	No	Yes	No	Yes	No	Yes	No	CTN	Non CTN	CTN	Non CTN
Confrontation should be used in addiction treatment	.47	.31	.56	.43	.00	.14	.50	.46	.38	.36	.37	.55
Methadone maintenance should be used with heroin addicts	.00	.00	.02	.00	.36	.00	.13	.68	.00	.01	.06	.00
Spirituality should be emphasized in addiction treatment	.79	.57	.75	.61	.37	.75	.72	.54	.53	.82	.59	.70
Research findings should be used in addiction treatment	.71	.48	.78	.63	.67	.77	.71	.71	.47	.74	.75	.67
Psychiatric medications should be used in addiction treatment	.37	.48	.45	.47	.00	.08	.39	.60	.47	.33	.53	.43
New approaches should be used in addiction treatment	.73	.53	.81	.80	.63	.67	.80	.82	.45	.80	.74	.85
Working in the addiction treatment field has been a rewarding experience	.63	.76	.77	.78	.72	.82	.84	.61	.66	.75	.65	.85
People who are unsuccessful in treatment don't want it bad enough	.00	.18	.21	.23	.00	.00	.21	.27	.06	.00	.23	.20
Addiction is really a disease	.47	.25	.49	.74	.63	.66	.65	.59	.36	.38	.63	.64
Recovering counselors make the best therapists	.52	.24	.51	.42	.00	.00	.29	.13	.26	.11	.45	.20
Noncompliant clients should be discharged	.19	.16	.27	.31	.00	.00	.27	.33	.24	.17	.37	.24
Mean rwg	.44	.36	.51	.49	.31	.35	.50	.52	.35	.41	.49	.48

NOTES. Year 1, recovery status yes N = 76, recovery status no N = 70, methadone treatment yes N = 41, methadone treatment no N = 115, research affiliated yes N = 68, research affiliated no N = 88; Year 2, recovery status yes N = 68, recovery status no N = 92, methadone treatment yes N = 36, methadone treatment no N = 127, research affiliated yes N = 55, research affiliated no N = 108,. r_wg of greater than .70 is interpreted as sufficient agreement to interpret aggregation to that level of analysis as justified.